A method and system for loading data required by a convolutional neural network accelerator operation processing unit

Abstract

The application belongs to the technical field of integrated circuit design, and particularly relates to a loading method for data required by an operation processing unit of a convolutional neural network accelerator, comprising: controlling the cache sequence of weight data and feature map data through a state machine, when controlling the cache of feature map data, first performing input feature map format conversion, performing sub-section division on the feature map data with an external channel-by-channel input and a bit width of A bits, storing the feature map data with an external channel-by-channel input and a bit width of A bits in a grouped staggered storage mode, and converting the stored feature map data into N-way parallel data with a bit width of B bits (B=A / N) by using a special readout mode, so that the B*N data stream can simultaneously complete the loading of feature map data of N storage units, the problem of mismatch between the bit width of off-chip data access and the bit width of data required in the operation processing unit is solved, and the bandwidth utilization of the convolutional neural network accelerator is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field related to integrated circuit design, and more particularly, to a data loading method for a convolutional neural network accelerator operation processing unit. BACKGROUND

[0002] In recent years, artificial intelligence technology dominated by deep learning has developed rapidly, and has made major breakthroughs in image recognition, autonomous driving, text generation, and dialogue. Convolutional neural networks, as an important model of deep learning, have better robustness, generalization ability, and feature learning ability than traditional algorithms, and have shown significant advantages in target matching tasks.

[0003] However, in order to meet the increasingly high algorithm accuracy requirements, the parameter quantity and calculation quantity of neural networks are constantly increasing, and the demand for computing power, storage capacity and storage bandwidth of network models on operation platforms is surging, requiring the use of specialized hardware acceleration platforms to provide computing power support for network training and inference.

[0004] In embedded platforms, special AI accelerators are often designed according to specific application scenarios to accelerate offline inference operations. By sacrificing some generality, higher energy efficiency can be achieved, such as using fixed data bit width, optimizing data flow for specific algorithms, etc. Specialized AI accelerators mainly have two implementation methods: Application Specific Integrated Circuit (ASIC) and Field Programmable Gate Array (FPGA). ASIC has higher energy efficiency, but has the characteristics of long development cycle, high chip cost and cannot be changed after chip. FPGA has better flexibility and reconfigurability, and is more adaptable to rapid iteration and upgrade of network structure. At the same time, FPGA can also be used as a prototype verification platform in AI chip development process to verify and evaluate the system prototype, effectively reducing chip development cost. Therefore, deploying convolutional neural networks on FPGA has important research significance.

[0005] In the design and implementation process of neural network accelerator, especially in the deployment and design of convolutional neural networks on FPGA, the mismatch between off-chip data access bit width and data bit width required in the operation processing unit often leads to low bandwidth utilization of the neural network accelerator. Therefore, designing an efficient and flexible data loading method and system for the operation processing unit of the convolutional neural network accelerator is an urgent problem to be solved. SUMMARY

[0006] In view of the above defects or improvement needs of the prior art, the present application provides a loading method for data required by an operation processing unit of a convolutional neural network accelerator, which aims to solve the technical problem of low bandwidth utilization of the neural network accelerator caused by the mismatch between the off-chip data access bit width and the data bit width required in the operation processing unit.

[0007] To achieve the above object, according to one aspect of the present application, a loading method for data required by an operation processing unit of a convolutional neural network accelerator is provided, comprising:

[0008] writing the working parameters of the convolutional neural network into the parameter register to complete parameter configuration; controlling the cache order of the weight data and the feature map data through a state machine, and after the feature map data and the weight data are completely written into the input cache of the operation processing unit of the convolutional neural network, alternately loading the weight data and the feature map data required for each operation in the operation stage of the operation processing unit;

[0009] In the control of the feature map data cache, first, input feature map format adaptation is performed, specifically: the feature map data input continuously frame by frame is segmented according to A bit to obtain a plurality of fields, and each field is converted into N parallel data with a B bit bit width; for each frame of feature map data, sequentially write into N BRAMs according to the field order, and the N parallel data of the same field are written into the same address of the N BRAMs, and after each field is written into the BRAM, skip n addresses and continue to write the next field, and n is the total number of frames of feature map data that can be cached by the N BRAMs; n fields with the same field number of every n frames of feature map data are stored in adjacent n addresses of the N BRAMs, and the n N parallel data are stored in the N BRAMs in a staggered manner between the adjacent n addresses, and the writing is completed frame by frame in this way; the same field of the same bit width B bit of the continuous n frames of feature map data is read from the N BRAMs in a reverse rotation reading mode in adjacent n fields, and the n B bit segments are spliced into n*B bit according to the frame sequence number order of the feature map data.

[0010] Further, in the input feature map format adaptation, two groups of block caches are configured, each group of block cache being N BRAMs, and the write-in and read-out of the feature map data are performed through a ping-pong cache mechanism, specifically:

[0011] After the first n frames of feature map data are stored in one of the two sets of block caches, the feature map data stream is switched to be written to the other set of block caches, while the data in the one set of block caches is read out; when the other set of block caches is completely loaded with n frames of feature map data, the feature map data stream is switched to be written to the one set of block caches, while the data in the other set of block caches is read out; in this way, the feature map data is alternately input to the two sets of block caches, until there is no feature map data input, and the remaining data in the block caches is read out.

[0012] Further, before the weight data and the feature map data after format adaptation are written to the input cache of the convolutional neural network operation processing unit, the feature map data and the weight data are prepared in the following manner:

[0013] According to the relationship between the convolution mode and the input channel of the feature map and the output channel of the convolutional neural network accelerator operation processing unit, one of copying and splicing operations is selected for the feature map data and the weight data, respectively, as the cache mode for writing the corresponding data to the input cache, so as to expand the bit width of the data sent to the operation processing unit;

[0014] Specifically, in the ordinary convolution or dilated convolution mode, if the number of output channels of the convolutional neural network accelerator operation processing unit is equal to the number of input channels of the feature map, the copying mode is selected, and specifically, the feature map and the weight data are copied zero times to complete the data preparation; if the number of output channels of the convolutional neural network accelerator operation processing unit is twice the number of input channels of the feature map, the copying mode is selected, and specifically, the feature map and the weight data are copied twice to complete the data preparation; if the number of output channels of the convolutional neural network accelerator operation processing unit is four times the number of input channels of the feature map, the copying mode is selected, and specifically, the feature map and the weight data are copied four times to complete the data preparation.

[0015] In the cross-correlation convolution mode, for the weight data, the copying mode is selected, and specifically, the weight data is copied zero times to complete the preparation of the weight data; for the feature map data, the splicing mode is selected, and specifically, n*B bit data is copied twice and spliced to complete the preparation of the feature map data.

[0016] Further, the input cache of the weight data is composed of Z BRAMs with a bit width of A bit;

[0017] When the weight data is written, the weight data stream is divided into weight write units of 512 bits according to the order of the weight data, each weight write unit is written to the same address of the Z BRAMs in turn from the first BRAM, and when the same address of the Z BRAMs is full, the address is jumped to the next one, and the corresponding writing is continued from the first BRAM, until all the weight data is stored.

[0018] In the weight data reading, all the weight data of the same address of the continuous multiple Z BRAMs are spliced and read, and the loading of the weight data required by all the convolutional neural network operation processing units is completed.

[0019] Further, the weight data loading is performed in the form of local multicast, that is, a single BRAM outputs weight data for multiple convolutional neural network operation processing units which are locally fixed.

[0020] Further, the BRAM in the input buffer of the weight data is a read-priority mode dual-port BRAM, which can optionally write all 0 to the current address while reading the weight data according to whether the weight data is reused in the next calculation, wherein when the weight data is not reused in the next calculation and 0 is selected to be written to the current address, the original data of the address is cleared, so as to realize the parallelization of data reading and buffer flushing; when the weight data is reused in the next calculation and 0 is not selected to be written to the current address, the original data of the address is retained, so as to avoid repeated loading of the weight.

[0021] Further, the loading mode of the feature map data required by each operation of the operation processing unit is as follows:

[0022] The hierarchical cache mechanism using registers and distributed RAMs is used to transform the feature map data required by the convolutional neural network operation processing unit, specifically, the input buffer of the feature map data loads multiple feature map row data required by the processing unit calculation to the row loading circuit according to the row number in time, wherein each feature map row data is a row of B bit data of the feature map data; one feature map data corresponds to one row loading circuit, and each row loading circuit is composed of (B / D-1) pre-loadable D bit shift registers and one distributed FIFO, the low D bit of each row is directly written into the distributed FIFO, and the high (B-D) bit of each row is loaded into the shift register, and the data of each shift register in each row loading circuit is sequentially sent into the distributed FIFO in each cycle; when all the FIFO water level signals are valid, the convolutional neural network operation processing unit simultaneously enables the read enable signal of all the FIFOs, and continuously reads the feature map data in each FIFO; the input buffer of the feature map data continuously supplies the interfaces of the row loading circuits in a polling manner until the entire feature map data is traversed.

[0023] Further, the feature map data loading is performed in the form of local multicast, that is, a single distributed FIFO outputs feature map data for multiple convolutional neural network operation processing units which are locally fixed.

[0024] According to another aspect of the present application, a loading system for data required by an operation processing unit of a convolutional neural network accelerator is provided for performing the loading method as described above, which comprises an input feature map format adaptation module, an input data loading control module, a weight data input buffer of the operation processing unit of the convolutional neural network, and a feature map data input buffer of the operation processing unit of the convolutional neural network.

[0025] The input data loading control module is configured to write working parameters of the convolutional neural network into a parameter register to complete parameter configuration, control the buffer order of the weight data and the feature map data through a state machine, and load the weight data and the feature map data alternately required for each operation in the operation stage of the operation processing unit after the feature map data and the weight data are completely written into the feature map data input buffer and the weight data input buffer of the operation processing unit.

[0026] The input feature map format adaptation module is configured to perform input feature map format adaptation first when the input data loading control module controls the feature map data buffer.

[0027] Overall, compared with the prior art, the scheme provided by the present application has the following beneficial effects:

[0028] 1. The present application provides a loading method for data required by an operation processing unit of a convolutional neural network accelerator, which first performs input feature map format conversion when controlling the feature map data buffer, divides the feature map data with an external channel input and a bit width of A bit into subsegments, stores the feature map data with an external channel input and a bit width of A bit in a group staggered storage mode, and converts the stored feature map data into N-way parallel data with a bit width of B bit (B=A / N) using a special readout mode. The data stream of BxN can simultaneously complete the feature map data loading of N storage units, solving the problem of mismatch between the data access bit width outside the chip and the data bit width required in the operation processing unit, and improving the bandwidth utilization rate of the convolutional neural network accelerator.

[0029] 2. The present application can select a specific replication and splicing input mode to control the storage of the feature map data and the weight data according to the relationship between the convolution operation mode and the feature map input channel and the output channel of the operation processing unit of the convolutional neural network accelerator. The present application supports the operation of ordinary convolution, dilated convolution, and cross-correlation convolution. In the ordinary convolution and dilated convolution mode, the feature map data and the weight data are stored in a replication mode. In the cross-correlation mode, the weight data is stored in a replication mode, and the feature map data is stored in a splicing mode. The above unique storage mode of the feature map data and the weight data expands the bit width of the feature map data and the weight data sent into the operation processing unit.

[0030] 3. The application designs a configurable read-refresh strategy for weight data, which can select whether to close read-refresh according to whether the weight is reused by the next calculation, and if the weight is reused by the next calculation, the read-refresh is closed to avoid repeated loading of the weight; if the weight is not reused by the next calculation, read-refresh is implemented to realize the parallelization of data readout and cache refresh, and improve the flexibility of the convolutional neural network accelerator.

[0031] 4. The application uses a hierarchical cache mechanism of registers cooperating with distributed RAM to reduce storage resource overhead. The loading of feature map data of each row is completed by a shift register and a distributed FIFO. In an FPGA, the distributed FIFO is implemented using distributed RAM, which occupies less LUT resource than BRAM, and is suitable for applications with low data bit width and small cache depth, thus reducing the resource overhead of the application. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is an input feature map format adaptation schematic diagram provided by an embodiment of the application;

[0033] Figure 2 is a feature map and weight loading control state machine schematic diagram provided by an embodiment of the application;

[0034] Figure 3 is a weight loading circuit schematic diagram provided by an embodiment of the application;

[0035] Figure 4 is a low-resource-overhead feature map row loading circuit schematic diagram provided by an embodiment of the application;

[0036] Figure 5 is a structural block diagram of a loading system for data required by an operation processing unit of a convolutional neural network accelerator provided by an embodiment of the application. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical scheme and advantages of the application clearer, the application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application. In addition, the technical features involved in each embodiment of the application described below can be combined with each other as long as they do not conflict with each other.

[0038] Embodiment one

[0039] A loading method for data required by an operation processing unit of a convolutional neural network accelerator, comprising:

[0040] The working parameters of the convolutional neural network are written into the parameter register to complete parameter configuration; the state machine controls the cache order of the weight data and the feature map data, and after the feature map data and the weight data are completely written into the input cache of the convolutional neural network operation processing unit, the required weight data and the feature map data for each operation are alternately loaded in the operation stage of the operation processing unit;

[0041] In the control of the feature map data cache, first, input feature map format adaptation is performed, specifically: the feature map data input continuously frame by frame is segmented according to A bits to obtain a plurality of fields, and each field is converted into N parallel data with a bit width of B bits; for each frame of feature map data, sequentially write into N BRAMs according to the field order, and the N parallel data of the same field are written into the same address of the N BRAMs, and after each field is written into the BRAM, skip n addresses and continue to write the next field, n is the total number of frames of feature map data that can be cached by the N BRAMs; n fields with the same field number of every n frames of feature map data are stored in adjacent n addresses of the N BRAMs, and the n N parallel data are stored in the N BRAMs in a staggered manner between the adjacent n addresses, and the writing is completed frame by frame in this way; the same bit width B bit sub-segment of the same field of the continuous n frames of feature map data is read from the N BRAMs in a reverse rotation reading mode in adjacent n fields, and the n B bit sub-segments are spliced into n*B bits according to the frame sequence number of the feature map data.

[0042] The embodiment provides a loading method for data required by an operation processing unit of a convolutional neural network accelerator, and first performs input feature map format conversion when controlling the feature map data cache. The feature map data input externally channel by channel and having a bit width of A bits is sub-segmented, the feature map data input externally channel by channel and having a bit width of A bits is stored in a grouped staggered storage mode, and the stored feature map data is converted into N parallel data with a bit width of B bits (B=A / N) by using a special readout mode. The B*N data stream can simultaneously complete feature map data loading of N storage units, solves the problem that the off-chip data access bit width and the required data bit width in the operation processing unit do not match, and improves the bandwidth utilization rate of the convolutional neural network accelerator.

[0043] As a preferred embodiment, in the input feature map format adaptation, two groups of block caches are configured, each group of block caches is N BRAMs, and the ping-pong cache mechanism is used for feature map data writing and reading, specifically:

[0044] After the first n frames of feature map data are stored in one of the two block caches, the feature map data stream is switched to be written to the other block cache, and a readout operation is performed on the data in the one block cache; when the other block cache is fully loaded with n frames of feature map data, the feature map data stream is switched to be written to the one block cache, and a readout operation is performed on the data in the other block cache; in this way, the feature map data is alternately input to the two block caches, and when there is no feature map data input, the remaining data in the block cache is read out.

[0045] As a preferred embodiment, before the weight data and the feature map data after format adaptation are written to the input buffer of the convolutional neural network operation processing unit, the following method is used for feature map data and weight data preparation operation:

[0046] According to the relationship between the convolution mode and the input channel of the feature map and the output channel of the convolutional neural network accelerator operation processing unit, one of the copy and splicing operations is selected for the feature map data and the weight data as the cache mode of the corresponding data written to the input buffer, so as to expand the bit width sent to the operation processing unit.

[0047] Specifically, in the ordinary convolution or dilated convolution mode, if the number of output channels of the convolutional neural network accelerator operation processing unit is equal to the number of input channels of the feature map, the copy mode is selected, and specifically, the feature map and weight data are copied zero times to complete the data preparation; if the number of output channels of the convolutional neural network accelerator operation processing unit is twice the number of input channels of the feature map, the copy mode is selected, and specifically, the feature map and weight data are copied twice to complete the data preparation; if the number of output channels of the convolutional neural network accelerator operation processing unit is four times the number of input channels of the feature map, the copy mode is selected, and specifically, the feature map and weight data are copied four times to complete the data preparation.

[0048] In the cross-correlation convolution mode, for the weight data, the copy mode is selected, and specifically, the weight data is copied 0 times to complete the preparation of the weight data, and for the feature map data, the splicing mode is selected, and specifically, n*B bit data is copied twice and spliced to complete the preparation of the feature map data.

[0049] The method of the embodiment can select a specific replication and splicing input mode to control the storage of feature map data and weight data according to the relationship between the convolution operation mode and the input channel of the feature map and the output channel of the operation processing unit of the convolutional neural network accelerator. The application supports the operation of ordinary convolution, dilated convolution and cross-correlation convolution. In the ordinary convolution and dilated convolution mode, the feature map data and the weight data are stored in the replication mode; in the cross-correlation mode, the weight data is stored in the replication mode, and the feature map data is stored in the splicing mode. The unique storage mode of the feature map data and the weight data expands the bit width of the feature map data and the weight data sent to the operation processing unit.

[0050] As a preferred embodiment, the input buffer of the weight data is composed of Z BRAMs with a bit width of A bit;

[0051] When the weight data is written, the weight data stream is divided into weight write units of 512 bits according to the order of the weight data, and each weight write unit is sequentially written into the same address of the Z BRAMs from the first BRAM. When the same address of the Z BRAMs is full, the address is jumped to the next one, and the corresponding writing is continued from the first BRAM, until all the weight data is stored.

[0052] When the weight data is read out, all the weight data of the same address of the Z BRAMs is spliced and read out, and the loading of the weight data required by all the convolutional neural network operation processing units is completed.

[0053] As a preferred embodiment, the weight data loading is performed in the form of local multicast, that is, a single BRAM outputs weight data for a plurality of convolutional neural network operation processing units which are fixed locally.

[0054] As a preferred embodiment, the BRAM in the input buffer of the weight data is a read-priority mode dual-port BRAM. When the weight data is read out, the next calculation of the weight data is reused, and optionally, all 0 is written to the current address. When the next calculation of the weight data is not reused, 0 is selected to be written to the current address, the original data of the address is cleared, and the parallelization of data reading and cache flushing is realized. When the next calculation of the weight data is reused, 0 is not selected to be written to the current address, and the original data of the address is retained, avoiding repeated loading of the weight.

[0055] The embodiment designs a configurable read-refresh strategy for the weight data. The strategy can select whether to close the read-refresh according to whether the weight is reused by the next calculation. If the weight is reused by the next calculation, the read-refresh is closed to avoid repeated loading of the weight. If the weight is not reused by the next calculation, the read-refresh is implemented to realize the parallelization of data reading and cache flushing, and the flexibility of the convolutional neural network accelerator is improved.

[0056] As a preferred embodiment, the loading mode of the feature map data required by each operation of the operation processing unit is as follows:

[0057] The hierarchical cache mechanism using registers and distributed RAM is used to convert the feature map data required by the convolutional neural network operation processing unit. Specifically, the input cache of the feature map data loads multiple feature map row data required by the processing unit calculation according to the row number in time, wherein each feature map row data is a row of B bit data of the feature map data; one feature map data corresponds to one row loading circuit, and each row loading circuit is composed of (B / D-1) pre-loadable D bit shift registers and a distributed FIFO. The low D bit of each row is directly written into the distributed FIFO, and the high (B-D) bit of each row is loaded into the shift register. The data of each shift register in each row loading circuit is sequentially sent into the distributed FIFO in each cycle. When all the FIFO water level signals are valid, the convolutional neural network operation processing unit simultaneously enables the read enable signals of all the FIFOs, and continuously reads the feature map data in each FIFO. The input cache of the feature map data continuously supplies each row loading circuit interface in a polling manner until the entire feature map data is traversed.

[0058] The embodiment uses the hierarchical cache mechanism using registers and distributed RAM to reduce the storage resource overhead. The loading of the feature map data of each row is completed by the shift register and the distributed FIFO. In the FPGA, the distributed FIFO is implemented using distributed RAM, which occupies less LUT resource than BRAM, and is suitable for applications with low data bit width and small cache depth, thereby reducing the resource overhead of the present application.

[0059] As a preferred embodiment, the feature map data loading is performed in the form of local multicast, that is, a single distributed FIFO outputs the feature map data for multiple convolutional neural network operation processing units which are locally fixed.

[0060] Embodiment two

[0061] A loading system for the data required by the operation processing unit of a convolutional neural network accelerator, for performing the loading method as described in embodiment one above, comprising: an input feature map format adaptation module, an input data loading control module, a weight data input cache of the convolutional neural network operation processing unit, and a feature map data input cache of the convolutional neural network operation processing unit;

[0062] The input data loading control module is configured to write working parameters of the convolutional neural network into a parameter register to complete parameter configuration, and control the cache order of the weight data and the feature map data through a state machine, and after the feature map data and the weight data are completely written into the feature map data input cache and the weight data input cache of an operation processing unit of the convolutional neural network, the weight data and the feature map data required for each operation are alternately loaded in an operation stage of the operation processing unit.

[0063] The input feature map format adaptation module is configured to adapt the input feature map format when the input data loading control module controls the cache of the feature map data.

[0064] Specifically, the input feature map format adaptation module is configured to convert feature map data of A bit in width input channel by channel (i.e. frame by frame) into N-way parallel data of B bit in width (B=A / N). The module segments the feature map data input channel by channel and continuously into A bit, and each field jumps n addresses to continue writing the next field after being written into an adapter unit (i.e. N BRAMs) at a certain address, where n is the total number of frames of the feature map in each block cache, and each field data is stored in a group staggered in the format adaptation module. When continuously writing, the write address constantly jumps and completes writing frame by frame; when continuously reading, the read address performs reverse rotation in adjacent C fields to complete reading. As a priority, two adapter units can be instantiated to implement ping-pong operation to provide data for the input feature map data input cache.

[0065] To better illustrate the input feature map format adaptation operation described in Embodiment One and Embodiment Two, an example is given as follows:

[0066] As shown in Figure 1 The process of input feature map format conversion is the working process of the input feature map format adaptation module described above. The 512-bit feature map data stream input channel by channel is converted into 8-channel parallel output, and each output channel data has a width of 64 bits, which is adapted to the width of the subsequent single block cache storage unit (i.e. single BRAM). The data stream of 64 bits x 8 can simultaneously complete the feature map data loading of 8 storage units (i.e. 8 BRAMs), i.e. the data stream of B x N can simultaneously complete the feature map data loading of N storage units, thereby eliminating the on-chip input bandwidth bottleneck.

[0067] As shown in Figure 1As shown, the above-mentioned adapter unit is composed of 8 blocks of 64-bit wide BRAM. The input feature map data is stored in the adapter unit in a staggered manner, and the Fmap label numbers represent the feature map frame number and the 64-bit data number in a single feature map frame, respectively. For example, Fmap1-1 represents the first 64-bit number of the first feature map frame, Fmap2-8 represents the eighth 64-bit number of the second feature map frame, and so on. When writing, the continuously input feature map data is segmented by 512 bits, and a plurality of fields are obtained, i.e., 8 consecutive 64-bit numbers form a field (one frame, field, and subsegment). For example, Fmap1-1, Fmap1-2, …, Fmap1-8 form the first field of the first frame, wherein each Fmap represents a field in the first segment, Fmap1-9, Fmap1-10, …, Fmap1-16 form the second field of the first frame, wherein each Fmap represents a field in the first segment, and so on. For a feature map frame, the fields are sequentially written, wherein each field is written to the adapter unit, then jumps n addresses, and then writes the next field. n is the total number of frames of the feature map in each block cache (i.e., the adapter unit), and in the legend, n = 8, i.e., the same field of 8 consecutive feature map frames is stored in 8 adjacent addresses of a block cache.

[0068] For the feature map writing mode, the continuously input data is segmented by A bits, i.e., N consecutive B bits form a field, and the first B bit number of the first frame is denoted as Fmap1-1. Fmap1-1, Fmap1-2, …, Fmap1-n are the first field of the first frame, Fmap1-(n+1), Fmap1-(n+2), …, Fmap1-2n are the second field of the first frame, and so on. Each field is written to the adapter unit, then jumps n addresses, and then writes the next field. n is the total number of frames of the feature map in each block cache, i.e., the same field of n consecutive feature map frames is stored in n adjacent addresses of the adapter.

[0069] When reading, 64-bit data numbers of the same data of 8 consecutive feature map frames are read in each cycle, i.e., the same color data in the figure, and then the data is spliced in the order of the feature map frame number to obtain 512 bits, i.e., 8-way simultaneous output of new feature map data.

[0070] The writing address is uniform when 8 adjacent 64-bit data of each feature map frame is written to 8 blocks of BRAM. When continuously writing, the writing address is constantly jumped by the above-mentioned jumping operation, and the writing is completed frame by frame in this way. The reading address of the BRAM is staggered, and when continuously reading, the reading address is reversely rotated in the adjacent 8 fields, for example, Figure 1In the left image, the 64-bit RBRAM in the first row outputs Fmap1-1 at address 0 in the first cycle, Fmap8-2 at address 7 in the second cycle, and so on. Similarly, the 64-bit memory in the second row outputs Fmap2-1 at address 1 in the first cycle, Fmap1-2 at address 0 in the second cycle, and so on. After reading the same field data from 8 consecutive frames of feature map data, the address jumps to the next field region. In general, this means reading a B-bit segment of the same field from n consecutive frames of feature map data from N BRAMs, and concatenating these n B-bit segments into an n*B-bit array according to the feature map data frame number.

[0071] In summary, the readout method for feature maps is as follows: the readout addresses between BRAMs are interleaved. During continuous reading, the read address rotates in reverse order within adjacent N fields. For example, BRAM1 outputs Fmap1-1 at address 0 in the first cycle, FmapN-2 at address n in the second cycle, and so on. BRAM2 outputs Fmap2-1 at address 1 in the first cycle, Fmap1-2 at address 0 in the second cycle, and so on. After reading the same field data from M consecutive frames of feature maps, the address jumps to the next field region.

[0072] It should also be noted that the grouped misalignment storage method in the input feature map format adaptation operation requires that multiple frames of feature maps be completely stored in the adapter unit before they can be read out in batches. Simultaneous read and write operations will cause read and write conflicts and data pollution. Therefore, preferably, the input feature map format adaptation module adopts a ping-pong caching mechanism to execute in parallel.

[0073] The ping-pong caching mechanism in read / write operations can be implemented as follows: the input feature map format adaptation module instantiates two adapter units, numbered X and Y respectively. For example... Figure 5 As shown, data is read from the interface asynchronous FIFO and written to adapter unit X via the input data stream selector. When a set of feature maps is fully loaded in X, the input data stream selector switches the data stream to write to Y. At the same time, the output data stream selector starts reading data from X and outputs it to the feature map block buffer module. When a set of feature maps is fully loaded in Y, the input selector switches to X, and the output selector switches to Y. Feature map data is alternately input to X and Y via the selectors until there is no new data in the asynchronous FIFO. At this point, the output data stream selector reads the remaining data from the adapter all at once.

[0074] Furthermore, to better illustrate the loading control operations described in Embodiment 1 and Embodiment 2, the following examples are provided:

[0075] like Figure 2As shown, the feature map and weight loading control logic is exemplarily implemented by a unified three-stage state machine. After reset, the circuit is in an idle (IDLE) state, enters a parameter configuration (PARA_CFG) state when a current layer calculation start signal is received, and waits for the main control module to write the working parameters (convolution type, image size, etc.) into the working parameter register. After the parameter configuration is completed, the circuit starts to wait for the feature map data and weight data to be completely input into the corresponding input buffer (this waiting and buffering state is referred to as WAIT INPUT), and enters the operation stage (including loading and operation while loading) of the convolutional neural network operation processing unit when the weight data and feature map data buffering are both completed. The weight data loading (LOAD_WEIGHT) and feature map data loading (LOAD_IFMAP) states are alternately jumped. Finally, after all the data are loaded, the circuit enters the loading completion (LOAD_END) state, and reenters the idle (IDLE) state until the calculation result is output.

[0076] An input data control module can be designed on the hardware to implement the above-mentioned logic, to write the working parameters of the convolutional neural network into the parameter register and complete the parameter configuration. The state machine is used to control the buffering sequence of the weight data and the feature map data, and the data preparation operation control can be first performed in the execution of the buffering operation. Specifically, a specific replication and splicing buffering mode can be selected according to the convolution operation mode to prepare the data, and the prepared data is written into the feature map data input buffer and the weight data input buffer of the convolutional neural network operation processing unit to expand the bit width sent into the operation processing unit. After the feature map data and the weight data are completely written into the feature map data input buffer and the weight data input buffer of the convolutional neural network operation processing unit, the weight data and the feature map data required for each operation in the operation stage of the operation processing unit are alternately loaded.

[0077] Therefore, the input data control module can include a weight and feature map buffering control module and a feature map and weight data buffering mode selection module.

[0078] The weight and feature map buffering control module is used to control the buffering sequence of the weight and the feature map, and the weight is first written into the weight buffering module, and then the feature map is written into the feature map buffering module through the state machine.

[0079] The feature map and weight data buffering mode selection module is used to control the buffering mode of the feature map and the weight data written into the input buffer. The buffering mode supports three convolution mode selections of normal convolution, empty convolution and cross-correlation convolution. According to the convolution mode and the relationship between the feature map input channel and the output channel of the operation processing unit of the convolutional neural network accelerator, one operation is selected from the replication and splicing operations as the buffering mode to expand the bit width sent into the operation processing unit.

[0080] Specifically, the selection of the caching mode is used for data preparation for writing to the feature map and weight cache. Specifically, in ordinary convolution or dilated convolution modes, if the number of output channels of the convolutional neural network accelerator processing unit equals the number of input channels of the feature map, then the copy mode is selected, specifically a copy mode that copies the feature map and weight data zero times; if the number of output channels of the convolutional neural network accelerator processing unit is twice the number of input channels of the feature map, then the copy mode is selected, specifically a copy mode that copies the feature map and weight data two times; if the number of output channels of the convolutional neural network accelerator processing unit is four times the number of input channels of the feature map, then the copy mode is selected, specifically a copy mode that copies the feature map and weight data four times. In cross-correlation convolution mode, for weight data, the copy mode is selected, specifically a copy mode that copies the weight data zero times; for feature map data, the concatenation mode is selected, copying the n*B bit data twice and concatenating the two copies into 2*n*B bit data for writing to the cache.

[0081] Furthermore, to better illustrate the weight data input cache described in Embodiments 1 and 2, the following examples are provided:

[0082] like Figure 3 The diagram illustrates the weight data cache. The weight block cache is also composed of BRAM groups. Specifically, the weight cache consists of nine BRAMs, each 512 bits wide and 1024 bits deep (4.5 Mib in total). These nine BRAMs are directly concatenated into a 4608-bit array at the output port. The 4608-bit data from four adjacent addresses are grouped together, with each group corresponding to all weights required for a single computation by a convolutional neural network (CNN) processing unit. During weight writing, 512 bits are written sequentially according to the numerical order shown in the diagram until all weights are written. During weight reading, one 4608-bit data point is read per cycle and loaded into the CNN processing unit. This allows for the local weight loading of 192 CNN processing units, and weight loading for all CNN processing units can be completed in four cycles. The process involves filling the same address in each BRAM before moving to the next address and writing sequentially. During reading, all weight data from the same address in the nine BRAMs are concatenated and read out.

[0083] For the weight data input cache: the weight data input cache module is composed of Z block BRAM with a bit width of 512 bits and a depth of 1024. The Z block BRAM is spliced into Z*512 bits at the output port, and the Z*512 bit data of adjacent t addresses is divided into a group. A single group of weights corresponds to all the weights required for a single calculation of the convolutional neural network accelerator operation processing unit. When writing weights, 512 bits are written at a time in sequence until all the weights are stored. When reading weights, a Z*512 bit data is loaded to the convolutional neural network accelerator operation processing unit every cycle, and all the weight loading can be completed in t cycles.

[0084] The weight data input cache outputs data to the convolutional neural network operation processing unit, that is, the weight data loading adopts a local multicast loading form. In hardware design, in order to improve the on-chip supply bandwidth, the weight BRAM output port data is grouped and spliced, and is sent to the convolutional neural network operation processing unit in the form of local multicast, that is, a single weight RAM provides weight data for several local fixed convolutional neural network accelerator operation processing units. Compared with global multicast, local multicast sacrifices a certain flexibility to obtain less logic and connection resource overhead, and at the same time obtains a higher loading rate.

[0085] In order to avoid the generation of unexpected partial sum data in the addition tree, it is necessary to ensure that the weight data of the convolutional neural network operation processing unit not participating in the operation is 0. Therefore, a configurable read refresh strategy can be designed as an option. A dual-port BRAM in read-first mode is selected, which writes all 0 to the current address while reading the weight data, clears the data at the original address, and realizes the parallelization of data reading and cache refreshing. The read refresh mode is optional. When the weight needs to be reused for the next calculation, the read refresh can be turned off to avoid repeated loading of the weight. For example, when the feature map is calculated by row and column block, the weights corresponding to different feature map blocks are the same group, and at this time, the read refresh mode of the weight cache does not need to be enabled.

[0086] Further, in order to better illustrate the feature map data input cache described in embodiment one and embodiment two, the following is illustrated:

[0087] First of all, it needs to be explained that the weight data loading is to read the data in the weight data input cache directly to the operation processing unit, and the difference between the weight data loading process is that the feature map data loading needs to transform the data read from the feature map data input cache, and load the transformed data to the operation processing unit. Therefore, compared with the hardware design of the weight data, a feature map data transformation module required by the convolutional neural network accelerator operation processing unit is additionally designed for the feature map data.

[0088] The feature map data storage module (i.e., feature map data input buffer) includes C basic storage units, each of which is composed of H BRAMs. Each W basic storage units form a local multicast group, and there are P (P = C / W) groups in total. Each local multicast group provides data for a specific convolutional neural network accelerator operation processing unit.

[0089] The transformation module of the feature map data required by the convolutional neural network operation processing unit uses a hierarchical cache mechanism of registers and distributed RAM to reduce the storage resource overhead. Specifically, the input buffer of the feature map data loads multiple feature map row data required by the processing unit calculation to the row loading circuit according to the row number in time, wherein each feature map row data is a row of B bit data of the feature map data; one feature map data corresponds to one row loading circuit, and each row loading circuit is composed of (B / D-1) pre-loadable D bit shift registers and a distributed FIFO. The low D bit of each row is directly written into the distributed FIFO, and the high (B-D) bit of each row is loaded into the shift register; the data of each shift register in each row loading circuit is sequentially sent into the distributed row buffer FIFO in each cycle, and when all the FIFO water level signals are valid, the convolutional neural network operation processing unit simultaneously enables the read enable signal of all FIFOs, and continuously reads the feature map data in each FIFO; the input buffer of the feature map data continuously supplies the interface of each row loading circuit in a polling manner until the entire feature map data is traversed.

[0090] As Figure 4The convolutional neural network operation processing unit shown needs a feature map data conversion module. The feature map row loading circuit uses a hierarchical cache mechanism of registers cooperating with distributed RAM to reduce storage resource overhead. The loading of each row of data of each frame is completed by 7 loadable 8-bit shift registers and an 8-bit x 16 depth distributed FIFO. In the FPGA, the distributed FIFO is implemented using distributed RAM, which occupies less LUT resource than block RAM (i.e. BRAM), and is more suitable for applications with low data bit width and small cache depth. It is worth noting that the row cache FIFO shown in the figure only needs to load 8 pixel points at a time, without loading the entire feature map row. The following describes the working process of the feature map row loading circuit: first, the feature map block cache sends 64-bit data (8 pixel points) to the 64-bit row loading circuit interface 1, where the low 8 bits are directly written to the distributed row cache FIFO, and the high 56 bits are preloaded to the shift register; then, the shift register starts to send data to the distributed row cache FIFO cycle by cycle, while the feature map block cache starts to send data to other row loading circuit interfaces; next, when all the FIFO water level signals (program full signals) are valid, the convolutional neural network operation processing unit simultaneously enables the read enable signals of all the FIFOs to continuously read the feature map data in the FIFOs; finally, the feature map block cache continuously supplies data to each row loading circuit interface in a polling manner until the entire feature map block is traversed.

[0091] The present application proposes a hierarchical cache strategy for the off-chip continuous access strategy and the streaming data flow mode of the convolutional neural network accelerator operation processing unit, which fully utilizes the locality characteristics of feature map and weight data during calculation in the convolutional neural network accelerator operation processing unit, improves data reuse rate, and realizes high-bandwidth off-chip memory access and stable data supply to on-chip operation modules. As for the hierarchical cache strategy, the feature map data is taken as an example, the off-chip data is cached in the input feature map format adaptation module, the new feature map data is cached in the input cache, and then cached in the distributed FIFO before being loaded into the operation processing unit. The locality characteristic refers to the reuse of weight data during convolution operation, which does not need to be loaded repeatedly, so the data reuse rate is improved. The bit width of the off-chip memory access in the present application is A bits, and the value of A is for example 512, which does not need to be set according to the 8-bit data required by the operation processing unit, so it is a high-bandwidth off-chip memory access, and can continuously supply data.

[0092] Those skilled in the art can easily understand that the above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for loading data required by a convolutional neural network accelerator operation processing unit, characterized in that, The application relates to a method for writing data into a convolutional neural network (CNN) accelerator, and belongs to the technical field of computer vision. The working parameters of the convolutional neural network are written into parameter registers to complete parameter configuration; the storage order of weight data and feature map data is controlled by a state machine, and after the feature map data and the weight data are completely written into the input buffer of the operation processing unit of the convolutional neural network, the weight data and the feature map data required for each operation are alternately loaded in the operation stage of the operation processing unit. In the control of the feature map data storage, first, input feature map format adaptation is performed, specifically: the continuously frame-by-frame input feature map data is segmented according to A bit to obtain a plurality of fields, and each field is converted into N parallel data with a B bit bit width; for each frame of feature map data, the N parallel data of the same field are written into the same address of the N BRAMs in sequence according to the field order, and after each field is written into the BRAM, n addresses are skipped, and the next field is continuously written, wherein n is the total number of frames of feature map data stored in the N BRAMs; n fields with the same field number of every n frames of feature map data are stored in adjacent n addresses of the N BRAMs, and n N parallel data are stored in the N BRAMs in a staggered manner between adjacent n addresses, and the writing is completed frame by frame in this way; the same field of the same bit width B bit of the continuous n frames of feature map data is read from the N BRAMs in a reverse rotation reading mode, and the n B bit segments are spliced into n*B bit according to the frame sequence number of the feature map data.

2. The loading method of claim 1, wherein, In the input feature map format adaptation, two groups of block caches are configured, each group of block caches is N BRAMs, and the ping-pong cache mechanism is used for writing and reading out the feature map data, specifically: After the first n frames of feature map data are stored in one of the block caches, the feature map data stream is switched to write into the other block cache, and the data in the one block cache is read out; when the other block cache is completely loaded with n frames of feature map data, the feature map data stream is switched to write into the one block cache, and the data in the other block cache is read out; in this way, the feature map data is alternately input into the two block caches until there is no feature map data input, and the remaining data in the block cache is read out.

3. The loading method of claim 1, wherein, Before the weight data and the feature map data after format adaptation are written into the input buffer of the convolutional neural network operation processing unit, the feature map data and the weight data are prepared in the following way: According to the relationship between the convolution mode and the input channel of the feature map and the output channel of the operation processing unit of the convolutional neural network accelerator, one of copying and splicing operations is selected for the feature map data and the weight data as the storage mode of the corresponding data written into the input buffer to expand the bit width sent into the operation processing unit. Specifically, in the ordinary convolution or the hole convolution mode, if the output channel number of the convolutional neural network accelerator operation processing unit is equal to the input channel number of the feature map, the copy mode is selected, and specifically, the feature map and the weight data are copied zero times respectively to complete the data preparation; if the output channel number of the convolutional neural network accelerator operation processing unit is twice the input channel number of the feature map, the copy mode is selected, and specifically, the feature map and the weight data are copied twice respectively to complete the data preparation; if the output channel number of the convolutional neural network accelerator operation processing unit is four times the input channel number of the feature map, the copy mode is selected, and specifically, the feature map and the weight data are copied four times to complete the data preparation. In the cross-correlation convolution mode, for the weight data, the copy mode is selected, and specifically, the weight data is copied zero times to complete the preparation of the weight data, and for the feature map data, the splicing mode is selected, and specifically, the n*B bit data is copied twice and spliced to complete the preparation of the feature map data.

4. The loading method of claim 1, wherein, The input buffer of the weight data is composed of Z BRAMs with Abit bit width; When the weight data is written, the weight data stream is divided into weight write units of 512 bits in sequence according to the weight data, each weight write unit is sequentially written into the same address of the Z BRAMs from the first BRAM, and when the same address of the Z BRAMs is full, the next address is jumped to and the corresponding writing is continued from the first BRAM, until all the weight data is stored; When the weight data is read out, all the weight data of the continuous multiple same addresses of the Z BRAMs is spliced and read out to complete the loading of the weight data required by all the convolutional neural network operation processing units.

5. The loading method of claim 1, wherein, The weight data loading is performed in the form of local multicast, that is, a single BRAM outputs weight data for a plurality of convolutional neural network operation processing units fixed locally.

6. The loading method of claim 4, wherein, The BRAM in the input buffer of the weight data is a read-priority mode dual-port BRAM, which can optionally write all 0 to the current address while reading out the weight data according to whether the weight data is reused next time, wherein when the weight data is not reused next time, 0 is written to the current address to clear the original data of the address, realizing the parallelization of data reading and buffer refreshing; when the weight data is reused next time, 0 is not written to the current address, and the original data of the address is retained to avoid repeated loading of the weight.

7. The loading method of claim 1, wherein, The loading mode of the feature map data required by each operation of the operation processing unit is: The hierarchical cache mechanism using registers cooperated with distributed RAM is used to transform feature map data required by the convolutional neural network operation processing unit. Specifically, the input cache of the feature map data is controlled to load multiple feature map row data required by the processing unit calculation to the row loading circuit according to the row number in time, wherein each feature map row data is a row of B bit data of the feature map data; one feature map data corresponds to one row loading circuit, and each row loading circuit is composed of (B / D-1) pre-loadable D bit shift registers and one distributed FIFO. The low D bit of each row is directly written into the distributed FIFO, and the high (B-D) bit of each row is loaded into the shift register. The data of each shift register in each row loading circuit is sequentially sent into the distributed FIFO in each cycle. When all the FIFO water level signals are valid, the convolutional neural network operation processing unit simultaneously enables the read enable signals of all the FIFOs to continuously read the feature map data in each FIFO. The input cache of the feature map data continuously supplies each row loading circuit interface in a polling manner until the entire feature map data is traversed.

8. The loading method of claim 7, wherein, The feature map data loading is performed in the form of local multicast, that is, a single distributed FIFO outputs feature map data for multiple convolutional neural network operation processing units which are locally fixed.

9. A loading system for data required by a convolutional neural network accelerator operation processing unit, characterized in that, The method comprises an input feature map format adaptation module, an input data loading control module, a weight data input cache of the convolutional neural network operation processing unit, and a feature map data input cache of the convolutional neural network operation processing unit. The input data loading control module is used to write the working parameters of the convolutional neural network into the parameter register to complete parameter configuration; the state machine is used to control the cache order of the weight data and the feature map data, and after the feature map data and the weight data are completely written into the feature map data input cache and the weight data input cache of the convolutional neural network operation processing unit, the required weight data and feature map data for each operation are alternately loaded in the operation stage of the operation processing unit. The input feature map format adaptation module is used to adapt the input feature map format when the input data loading control module controls the cache of the feature map data.

Images