Method and system for loading data required by convolutional neural network accelerator operation processing unit

Abstract

The invention belongs to the related technical field of integrated circuit design, and particularly relates to a method for loading data required by an accelerator operation processing unit of a convolutional neural network, which comprises the following steps of: controlling a cache sequence of weight data and feature map data through a state machine, when controlling the cache of the feature map data, firstly performing input feature map format conversion, the method comprises the following steps: performing sub-segment division on external channel-by-channel input feature map data with the bit width of A bit, storing the external channel-by-channel input feature map data with the bit width of A bit in a grouping dislocation storage mode, and converting the stored feature map data into N paths of parallel data with the bit width of B bit (B = A / N) in a special reading mode, the B * N data flow can complete feature map data loading of N storage units at the same time, the problem that the off-chip data access bit width is not matched with the data bit width needed in the operation processing unit is solved, and the bandwidth utilization rate of the convolutional neural network accelerator is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field related to integrated circuit design, and more specifically, relates to a method for loading data required by a convolutional neural network accelerator operation processing unit. Background Art

[0002] In recent years, artificial intelligence technology led by deep learning has flourished, and has made major breakthroughs in applications such as image recognition, autonomous driving, text generation and dialogue. As an important model of deep learning, convolutional neural networks are generally superior to traditional algorithms in terms of robustness, generalization ability and feature learning ability, and have shown significant advantages in target matching tasks.

[0003] However, in order to meet the increasingly high requirements for algorithm accuracy, the number of parameters and computing power of neural networks continues to increase, and the network model's demand for the computing power, storage capacity, and storage bandwidth of the computing platform has increased sharply, requiring the use of specialized hardware acceleration platforms to provide computing power support for network training and reasoning.

[0004] In embedded platforms, dedicated AI accelerators are often designed according to specific application scenarios to accelerate offline reasoning operations. Higher energy efficiency can be achieved at the cost of sacrificing certain versatility, such as using fixed data bit widths and optimizing data flows for specific algorithms. Dedicated AI accelerators are mainly implemented in two ways: Application Specific Integrated Circuit (ASIC) and Field Programmable Gate Array (FPGA). ASIC has higher energy efficiency, but has the characteristics of long development cycle, high tape-out cost and cannot be changed after tape-out. FPGA has better flexibility, reconfigurability, and stronger adaptability to rapid iteration and upgrading of network structure. At the same time, FPGA can also be used as a prototype verification platform in the AI ​​chip development process to verify and evaluate the system prototype, effectively reducing the cost of chip R&D. Therefore, it is of great research significance to deploy convolutional neural networks on FPGA.

[0005] In the process of designing and implementing neural network accelerators, especially in the design and implementation of deploying convolutional neural networks on FPGAs, the bandwidth utilization of neural network accelerators is often low due to the mismatch between the off-chip data access bit width and the required data bit width in the operation processing unit. Therefore, designing an efficient and flexible data loading method and system for the operation processing unit of a convolutional neural network accelerator is an urgent problem to be solved. Summary of the invention

[0006] In view of the above defects or improvement needs of the prior art, the present invention provides a method for loading data required by a convolutional neural network accelerator operation processing unit, which aims to solve the technical problem of low bandwidth utilization of the neural network accelerator caused by the mismatch between the off-chip data access bit width and the required data bit width in the operation processing unit.

[0007] To achieve the above object, according to one aspect of the present invention, a method for loading data required by a convolutional neural network accelerator operation processing unit is provided, comprising:

[0008] The working parameters of the convolutional neural network are written into the parameter register to complete the parameter configuration; the cache order of the weight data and the feature map data is controlled by the state machine, and after the feature map data and the weight data are completely written into the input cache of the convolutional neural network operation processing unit, the weight data and the feature map data required for each operation are alternately loaded in the operation stage of the operation processing unit;

[0009] Among them, when controlling the feature map data cache, the input feature map format is first adapted, specifically: the feature map data input continuously frame by frame is segmented according to A bit to obtain multiple fields, and each field is converted into N sub-segments with a bit width of Bbit, that is, N parallel data; for each frame of feature map data, the N BRAMs are written in sequence according to the field order, and the N parallel data of the same field are correspondingly written to the same address of the N BRAMs, and after each field is written to the BRAM, n addresses are jumped to continue writing the next field, and n is the total number of frames of feature map data that can be cached by the N BRAMs; the n fields with the same field number of each consecutive n frames of feature map data are stored in the adjacent n addresses of the N BRAMs, and the n N parallel data are staggered between the adjacent n addresses and stored in the N BRAMs, so that the writing is completed frame by frame in this way; the sub-segments with a bit width of B bit of the same field of the consecutive n frames of feature map data are read from the N BRAMs in a reverse rotation reading method in the adjacent n fields, and the n B The bit sub-segments are concatenated into n*B bits in the order of the feature map data frame numbers.

[0010] Furthermore, in the input feature map format adaptation, two groups of block caches are configured, each group of block caches is N BRAMs, and a ping-pong cache mechanism is used to write and read feature map data, specifically:

[0011] After the first n frames of feature map data are stored in one group of block caches, the feature map data stream is switched to be written to the other group of block caches, and a read operation is performed on the data in the one group of block caches; when the other group of block caches is completely loaded with n frames of feature map data, the feature map data stream is switched to be written to the one group of block caches, and a read operation is performed on the data in the other group of block caches; in this way, feature map data are alternately input into the two groups of block caches until no feature map data is input, and a read operation is performed on the remaining data in the block caches.

[0012] Furthermore, before writing the weight data and the feature map data after format adaptation into the input buffer of the convolutional neural network operation processing unit, the feature map data and weight data preparation operation is performed in the following manner:

[0013] According to the relationship between the convolution mode and the feature map input channel and the output channel of the convolutional neural network accelerator operation processing unit, one operation is selected from the copying and splicing operations for the feature map data and the weight data, respectively, as the cache mode for writing the corresponding data into the input cache, so as to expand the bit width sent to the operation processing unit;

[0014] Specifically, in the normal convolution or dilated convolution mode, if the number of output channels of the convolutional neural network accelerator operation processing unit is equal to the number of input channels of the feature map, the copy mode is selected, and specifically, the feature map and weight data are copied zero copies respectively to complete the data preparation; if the number of output channels of the convolutional neural network accelerator operation processing unit is twice the number of input channels of the feature map, the copy mode is selected, and specifically, the feature map and weight data are copied twice respectively to complete the data preparation; if the number of output channels of the convolutional neural network accelerator operation processing unit is four times the number of input channels of the feature map, the copy mode is selected, and specifically, the feature map and weight data are copied four times respectively to complete the data preparation;

[0015] In the cross-correlation convolution mode, for the weight data, the copy mode is selected, and specifically, the weight data is copied zero times to complete the preparation of the weight data. For the feature map data, the splicing mode is selected, and specifically, the n*B bit data is copied twice and spliced ​​to complete the preparation of the feature map data.

[0016] Further, the input buffer of the weight data is composed of Z BRAMs with a bit width of A bit;

[0017] When writing weight data, the weight data stream is divided into 512-bit weight writing units according to the weight data sequence, and each weight writing unit is sequentially written to the same address of Z BRAMs starting from the first BRAM. After the same address of Z BRAMs is full, jump to the next address, and continue to write from the first BRAM until all weight data are stored;

[0018] When the weight data is read out, all the weight data of multiple consecutive addresses of Z BRAMs are spliced ​​and read out to complete the loading of the weight data required by all convolutional neural network operation processing units.

[0019] Furthermore, weight data loading is performed in the form of local multicast, that is, a single BRAM outputs weight data for multiple locally fixed convolutional neural network operation processing units.

[0020] Furthermore, the BRAM in the input cache of the weight data is a dual-port BRAM in read priority mode. When reading out the weight data, all 0s can be optionally written to the current address based on whether the weight data will be reused in the next calculation. When the weight data is not reused in the next calculation and 0 is chosen to be written to the current address, the original data at the address is cleared to achieve parallelization of data reading and cache refresh; when the weight data is reused in the next calculation and 0 is chosen not to be written to the current address, the original data at the address is retained to avoid repeated loading of weights.

[0021] Furthermore, the method of loading the feature map data required for each operation of the operation processing unit is as follows:

[0022] The hierarchical cache mechanism of registers and distributed RAM is used to transform the feature map data required by the convolutional neural network operation processing unit. Specifically, the input cache of the feature map data is controlled to load multiple feature map row data required for the processing unit calculation into the row loading circuit in time-sharing according to the row sequence number, wherein each feature map row data is a row of B bit data of the feature map data; one feature map data corresponds to one row loading circuit, each row loading circuit is composed of (B / D-1) preloadable D bit shift registers and a distributed FIFO, the low-order D bits of each row are directly written into the distributed FIFO, the high-order (BD) bits of each row are loaded into the shift register, and the data of each shift register in each row loading circuit is sequentially sent to the distributed FIFO cycle by cycle; when all FIFO water level signals are valid, the convolutional neural network operation processing unit simultaneously enables the read valid signals of all FIFOs to continuously read the feature map data in each FIFO; the input cache of the feature map data maintains continuous supply of data to each row loading circuit interface in a polling manner until the entire feature map data is traversed.

[0023] Furthermore, feature map data loading is performed in the form of local multicast, that is, a single distributed FIFO outputs feature map data for multiple locally fixed convolutional neural network operation processing units.

[0024] According to another aspect of the present invention, a system for loading data required by a convolutional neural network accelerator operation processing unit is provided, which is used to execute the loading method as described above, and comprises: an input feature map format adaptation module, an input data loading control module, a weight data input cache of the convolutional neural network operation processing unit, and a feature map data input cache of the convolutional neural network operation processing unit;

[0025] The input data loading control module is used to write the working parameters of the convolutional neural network into the parameter register to complete the parameter configuration; the cache order of the weight data and the feature map data is controlled by the state machine, and after the feature map data and the weight data are completely written into the feature map data input cache and the weight data input cache of the convolutional neural network operation processing unit, the weight data and the feature map data required for each operation are alternately loaded in the operation stage of the operation processing unit;

[0026] The input feature map format adaptation module is used to first perform input feature map format adaptation when the input data loading control module controls the feature map data cache.

[0027] In general, compared with the prior art, the above technical solution conceived by the present invention has the following beneficial effects:

[0028] 1. The present invention proposes a method for loading data required by a convolutional neural network accelerator operation processing unit. When controlling the feature map data cache, the input feature map format is first converted, the feature map data with a bit width of A bit input channel by channel from the outside is divided into sub-segments, and the feature map data with a bit width of A bit input channel by channel from the outside is stored in a grouped staggered storage manner, and the stored feature map data is converted into N-channel parallel data with a bit width of B bit (B=A / N) in a special read-out manner. The B×N data stream can simultaneously complete the loading of feature map data for N storage units, thereby solving the problem of mismatch between the off-chip data access bit width and the data bit width required in the operation processing unit, and improving the bandwidth utilization of the convolutional neural network accelerator.

[0029] 2. The present invention can select specific copy and splice input modes to control the storage of feature map data and weight data according to the convolution operation mode and the relationship between the feature map input channel and the output channel of the convolutional neural network accelerator operation processing unit. The present invention supports the operation of ordinary convolution, dilated convolution and cross-correlation convolution. In ordinary convolution and dilated convolution modes, feature map data and weight data are stored in copy mode; in cross-correlation mode, weight data is stored in copy mode and feature map data is stored in splicing mode. The above unique storage method of feature map data and weight data expands the bit width of the feature map data and weight data sent to the operation processing unit.

[0030] 3. The present invention designs a configurable read refresh strategy for weight data, which can choose whether to turn off the read refresh based on whether the weight is reused in the next calculation. If the weight is reused in the next calculation, the read refresh is turned off to avoid repeated loading of the weight; if the weight is not reused in the next calculation, the read refresh is implemented to achieve parallelization of data reading and cache refresh, thereby improving the flexibility of the convolutional neural network accelerator.

[0031] 4. The present invention uses a hierarchical cache mechanism that combines registers and distributed RAM to reduce storage resource overhead. The loading of feature map data for each row is completed by the shift register and distributed FIFO. In FPGA, distributed FIFO is implemented using distributed RAM. Unlike BRAM, distributed RAM occupies less LUT resources and is suitable for applications with low data bit width and small cache depth, thus reducing the resource overhead of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a schematic diagram of input feature map format adaptation provided by an embodiment of the present invention;

[0033] Figure 2 It is a schematic diagram of a characteristic graph and a weight loading control state machine provided by an embodiment of the present invention;

[0034] Figure 3 is a schematic diagram of a weight loading circuit provided by an embodiment of the present invention;

[0035] Figure 4 is a schematic diagram of a characteristic graph row loading circuit with low resource overhead provided by an embodiment of the present invention;

[0036] Figure 5 It is a structural block diagram of a system for loading data required by a convolutional neural network accelerator operation processing unit provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0038] Embodiment 1

[0039] A method for loading data required by a convolutional neural network accelerator operation processing unit, comprising:

[0040] The working parameters of the convolutional neural network are written into the parameter register to complete the parameter configuration; the cache order of the weight data and the feature map data is controlled by the state machine, and after the feature map data and the weight data are completely written into the input cache of the convolutional neural network operation processing unit, the weight data and the feature map data required for each operation are alternately loaded in the operation stage of the operation processing unit;

[0041] Among them, when controlling the feature map data cache, the input feature map format is first adapted, specifically: the feature map data input continuously frame by frame is segmented according to A bit to obtain multiple fields, and each field is converted into N sub-segments with a bit width of Bbit, that is, N parallel data; for each frame of feature map data, the N BRAMs are written in sequence according to the field order, and the N parallel data of the same field are correspondingly written to the same address of the N BRAMs, and after each field is written to the BRAM, n addresses are jumped to continue writing the next field, and n is the total number of frames of feature map data that can be cached by the N BRAMs; the n fields with the same field number of each consecutive n frames of feature map data are stored in the adjacent n addresses of the N BRAMs, and the n N parallel data are staggered between the adjacent n addresses and stored in the N BRAMs, so that the writing is completed frame by frame in this way; the sub-segments with a bit width of B bit of the same field of the consecutive n frames of feature map data are read from the N BRAMs in a reverse rotation reading method in the adjacent n fields, and the n B The bit sub-segments are concatenated into n*B bits in the order of the feature map data frame numbers.

[0042] This embodiment proposes a method for loading data required by a convolutional neural network accelerator operation processing unit. When controlling the feature map data cache, the input feature map format is first converted, the feature map data with a bit width of A bit input channel by channel from the outside is divided into sub-segments, and the feature map data with a bit width of A bit input channel by channel from the outside is stored in a grouped staggered storage manner, and the stored feature map data is converted into N-channel parallel data with a bit width of B bit (B=A / N) in a special read-out manner. The B×N data stream can complete the loading of feature map data for N storage units at the same time, solving the problem of mismatch between the off-chip data access bit width and the data bit width required in the operation processing unit, and improving the bandwidth utilization of the convolutional neural network accelerator.

[0043] As a preferred implementation, in the input feature map format adaptation, two groups of block caches are configured, each group of block caches is N BRAMs, and a ping-pong cache mechanism is used to write and read feature map data, specifically:

[0044] After the first n frames of feature map data are stored in one group of block caches, the feature map data stream is switched to be written to the other group of block caches, and a read operation is performed on the data in the one group of block caches; when the other group of block caches is completely loaded with n frames of feature map data, the feature map data stream is switched to be written to the one group of block caches, and a read operation is performed on the data in the other group of block caches; in this way, feature map data are alternately input into the two groups of block caches until no feature map data is input, and a read operation is performed on the remaining data in the block caches.

[0045] As a preferred implementation, before writing the weight data and the feature map data after format adaptation into the input buffer of the convolutional neural network operation processing unit, the feature map data and weight data preparation operation is performed in the following manner:

[0046] According to the relationship between the convolution mode and the feature map input channel and the output channel of the convolutional neural network accelerator operation processing unit, one operation is selected from the copying and splicing operations for the feature map data and the weight data, respectively, as the cache mode for writing the corresponding data into the input cache, so as to expand the bit width sent to the operation processing unit;

[0047] Specifically, in the normal convolution or dilated convolution mode, if the number of output channels of the convolutional neural network accelerator operation processing unit is equal to the number of input channels of the feature map, the copy mode is selected, and specifically, the feature map and weight data are copied zero copies respectively to complete the data preparation; if the number of output channels of the convolutional neural network accelerator operation processing unit is twice the number of input channels of the feature map, the copy mode is selected, and specifically, the feature map and weight data are copied twice respectively to complete the data preparation; if the number of output channels of the convolutional neural network accelerator operation processing unit is four times the number of input channels of the feature map, the copy mode is selected, and specifically, the feature map and weight data are copied four times respectively to complete the data preparation;

[0048] In the cross-correlation convolution mode, for the weight data, the copy mode is selected, and specifically, the weight data is copied 0 times to complete the preparation of the weight data. For the feature map data, the splicing mode is selected, and specifically, the n*B bit data is copied twice and spliced ​​to complete the preparation of the feature map data.

[0049] The method of this embodiment can select a specific copy and splice input mode to control the storage of feature map data and weight data according to the convolution operation mode and the relationship between the feature map input channel and the output channel of the convolutional neural network accelerator operation processing unit. The present invention supports the operation of ordinary convolution, hole convolution and cross-correlation convolution. In ordinary convolution and hole convolution modes, feature map data and weight data are stored in copy mode; in cross-correlation mode, weight data is stored in copy mode and feature map data is stored in splicing mode. The above unique storage method of feature map data and weight data expands the bit width of feature map data and weight data sent to the operation processing unit.

[0050] As a preferred implementation, the input buffer of the weight data is composed of Z BRAMs with a bit width of A bit;

[0051] When writing weight data, the weight data stream is divided into 512-bit weight writing units according to the weight data sequence, and each weight writing unit is sequentially written to the same address of Z BRAMs starting from the first BRAM. After the same address of Z BRAMs is full, jump to the next address, and continue to write from the first BRAM until all weight data are stored;

[0052] When the weight data is read out, all the weight data of multiple consecutive addresses of Z BRAMs are spliced ​​and read out to complete the loading of the weight data required by all convolutional neural network operation processing units.

[0053] As a preferred implementation, weight data loading is performed in the form of local multicast, that is, a single BRAM outputs weight data for multiple locally fixed convolutional neural network operation processing units.

[0054] As a preferred implementation, the BRAM in the input cache of the weight data is a dual-port BRAM in read priority mode. While reading out the weight data, all 0s can be optionally written to the current address based on whether the weight data will be reused in the next calculation. When the weight data is not reused in the next calculation and 0 is chosen to be written to the current address, the original data at the address is cleared to achieve parallelization of data reading and cache refresh; when the weight data is reused in the next calculation and 0 is chosen not to be written to the current address, the original data at the address is retained to avoid repeated loading of weights.

[0055] This implementation method designs a configurable read refresh strategy for weight data, which can choose whether to turn off the read refresh based on whether the weight will be reused in the next calculation. If the weight is reused in the next calculation, the read refresh is turned off to avoid repeated loading of the weight; if the weight is not reused in the next calculation, the read refresh is implemented to achieve parallelization of data reading and cache refresh, thereby improving the flexibility of the convolutional neural network accelerator.

[0056] As a preferred implementation, the method of loading the feature map data required for each operation of the operation processing unit is as follows:

[0057] The hierarchical cache mechanism of registers and distributed RAM is used to transform the feature map data required by the convolutional neural network operation processing unit. Specifically, the input cache of the feature map data is controlled to load multiple feature map row data required for the processing unit calculation into the row loading circuit in time-sharing according to the row sequence number, wherein each feature map row data is a row of B bit data of the feature map data; one feature map data corresponds to one row loading circuit, each row loading circuit is composed of (B / D-1) preloadable D bit shift registers and a distributed FIFO, the low-order D bits of each row are directly written into the distributed FIFO, the high-order (BD) bits of each row are loaded into the shift register, and the data of each shift register in each row loading circuit is sequentially sent to the distributed FIFO cycle by cycle; when all FIFO water level signals are valid, the convolutional neural network operation processing unit simultaneously enables the read valid signals of all FIFOs to continuously read the feature map data in each FIFO; the input cache of the feature map data maintains continuous supply of data to each row loading circuit interface in a polling manner until the entire feature map data is traversed.

[0058] This embodiment uses a hierarchical cache mechanism that cooperates with registers and distributed RAM to reduce storage resource overhead. The feature map data loading of each row is completed by the shift register and the distributed FIFO. In FPGA, the distributed FIFO is implemented using distributed RAM. Unlike BRAM, distributed RAM occupies less LUT resources and is suitable for applications with low data bit width and small cache depth, thereby reducing the resource overhead of the present invention.

[0059] As a preferred implementation, local multicast is used to load feature map data, that is, a single distributed FIFO outputs feature map data for multiple locally fixed convolutional neural network operation processing units.

[0060] Embodiment 2

[0061] A system for loading data required by a convolutional neural network accelerator operation processing unit, used to execute the loading method described in the first embodiment, comprising: an input feature map format adaptation module, an input data loading control module, a weight data input cache of the convolutional neural network operation processing unit, and a feature map data input cache of the convolutional neural network operation processing unit;

[0062] Among them, the input data loading control module is used to write the working parameters of the convolutional neural network into the parameter register to complete the parameter configuration; the cache order of the weight data and the feature map data is controlled by the state machine, and after the feature map data and the weight data are completely written into the feature map data input cache and the weight data input cache of the convolutional neural network operation processing unit, the weight data and the feature map data required for each operation are alternately loaded in the operation stage of the operation processing unit;

[0063] The input feature map format adaptation module is used to first perform input feature map format adaptation when the input data loading control module controls the feature map data cache.

[0064] Specifically, the input feature map format adaptation module is used to convert the feature map data with a bit width of Abit input channel by channel (i.e. frame by frame) into N parallel data with a bit width of B bit (B = A / N). The module segments the feature map data input continuously channel by channel by Abit. After each field is written to a certain address of the adapter unit (i.e. N BRAMs), it jumps n addresses to continue writing to the next field. n is the total number of frames of the feature map inside each block cache, and each field data is stored in a staggered manner in groups inside the format adaptation module. When writing continuously, the write address jumps continuously and completes the writing frame by frame; when reading continuously, the read address rotates in reverse in the adjacent C fields to complete the reading. It can be used as a priority, and two adapter units are instantiated to implement ping-pong operations to provide data for the input feature map data input cache.

[0065] In order to better illustrate the input feature map format adaptation operation described in Embodiment 1 and Embodiment 2, an example is given as follows:

[0066] like Figure 1 As shown in the figure, the process of input feature map format conversion is the working process of the above-mentioned input feature map format adaptation module. The 512-bit feature map data stream input channel by channel is converted into 8-channel parallel output, and each output channel data bit width is 64 bits, which is adapted to the bit width of the subsequent single block cache storage unit (i.e., a single BRAM). The 64-bit×8 data stream can complete the loading of feature map data for 8 storage units (i.e., 8 BRAMs) at the same time, that is, the B×N data stream can complete the loading of feature map data for N storage units at the same time, thereby eliminating the on-chip input bandwidth bottleneck.

[0067] like Figure 1As shown, the adapter unit is composed of 8 BRAMs with a width of 64 bits. The input feature map data is stored in groups and staggered inside the adapter unit. The Fmap number indicates the feature map frame number and the 64-bit data number in the single-frame feature map, for example, Fmap1-1 indicates the first 64-bit number of the feature map of the first frame, and Fmap2-8 indicates the eighth 64-bit number of the feature map of the second frame, and so on. When writing, the feature map input continuously channel by channel is segmented by 512 bits to obtain multiple fields, that is, 8 consecutive 64-bit numbers form a field (one frame, field, sub-segment), for example, Fmap1-1, Fmap1-2, ..., Fmap1-8 constitute the first field of the first frame, where each Fmap represents a field in the first segment, Fmap1-9, Fmap1-10, ..., Fmap1-16 constitute the second field of the first frame, where each Fmap represents a field in the first segment, and so on. For a frame of feature map data, it is written in sequence according to the field order. After each field is written into the adapter unit, it jumps n addresses and continues to write the next field. n is the total number of frames of the feature map inside each block cache (i.e., the adapter unit). In the figure, n=8, that is, the same field of every 8 consecutive frames of feature maps will be stored in 8 adjacent addresses of a group of block caches.

[0068] To summarize the feature map writing method: segment the data input channel by channel continuously by A bit, that is, N consecutive B bits form a field, and the first B bit of the first frame is recorded as Fmap1-1. Fmap1-1, Fmap1-2, ..., Fmap1-n are the first field of the first frame, Fmap1-(n+1), Fmap1-(n+2), ..., Fmap1-2n are the second field of the first frame, and so on. After each field is written to the adapter unit, it jumps n addresses to continue writing the next field. n is the total number of frames of the internal feature map of each block cache, that is, the same field of each consecutive n-frame feature map will be stored in the n adjacent addresses of the adapter.

[0069] When reading out, the data with the same 64-bit data sequence number in 8 consecutive feature maps, that is, the data of the same color in the map, are read in each cycle, and are spliced ​​into 512 bits in the order of the feature map frame sequence number to obtain a new feature map with 8 channels output simultaneously.

[0070] Each time when writing 8 adjacent 64-bit data of each frame feature map into 8 BRAMs, the write address is unified. When writing continuously, the write address jumps continuously through the above jump operation, and the writing is completed frame by frame in this way. The read addresses between BRAMs are staggered. When reading continuously, the read address rotates in reverse in the adjacent 8 fields, for example, Figure 2The 64-bit RBAM in the first row of the left figure outputs Fmap1-1 at address 0 in the first cycle, Fmap8-2 at address 7 in the second cycle, and so on, while the 64-bit memory in the second row outputs Fmap2-1 at address 1 in the first cycle, Fmap1-2 at address 0 in the second cycle, and so on. After reading the same field data of 8 consecutive frames of feature maps, the address jumps to the next field area. In general, that is, the same way of the same field of the same field of consecutive n frames of feature map data is read from N BRAMs with a width of B bits, and the n B bit sub-segments are spliced ​​into n*Bbit according to the sequence number of the feature map data frame.

[0071] To summarize the feature map readout method: the readout addresses between BRAMs are staggered. When reading continuously, the read addresses rotate in reverse in the adjacent N fields. For example, BRAM1 outputs Fmap1-1 at address 0 in the first cycle, FmapN-2 at address n in the second cycle, and so on. BRAM2 outputs Fmap2-1 at address 1 in the first cycle, Fmap1-2 at address 0 in the second cycle, and so on. After reading the same field data of the feature map of consecutive M frames, the address jumps to the next field area.

[0072] It should also be noted that the grouped staggered storage method in the input feature map format adaptation operation requires that multiple frames of feature maps be completely stored in the adapter unit before they can be read out in batches. Simultaneous read and write operations will cause read-write conflicts and data pollution. Therefore, preferably, the input feature map format adaptation module adopts a ping-pong cache mechanism for parallel execution.

[0073] The ping-pong cache mechanism can be reflected in the implementation of read and write operations as follows: the input feature map format adaptation module instantiates two adapter units, numbered X and Y respectively. Figure 5 As shown, data is read from the interface asynchronous FIFO and written into the adapter unit X through the input data stream selector; when a set of feature maps is completely loaded in X, the input data stream selector switches the data stream to write to Y, and at the same time, the output data stream selector starts to read the data in X and outputs it to the feature map block cache module; when a set of feature maps is completely loaded in Y, the input selector switches to X, and the output selector switches to Y; the feature map data is alternately input to X and Y through the selector until there is no new data in the asynchronous FIFO, at which time the output data stream selector reads the remaining data in the adapter at one time.

[0074] Further, in order to better illustrate the loading control operation described in the first and second embodiments, an example is given as follows:

[0075] like Figure 2As shown, the feature map and weight loading control logic can be implemented by a unified three-stage state machine. After reset, the circuit is in the idle (IDLE) state. When the current layer calculation start signal is received, it enters the parameter configuration (PARA_CFG) state, waiting for the main control module to write the working parameters (convolution type, image size, etc.) into the working parameter register. After the parameter configuration is completed, the circuit starts to wait for the feature map data and weight data to be completely input into the corresponding input buffer (this waiting and caching state is called WAIT_INPUT). When the weight data and feature map data cache are completed, it enters the operation stage of the convolutional neural network operation processing unit (including loading, loading and operation), and the weight data loading (LOAD_WEIGHT) and feature map data loading (LOAD_IFMAP) states alternately jump. Finally, after all data are loaded, it enters the loading completion (LOAD_END) state until the calculation results are output and re-enters the idle (IDLE) state.

[0076] An input data control module can be designed in hardware to implement the above logic, which is used to write the working parameters of the convolutional neural network into the parameter register to complete the parameter configuration; the cache order of the weight data and the feature map data can be controlled by the state machine, and the data preparation operation control can be performed first in the execution of the cache operation. Specifically, according to the convolution operation mode, a specific copy and splicing cache mode can be selected to prepare data, and the prepared data can be controlled to be written into the feature map data input cache and the weight data input cache of the convolutional neural network operation processing unit to expand the bit width sent to the operation processing unit. After the feature map data and the weight data are completely written into the feature map data input cache and the weight data input cache of the convolutional neural network operation processing unit, the weight data and the feature map data required for each operation are alternately loaded in the operation stage of the operation processing unit.

[0077] Therefore, the input data control module may include a weight and feature map cache control module and a feature map and weight data cache mode selection module.

[0078] The weight and feature map cache control module is used to control the cache order of weights and feature maps. The weights are first written into the weight cache module through the state machine, and then the feature maps are written into the feature map cache module.

[0079] The feature map and weight data cache mode selection module is used to control the cache mode of writing feature maps and weight data into the input cache. The cache mode supports three convolution mode selections: normal convolution, dilated convolution, and cross-correlation convolution. According to the relationship between the convolution mode and the feature map input channel and the output channel of the convolutional neural network accelerator operation processing unit, one operation is selected from the copy and splice operations as the cache mode to expand the bit width sent to the operation processing unit.

[0080] Specifically, the selection of cache mode is used to prepare data for writing feature maps and weight caches. Specifically, in normal convolution or dilated convolution mode, if the number of output channels of the convolutional neural network accelerator operation processing unit is equal to the number of input channels of the feature map, the copy mode is selected, and specifically, the copy mode of copying zero copies of the feature map and weight data; if the number of output channels of the convolutional neural network accelerator operation processing unit is twice the number of input channels of the feature map, the copy mode is selected, and specifically, the copy mode of copying two copies of the feature map and weight data; if the number of output channels of the convolutional neural network accelerator operation processing unit is four times the number of input channels of the feature map, the copy mode is selected, and specifically, the copy mode of copying four copies of the feature map and weight data. In the cross-correlation convolution mode, for weight data, the copy mode is selected, and specifically, the copy mode of copying zero copies of the weight data, and for feature map data, the splicing mode is selected, and the n*B bit data is copied twice, and the two copies are spliced ​​into 2*n*B bit data and written into the cache.

[0081] Further, in order to better illustrate the weight data input cache described in the first and second embodiments, an example is given as follows:

[0082] like Figure 3 The weight data cache diagram is shown in FIG. The weight block cache is also composed of a BRAM group. Specifically, the weight cache is composed of 9 BRAMs with a bit width of 512 bits and a depth of 1024 (a total of 4.5 Mib). The 9 BRAMs are directly spliced ​​into 4608 bits at the output port, and the 4608-bit data of the adjacent 4 addresses are grouped into a group. A single group of weights corresponds to all weights required for a single calculation of the convolutional neural network operation processing unit. When writing weights, 512 bits are written each time in the order of the digital numbers in the figure until all weights are written; when reading weights, a 4608-bit data is read out each cycle and loaded into the convolutional neural network operation processing unit, which can complete the local weight loading of 192 convolutional neural network operation processing units, and the weight loading of all convolutional neural network operation processing units can be completed in 4 cycles. Fill the same address of each BRAM, and then jump to the next address of each BRAM and write in sequence. When reading, all weight data of the same address of 9 BRAMs are spliced ​​and read out.

[0083] To summarize the weight data input cache: the weight data input cache module is composed of Z blocks of 512-bit BRAMs with a bit width of 1024. The Z blocks of BRAM are spliced ​​into Z*512 bits at the output port, and the Z*512-bit data of t adjacent addresses are grouped into a group. A single group of weights corresponds to all weights required for a single calculation of the convolutional neural network accelerator operation processing unit. When writing weights, 512 bits are written in sequence until all weights are stored; when reading weights, a Z*512-bit data is read out each cycle and loaded into the convolutional neural network accelerator operation processing unit. All weight loading can be completed in t cycles.

[0084] The weight data input cache outputs data to the convolutional neural network operation processing unit, that is, the weight data loading adopts the loading form of local multicast. In hardware design, in order to improve the on-chip data supply bandwidth, the weight BRAM output port data is grouped and spliced, and sent to the convolutional neural network operation processing unit in the form of local multicast, that is, a single weight RAM provides weight data for several local fixed convolutional neural network accelerator operation processing units. Compared with global multicast, local multicast sacrifices a certain degree of flexibility in exchange for less logic and wiring resource overhead, while obtaining a higher loading rate.

[0085] In order to avoid the generation of unexpected parts and data in the addition tree, it is necessary to ensure that the weight data of the convolutional neural network operation processing unit that does not participate in the operation is set to 0. For this purpose, a configurable read refresh strategy can be designed as an optimal choice. Select a dual-port BRAM in read first mode, write all 0s to the current address while reading the weight data, and clear the original data at the address to achieve parallel data reading and cache refresh. The read refresh mode is optional. When the weight needs to be reused in the next calculation, you can choose to turn off the read refresh to avoid repeated loading of the weight. For example, when calculating the feature map in rows and columns, the weights corresponding to different feature map blocks are the same group. At this time, there is no need to enable the read refresh mode of the weight cache.

[0086] Further, in order to better illustrate the feature map data input cache described in the first and second embodiments, an example is given as follows:

[0087] First of all, it should be noted that weight data loading is to directly read the data in the weight data input cache to the operation processing unit. However, unlike the weight data loading process, feature map data loading requires the data read from the feature map data input cache to be converted and loaded into the operation processing unit. Therefore, compared with the hardware design of weight data, a conversion module for the feature map data required by the convolutional neural network accelerator operation processing unit is designed for feature map data.

[0088] The feature map data storage module (i.e., feature map data input cache) includes C basic storage units, each of which is composed of H BRAMs. Each W basic storage units are grouped into a local multicast group, with a total of P (P = C / W) groups, and each local multicast group provides data for a specific convolutional neural network accelerator operation processing unit.

[0089] The conversion module of the feature map data required by the convolutional neural network operation processing unit uses a hierarchical cache mechanism that cooperates with registers and distributed RAM to reduce storage resource overhead. Specifically, the input cache of the feature map data loads the multiple feature map row data required for the processing unit calculation into the row loading circuit according to the row sequence number in a time-sharing manner, wherein each feature map row data is a row of B bit data of the feature map data; one feature map data corresponds to a row loading circuit, and each row loading circuit is composed of (B / D-1) preloadable D bit shift registers and a distributed FIFO, and the low-order D bit of each row is directly written into the distributed FIFO, and the high-order (BD) bit of each row is loaded into the shift register; the data of each shift register in each row loading circuit is sequentially sent to the distributed row cache FIFO cycle by cycle, and when all FIFO water level signals are valid, the convolutional neural network operation processing unit simultaneously enables the read valid signals of all FIFOs to continuously read the feature map data in each FIFO; the input cache of the feature map data maintains continuous supply of data to each row loading circuit interface in a polling manner until the entire feature map data is traversed.

[0090] like Figure 4The conversion module of the feature map data required by the convolutional neural network operation processing unit shown in the figure. The feature map row loading circuit uses a hierarchical cache mechanism that cooperates with registers and distributed RAM to reduce storage resource overhead. The loading of each row of data in each frame is completed by 7 loadable 8-bit shift registers and an 8-bit×16 deep distributed FIFO. In FPGA, distributed FIFO is implemented using distributed RAM. Unlike block RAM (i.e. BRAM), distributed RAM occupies less LUT resources and is more suitable for applications with low data bit width and small cache depth. It is worth noting that the row cache FIFO shown in the figure only needs to load 8 pixels at a time, without the need to load the entire feature map row. The following describes the workflow of the feature map row loading circuit: first, the feature map block cache sends 64-bit data (8 pixels) to the 64-bit row loading circuit interface 1, of which the low-order 8 bits are directly written into the distributed row cache FIFO, and the high-order 56 bits are preloaded into the shift register; then, the shift register begins to send the data into the distributed row cache FIFO cycle by cycle, and at the same time, the feature map block cache begins to send data to other row loading circuit interfaces; then, when all FIFO water level signals (programming full signals) are valid, the convolutional neural network operation processing unit simultaneously enables the read valid signals of all FIFOs to continuously read the feature map data in the FIFO; finally, the feature map block cache maintains continuous supply of data to each row loading circuit interface in a polling manner until the entire feature map block is traversed.

[0091] The present invention proposes a hierarchical caching strategy for the off-chip continuous access strategy and the streaming data flow mode of the convolutional neural network accelerator operation processing unit, which makes full use of the local characteristics of feature maps and weight data when calculated in the convolutional neural network accelerator operation processing unit, improves the data reuse rate, and realizes high-bandwidth off-chip memory access and stable data supply to the on-chip operation module. Among them, regarding the above-mentioned hierarchical caching strategy, taking feature map data as an example, the off-chip data is cached in the input feature map format adaptation module, the new feature map data is cached in the input cache, and then cached in the distributed FIFO before loading into the operation processing unit; regarding the above-mentioned local characteristics, it refers to the reuse of weight data in the convolution operation process, and at this time, the weight data does not need to be repeatedly loaded, so the data reuse rate is improved; since the bit width of the off-chip memory access method of the present invention is Abit, the value of A is 512, for example, and its value does not need to be set according to the 8bit data required by the operation processing unit, it is a high-bandwidth off-chip memory access and can continuously supply data.

[0092] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for loading data required by a convolutional neural network accelerator computing processing unit, characterized in that: include: The working parameters of the convolutional neural network are written into the parameter register to complete the parameter configuration; the cache order of the weight data and the feature map data is controlled by the state machine, and after the feature map data and the weight data are completely written into the input cache of the convolutional neural network operation processing unit, the weight data and the feature map data required for each operation are alternately loaded in the operation stage of the operation processing unit; Among them, when controlling the feature map data cache, the input feature map format is first adapted, specifically: the feature map data input continuously frame by frame is segmented according to A bit to obtain multiple fields, and each field is converted into N sub-segments with a bit width of B bit, that is, N parallel data; for each frame of feature map data, the N BRAMs are written in sequence according to the field order, and the N parallel data of the same field are correspondingly written to the same address of the N BRAMs, and after each field is written to the BRAM, n addresses are jumped to continue writing the next field, and n is the total number of frames of the N BRAM cache feature map data; the n fields with the same field number of each consecutive n frames of feature map data are stored in the adjacent n addresses of the N BRAMs, and the n N parallel data are staggered between the adjacent n addresses and stored in the N BRAMs, and the writing is completed frame by frame in this way; the sub-segments with a bit width of Bbit of the same field of the consecutive n frames of feature map data are read from the N BRAMs in a reverse rotation reading method in the adjacent n fields, and the n B The bit sub-segments are concatenated into n*B bits in the order of the feature map data frame numbers.

2. The loading method according to claim 1, characterized in that: In the input feature map format adaptation, two groups of block caches are configured, each group of block caches is N BRAMs, and a ping-pong cache mechanism is used to write and read feature map data, specifically: After the first n frames of feature map data are stored in one group of block caches, the feature map data stream is switched to be written to the other group of block caches, and a read operation is performed on the data in the one group of block caches; when the other group of block caches is completely loaded with n frames of feature map data, the feature map data stream is switched to be written to the one group of block caches, and a read operation is performed on the data in the other group of block caches; in this way, feature map data are alternately input into the two groups of block caches until no feature map data is input, and a read operation is performed on the remaining data in the block caches.

3. The loading method according to claim 1, characterized in that: Before writing the weight data and the feature map data after format adaptation into the input buffer of the convolutional neural network operation processing unit, the feature map data and weight data preparation operation is performed in the following manner: According to the relationship between the convolution mode and the feature map input channel and the output channel of the convolutional neural network accelerator operation processing unit, one operation is selected from the copying and splicing operations for the feature map data and the weight data, respectively, as the cache mode for writing the corresponding data into the input cache, so as to expand the bit width sent to the operation processing unit; Specifically, in the normal convolution or dilated convolution mode, if the number of output channels of the convolutional neural network accelerator operation processing unit is equal to the number of input channels of the feature map, the copy mode is selected, and specifically, the feature map and weight data are copied zero copies respectively to complete the data preparation; if the number of output channels of the convolutional neural network accelerator operation processing unit is twice the number of input channels of the feature map, the copy mode is selected, and specifically, the feature map and weight data are copied twice respectively to complete the data preparation; if the number of output channels of the convolutional neural network accelerator operation processing unit is four times the number of input channels of the feature map, the copy mode is selected, and specifically, the feature map and weight data are copied four times respectively to complete the data preparation; In the cross-correlation convolution mode, for the weight data, the copy mode is selected, and specifically, the weight data is copied zero times to complete the preparation of the weight data. For the feature map data, the splicing mode is selected, and specifically, the n*B bit data is copied twice and spliced ​​to complete the preparation of the feature map data.

4. The loading method according to claim 1, characterized in that: The input buffer of the weight data is composed of Z BRAMs with a bit width of Abit; When writing weight data, the weight data stream is divided into 512-bit weight writing units according to the weight data sequence, and each weight writing unit is sequentially written to the same address of Z BRAMs starting from the first BRAM. After the same address of Z BRAMs is full, jump to the next address, and continue to write from the first BRAM until all weight data are stored; When the weight data is read out, all the weight data of multiple consecutive addresses of Z BRAMs are spliced ​​and read out to complete the loading of the weight data required by all convolutional neural network operation processing units.

5. The loading method according to claim 1, characterized in that: The weight data is loaded in the form of local multicast, that is, a single BRAM outputs weight data for multiple locally fixed convolutional neural network operation processing units.

6. The loading method according to claim 4, characterized in that: The BRAM in the input cache of the weight data is a dual-port BRAM in read priority mode. When reading out the weight data, all 0s can be optionally written to the current address based on whether the weight data will be reused in the next calculation. When the weight data is not reused in the next calculation, 0 is written to the current address to clear the original data at the address, thereby realizing the parallelization of data reading and cache refresh. When the weight data is reused in the next calculation, 0 is not written to the current address to retain the original data at the address to avoid repeated loading of weights.

7. The loading method according to claim 1, characterized in that: The method of loading the feature map data required for each operation of the operation processing unit is as follows: The hierarchical cache mechanism of registers and distributed RAM is used to transform the feature map data required by the convolutional neural network operation processing unit. Specifically, the input cache of the feature map data is controlled to load multiple feature map row data required for the processing unit calculation into the row loading circuit in time-sharing according to the row sequence number, wherein each feature map row data is a row of B bit data of the feature map data; one feature map data corresponds to one row loading circuit, each row loading circuit is composed of (B / D-1) preloadable D bit shift registers and a distributed FIFO, the low-order D bits of each row are directly written into the distributed FIFO, the high-order (BD) bits of each row are loaded into the shift register, and the data of each shift register in each row loading circuit is sequentially sent to the distributed FIFO cycle by cycle; when all FIFO water level signals are valid, the convolutional neural network operation processing unit simultaneously enables the read valid signals of all FIFOs to continuously read the feature map data in each FIFO; the input cache of the feature map data maintains continuous supply of data to each row loading circuit interface in a polling manner until the entire feature map data is traversed.

8. The loading method according to claim 7, characterized in that: The feature map data is loaded in the form of local multicast, that is, a single distributed FIFO outputs feature map data for multiple locally fixed convolutional neural network operation processing units.

9. A system for loading data required by a convolutional neural network accelerator computing processing unit, characterized in that: Used to execute the loading method according to any one of claims 1 to 8, comprising: an input feature map format adaptation module, an input data loading control module, a weight data input cache of a convolutional neural network operation processing unit, and a feature map data input cache of a convolutional neural network operation processing unit; The input data loading control module is used to write the working parameters of the convolutional neural network into the parameter register to complete the parameter configuration; the cache order of the weight data and the feature map data is controlled by the state machine, and after the feature map data and the weight data are completely written into the feature map data input cache and the weight data input cache of the convolutional neural network operation processing unit, the weight data and the feature map data required for each operation are alternately loaded in the operation stage of the operation processing unit; The input feature map format adaptation module is used to first perform input feature map format adaptation when the input data loading control module controls the feature map data cache.

Images