A convolutional neural network coprocessor for one-dimensional convolution

Abstract

A convolutional neural network coprocessor for one-dimensional convolution includes a central controller, on-chip RAM and a multiply-accumulator array, the multiply-accumulator array includes KoutXin multiply-accumulators, divided into Kout layers and Xin columns, each multiply-accumulator receives one channel of input feature map data and one channel of convolution kernel weight data, performs one-dimensional convolution kernel and one-dimensional input feature map convolution operation, and the same column of multiply-accumulators receives the same channel of input feature map data, each layer of multiply-accumulators corresponds to a convolution kernel, and each multiply-accumulator receives one channel of weight data of the convolution kernel corresponding to the layer where the multiply-accumulator is located, so that the multiply-accumulator array realizes parallel calculation among Xin channels of input feature maps and Kout channels of output feature maps, and the input feature map data is shared among the multiply-accumulator layers, reducing the number of memory accesses, and effectively accelerating the operation of the one-dimensional convolutional neural network.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically to a convolutional neural network coprocessor for one-dimensional convolution. Background Technology

[0002] In many engineering applications, one-dimensional signals, such as speech signals and bioelectrical signals, need to be processed. Since the voltage amplitude of speech and bioelectrical signals changes over time, they can be recorded as a one-dimensional array arranged sequentially along a time axis, thus forming a one-dimensional signal. Currently, the application of artificial intelligence (AI) is expanding, playing a crucial role in one-dimensional signal processing. For example, AI algorithms can be used for semantic analysis of speech signals; and AI algorithms can be used to classify and process collected bioelectrical signals to analyze human health information. Compared to traditional data analysis, AI demonstrates significant advantages in processing efficiency and recognition accuracy. Currently, the demand for one-dimensional signal analysis in portable devices, such as wearable devices, is growing. For instance, applying AI algorithms to portable health monitoring devices, such as smartwatches, to achieve real-time and accurate health monitoring has become a new development trend. Due to the limited battery capacity of portable devices, power consumption requirements are very stringent. AI algorithms generally involve large amounts of data and computation; using traditional general-purpose microcontrollers to perform AI algorithms results in unacceptable low computational efficiency and high power consumption. In the biomedical field, many research teams at home and abroad have proposed specialized bioelectric signal processors based on the characteristics of artificial intelligence algorithms to reduce the power consumption and latency when artificial intelligence algorithms process bioelectric signals.

[0003] Due to the strict requirements of power consumption and hardware cost for portable devices, current research mainly proposes corresponding hardware acceleration architectures based on artificial intelligence algorithms with low complexity, such as SVM (Support Vector Machine) or small-scale NN (Neural Network) algorithms. Since SVM-based bioelectric signal processors are simple to implement and have high accuracy in some applications, they have been widely used in early research. Hsu et al. proposed a low-power machine learning-assisted ECG (Electrocardiogram) processor for mobile health applications [1]. It designed a functionally switchable classification engine that can switch between SVM and MLC (Maximum Likelihood Classification). The features extracted by the front-end processing engine are input into the classification engine to complete the classification. In 2018, Zhejiang University proposed a low-power ECG processor for arrhythmia detection [2], which also included an SVM classifier. In order to reduce the complexity of the SVM classifier, principal component analysis was used for feature dimensionality reduction.

[0004] Compared with support vector machines, neural networks can achieve higher classification accuracy. Currently, CNN (Convolutional Neural Networks) is increasingly used in one-dimensional signal processing due to its strong robustness to noise and its ability to fully distinguish morphological differences in subtle signals when the data is noisy. For example, one of the most commonly used algorithms for ECG arrhythmia detection in biomedicine is CNN [3]. Since CNN algorithms are computationally intensive and involve a large number of weights, current processors still need to improve their computing speed, and a processor that can effectively accelerate CNN algorithms is needed.

[0005] References:

[0006] [1] S.-Y.Hsu, Y.Ho, P.-Y.Chang, C.Su, and C.-Y.Lee, "A 48.6-to-105.2μWMachine Learning Assisted Cardiac Sensor SoC for Mobile Healthcare Applications," IEEE Journal of Solid-State Circuits, vol.49, no.4, Art.no.4, Apr.2014.

[0007] [2]Z.Chen et al., "An Energy-Efficient ECG Processor With Weak-StrongHybrid Classifier for Arrhythmia Detection," IEEE Transactions on Circuits andSystems II: Express Briefs, vol.65, no.7, Art.no.7, Jul.2018.

[0008] [3]Shu Lih Oh, et al. "Automated beat-wise arrhythmia diagnosis using modified U-net on extended electrocardiographic recordings with heterogeneousarrhythmia types." Computers in Biology and Medicine, vol.105, pp.92-101, Dec.2018. Summary of the Invention

[0009] The main technical problem addressed by this invention is how to effectively accelerate CNN algorithms in one-dimensional signal processing.

[0010] One embodiment of the present invention provides a convolutional neural network coprocessor for one-dimensional convolution, comprising:

[0011] The central controller is used to generate data addresses and control the operation of various components in the convolutional neural network coprocessor.

[0012] On-chip RAM, which is connected to the central controller, is used to store input feature map data and convolution kernel weight data;

[0013] A multiply-accumulator array, connected to the central controller and the on-chip RAM, is used to read input feature map data and convolution kernel weight data from the on-chip RAM according to the data address, and perform convolution operation between the input feature map and the convolution kernel. The multiply-accumulator array includes Kout×Xin multiply-accumulators, where Kout and Xin are both natural numbers greater than 0, Kout is the number of layers, and Xin is the number of columns. Each multiply-accumulator is used to perform convolution operation between a single-channel one-dimensional convolution kernel and a one-dimensional input feature map.

[0014] During convolution operations, the on-chip RAM broadcasts the input feature map data and the weight data of the convolution kernel to all multiply-accumulate units in parallel. Each multiply-accumulate unit receives one channel of input feature map data, and multiply-accumulate units in the same column of different layers share one channel of input feature map data. Each layer of multiply-accumulate unit corresponds to one convolution kernel, and each multiply-accumulate unit receives the weight data of one channel of the convolution kernel corresponding to its layer.

[0015] The convolutional neural network coprocessor for one-dimensional convolution according to the above embodiment includes a central controller, on-chip RAM, and a multiply-accumulate array. The multiply-accumulate array includes Kout × Xin multiply-accumulates, divided into Kout layers and Xin columns. Each multiply-accumulate receives one channel of input feature map data and one channel of convolution kernel weight data, performing convolution operations between a single-channel one-dimensional convolution kernel and the one-dimensional input feature map. Furthermore, multiply-accumulates in the same column receive input feature map data from the same channel. Each layer of multiply-accumulates corresponds to one convolution kernel, and each multiply-accumulate receives one channel of weight data from the convolution kernel corresponding to its layer. Since the number of channels in the output feature map is the same as the number of convolution kernels, the multiply-accumulate array effectively implements the input feature map... Figure X Parallel computation between the in channels and between the output feature map and the Kout channels, while the input feature map data is shared between the multiply-accumulate layers, reduces the number of memory accesses and effectively accelerates the operation of the one-dimensional convolutional neural network. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the structure of a convolutional neural network;

[0017] Figure 2 This is a schematic diagram of the structure of a convolutional neural network coprocessor for one-dimensional convolution, according to one embodiment.

[0018] Figure 3 This is a schematic diagram of the structure of a multiply-accumulate array according to one embodiment;

[0019] Figure 4 This is a mapping diagram showing the relationship between the multiplier-accumulator and the input feature map channels and the convolution kernel channels;

[0020] Figure 5 This is a schematic diagram of the structure of a multiply-accumulate unit according to one embodiment;

[0021] Figure 6 This is a schematic diagram illustrating the one-dimensional convolution process performed by the multiplication accumulator in one embodiment;

[0022] Figure 7 This is a schematic diagram of the structure of a multiply-accumulate array according to another embodiment;

[0023] Figure 8This is a schematic diagram of the output feature map data arrangement format stored in on-chip RAM in one embodiment;

[0024] Figure 9 This is a schematic diagram of the structure of a convolutional neural network coprocessor for one-dimensional convolution, according to another embodiment.

[0025] Figure 10 This is a schematic diagram of the structure of a multiply-accumulate array according to another embodiment;

[0026] Figure 11 This is a schematic diagram of the structure of a pooling operation unit in one embodiment;

[0027] Figure 12 This describes the arrangement format of the input feature map data of the fully connected layer in one embodiment, the arrangement format after weight block division, and the correspondence between the two.

[0028] Figure 13 This is a schematic diagram illustrating the format of operation instructions in one embodiment. Detailed Implementation

[0029] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of this application. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to this application are not shown or described in the specification. This is to avoid obscuring the core parts of this application with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0030] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0031] The serial numbers assigned to components in this document, such as "first" and "second," are used only to distinguish the described objects and have no sequential or technical meaning. The terms "connection" and "linkage" used in this application, unless otherwise specified, include both direct and indirect connections (linkages).

[0032] Definitions of some terms used in this application:

[0033] 1D-CNN: One-Dimension Convolutional Neural Networks, which are convolutional neural networks whose input feature maps are one-dimensional feature maps;

[0034] MAC: multiplier-accumulator;

[0035] RAM: Random Access Memory;

[0036] DMA: Direct Memory Access;

[0037] FIFO: First In First Out.

[0038] Convolutional Neural Networks (CNNs) are robust, capable of extracting useful features from noisy signals, and have good ability to distinguish subtle changes between signals, ensuring high accuracy in multi-classification tasks. Due to these advantages, CNNs are increasingly used in one-dimensional signal processing. For example, in medicine, 1D-CNNs are increasingly applied to bioelectrical signal classification, such as myocardial ischemia detection, arrhythmia detection, and epilepsy detection.

[0039] In recent years, many research teams have proposed deep neural network accelerators to speed up CNN operations, such as Cambricon's DianNao series, the UNPU from the Korea Advanced Institute of Science and Technology (KAIST), and Tsinghua University's Thinker. However, these are designed for image processing applications, with complex hardware structures and high power consumption, making them unsuitable for portable health monitoring devices. Some research teams have proposed CNN accelerator hardware architectures for biosignal processing. National Chiao Tung University in Taiwan proposed a CNN coprocessor for real-time epilepsy detection. A RISC-V core interconnects with the coprocessor via a custom interface to configure the coprocessor and transmit data. This CNN coprocessor is controlled by a state machine composed of multiple microcontrollers. The computation of the CNN algorithm is completed by a single PE (Processing Element), and all layers of the CNN share this PE. Depending on the different computational stages, the corresponding microcontroller is selected to control the execution of the PE. This method saves hardware area but has low computational efficiency. Furthermore, since the microcontroller is determined based on the algorithm steps, this coprocessor is only suitable for accelerating CNN algorithms with fixed structures, limiting its application scenarios.

[0040] In summary, existing research on biosignal classification processors has revealed that SVM classifiers have low accuracy, while CNN accelerators are designed for specific algorithms in particular applications and cannot provide configurable CNN algorithm parameters. Furthermore, for CNN algorithms with different parameters, issues such as low computational efficiency arise.

[0041] This invention proposes a novel convolutional neural network (CNN) coprocessor that achieves a good trade-off between computational accuracy, throughput, and hardware utilization, thus addressing at least some of the aforementioned issues. Because the hardware architecture of this CNN coprocessor is designed for accelerating CNN algorithms, it maintains good accuracy across various application scenarios. To achieve higher throughput, this invention proposes a MAC array for accelerating 1D-CNN convolution operations, enabling parallel computation between input and output feature map channels. Furthermore, considering the need to adapt to convolutional kernels of different sizes, the MAC array does not perform parallel computation between kernel weights; instead, each MAC performs a single-channel one-dimensional convolution operation, avoiding the hardware resource waste that occurs when the CNN algorithm has small convolutional kernel sizes. This coprocessor can be applied to devices such as computers, smartphones, and smartwatches, connecting to the main processor (e.g., CPU) in these devices and assisting in CNN operations under the main processor's instructions.

[0042] To better understand the technical solution of this invention, a brief introduction to Convolutional Neural Networks (CNNs) is given first. Please refer to... Figure 1 Convolutional neural networks (CNNs) are mainly composed of these types of layers: convolutional layers, activation layers, pooling layers, and fully connected layers (fully connected layers are the same as those in conventional neural networks). In practical applications, convolutional layers and activation layers are often collectively referred to as convolutional layers. The feature maps input to the CNN undergo convolution and activation operations, followed by pooling operations, and finally fully connected layer operations to output the inference result. For example, for classification tasks, the inference result includes the probability of each class. Convolution, activation, and pooling operations can be performed multiple times; therefore, a CNN can include multiple convolutional layers, activation layers, and pooling layers. Figure 1 The layers are represented by n layers, where n is a natural number greater than 0. The convolutional neural network coprocessor of this invention is mainly designed for one-dimensional signals. Therefore, the input to the CNN is a one-dimensional feature map, that is, a feature map of size I_len×1, where I_len is the length of the feature map.

[0043] Please refer to Figure 2 In one embodiment of the present invention, the convolutional neural network coprocessor for one-dimensional convolution includes a central controller 1, an on-chip RAM 2, and a multiply-accumulate array (hereinafter referred to as "MAC array") 3, which are described below.

[0044] The central controller 1 receives operation instructions sent by the main processor, decodes the instructions, and generates data addresses and corresponding control signals to control the operation of each component in the convolutional neural network coprocessor, ensuring accurate data reading and writing and the execution of calculations. For example, for on-chip RAM 2 and MAC array 3, the central controller 1 controls the writing and reading of data in on-chip RAM 2 and controls MAC array 3 to perform accurate calculations.

[0045] The on-chip RAM2 is connected to the central controller 1 and is used to store the input feature map data and the weight data of the convolution kernel.

[0046] The input feature map can have multiple channels, and there can also be multiple convolutional kernels. The number of channels in each convolutional kernel is the same as the number of channels in the input feature map, and the number of channels in the output feature map is the same as the number of convolutional kernels. Here, the input and output feature maps are relative to each layer; the input feature map of each layer is different, and the output feature map of the previous layer is the input feature map of the next layer. For example, when performing convolution operations, on-chip RAM2 can store the input feature map of the convolutional layer; when performing pooling operations, on-chip RAM2 can store the input feature map of the pooling layer, which is the output feature map of the activation layer.

[0047] The multiplier-accumulator array 3 is connected to the central controller 1 and the on-chip RAM 2. It is used to read the input feature map data and the weight data of the convolution kernel from the on-chip RAM 2 according to the data address generated by the central controller 1, and to perform convolution operations between the input feature map and the convolution kernel. Please refer to [reference needed]. Figure 3 The multiply-accumulator array 3 includes Kout×Xin multiply-accumulators, where Kout and Xin are both natural numbers greater than 0. Kout is the number of layers and Xin is the number of columns. That is, the multiply-accumulator array 3 is divided into Kout layers, and each layer has Xin multiply-accumulators. Each multiply-accumulator is used to perform convolution operations between a single-channel one-dimensional convolution kernel and a one-dimensional input feature map.

[0048] The MAC array proposed in this invention is mainly used to accelerate the computation of convolutional layers in the 1D-CNN algorithm. The convolution calculation part of the 1D-CNN algorithm can be divided into 4 loops, and the calculation process is represented by pseudocode as follows:

[0049]

[0050]

[0051] In the pseudocode above, k represents the index of the output feature map, ofmap_num represents the number of channels in the output feature map, i.e., the number of convolutional kernels; y represents the index of an element in the output feature map of one channel, ofmap_size represents the length of the output feature map; x represents the index of the input feature map, ofmap_num represents the number of channels in the input feature map; i represents the index of the weights within the convolutional kernel, K_size represents the length of the convolutional kernel (i.e., the kernel size), meaning the convolutional kernel contains K_size weights; stride represents the stride, bias represents the bias value, and ReLU is the activation function.

[0052] The MAC array proposed in this invention performs parallel computation on loops 2 and 4 in the aforementioned pseudocode to accelerate the convolution operation process. The number of layers (Kout) and columns (Xin) of the MAC array 3 correspond to the parallel computation parameters after expanding loops 4 and 2 of the 1D-CNN algorithm, respectively. Kout and Xin can be set according to the characteristics and actual needs of the one-dimensional feature maps processed by the convolutional neural network, and are not limited here. The inventors found through research that many one-dimensional signals, such as bioelectrical signals, generally have a number of channels and a multiple of 4 for the number of channels in the convolution kernel. Therefore, it is preferable that Kout and Xin are multiples of 4, for example, Kout = 4, Xin = 4.

[0053] Please refer to Figure 3 During convolution operations, on-chip RAM2 broadcasts the input feature map data and convolution kernel weights to all MACs in parallel. Each MAC receives one channel of input feature map data, and MACs in the same column across different layers share the same channel of input feature map data; that is, MACs in the same column receive the same channel of input feature map data. Each MAC layer corresponds to one convolution kernel, and each MAC receives the weights of one channel of the convolution kernel corresponding to its layer. Taking Kout=4 and Xin=4 as an example, the mapping relationship between each MAC and the input feature map channels and convolution kernel channels is as follows: Figure 4 As shown, I_Cx represents the x-th channel of the input feature map, and Kk_Cx represents the x-th channel of the k-th convolutional kernel.

[0054] It can be seen that each layer has Xin MACs, which can achieve processing of input features. Figure X The parallel computation between *in* channels corresponds to the acceleration of loop 2; the Kout layer MAC array enables parallel computation between *Kout* convolutional kernels, corresponding to the acceleration of loop 4. The entire MAC array 3 can perform *Kout*×*in* single-channel one-dimensional convolutional operations in parallel. It is evident that MAC array 3 realizes the input features... Figure X Parallel computation between the in channels and between the output feature map and the Kout channels, while the input feature map data is shared between MAC layers, reduces the number of memory accesses and effectively accelerates the operation of the one-dimensional convolutional neural network.

[0055] The hardware components of a Mac are as follows Figure 5 As shown. Each MAC includes a multiplier 301, an adder 302, a third multiplexer 303, a partial sum register 304, a fourth multiplexer 305, and an output register 306. The first and second input terminals of multiplier 301 are used to input weight data and feature map data, respectively, and its output terminal is connected to the first input terminal of adder 302. The second input terminal of adder 302 is connected to the output terminal of third multiplexer 303, and its output terminal is connected to the input terminal of partial sum register 304. The first input terminal of third multiplexer 303 is connected to the output terminal of partial sum register 304, its second input terminal can be set to 0, and its strobe terminal SEL0 is connected to central controller 1. Partial sum register 304 is used to temporarily store the partial sum of the intermediate process of single-channel one-dimensional convolution operation. The first input terminal of fourth multiplexer 305 is connected to the output terminal of output register 306, its second input terminal is connected to the output terminal of partial sum register 304, and its strobe terminal SEL1 is connected to central controller 1. The input terminal of output register 306 is connected to the output terminal of fourth multiplexer 305, and the convolution operation result Psum (which is also the partial sum of the entire MAC layer) is stored in output register 306 for output to proceed to the next calculation.

[0056] The following example uses a 5*1 input feature map and a 3*1 convolution kernel to illustrate the detailed calculation process of MAC. This single-channel one-dimensional convolution process is as follows: Figure 6As shown, the stride of the convolution is 1, the kernel line is one channel of the convolution kernel, and the input feature map line is one channel of the input feature map. In the first clock cycle after the convolution begins, the central controller 1 sets the strobe SEL0 = 1 and SEL1 = 0, and the MAC calculates psum0 = 0 + A1 * W1 and stores it in the partial sum register 304; in the second clock cycle, the central controller 1 sets the strobe SEL0 = 0 and SEL1 = 0, and the MAC calculates psum1 = psum0 + A2 * W2 and stores it in the partial sum register 304; in the third clock cycle, the central controller 1 sets the strobe SEL0 = 0 and SEL1 = 1, and the MAC calculates psum2 = p sum1 + A3 * W3 = P1, and the obtained single-channel convolution feature value P1 is updated to the output register 306 for output to the next component for the next calculation. Then the convolution kernel needs to slide by stride 1 to calculate the next feature value P2. Therefore, in the fourth clock cycle, the central controller 1 sets the strobe terminal SEL0 = 1 and SEL1 = 0 again, and the MAC calculates psum0 = 0 + A2 * W1. Subsequent calculations are carried out in the same way until the calculation of feature value P2 is completed. The calculation of feature value P3 is the same as the above process, and finally the complete partial sum Psum is obtained. As can be seen, every K_size clock cycles, the central controller 1 sets SEL0 to 1 to start the calculation of a feature value, and then sets SEL0 to 0 to accumulate partial sums to calculate the feature value. The partial sums that need to be accumulated are stored in the partial sum register 304. In addition, every K_size clock cycles, the central controller 1 sets SEL1 to 1 to update the feature value output by the partial sum register 304 into the output register 306 for the next calculation, and then sets SEL1 to 0 to keep the value in the output register 306 unchanged.

[0057] The convolution results of all MACs in each layer need to be summed, and usually a bias value also needs to be added, before activation operations are performed. Please refer to [reference needed]. Figure 7 and Figure 9 In one embodiment, the MAC array 3 further includes an adder tree 31 and an activation operation unit 32. Each MAC layer has an adder tree 31, which is connected to each MAC layer to accumulate the convolution operation results of all MAC layers in that layer to obtain a convolution partial sum. For example, if there are three MAC layers and the three convolution operation results are [P11,P12,P13], [P21,P22,P23], and [P31,P32,P33], then the convolution partial sum obtained after accumulation by the adder tree 31 is [P11+P21+P31,P12+P22+P32,P13+P23+P33].

[0058] The on-chip RAM2 is also used to store the bias values ​​of the convolution kernels and the output feature map data. The adder tree 31 of each layer is also connected to the on-chip RAM2 to obtain the bias value of the convolution kernel corresponding to that layer from the on-chip RAM2, and add the convolution part to the bias value to obtain the output feature map data to be activated.

[0059] The activation unit 32 is connected to the adder tree 31 and on-chip RAM 2. It performs function activation processing on the output feature map data to be activated to obtain the output feature map data, which is then written into on-chip RAM 2, awaiting the next layer's computation. The output feature map data to be activated in each MAC layer becomes the output feature map data for one channel after function activation processing. The Kout layer can obtain Kout channels of output feature map data. The activation operation can use the ReLU function as the activation function.

[0060] Since the MAC array 3 can only perform convolution operations on Kout convolutional kernels and Xin channels of each kernel in one round of computation, block or group computation is required in the case of multiple convolutional kernels and multiple channels. Specifically, when the number of channels Chi of the input feature map is greater than Xin and / or the number of convolutional kernels Cho is greater than Kout, the input feature map data can be divided into Chi / Xin groups, each group including the input feature map data of Xin channels, and the weight data of the convolutional kernels can be divided into Cho / Kout groups, each group including the weight data of Kout convolutional kernels, where Chi and Cho are both natural numbers greater than 0.

[0061] During convolution operations, MAC array 3 reads each set of weight data sequentially from on-chip RAM 2. For each set of weight data, MAC array 3 reads each set of input feature map data sequentially from on-chip RAM 2 and performs convolution operations with that set of weight data respectively. A total of (Cho / Kout)×(Chi / Xin) rounds of convolution operations are performed to complete the convolution operations between the input feature map data of all channels and all convolution kernels.

[0062] Please refer to Figure 7To accommodate block or grouped computations, each adder tree 31 can also be connected to a FIFO memory 33, which temporarily stores the convolutional partial sums. In each round of convolution, the convolutional partial sum obtained by the adder tree 31 is added to the convolutional partial sum stored in the FIFO memory 33, and then rewritten to the FIFO memory 33 to update the convolutional partial sum in the FIFO memory 33. This process is repeated in the next round of convolution. Whenever a set of weight data completes its convolution with all input feature map data sets, the FIFO memory 33 accumulates the partial sums of all channels. The adder tree 31 retrieves the bias value of the convolution kernel corresponding to its layer from the on-chip RAM 2, adds the convolutional partial sum in the FIFO memory 33 to this bias value to obtain the output feature map data to be activated, and then clears the FIFO memory 33. The output feature map data to be activated is processed by the activation operation unit 32 to obtain the output feature map data. Afterwards, the MAC array 3 continues to read the next set of weight data and performs the same processing.

[0063] Taking a convolutional layer with Kout=4, Xin=4, and the following parameters as an example: input feature map channels Chi=8, length I_len, number of convolutional kernels (i.e., output feature map channels) Cho=16, kernel size K_size, and stride=1. The input feature map data is divided into 2 groups, each containing 4 channels of input feature map data. The kernel weights are divided into 4 groups, each containing 4 kernel weights. This convolutional layer will complete the computation in 8 rounds. The first round of computation completes the calculation of the first set of convolutional kernels and the first set of input feature map data. The resulting convolutional partial sum is stored in FIFO memory 33. The second round of computation completes the calculation of the first set of convolutional kernels and the second set of input feature map data. At this time, FIFO memory 33 has accumulated the partial sums of all channels. The activated operation unit 32 stores the output feature map in on-chip RAM 2. Since there are 4 convolutional kernels, output feature maps of 4 channels can be obtained, namely the output feature maps of channels 1-4. Similarly, the third and fourth rounds of computation complete the calculation of the output feature maps of channels 5-8. The calculation of the remaining rounds follows the same pattern. After 8 rounds of computation are completed, the output feature maps of all channels are stored in on-chip RAM 2, and the operation of this convolutional layer ends.

[0064] Since most one-dimensional signals have relatively small input data volumes and a small number of convolutional layer parameters, the output feature maps of the convolutional layer can be temporarily stored in on-chip RAM2 without transferring data to external memory. If the convolutional neural network coprocessor performs operations based on 16-bit fixed-point numbers, and on-chip RAM2 has a bandwidth of 64 bits, then according to the above-mentioned grouping calculation strategy, the data arrangement format of the 16-channel output feature maps of this convolutional layer stored in on-chip RAM2 is as follows: Figure 8As shown, ofmap_cx represents the x-th channel of the output feature map. It can be seen that the output feature maps of channels 1-4 occupy the first I_len-K_size+1 64-bit storage space of the cache. The output feature maps of the remaining channels are also written into on-chip RAM2 in a manner where every 4 channels occupy I_len-K_size+1 64-bit storage space. The output feature maps temporarily stored in on-chip RAM2 can be directly sent to the next component for operation, such as directly to the pooling operation unit to perform downsampling operations. This method avoids redundant read and write operations to external memory, improves computational efficiency, and reduces the power consumption of reading and writing memory.

[0065] Please refer to Figure 9 and Figure 10 In one embodiment, the on-chip RAM 2 includes an input RAM 21, a weight RAM 22, a bias RAM 23, and an output RAM 24.

[0066] Input RAM 21 is connected to MAC array 3 and is used to store input feature map data and broadcast input feature map data to all MACs in parallel. Specifically, input RAM 21 can be used to store input feature map data before each convolution operation and to broadcast input feature map data to all MACs during the convolution operation.

[0067] The weight RAM 22 is connected to the MAC array 3 and is used to store the weight data of the convolution kernels and to broadcast the weight data of the convolution kernels to all MACs in parallel. The weight RAM 22 includes Kout partitions, with one partition corresponding to each MAC layer. Each partition stores the weight data of all channels of a convolution kernel, and each partition is used to broadcast the stored weight data to the MAC of the corresponding layer.

[0068] Figure 9 and Figure 10 Taking Kout=4 and Xin=4 as an example, then weight RAM22 includes 4 partitions, denoted as Weight RAM[0], Weight RAM[1], Weight RAM[2] and Weight RAM[3] respectively. Figure 10As shown, if the convolutional neural network coprocessor operates on 16-bit fixed-point numbers, each partition has a 64-bit output bandwidth. During convolution, each partition broadcasts the weight data of four channels within a convolution kernel to the MAC of its layer in parallel, and each MAC receives the weight data of one channel. Similarly, the input RAM21 also has a 64-bit output bandwidth, broadcasting the input feature map data of four channels in parallel to all MACs. MACs in the same column of different layers share the input feature map data of one channel.

[0069] The bias RAM 23 is connected to the adder tree 31 and is used to store the bias values ​​of each convolution kernel.

[0070] The output RAM 24 is connected to the activation operation unit 32 and is used to store the output feature map data.

[0071] Please refer to Figure 9 In one embodiment of the present invention, the convolutional neural network coprocessor for one-dimensional convolution further includes a pooling operation unit 4. The pooling operation unit 4 is connected to the input RAM 21 and the output RAM 24, and is responsible for pooling layer operations. It performs pooling operations on the output feature map data stored in the output RAM 24 and stores it in the input RAM 21 as input feature map data for the next convolution operation or fully connected layer operation. The pooling operation performed by the pooling operation unit 4 can be max pooling, average pooling, etc.

[0072] One embodiment of the present invention provides a pooling operation unit for performing max pooling. Please refer to [link / reference]. Figure 11 In this embodiment, the pooling operation unit 4 is mainly composed of a comparator CMP to complete the max pooling operation.

[0073] The number of comparators C_num in pooling operation unit 4 can be determined according to actual needs. For ease of explanation, the following explanation uses 4 comparators CMP as an example. Figure 11As shown, in one embodiment, the signal ports of the comparator CMP include a clock terminal CLK, a control terminal CE, a strobe terminal SEL, an input terminal A, and an output terminal Y, with each input / output terminal having a bit width of 16 bits. Assuming the output RAM 24 has an output bandwidth of 4*16 bits, it can output four 16-bit output feature map data from different channels in parallel. These data are fed into four 16-bit comparators CMP to perform max pooling operations. Each comparator CMP performs pooling operations on the output feature map data of one channel. The number of comparisons required for the comparator CMP to output a feature value is equal to the pooling size P_size. When SEL = 0, A is compared with 0, and the value of A is assigned to Y; when SEL = 1, A is compared with Y, and the larger value is assigned to Y. In every P_size clock cycles, SEL = 0 in the first clock cycle and SEL = 1 in subsequent clock cycles. Every P_size clock cycles, the comparator CMP outputs a feature value and writes it into the input RAM 21.

[0074] During pooling operations, the input RAM21 is converted into an output buffer for the pooling layer operations. The output feature map after pooling is temporarily stored in the input RAM21 as the input feature map for the next convolution operation or fully connected layer operation. This also helps to avoid redundant read and write operations to external memory, improves computational efficiency, and reduces power consumption.

[0075] Similarly, for pooling layer operations, if the number of channels in the output feature map stored in the output RAM24 is greater than C_num, block operations are also required, i.e., divided into Cho / C_num blocks, and Cho / C_num rounds of pooling operations are performed, with each round pooling operation on C_num channels. The data arrangement format in the input RAM21 is similar to that in the output RAM24. For example, if the output feature map data arrangement format in the output RAM24 is... Figure 8 As shown, the arrangement format of the output feature map data in input RAM21 after pooling operation is also as follows. Figure 8 As shown, the only difference is the amount of storage space they occupy.

[0076] The MAC array of this invention can also be used to accelerate the computation of fully connected layers in 1D-CNN algorithms. For fully connected layer computation, it is typically necessary to flatten the output feature map after pooling in the upper layer into a one-dimensional feature vector, and then perform an inner product operation with the weights of the fully connected layer. Performing fully connected operations according to this rule is equivalent to performing convolution operations on a single-channel input feature map, which will result in a significant waste of the output bandwidth of on-chip RAM2, and a large portion of the MAC array will be idle. To solve this problem, this invention proposes a block-based computation method for fully connected layers.

[0077] When performing fully connected layer operations, firstly, the input feature map stored in input RAM21 after pooling is not flattened and its original storage arrangement is maintained; instead, the weights of the fully connected layer are stored in blocks, and the number of blocks is equal to the number of channels of the upper layer output feature map, that is, the number of channels Chi of the input feature map data in input RAM21.

[0078] Specifically, the weight data corresponding to each output feature value of the fully connected layer needs to be divided into Chi blocks. Then, the weight data of the fully connected layer is used as the weight data of the convolution kernel. That is, the weight data corresponding to each output feature value is used as the weight data of a convolution kernel, and each block is used as a channel of the convolution kernel. Assuming that the fully connected layer has K output feature values, we can obtain the weight data of K convolution kernels. The weight data of these K convolution kernels is stored in weight RAM22, where K is a natural number greater than 0.

[0079] The MAC array 3 reads input feature map data from input RAM 21 and weight data from weight RAM 22, performs convolution operation between the input feature map and the K convolution kernels, thereby completing the operation of the fully connected layer, obtaining the inference result, and storing the inference result in output RAM 24.

[0080] Assuming the pooling layer output feature map has 8 channels and a length of I_len, the weights corresponding to one output feature value are also divided into 8 blocks for storage. Each block contains weights equal to I_len, which is equivalent to cutting the weight vector into an 8-channel convolutional kernel. The fully connected weights corresponding to the remaining output feature values ​​are also divided into blocks in the same way. Figure 12 The diagram illustrates the storage of the fully connected weights after the first output feature value is divided into blocks in Weight RAM[0], where fcw_cx represents the x-th block, and each block corresponds to a channel of the input feature map. After this block division, the computation of the fully connected layer is equivalent to performing a convolution operation between an input feature map with 8 channels and a length I_len and a convolution kernel of length I_len. This equivalent substitution process can be expressed by the formula:

[0081]

[0082] Where OUT[0] represents the first output feature value, ifmap[i] represents the i-th feature value of the input feature map, fcw[i] represents the i-th weight corresponding to the first output feature value, ifmap[chi] represents the i-th channel of the input feature map, fcw[chi] represents the i-th block of the weight data corresponding to the first output feature value, and i is a natural number greater than 0.

[0083] The difference from convolutional layers is that the convolution kernel and the input feature map have the same length, I_len. In this case, the SEL signal of the multiplexer 302 controls the MAC to sum the intermediate parts of the convolution operation I_len-1 times, and then the result is fed into the adder tree 31. If Kout = 4, as can be seen from the grouped calculations of the previous convolutional layer operations, this output feature value requires two rounds of calculation. The first round completes the convolution of channels 1-4, and the second round completes the convolution of channels 5-8. By using this block-based strategy, the entire MAC array 3 can perform parallel calculations of Kout output feature values, improving the efficiency of the fully connected layer operations.

[0084] It should be noted that, Figure 8 , Figures 10 to 12 The example given is that the convolutional neural network coprocessor can process 16-bit data and the on-chip RAM2 has a 64-bit input / output bandwidth. This is not an example of limitation, and other bit lengths and bandwidths can be set according to actual needs.

[0085] Please refer to Figure 9 In one embodiment of the present invention, the convolutional neural network coprocessor for one-dimensional convolution further includes a direct memory access module (DMA) 5, a first multiplexer (Switch 1) 6, and a second multiplexer (Switch 2) 7. The pooling operation unit 4 includes an output terminal and an input terminal.

[0086] The direct storage access module 5 is used to read input feature map data, weight data and bias values ​​from external memory and write them into input RAM 21, weight RAM 22 and bias RAM 23 respectively. It is also used to write the inference results stored in output RAM 24 back to external memory.

[0087] The first multiplexer 6 includes an input terminal, a first output terminal, and a second output terminal. The input terminal is connected to the output RAM 24, the first output terminal is connected to the direct memory access module 5, and the second output terminal is connected to the input terminal of the pooling operation unit 4.

[0088] The second multiplexer 7 includes a first input terminal, a second input terminal, and an output terminal. The first input terminal is connected to the output terminal of the pooling operation unit 4, the second input terminal is connected to the direct memory access module 5, and the output terminal is connected to the input RAM 21.

[0089] The direct memory access module 5 can be connected to the weighted RAM 22, the bias RAM 23, the first output of the first multiplexer 6, and the second input of the second multiplexer 7 via the data bus.

[0090] When the convolutional neural network coprocessor reads input feature map data from external memory, the second input of the second multiplexer 7 is turned on, so that the direct memory access module 5 writes the input feature map data into the input RAM 21. When performing pooling operations, the second output of the first multiplexer 6 and the first input of the second multiplexer 7 are turned on, so that the pooling operation unit 4 reads the output feature map data from the output RAM 24, performs pooling operations, and stores it into the input RAM 21. When outputting the inference result, the first output of the first multiplexer 6 is turned on, so that the direct memory access module 5 reads the inference result from the output RAM 24 and writes it back to the external memory.

[0091] As can be seen, the input RAM 21 has two data sources. During the stage of reading input feature map data from external memory, the data source is the data bus connected to the direct memory access module 5, which writes data to the input RAM 21 via the data bus. During the pooling layer operation stage, the data source is the pooling operation unit 4. The output RAM 24 outputs data in two directions. During the pooling layer operation stage, data is output to the pooling operation unit 4; during the inference result output stage, data is output to the data bus connected to the direct memory access module 5, which writes the inference result back to the external memory. That is, during the pooling operation, the functions of the input RAM 21 and the output RAM 24 are interchanged. The output RAM 24, which stores the output feature map of the convolutional layer operation, provides the input feature map in the pooling layer operation, while the output feature map of the pooling layer operation is stored in the input RAM 21. This fully utilizes both the input RAM 21 and the output RAM 24, reducing the number of read / write operations to external memory, improving computational efficiency, and reducing the power consumption of reading and writing to memory.

[0092] Please refer to Figure 9 In one embodiment of the present invention, the convolutional neural network coprocessor for one-dimensional convolution further includes an instruction memory (INST MEM) 8. The instruction memory 8 is used to store operation instructions, which include opcodes and operation parameters. The opcodes are used to identify the instruction type, which includes input instructions, convolution operation instructions, pooling operation instructions, fully connected layer operation instructions, and output instructions. The central controller 1 is also connected to the instruction memory 8 and is used to read operation instructions from the instruction memory 8, identify the instruction type through the opcode and decode it, and perform corresponding control operations according to the operation parameters.

[0093] To achieve high flexibility and configurability in running the 1D-CNN algorithm on the convolutional neural network coprocessor of this invention, a set of operation instructions is defined based on the algorithm's characteristics. These instructions are used to precisely control the computation of each layer. Please refer to... Figure 13An embodiment of the present invention includes operation instructions including an input instruction (INPUT), a convolution operation instruction (CONV), a pooling operation instruction (POOL), a fully connected layer operation instruction (FC), and an output instruction (OUTPUT). The opcode length can be 3 bits, and the length of the operation parameters is variable. Each operation instruction may include multiple operation parameters, the length and number of which vary depending on the type of operation instruction. The following describes the process in conjunction with... Figure 13 Each operation instruction is explained in detail.

[0094] The operation parameters for input instructions include the starting address D_addr of the input feature map data in external memory, the length I_len of the input feature map, and the number of channels Chi of the input feature map. The operation parameters for convolution operation instructions include the starting address W_addr of the weight data in external memory, the length I_len of the input feature map, the number of channels Chi of the input feature map, the length K_size of the convolution kernel, and the number of channels Cho of the output feature map. The operation parameters for pooling operation instructions include the length I_len of the input feature map, the number of channels Chi of the input feature map, and the pooling size P_size. The operation parameters for fully connected layer operation instructions include the starting address W_addr of the weight data in external memory, the length I_len of the input feature map, and the length O_len of the output feature map. The operation parameters for output instructions include the write address D_addr of the output feature map data in external memory, the length O_len of the output feature map, and the number of channels Cho of the output feature map.

[0095] The 1D-CNN algorithm to be executed is first mapped into an instruction stream and a data stream consisting of the aforementioned operation instructions, and then the main processor writes the instructions into the instruction memory 8. When executing the 1D-CNN algorithm, the central controller 1 reads the operation instructions from the instruction memory 8, identifies the instruction type through the opcode, and decodes it.

[0096] When an input instruction is read, the central controller 1 controls the second input terminal of the second multiplexer 7 to be turned on, so that the input port of the input RAM 21 is connected to the data bus. According to the operation parameters of the input instruction, the direct storage access module 5 is controlled to read the input feature map data from the external memory and write it into the input RAM 21.

[0097] When a convolution operation instruction is read, the central controller 1 controls the direct memory access module 5 to read the weight data and bias value of the convolution kernel from the external memory according to the operation parameters of the convolution operation instruction, and writes them into the weight RAM 22 and bias RAM 23 respectively. During the reading, the operation parameters need to be sent to the direct memory access module 5. Then, the central controller 1 generates the data addresses of the weight RAM 22 and bias RAM 23 according to the operation parameters of the convolution operation instruction, and controls the MAC array 3 to read the input feature map data from the input RAM 21, the weight data from the weight RAM 22, and the bias value from the bias RAM 23 to perform convolution operation to obtain the output feature map data, and writes the output feature map data into the output RAM 24.

[0098] When a pooling operation instruction is read, the central controller 1 controls the second output terminal of the first multiplexer 6 and the first input terminal of the second multiplexer 7 to be turned on, so that the input port of the input RAM 21 is connected to the output terminal of the pooling operation unit 4, and the output port of the output RAM 24 is connected to the input terminal of the pooling operation unit 4. Then, according to the operation parameters of the pooling operation instruction, the pooling operation unit 4 is controlled to read the output feature map data in the output RAM 24, perform pooling operation, and store it in the input RAM 21.

[0099] When a fully connected layer operation instruction is read, the central controller 1 controls the direct memory access module 5 to read the weight data of the fully connected layer from the external memory and write it into the weight RAM 22 according to the operation parameters of the fully connected layer operation instruction. During the reading, the operation parameters need to be sent to the direct memory access module 5. Then, the data address of the weight RAM 22 is generated according to the operation parameters of the fully connected layer operation instruction, and the MAC array 3 is controlled to read the input feature map data from the input RAM 21 and the weight data from the weight RAM 22 to perform convolution operation to obtain the inference result. The inference result is then written into the output RAM 24.

[0100] When an output instruction is read, the central controller 1 controls the first output terminal of the first multiplexer 6 to be turned on, so that the output port of the output RAM 24 is connected to the data bus, and controls the direct memory access module 5 to read the inference result from the output RAM 24 and write it back to the external memory. When writing, the operation parameters need to be sent to the direct memory access module 5.

[0101] It should be noted that, Figure 13 The number of bits for the opcode and operation parameters is shown, but this is only one implementation method and is not limited to this. Other numbers can be set according to actual needs.

[0102] Please refer to Figure 9In one embodiment, the instruction memory 8 is connected to the main processor via an AXI-Lite bus. The 1D-CNN algorithm to be run is first mapped into an instruction stream and a data stream consisting of the above-mentioned operation instructions. Then, the main processor writes the instructions to the instruction memory 8 via the AXI-Lite bus. The direct memory access module 5 is connected to the external memory via the AXI bus. The direct memory access module 5 writes the inference results to a specified address in the external memory via the AXI bus.

[0103] According to the above embodiment of the convolutional neural network coprocessor for one-dimensional convolution, since the MAC array includes Kout×Xin MACs, divided into Kout layers and Xin columns, each MAC receives one channel of input feature map data and one channel of convolution kernel weight data, performing convolution operation between a single-channel one-dimensional convolution kernel and a one-dimensional input feature map. Furthermore, MACs in the same column receive input feature map data from the same channel. Each MAC layer corresponds to one convolution kernel, and each MAC receives one channel of weight data from the convolution kernel corresponding to its layer. Therefore, the input feature map is realized. Figure X Parallel computation between the in channels and between the output feature map and the Kout channels, while the input feature map data is shared between MAC layers, effectively accelerates the operation of the one-dimensional convolutional neural network, reduces the number of memory accesses, improves computational efficiency, and reduces the power consumption of reading and writing memory.

[0104] Meanwhile, this invention makes a trade-off between computational throughput and hardware cost. Each MAC can complete a one-dimensional convolution operation in a single channel without using parallel operations between convolution kernel weights. It can adapt to convolution kernels of different sizes, avoid the situation of idle hardware resources when the convolution kernel size is small, and reduce the waste of hardware resources.

[0105] In one embodiment, for convolution operations with a large number of input channels and a large number of convolution kernels, the MAC array can accelerate the entire convolution process by using partial and temporary storage and block computation methods to complete the entire convolution process in multiple rounds. This allows the hardware scale of the MAC array to be smaller, enabling the completion of large-scale algorithms with limited hardware and reducing hardware costs.

[0106] In one embodiment, by swapping the functions of the input RAM and the output RAM during the pooling operation, the number of read and write operations to the external memory is reduced, the operation efficiency is improved, and the power consumption of reading and writing to the memory is reduced.

[0107] In one embodiment, the weight data of the fully connected layer is divided into blocks and converted into convolution operations. Parallel computation is then performed using the MAC array of the present invention, thereby accelerating the operation of the fully connected layer and making full use of hardware resources.

[0108] The hardware architecture of the convolutional neural network coprocessor of this invention is designed to accelerate CNN algorithms, thus maintaining good accuracy in various application scenarios. Furthermore, in some embodiments of this invention, the convolutional neural network coprocessor can be made compatible with CNN algorithms in different application scenarios in two ways: First, the computational parameters are configurable in the form of operation instructions. By sending operation instructions to the convolutional neural network coprocessor, some parameters during computation, such as the convolutional kernel size, the number of input feature map channels, and the pooling size, can be adjusted, enabling compatibility with CNN algorithms with different parameters. Second, for larger-scale CNN algorithms, the computation can be completed in a multi-round block computation manner, thereby greatly increasing the flexibility and applicability of the coprocessor.

[0109] The above examples illustrate the present invention only to aid in understanding it and are not intended to limit the scope of the invention. Those skilled in the art can make various simple deductions, modifications, or substitutions based on the principles of this invention.

Claims

1. A convolutional neural network coprocessor for one-dimensional convolution, characterized in that, include: The central controller is used to generate data addresses and control the operation of various components in the convolutional neural network coprocessor. On-chip RAM, which is connected to the central controller, is used to store input feature map data and convolution kernel weight data; A multiply-accumulator array, connected to the central controller and the on-chip RAM, is used to read input feature map data and convolution kernel weight data from the on-chip RAM according to the data address, and perform convolution operation between the input feature map and the convolution kernel. The multiply-accumulator array includes Kout×Xin multiply-accumulators, where Kout and Xin are both natural numbers greater than 0, Kout is the number of layers, and Xin is the number of columns. Each multiply-accumulator is used to perform convolution operation between a single-channel one-dimensional convolution kernel and a one-dimensional input feature map. During convolution operations, the on-chip RAM broadcasts the input feature map data and the weight data of the convolution kernel to all multiply-accumulate units in parallel. Each multiply-accumulate unit receives one channel of input feature map data, and multiply-accumulate units in the same column of different layers share one channel of input feature map data. Each layer of multiply-accumulate unit corresponds to one convolution kernel, and each multiply-accumulate unit receives the weight data of one channel of the convolution kernel corresponding to its layer.

2. The convolutional neural network coprocessor as described in claim 1, characterized in that, The on-chip RAM is also used to store the bias values ​​of the convolution kernels and the output feature map data; The multiply-accumulator array also includes an adder tree and activation operation units, wherein each layer of multiply-accumulator has an adder tree, and the adder tree of each layer is connected to each multiply-accumulator of that layer, for accumulating the convolution operation results of all multiply-accumulators in that layer to obtain the convolution part sum; The adder tree of each layer is also connected to the on-chip RAM to obtain the bias value of the convolution kernel corresponding to that layer from the on-chip RAM, and add the convolution part to the bias value to obtain the output feature map data to be activated; The activation operation unit is connected to the adder tree and the on-chip RAM, and is used to perform function activation processing on the output feature map data to be activated to obtain the output feature map data, and write it into the on-chip RAM.

3. The convolutional neural network coprocessor as described in claim 2, characterized in that, Each adder tree is connected to a FIFO memory; When the number of channels Chi of the input feature map is greater than Xin and / or the number of convolutional kernels Cho is greater than Kout, the input feature map data is divided into Chi / Xin groups, each group including the input feature map data with Xin channels, and the weight data of the convolutional kernels is divided into Cho / Kout groups, each group including the weight data of Kout convolutional kernels, where Chi and Cho are both natural numbers greater than 0. The multiply-accumulator array reads each set of weight data sequentially from the on-chip RAM. For each set of weight data, the multiply-accumulator array reads each set of input feature map data sequentially from the on-chip RAM and performs convolution operations with the set of weight data respectively, thereby completing the convolution operation between the input feature map data of all channels and all convolution kernels. The FIFO memory is used to temporarily store the convolutional partial sums. In each round of convolution operation, the convolutional partial sum obtained by the adder tree is added to the convolutional partial sum stored in the FIFO memory, and then written back to the FIFO memory to update the convolutional partial sum in the FIFO memory. Whenever a set of weight data completes the convolution operation with all input feature map data sets, the adder tree obtains the bias value of the convolution kernel corresponding to its layer from the on-chip RAM, adds the convolutional partial sum in the FIFO memory to the bias value to obtain the output feature map data to be activated, and then clears the FIFO memory.

4. The convolutional neural network coprocessor as described in claim 2 or 3, characterized in that, The on-chip RAM includes input RAM, weight RAM, bias RAM, and output RAM; The input RAM is connected to the multiply-accumulate array and is used to store input feature map data and broadcast the input feature map data to all multiply-accumulates in parallel. The weight RAM is connected to the multiply-accumulate array and is used to store the weight data of the convolution kernel and to broadcast the weight data of the convolution kernel to all multiply-accumulates in parallel. The weight RAM includes Kout partitions, each multiply-accumulate layer has a corresponding partition, each partition stores the weight data of one convolution kernel, and each partition is used to broadcast the stored weight data to the multiply-accumulate layer of the corresponding layer. The bias RAM is connected to the adder tree and is used to store the bias values ​​of the convolution kernel; The output RAM is connected to the activation operation unit and is used to store output feature map data.

5. The convolutional neural network coprocessor as described in claim 4, characterized in that, It also includes a pooling operation unit, which is connected to the input RAM and the output RAM. The pooling operation unit is used to perform pooling operations on the output feature map data stored in the output RAM and store it in the input RAM as input feature map data for the next convolution operation or fully connected layer operation.

6. The convolutional neural network coprocessor as described in claim 5, characterized in that, The weight RAM is also used to store the weight data of the fully connected layer during fully connected layer operations; When performing fully connected layer operations, the weight data corresponding to an output feature value of the fully connected layer is divided into Chi blocks, where Chi is the number of channels of the input feature map data stored in the input RAM. The weight data corresponding to each output feature value is used as the weight data of a convolution kernel, and each block is used as a channel of the convolution kernel. The weight data of the resulting K convolution kernels are stored in the weight RAM, where K is a natural number greater than 0, representing the number of output feature values ​​of the fully connected layer. The multiply-accumulator array reads input feature map data from the input RAM and weight data from the weight RAM, performs convolution operations between the input feature map and the K convolution kernels, thereby completing the operation of the fully connected layer and obtaining the inference result, which is stored in the output RAM.

7. The convolutional neural network coprocessor as described in claim 6, characterized in that, It also includes a direct memory access module, a first multiplexer, and a second multiplexer; the pooling operation unit includes an output terminal and an input terminal; The direct storage access module is used to read input feature map data, weight data and bias value from external memory and write them into the input RAM, the weight RAM and the bias RAM respectively. It is also used to write the inference result stored in the output RAM back to the external memory. The first multiplexer includes an input terminal, a first output terminal, and a second output terminal. The input terminal of the first multiplexer is connected to the output RAM, the first output terminal is connected to the direct memory access module, and the second output terminal is connected to the input terminal of the pooling operation unit. The second multiplexer includes a first input terminal, a second input terminal, and an output terminal. The first input terminal is connected to the output terminal of the pooling operation unit, the second input terminal is connected to the direct memory access module, and the output terminal of the second multiplexer is connected to the input RAM. When reading input feature map data from external memory, the second input of the second multiplexer is turned on, so that the direct memory access module writes the input feature map data into the input RAM; During pooling operations, the second output of the first multiplexer and the first input of the second multiplexer are connected, so that the pooling operation unit reads the output feature map data in the output RAM, performs pooling operations, and stores it in the input RAM. When outputting the inference result, the first output terminal of the first multiplexer is turned on, so that the direct memory access module reads the inference result from the output RAM and writes it back to the external memory.

8. The convolutional neural network coprocessor as described in claim 7, characterized in that, It also includes an instruction memory for storing operation instructions, which include an opcode and operation parameters. The opcode is used to identify the instruction type, which includes input instructions, convolution operation instructions, pooling operation instructions, fully connected layer operation instructions, and output instructions. The central controller is used to read operation instructions from the instruction memory, identify the instruction type by the opcode and decode it, and perform corresponding control operations according to the operation parameters.

9. The convolutional neural network coprocessor as described in claim 8, characterized in that, The operation parameters of the input instruction include the starting address of the input feature map data in the external memory, the length of the input feature map, and the number of input feature map channels; The operation parameters of the convolution operation instruction include the starting address of the weight data in the external memory, the length of the input feature map, the number of input feature map channels, the length of the convolution kernel, and the number of output feature map channels; The operation parameters of the pooling operation instruction include the input feature map length, the number of input feature map channels, and the pooling size; The operation parameters of the fully connected layer operation instructions include the starting address of the weight data in the external memory, the length of the input feature map, and the length of the output feature map. The operation parameters of the output instruction include the write address of the output feature map data in the external memory, the length of the output feature map, and the number of output feature map channels; When the input instruction is read, the central controller controls the second input terminal of the second multiplexer to be turned on, and controls the direct memory access module to read input feature map data from external memory and write it into the input RAM according to the operation parameters of the input instruction; When the convolution operation instruction is read, the central controller controls the direct memory access module to read the weight data and bias value of the convolution kernel from the external memory according to the operation parameters of the convolution operation instruction, and writes them into the weight RAM and bias RAM respectively. The central controller also generates the data addresses of the weight RAM and bias RAM according to the operation parameters of the convolution operation instruction, and controls the multiplier-accumulator array to read the input feature map data from the input RAM, read the weight data from the weight RAM, and read the bias value from the bias RAM to perform convolution operation to obtain the output feature map data, and writes the output feature map data into the output RAM. When the pooling operation instruction is read, the central controller controls the second output terminal of the first multiplexer and the first input terminal of the second multiplexer to be turned on. According to the operation parameters of the pooling operation instruction, the central controller controls the pooling operation unit to read the output feature map data in the output RAM, perform pooling operation, and store it in the input RAM. When the fully connected layer operation instruction is read, the central controller controls the direct memory access module to read the weight data of the fully connected layer from the external memory and write it into the weight RAM according to the operation parameters of the fully connected layer operation instruction. The central controller generates the data address of the weight RAM according to the operation parameters of the fully connected layer operation instruction, and controls the multiply-accumulate array to read the input feature map data from the input RAM and the weight data from the weight RAM to perform convolution operation to obtain the inference result, and writes the inference result into the output RAM. When the output instruction is read, the central controller controls the first output terminal of the first multiplexer to be turned on, and controls the direct memory access module to read the inference result from the output RAM and write it back to the external memory.

10. The convolutional neural network coprocessor as described in claim 9, characterized in that, The instruction memory is connected to the main processor via an AXI-Lite bus, and the main processor writes instructions to the instruction memory via the AXI-Lite bus. The direct memory access module is connected to the external memory via an AXI bus, and the direct memory access module writes the inference result to a specified address in the external memory via the AXI bus.

Images