CNN hardware acceleration method and accelerator based on dynamic reconfigurable parallel computing

Abstract

The application discloses a CNN hardware acceleration method and accelerator based on dynamic reconfigurable parallel calculation. 总 First, the total calculation parallel degree P of the CNN is set according to actual hardware calculation resources used for running the CNN 总 Then, the values of Pc and Pf corresponding to the convolution layer are dynamically determined according to the input channel number C and the output channel number F of the CNN under the condition of Pc*Pf=P 总 The parallel convolution calculation between different input channels of the input feature map of the convolution layer is performed by using the Pc MAC units, and the parallel convolution calculation between different output channels of the convolution layer is simultaneously performed by using the Pf MAC units. The application can dynamically adjust the parallel degrees of the input channel and the output channel in two directions according to the input and output channel numbers of different convolution layers, and improve the overall calculation efficiency of the CNN.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a hardware acceleration method and accelerator for CNNs based on dynamically reconfigurable parallel computing. Background Technology

[0002] Convolutional Neural Networks (CNNs) have achieved great success in recent years and have been widely used in real-time processing systems such as image classification [1], object recognition [2], face detection [3], and semantic segmentation [4]. However, due to the large computational cost of CNN algorithms compared to traditional algorithms, their application in edge computing and real-time systems is severely limited, especially for FPGA devices with limited computing resources.

[0003] Convolutional neural networks mainly consist of convolutional layers and nonlinear layers (pooling layers, activation layers, etc.). The main computational load is concentrated in the convolutional layers, and the computation of the convolutional layers is highly parallel. The method of using parallel computing architecture to accelerate convolutional neural networks has received widespread attention and application.

[0004] The pseudocode for convolutional layer operations is as follows: Figure 1 The basic idea behind parallel computing is to expand and parallelize the for loops to reduce the algorithm's computation time. Therefore, the main parallel computing directions that can be adopted include: 1) Parallel computation of convolutional kernels (Pk), corresponding to expanding loops 1 and 2 to compute a convolutional window in parallel; 2) Parallel computation of the height or width of the output feature map data (Pv), corresponding to expanding loops 3 or 4 to compute multiple output data in parallel; 3) Parallel computation between different channels of the input feature map (Pc), corresponding to expanding loop 5 to compute data from different input channels in parallel; 4) Parallel computation between different output channels (Pf), corresponding to expanding loop 6 to compute data from different output channels in parallel.

[0005] Due to limitations in computing resources, existing design methods often employ only 1 to 3 degrees of parallelism for computational acceleration. For example, references [5, 6, 7] use parallelism of Pc and Pf, references [8, 9] use parallelism of Pk, Pv, and Pf, reference

[10] uses parallelism of Pk, Pc, and Pf, and reference

[11] uses parallelism in the Pv and Pf directions. The degree of parallelism designed in existing technologies mainly depends on the number of actual hardware computing resources.

[0006] Existing parallel computing solutions mainly suffer from the following problems: 1) Parallelism in the Pk direction is only suitable for convolutional kernels of fixed size (e.g., 3x3), and cannot be applied to CNN networks using different convolutional kernels, nor to different types of CNNs (e.g., VGG, AlexNet, etc.); 2) Using parallelism in certain dimensions can easily lead to a waste of computational resources. For example, in a Pv design, since the width and height of the convolutional layer are often not divisible by Pv, computational resources are wasted, resulting in low computational efficiency; in a Pc design, since the first layer C=3, the parallelism is low. The inability to fully utilize resources will also lead to a decrease in computational efficiency in the first layer; 3) The parallelism of existing methods is fixed in each computational dimension. Once the hardware design is completed, its parallelism cannot be changed. However, the data size (H, W, C, F) of each layer of a convolutional neural network is different. Generally speaking, the width and height directions decrease as the number of layers increases. Meanwhile, the input channels increase layer by layer. Therefore, existing solutions only have high computational efficiency in a few layers, while the computational efficiency is very low in other layers or other networks, resulting in low overall computational efficiency or lack of universality in different network structures.

[0007] Overall, existing solutions based on parallel computing technology adopt a fixed degree of parallelism design. Due to the characteristics of the convolutional neural network structure, it is impossible to achieve high efficiency at every layer of the CNN, resulting in the underutilization of computing resources. Furthermore, the computing efficiency varies greatly across different network structures and different FPGA devices. Summary of the Invention

[0008] This invention provides a CNN hardware acceleration method and accelerator based on dynamic reconfigurable parallel computing. It can dynamically adjust the parallelism in both Pc and Pf directions according to the number of input and output channels of different convolutional layers, thereby improving the overall computational efficiency of CNN.

[0009] To achieve the above technical objectives, the present invention adopts the following technical solution:

[0010] A hardware acceleration method for CNNs based on dynamically reconfigurable parallel computing is proposed. The reconfigurable parallel computing method involves first setting the total computational parallelism P of the CNN based on the actual hardware computing resources used to run the CNN. 总 Then, satisfying Pc*Pf=P 总 Under these conditions, based on the number of input channels C and output channels F of each convolutional layer in the CNN, the values ​​of Pc and Pf corresponding to that convolutional layer are dynamically determined, where Pc = 2. m ≤C、Pf=2 nIf F ≤ F, m, n ≥ 0, then Pc MAC units are used to perform parallel convolution calculations between different input channels of the input feature map of the convolutional layer. At the same time, Pf MAC units are used for each input channel to perform parallel convolution calculations between different output channels of the convolutional layer.

[0011] In superior CNN hardware acceleration methods, all P 总 Each MAC unit uses the FPGA's DSP as a building block, arranged in RoF rows to form a multiply-accumulate array, where Pc max This represents the maximum value of Pc among all dynamically changing values ​​in all convolutional layers of the CNN. The multiply-accumulate array is hierarchically managed to adapt to the dynamic changes of Pc and Pf in each convolutional layer. The specific hierarchical management method is as follows:

[0012] (1) Set a selector for each MAC cell in each row of the multiply-accumulate array as the 0th layer;

[0013] (2) Group all selectors set in the previous layer into pairs according to the adjacent rule, set an adder for each pair of selectors in each group, and follow the adder with a selector; this is the i-th layer;

[0014] (3) Repeat step (2); when there are only 2 selectors in the upper layer, set an adder for the 2 selectors;

[0015] (4) If 0 < Pc < Pc max The selector before the log2Pc layer controls its input data to enter the adder of the next layer for calculation, while the selector of the log2Pc layer controls its input data to be stored directly.

[0016] If Pc = Pc max Then all selectors control their input numbers to enter the next layer of adders for accumulation calculation, and store the output data of the lowest layer adder.

[0017] In a more efficient CNN hardware acceleration solution, the input and output buffers are stored as follows: each Pc data points along the input channel dimension constitute one storage unit, and the storage width of one storage unit is P. 总 For input feature maps comprising width, height, and input channels, the input buffer prioritizes storing data in the input channel dimension, followed by the width and height dimensions. Based on the K×K×Tc cycles of convolution operations on the convolutional layer, Pf results are output, where Tc = C / Pc. A buffer of size P is added before the output buffer. 总 The FIFO stores Pf intermediate results output by the computing engine each time. When the FIFO is full, it is stored in the output buffer once for input data of the next layer of the CNN.

[0018] In a more efficient CNN hardware acceleration solution, the weight buffer is stored as follows: each storage unit consists of Pc data points along the channel dimension, and the storage width of each storage unit is P. 总 The weights first store the input channel dimension, then store the width and height dimensions, and the weights of multiple convolutional kernels are fed into the computation engine in parallel.

[0019] A CNN hardware accelerator based on dynamically reconfigurable parallel computing includes: external memory, DMA, input buffer, weight buffer, computing engine, output buffer, partial sum buffer, adder array, pooling and activation module; the computing engine includes P... 总 Each MAC unit uses the FPGA's DSP as a building block and is arranged in a RoF row to form a multiply-accumulate array. Pc max The maximum value of Pc among all dynamically changing values ​​in all convolutional layers of the CNN; hierarchical management of the multiply-accumulate array to adapt to the dynamic changes of Pc and Pf in each convolutional layer;

[0020] Wherein, Pc of the convolutional layer refers to the parallelism of the convolutional layer in performing parallel convolution calculations between different input channels of the input feature map, and Pf refers to the parallelism of the convolutional layer in performing parallel convolution calculations between different output channels, satisfying Pc*Pf=P 总 P 总 This represents the total computational parallelism of the CNN.

[0021] In a superior CNN hardware accelerator solution, the hierarchical management structure of the multiply-accumulate array is as follows:

[0022] (1) Set a selector for each MAC cell in each row of the multiply-accumulate array as the 0th layer;

[0023] (2) Group all selectors set in the previous layer into pairs according to the adjacent rule, set an adder for each pair of selectors in each group, and follow the adder with a selector; this is the i-th layer;

[0024] (3) Repeat step (2); when there are only 2 selectors in the upper layer, set an adder for the 2 selectors;

[0025] (4) If 0 < Pc < Pc max The selector before the log2Pc layer controls its input data to enter the adder of the next layer for calculation, while the selector of the log2Pc layer controls its input data to be stored directly.

[0026] If Pc = Pc maxThen all selectors control their input numbers to enter the next layer of adders for accumulation calculation, and store the output data of the lowest layer adder.

[0027] In a more optimized CNN hardware accelerator solution, the storage structure of the input and output buffers is as follows: the input buffer uses Pc data points along the input channel dimension as one storage unit, and the storage width of one storage unit is P. 总 For input feature maps comprising width, height, and input channels, the input buffer prioritizes storing data in the input channel dimension, followed by the width and height dimensions. Based on the K×K×Tc cycles of convolution operations on the convolutional layer, Pf results are output, where Tc = C / Pc. A buffer of size P is added before the output buffer. 总 The FIFO stores Pf intermediate results output by the computing engine each time. When the FIFO is full, it is stored in the output buffer once for input data of the next layer of the CNN.

[0028] In a more efficient CNN hardware accelerator solution, the weight buffer's storage structure is as follows: each storage unit consists of Pc data points along the channel dimension, and the storage width of each storage unit is P. 总 The weights are first stored in the input channel dimension, followed by the data in the width and height dimensions. The weights of multiple convolutional kernels are fed into the computation engine in parallel.

[0029] Beneficial effects

[0030] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0031] 1. By adopting parallelism design and dynamic reconfiguration technology in different directions, by abandoning parallel computing in the Pv direction and reducing the parallelism of Pc in certain layers, and instead supporting parallelism in other directions such as Pf, the problem of low computational efficiency in specific layers of traditional design methods is effectively avoided.

[0032] 2. By dynamically selecting the parallelism in different directions based on the data size of different convolutional layers, the parallel computing acceleration method of this invention can achieve efficient computation of computing resources, thereby obtaining higher computational efficiency and acceleration ratio compared with traditional methods, and is adaptable to different CNN network structures.

[0033] 3. Dynamically reconfigurable parallel computing is flexible and can effectively utilize the characteristics of CNN network structure and hardware board resources. Therefore, it can be applied to FPGA boards with different resource numbers and can achieve high computing efficiency under different network structures and computing resources. Attached Figure Description

[0034] Figure 1 The pseudocode for the convolutional layer operation is shown in the figure;

[0035] Figure 2 This is a distribution diagram of the various modules of the accelerator in an embodiment of this application;

[0036] Figure 3 This is a diagram of the computing engine architecture according to an embodiment of this application;

[0037] Figure 4 This is a schematic diagram of the input and output buffers in an embodiment of this application;

[0038] Figure 5 This is a schematic diagram of the weighted cache in an embodiment of this application;

[0039] Figure 6 This is a diagram illustrating the overall hardware architecture of the CNN implemented in this application.

[0040] Figure 7 and Figure 8 These are all comparison charts showing the computational efficiency of embodiments of this application and several traditional parallel computing solutions on different Intel FPGA devices. Figure 7 The CNN model is GoogleNet. Figure 8 The CNN model used is AlexNet. Detailed Implementation

[0041] The embodiments of the present invention will be described in detail below. These embodiments are based on the technical solutions of the present invention and provide detailed implementation methods and specific operation processes to further explain the technical solutions of the present invention.

[0042] This invention provides a CNN hardware acceleration method based on dynamically reconfigurable parallel computing. The main idea is that while the total computing resources of the actual hardware (such as an FPGA) used in the CNN design are fixed, these resources can dynamically support parallel computing in different dimensions. That is, given a fixed overall parallelism (Pc*Pf), the parallelism (Pc, Pf) of each dimension changes dynamically during computation at different layers. This invention primarily addresses the following problems:

[0043] 1) Traditional design methods using Pv or Pc parallel computing suffer from low computational efficiency in certain layers due to low parallelism.

[0044] 2) Traditional design methods, when computational resources are sufficient, use a fixed degree of computational parallelism for each dimension, resulting in significant differences in computational efficiency between different computational layers and different network structures, thus leading to low overall computational efficiency.

[0045] 3) The problem of significant differences in computational efficiency under different board resources in traditional design methods.

[0046] The reconfigurable parallel computing method of this invention is as follows: First, based on the actual hardware computing resources used to run the CNN, the total computational parallelism P of the CNN is set. 总 Then, satisfying Pc*Pf=P 总 Under these conditions, based on the number of input channels C and output channels F of each convolutional layer in the CNN, the values ​​of Pc and Pf corresponding to that convolutional layer are dynamically determined, where Pc = 2. m ≤C、Pf=2 n If F ≤ m, n ≥ 0, then Pc MAC units are used to perform parallel convolution calculations between different input channels of the input feature map of the convolutional layer. Simultaneously, for each input channel, Pf MAC units are used to perform parallel convolution calculations between different output channels of the convolutional layer. This ensures high computational efficiency at each layer of the CNN and adaptability to different CNN networks and FPGA boards.

[0047] The further computational architecture and resource arrangement are as follows: the total computational parallelism P is determined based on device resources. 总 Then, the computational parallelism in that direction is achieved by instantiating multipliers of the corresponding parallel dimension. The computational architecture is as follows: Figure 3 As shown, the convolution calculation engine uses P 总 Each MAC unit performs a convolution operation. These MAC units use the FPGA's DSP chips as building blocks, and each MAC unit performs one multiply-accumulate operation.

[0048] All P 总 Each MAC unit uses the FPGA's DSP as a building block, arranged in RoF rows to form a multiply-accumulate array, where Pc max This is the maximum value among the dynamically changing Pc values ​​in all convolutional layers of a CNN, and it is mainly determined by device resources and the structure of the convolutional neural network. For example... Figure 3 As shown, since the Pc direction is calculated in each row, the input data for all rows is the same, and the weights for the second row and thereafter come from different convolution kernels.

[0049] The multiply-accumulate array is hierarchically managed to adapt to the dynamic changes of Pc and Pf in each convolutional layer. The specific hierarchical management method is as follows:

[0050] (1) Set a selector for each MAC cell in each row of the multiply-accumulate array as the 0th layer;

[0051] (2) Group all selectors set in the previous layer into pairs according to the adjacent rule, set an adder for each pair of selectors in each group, and follow the adder with a selector; this is the i-th layer;

[0052] (3) Repeat step (2); when there are only 2 selectors in the upper layer, set an adder for the 2 selectors;

[0053] (4) If 0 < Pc < Pc max The selector before the log2Pc layer controls its input data to enter the adder of the next layer for calculation, while the selector of the log2Pc layer controls its input data to be stored directly.

[0054] If Pc = Pc max Then all selectors control their input numbers to enter the next layer of adders for accumulation calculation, and store the output data of the lowest layer adder.

[0055] This computing architecture differs from traditional design methods in that it uses hierarchical management of the multiply-accumulate array to adapt to changes in Pc and Pf, while keeping computing resources busy for useful computations. The following example illustrates how the computing engine adapts its computation and storage operations after changes in parallelism. When P... 总 =Pc max When the multiplier is 8, the computation engine consists of 8 MAC units per row. The specific multiply-accumulate tree arrangement is shown in the figure, divided into three levels of management. A selector is added after each adder to control whether the data is processed or stored. In shallower layers of the CNN, the channel dimension C is smaller, and Pc is typically 2, with a corresponding Pf value of 4. Mapped to the hardware structure, (each row of two multipliers forms a group for parallel computation with Pc=2, and each row simultaneously supports Pc=2 and Pf=4). Simultaneously, the first-level selector controls the direct storage of data without transferring it to the next layer's adders. After a certain period, the input data is convolved and traversed, and non-linear and pooling processing is completed before being passed to the next layer. Based on the characteristics of neural networks, the channel dimension increases, and the corresponding Pc value can be larger, while Pf adaptively decreases to 2. In hardware design, this is described as the first-level selector controlling the data transmission to the next layer for accumulation with another group, while the second-level selector controls the data storage without transmission.

[0056] The data storage method corresponding to this embodiment is:

[0057] (1) Input and output buffers

[0058] To support parallel computing across different dimensions and dynamic configuration, the data storage method is as follows: input consists of Pc data points along the channel dimension as one storage unit, and the storage width of one storage volume is P. 总For input feature maps comprising width, height, and input channels, the input buffer prioritizes storing data in the input channel dimension, followed by data in the width and height dimensions. The convolution operation outputs Pf results after K×K×Tc cycles, where Tc = C / Pc. Since Pf is generally less than or equal to the storage width P of the buffer... 总 Therefore, a FIFO is added after the accumulator (i.e., before the output buffer) to store the Pf intermediate results output by the computing engine each time. The storage size of the FIFO is equal to P. 总 When the FIFO is full, the data is stored in the output buffer for the next layer's input. The specific storage method is as follows: Figure 4 As shown.

[0059] (2) Weighted buffer

[0060] The design of the weight buffer is as follows Figure 5 As shown, similar to the input buffer, a memory bank stores P. 总 Each storage unit stores Pc data points in the channel dimension, so a single storage unit can store the weights of Pf convolutional kernels. For example... Figure 5 As shown, the weights first store the input channel dimension, and then store the data for the width and height dimensions. The weights of multiple convolutional kernels are fed into the computation engine in parallel.

[0061] An input image of size 256*256*16 undergoes convolution operations with 32 convolution kernels of size 3*3*16, nonlinear operations, and pooling operations, outputting a result of size 128*128*32. The next layer uses 64 convolution kernels of size 3*3*32 for computation. The selected device supports a maximum parallelism P of 128. The parallelism of the first operation is Pc×Pf=4×32, and after dynamic configuration, the parallelism of the second operation is Pc×Pf=8×16.

[0062] The buffer design, satisfying the initial conditions described above, uses a single memory bank storing 128 data points. Before the first convolutional layer calculation, all 128 data points at one address are read out and temporarily stored in the register buffer. Based on the configuration data from the CNN high-level description, Pc is 4 in the first convolutional layer operation, and only the first four data points in the register are transmitted to the convolutional module for calculation. Every 3*3*4 cycles, 32 intermediate data points are obtained. These data points are temporarily stored in a partial sum register for later accumulation with the data calculated from the remaining convolutional kernels. These 32 output data points are processed and first stored in a FIFO, waiting for the remaining 96 data points to be stored together in the output register, ready to be used as input data for the next layer. The 32 output data points are all in the channel dimension, the same as the input buffer data storage method, conforming to the unified input / output memory read / write mechanism. During the next layer calculation, the image size and Pc and Pf change, and the configuration data is adjusted accordingly to meet the data transmission and storage requirements of each CNN layer.

[0063] For accelerators corresponding to the aforementioned CNN hardware acceleration methods based on dynamically reconfigurable parallel computing, refer to... Figure 2 As shown, it specifically includes: external memory, DMA, input buffer, weight buffer, computing engine, output buffer, partial sum buffer, adder array, pooling and activation module; the computing engine includes P 总 Each MAC unit performs multiplication and accumulation operations, supporting parallel processing. The internal structure of the computing engine and its working principle with other modules in the accelerator are the same as the aforementioned acceleration methods.

[0064] refer to Figure 6 As shown, the controller reads data from the input buffer at the corresponding address and stores it in a register. Only Pc data points are needed for a single parallel computation; the remaining data is temporarily stored in the register. According to the convolution operation order, the channel dimension is read first for calculation, followed by the width and height. The convolution module retrieves Pc data points and loads them into Pc multipliers. The remaining multipliers reuse Pc data points for parallel computation between output channels (PF dimension). The required weights are provided all at once by the data stored at the corresponding address in the weight buffer. In the next cycle, the input data is transferred from the register as a second Pc value, and the weights come from the next address in the buffer. After K×K×Tc cycles, Pf data points are output, followed by non-linear and pooling processing, and then stored in a FIFO. The data is then stored back in the buffer when the FIFO is full. The specific implementation steps are as follows:

[0065] The trained weights and input data of each layer of S1 are stored in external memory, and the high-level description of the CNN is converted into configuration data by the network parser and stored in the configuration unit, waiting for execution.

[0066] S2 reads the input image and the first-layer weights into the input buffer and weight buffer. The controller obtains configuration data from the configuration unit to understand the attributes of the CNN layers and generates control signals accordingly to configure each module. The first-layer convolutional network is executed through the convolution calculation module, and the result is stored in the output buffer.

[0067] S3: After S2 is completed, the convolution calculation module is idle, and all the results have been stored in the output buffer. The execution of the second layer convolution module can begin. At this time, the input data is provided by the output buffer.

[0068] S4 executes S2 repeatedly until all convolutional layers have been executed, and the final result is written back to external memory.

[0069] S5 performs the calculation of the fully connected layers and outputs the final image classification results from the host computer.

[0070] like Figure 7 , 8 As shown, a comparison of the computational efficiency of the present invention with several traditional parallel computing solutions on different Intel FPGA devices is presented.

[0071] The above embodiments are preferred embodiments of this application. Those skilled in the art can make various changes or improvements based on them. Without departing from the overall concept of this application, these changes or improvements should fall within the scope of protection claimed in this application.

[0072] References:

[0073] [1] A.Krizhevsky, I.Sutskever, and GEHinton, “Imagenet classification with deep convolutional neural networks,” in Advances in neural informationprocessing systems, 2012, pp.1097–1105.

[0074] [2] K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," arXiv preprint arXiv:1409.1556, 2014.

[0075] [3]J.Dai,Y.Li,K.He,and J.Sun,“R-fcn:Object detection via regionbasedfully convolutional networks,”in Advances in neural information processingsystems,2016,pp.379–387.

[0076] [4]V.Badrinarayanan,A.Kendall,and R.Cipolla,“Segnet:A deepconvolutional encoder-decoder architecture for image segmentation,”IEEETrans.pattern analysis and machine intelligence.,vol.39,no.12,pp.2481–2495,2017.

[0077] [5]Zhang,Chen,et al."Optimizing fpga-based accelerator design fordeep convolutional neural networks."Proceedings of the 2015 ACM / SIGDAinternational symposium on field-programmable gate arrays.2015.

[0078] [6]Liu,Shuanglong,and Wayne Luk."Towards an efficient accelerator forDNN-based remote sensing image segmentation on FPGAs."2019 29th InternationalConference on Field Programmable Logic andApplications(FPL).IEEE,2019.

[0079] [7]Liu,Shuanglong,et al."Toward Full-Stack Acceleration of DeepConvolutional Neural Networks on FPGAs."IEEE Transactions on Neural Networksand Learning Systems(2021).

[0080] [8]Zhao,Ruizhe,et al."Optimizing CNN-based object detectionalgorithms on embedded FPGA platforms."International Symposium on AppliedReconfigurable Computing.Springer,Cham,2017.

[0081] [9]Liu,Shuanglong,et al."Optimizing CNN-based segmentation withdeeply customized convolutional and deconvolutional architectures on FPGA."ACM Transactions on Reconfigurable Technology and Systems(TRETS)11.3(2018):1-22.

[0082]

[10] Qiu,Jiantao,et al."Going deeper with embedded fpga platform forconvolutional neural network."Proceedings ofthe 2016 ACM / SIGDA InternationalSymposium on Field-Programmable Gate Arrays.2016.

[0083]

[11] Ma,Yufei,et al."An automatic RTL compiler for high-throughputFPGA implementation of diverse deep convolutional neural networks."201727thInternational Conference on Field Programmable Logic andApplications(FPL).IEEE,2017.

Claims

1. A hardware acceleration method for CNNs based on dynamically reconfigurable parallel computing, characterized in that, The reconfigurable parallel computing method is as follows: First, based on the actual hardware computing resources used to run the CNN, the total computational parallelism P of the CNN is set. 总 Then, satisfying Pc*Pf=P 总 Under these conditions, based on the number of input channels C and output channels F of each convolutional layer in the CNN, the values ​​of Pc and Pf corresponding to that convolutional layer are dynamically determined, where Pc = 2. m ≤C、Pf=2 n ≤F, m,n≥0, and then Pc MAC units are used to perform parallel convolution calculations between different input channels of the input feature map of the convolutional layer. At the same time, Pf MAC units are used for each input channel to perform parallel convolution calculations between different output channels of the convolutional layer. All P 总 Each MAC unit uses the FPGA's DSP as a building block, arranged in RoF rows to form a multiply-accumulate array, where Pc max This represents the maximum value of Pc among all dynamically changing values ​​in all convolutional layers of the CNN. The multiply-accumulate array is hierarchically managed to adapt to the dynamic changes of Pc and Pf in each convolutional layer. The specific hierarchical management method is as follows: (1) Set a selector for each MAC cell in each row of the multiply-accumulate array as the 0th layer; (2) Group all selectors set in the previous layer into pairs according to the adjacent rule, set an adder for each pair of selectors in each group, and follow the adder with a selector; this is the i-th layer; (3) Repeat step (2); when there are only 2 selectors in the upper layer, set an adder for the 2 selectors; (4) If 0 < Pc < Pc max Then, the selector before the log2Pc layer controls its input data to enter the adder of the next layer for calculation, and the selector of the log2Pc layer controls its input data to be stored directly. If Pc = Pc max Then all selectors control their input data to enter the next layer adder for accumulation calculation, and store the output data of the lowest layer adder; Pc refers to the parallelism of the convolutional layer in performing parallel convolution calculations between different input channels of the input feature map, while Pf refers to the parallelism of the convolutional layer in performing parallel convolution calculations between different output channels.

2. The CNN hardware acceleration method based on dynamically reconfigurable parallel computing according to claim 1, characterized in that, The storage method for input and output buffers is as follows: each storage unit consists of Pc data points along the input channel dimension, and the storage width of each storage unit is P. 总 For input feature maps comprising width, height, and input channels, the input buffer prioritizes storing data in the input channel dimension, followed by the width and height dimensions. Based on the K×K×Tc cycles of convolution operations on the convolutional layer, Pf results are output, where Tc = C / Pc. A buffer of size P is added before the output buffer. 总 The FIFO stores Pf intermediate results output by the computing engine each time. When the FIFO is full, it is stored in the output buffer once for input data of the next layer of the CNN.

3. The CNN hardware acceleration method based on dynamically reconfigurable parallel computing according to claim 1, characterized in that, The weighted buffer is stored as follows: each storage unit consists of Pc data points along the channel dimension, and the storage width of each storage unit is P. 总 The weights first store the input channel dimension, then store the width and height dimensions, and the weights of multiple convolutional kernels are fed into the computation engine in parallel.

4. A CNN hardware accelerator based on dynamically reconfigurable parallel computing, characterized in that, include: External memory, DMA, input buffer, weight buffer, computing engine, output buffer, partial and sum buffer, adder array, pooling and activation module; The computing engine includes P 总 Each MAC unit uses the FPGA's DSP as a building block and is arranged in a RoF row to form a multiply-accumulate array. Pc max The maximum value of Pc among all dynamically changing values ​​in all convolutional layers of the CNN; hierarchical management of the multiply-accumulate array to adapt to the dynamic changes of Pc and Pf in each convolutional layer; Wherein, Pc of the convolutional layer refers to the parallelism of the convolutional layer in performing parallel convolution calculations between different input channels of the input feature map, and Pf refers to the parallelism of the convolutional layer in performing parallel convolution calculations between different output channels, satisfying Pc*Pf=P 总 P 总 This represents the total computational parallelism of the CNN. The hierarchical management structure for the multiply-accumulate array is as follows: (1) Set a selector for each MAC cell in each row of the multiply-accumulate array as the 0th layer; (2) Group all selectors set in the previous layer into pairs according to the adjacent rule, set an adder for each pair of selectors in each group, and follow the adder with a selector; this is the i-th layer; (3) Repeat step (2); when there are only 2 selectors in the upper layer, set an adder for the 2 selectors; (4) If 0 <Pc<Pc max Then, the selector before the log2Pc layer controls its input data to enter the adder of the next layer for calculation, and the selector of the log2Pc layer controls its input data to be stored directly. If Pc = Pc max Then all selectors control their input data to enter the next layer adder for accumulation calculation, and store the output data of the lowest layer adder.

5. The CNN hardware accelerator based on dynamically reconfigurable parallel computing according to claim 4, characterized in that, The storage structure of the input and output buffers is as follows: the input buffer uses Pc data points along the input channel dimension as one storage unit, and the storage width of one storage unit is P. 总 For input feature maps comprising three dimensions—width, height, and input channels—the input buffer prioritizes storing data in the input channel dimension, followed by data in the width and height dimensions. Based on the K×K×Tc cycles of convolution operations in the convolutional layers, Pf results are output, where Tc = C / Pc, and C represents the number of input channels in each convolutional layer of the CNN. A buffer of size P is added before the output buffer. 总 The FIFO stores Pf intermediate results output by the computing engine each time. When the FIFO is full, it is stored in the output buffer once for input data of the next layer of the CNN.

6. The CNN hardware accelerator based on dynamically reconfigurable parallel computing according to claim 4, characterized in that, The storage structure of the weighted buffer is as follows: each storage unit consists of Pc data points along the channel dimension, and the storage width of each storage unit is P. 总 The weights first store the input channel dimension, then store the width and height dimensions, and the weights of multiple convolutional kernels are fed into the computation engine in parallel.

Images