A dynamically reconfigurable convolutional neural network multi-core accelerator

Abstract

The present invention relates to a dynamically reconfigurable convolutional neural network multi-core accelerator, comprising an instruction distribution module and a loading module, a routing module, a computing module and an unloading module connected thereto and controlled therefrom; the loading module loads input feature data and convolution kernel data from the outside; the routing module distributes the loaded data to the computing module for convolutional neural network acceleration processing; the computing module outputs the result through the unloading module; the loading module, the routing module, the computing module and the unloading module include multiple loaders, routers, computing cores and unloaders; in the convolutional neural network processing process, the instruction distribution module performs group dynamic reconstruction on the multiple loaders, routers, computing cores and unloaders, loads feature data and convolution kernel data in the computing core, and performs parallel convolutional neural network acceleration processing. The present invention utilizes dynamic reconstruction to achieve parallelism between multiple computing cores, thereby improving the large dynamic adaptability of the convolutional neural network.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning acceleration circuits, and in particular relates to a dynamically reconfigurable convolutional neural network multi-core accelerator. Background Art

[0002] At present, CNN (Convolutional Neural Network) has been widely used in aerospace, security, medical, mobile robots and other fields. With the continuous improvement of the accuracy of convolutional neural network algorithms, their depth, computational complexity and parameter quantity are also increasing, which brings great challenges to the real-time performance of embedded applications. For example, the target detection algorithms based on convolutional neural networks commonly used in the aerospace field, such as SSD, YOLO, etc., have a computational complexity of more than 30G. In order to meet the strong real-time requirements of aerospace applications, high-parallel computing devices must be used to accelerate them. However, due to the large dynamic range of the number of channels and feature map area size of the convolutional neural network algorithm, the actual computing efficiency of many existing convolutional neural network accelerators with fixed parallelism is not ideal. Therefore, the parallelism that can be adjusted at runtime is used to better adapt to the large dynamic range of the number of channels and feature map area, thereby improving the actual computing efficiency of the high-parallel convolutional neural network accelerator. Summary of the invention

[0003] In view of the above analysis, the present invention aims to disclose a dynamically reconfigurable convolutional neural network multi-core accelerator, which adjusts the parallelism at runtime to adapt to the large dynamic range of the number of channels and the area of ​​the feature map, thereby improving the actual computing efficiency of the high-parallelism convolutional neural network accelerator.

[0004] The present invention discloses a dynamically reconfigurable convolutional neural network multi-core accelerator, comprising an instruction distribution module and a loading module, a routing module, a computing module and an unloading module connected to and controlled by the instruction distribution module;

[0005] The loading module loads input feature data and convolution kernel data from the outside; the routing module distributes the loaded data to the computing module for convolution neural network acceleration processing; the computing module outputs the result through the unloading module;

[0006] The loading module, routing module, computing module and unloading module include multiple loaders, routers, computing cores and unloaders;

[0007] During the convolutional neural network processing, the instruction distribution module dynamically reconstructs the multi-way loaders, routers, computing cores and unloaders in groups, loads feature data and convolution kernel data in the computing core, and performs parallel convolutional neural network acceleration processing.

[0008] Furthermore, the number of computing cores is s, the number of routers is s', the number of loaders is s+s', and the number of unloaders is s;

[0009] The computing cores are divided into s' groups by grouping; each group of computing cores is connected to a router; each router is connected to s / s'+1 loaders; each computing core is connected to an unloader.

[0010] Furthermore, by dynamically reconstructing the groups, the parallel operation is performed in the feature graph within the group and the feature graph between groups.

[0011] Under the allocation of each router, the convolution kernel data loaded by each computing core of all groups is the same convolution kernel data; the feature data loaded by each computing core is the feature data at different positions in the feature map; the computing cores of all groups jointly and parallelly execute the convolution neural network acceleration processing.

[0012] Furthermore, through group dynamic reconstruction, when working in the mode of parallel output channels within the group and parallel output channels between groups,

[0013] Under the allocation of each router, the convolution kernel data loaded by each computing core of all groups is different convolution kernel data; the feature data loaded by each computing core is the feature data at the same position in the feature map; the computing cores of all groups jointly and parallelly execute convolutional neural network acceleration processing.

[0014] Furthermore, by dynamically reconstructing the groups, the parallel operation of the feature graphs within the group and the parallel output channels between groups is achieved.

[0015] Under the allocation of each router, the convolution kernel data loaded by the computing cores in each group are the same, and the feature data loaded by the computing cores in each group are feature data at different positions in the feature map; while between computing core groups, the loaded feature data are the same, and the loaded convolution kernel data are different; the computing cores of all groups jointly and parallelly execute convolutional neural network acceleration processing.

[0016] Furthermore, by dynamically reconstructing the groups, the output channels in the groups are parallelized and the feature maps between the groups are parallelized.

[0017] Under the allocation of each router, the convolution kernel data loaded by the computing cores in each group are different convolution kernel data, and the feature data loaded by the computing cores in each group are feature data at the same position in the feature map; between computing core groups, the feature data loaded are feature data at different positions in the feature map, and the loaded convolution kernel data are the same convolution kernel data; the computing cores of all groups jointly and parallelly execute convolutional neural network acceleration processing.

[0018] Furthermore, the router in the routing module includes data replicators U0 to U s / s’ , data register R0~R s / s’-3 , data selector S0~S s / s’-1 ;

[0019] Among them, the input end of replicator U0 is connected to the output end of feature loader, and replicators U1~U s / s’-1 Connect a data loader output, a replicator U s / s’ The input end is connected to the output end of the weight loader;

[0020] The output of replicator U0 is connected to data registers R0 and R 4(i-1)-3 Input terminal;

[0021] ReplicatorU i The output terminals are connected to the data register R 4(i-1)-1 and R 4(i-1)-2 ;

[0022] ReplicatorU s / s The output terminals are connected to the data register R s / s’-3 and R 4(i-1) Input terminal;

[0023] The first input terminal of the data selector S0 is connected to the output terminal of the register R1, and the second input terminal is connected to the output terminal of the register R2; the data selector S i-1 The first input terminal is connected to register R 4(i-1)-1 Output terminal, the second input terminal is connected to register R 4(i-1) Output terminal; i = 2, ..., s / s';

[0024] Registers R0 and R s / s’-3 The output terminal and data selector S0~S s / s’-1 As the output end of the routing module; wherein the output end of the register R0 and the data selector S1 is used as the input end of the first computing core; the data selector S s / s’-2 and register R 4s / s’-3 The output end of is used as the input end of the s / s' computing core; the data selector S i-1 and data selector S i+1 The output end of is used as the input end of the i-th computing core, i=2,…,s / s'-1.

[0025] Furthermore, in parallel mode within the feature graph of the computing core group, data selectors S0~S s / s’-1 Select the data of the second input terminal for output;

[0026] In the parallel mode of the output channels in the computing core group, data selectors S0~S s / s’-1 Select the data of the first input terminal for output.

[0027] Further, the computing core includes a preprocessing module, a convolution module, a pooling module, a first FIFO-B, a second FIFO-B, a third FIFO-B, a fourth FIFO-B, a FIFO-A, a first bypass, a second bypass and a third bypass;

[0028] The input ends of the first FIFO-B and the second FIFO-B are connected to the output ends of the corresponding redundant modules as the input ends of the computing core; the output end of the first FIFO-B is connected to the input ends of the preprocessing module and the first bypass respectively; the output end of the second FIFO-B is connected to the first input end of the convolution module;

[0029] The output ends of the preprocessing module and the first bypass are simultaneously connected to the input end of the third FIFO-B; the output end of the third FIFO-B is connected to the second input end of the convolution module; the second bypass is connected between the second input end and the output end of the convolution module;

[0030] The output end of the convolution module is connected to the input end of the fourth FIFO-B, and the output end of the fourth FIFO-B is connected to the input end of the pooling module and the third bypass;

[0031] The output ends of the pooling module and the third bypass are simultaneously connected to the input end of FIFO-A; the output end of FIFO-A is connected to the input end of the redundancy module as the output end of the computing core;

[0032] The preprocessing module is used to load input features and perform preprocessing of the convolutional neural network under the control of the instruction distribution control module;

[0033] The convolution module is used to load convolution kernel data and input features to perform calculations of convolution layers, fully connected layers or nonlinear activation layers under the control of the instruction distribution control module;

[0034] The pooling module is used to load the calculation result of the convolution module under the control of the instruction distribution control module to execute the calculation output of the pooling layer;

[0035] The first bypass, the second bypass and the third bypass are connected or disconnected under the control of the instruction distribution control module.

[0036] Furthermore, under the control of the instruction distribution control module, the preprocessing module, the convolution module and the pooling module of the computing core are dynamically connected in a streaming structure; the convolution neural network is accelerated to process the input data according to the structure of the convolution neural network;

[0037] In the flow structure connection, including;

[0038] 1) The preprocessing module is connected to the convolution module, and the pooling module is bypassed through the third bypass; and the preprocessing module is configured as a preprocessing layer, and the convolution module is configured as a convolution layer or a convolution layer + a nonlinear activation layer;

[0039] 2) Connecting the convolution module to the pooling module, bypassing the preprocessing module through the first bypass; configuring the convolution module as a convolution layer or a convolution layer + a nonlinear activation layer, and configuring the pooling module as a pooling layer;

[0040] 3) The convolution module is retained, and the preprocessing module and the pooling module are bypassed through the first bypass and the third bypass, and the convolution module is configured as a convolution layer, a fully connected layer, a convolution layer + a non-linear activation layer, or a fully connected layer + a non-linear activation layer.

[0041] The present invention can achieve at least one of the following beneficial effects:

[0042] The present invention uses dynamic reconstruction to achieve parallelism among multiple computing cores, improving the adaptability of the convolutional neural network accelerator to the large dynamic range of the number of convolutional neural network channels and feature map sizes. This further improves the actual computing efficiency of the high-parallelism convolutional neural network accelerator, thereby meeting the real-time requirements of deep algorithms with high computational complexity running on embedded devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings are only for the purpose of illustrating specific embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0044] Figure 1 It is a principle block diagram of a dynamically reconfigurable convolutional neural network multi-core accelerator in an embodiment of the present invention;

[0045] Figure 2 A schematic diagram showing the connection principle of the routing module in the embodiment of the present invention;

[0046] Figure 3 A block diagram showing the connection principle of the computing core components in an embodiment of the present invention;

[0047] Figure 4 Schematic diagram of a typical convolutional neural network structure in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used to illustrate the principles of the present invention together with the embodiments of the present invention.

[0049] An embodiment of the present invention discloses a dynamically reconfigurable convolutional neural network multi-core accelerator, such as Figure 1As shown, it includes an instruction distribution module and a loading module, a routing module, a calculation module and an unloading module connected to and controlled by the instruction distribution module;

[0050] The loading module loads input feature data and convolution kernel data from the outside; the routing module distributes the loaded data to the computing module for convolution neural network acceleration processing; the computing module outputs the result through the unloading module;

[0051] The loading module, routing module, computing module and unloading module include multiple loaders, routers, computing cores and unloaders;

[0052] During the convolutional neural network processing, the instruction distribution module dynamically reconstructs the multi-way loaders, routers, computing cores and unloaders in groups, loads feature data and convolution kernel data in the computing core, and performs parallel convolutional neural network acceleration processing.

[0053] The loader loads data from the external DDR through the AXI interface; the unloader unloads data to the external DDR through the AXI interface.

[0054] The function of the loader and unloader is similar to a dedicated DMA (Direct Memory Access), that is, automatically moving a certain amount of data from the source location to the target location. It can be configured to the type of data required to be transferred according to actual needs.

[0055] Before the loader runs, it obtains two types of configuration information from the instruction distribution module: 1. The address of the target data in the external DDR; 2. The amount of data transmission. During operation, the loader will load data from the external DDR to its lower-level FIFO-A through the AXI interface. When FIFO-A is full, the loader temporarily enters a blocked state; the unloader will write the data in its upper-level FIFO-A to the external DDR. When FIFO-A is empty, the unloader temporarily enters a blocked state.

[0056] The instruction distribution control module obtains instruction words from the master control (or external storage) through the AXI-Lite interface, realizing the separation of data and instructions; the instruction distribution control module distributes the instruction flag words to each controlled module to configure the control mode of the entire system. In addition, this module is responsible for receiving the start signal of the master control and sending interrupt signals, end signals, etc. to the master control.

[0057] Specifically, the number of computing cores is s, the number of routers is s', the number of loaders is s+s', and the number of unloaders is s;

[0058] The computing cores are divided into s' groups through grouping, preferably s is an integer multiple of s'; each group of computing cores is connected to a router, that is, one router is responsible for data distribution of s / s' computing cores; each router is connected to s / s'+1 loaders, and the loaded data of s / s'+1 loaders are distributed to s / s' computing cores to perform multi-core acceleration processing of convolutional neural networks in parallel; each computing core is connected to an unloader, and s / s' computing cores output s / s' processing results and output them through s / s' unloaders.

[0059] The s / s'+1 loaders in the group are numbered, where the first loader is a feature loader, used to load input feature data; the s / s'th loader is a weight loader, used to load convolution kernel data; the other loaders are data loaders; they load input feature data or convolution kernel data according to instructions; the s / s'th unloader is a feature unloader, used to unload s / s'th computing cores and output s / s'th processing results.

[0060] In this embodiment, during the convolutional neural network processing, the instruction distribution module dynamically reconstructs the grouping of multi-way loaders, routers, computing cores and unloaders. According to factors such as the feature map size of the comprehensive input features and the number of convolutional layer channels in the convolutional neural network, the working mode within and between groups of computing cores in the computing module is dynamically reconstructed, and the corresponding feature data and convolution kernel data are loaded in the computing core to perform parallel convolutional neural network acceleration processing.

[0061] Specifically, through group dynamic reconstruction, the following four working modes are obtained:

[0062] 1) Parallelism within the feature map within a group - Parallelism within the feature map between groups;

[0063] In this working method,

[0064] Under the allocation of each router, the convolution kernel data loaded by each computing core of all groups is the same convolution kernel data; the feature data loaded by each computing core is the feature data at different positions in the feature map; the computing cores of all groups jointly and parallelly execute the convolution neural network acceleration processing.

[0065] Specifically, the s / s'+1 loaders corresponding to each group of computing cores respectively load one convolution kernel data and feature data at different positions in s / s' feature maps to the corresponding routers; the s / s'th loader in the group loads one convolution kernel data, and the other s / s' loaders respectively load feature data at different positions in s / s' feature maps; the routers respectively load feature data at different positions in s / s' feature maps into s / s' computing cores in the group, and respectively load one convolution kernel data into s / s' computing cores in the group, to perform convolution neural network acceleration processing;

[0066] The convolution kernel data loaded between the calculation core groups are the same convolution kernel data, and the loaded input features are the feature data at different positions in the feature map.

[0067] 2) Intra-group output channel parallelism - inter-group output channel parallelism;

[0068] In this working method,

[0069] Under the allocation of each router, the convolution kernel data loaded by each computing core of all groups is different convolution kernel data; the feature data loaded by each computing core is the feature data at the same position in the feature map; the computing cores of all groups jointly and parallelly execute convolutional neural network acceleration processing.

[0070] Specifically, the s / s'+1 loaders corresponding to each group of computing cores respectively load the feature data of a point in a feature map and s / s' different convolution kernel data to the corresponding router; the first loader in the group loads the feature data of a point in a feature map, and the other s / s' loaders respectively load s / s' different convolution kernel data; the router loads the feature data of a point in a feature map into the s / s' computing cores in the group, and loads the s / s' different convolution kernel data into the s / s' computing cores in the group, to perform convolution neural network acceleration processing;

[0071] The convolution kernel data loaded between the calculation core groups are different convolution kernel data, and the loaded input features are the feature data at the same position in the feature map.

[0072] 3) Parallelism within feature maps within a group - parallelism of output channels between groups;

[0073] In this working method,

[0074] Under the allocation of each router, the convolution kernel data loaded by the computing cores in each group are the same, and the feature data loaded by the computing cores in each group are feature data at different positions in the feature map; while between computing core groups, the loaded feature data are the same, and the loaded convolution kernel data are different; the computing cores of all groups jointly and parallelly execute convolutional neural network acceleration processing.

[0075] The s / s'+1 loaders corresponding to each group of computing cores respectively load one convolution kernel data and feature data at different positions in the s / s' feature map to the corresponding router; the s / s'th loader in the group loads one convolution kernel data, and the other s / s' loaders respectively load feature data at different positions in the feature map; the router loads the feature data at different positions in the s / s' feature map into the s / s' computing cores in the group, and loads one convolution kernel data into the s / s' computing cores in the group, to perform convolution neural network acceleration processing;

[0076] The convolution kernel data loaded between the computing core groups are different convolution kernel data, and the loaded input features are feature data at different positions in the same set of feature maps.

[0077] 4) Parallel output channels within a group - parallel feature maps between groups;

[0078] In this working method,

[0079] Under the allocation of each router, the convolution kernel data loaded by the computing cores in each group are different convolution kernel data, and the feature data loaded by the computing cores in each group are feature data at the same position in the feature map; between computing core groups, the feature data loaded are feature data at different positions in the feature map, and the loaded convolution kernel data are the same convolution kernel data; the computing cores of all groups jointly and parallelly execute convolutional neural network acceleration processing.

[0080] The s / s'+1 loaders corresponding to each group of computing cores respectively load the feature data of a point in the feature map and s / s' different convolution kernel data to the corresponding routers; the first loader in the group loads the feature data of a point in the feature map, and the other s / s' loaders respectively load s / s' different convolution kernel data; the router loads the feature data of a point in the feature map into the s / s' computing cores in the group, and loads the s / s' different convolution kernel data into the s / s' computing cores in the group, and performs convolution neural network acceleration processing;

[0081] The convolution kernel data loaded between the calculation core groups are the same convolution kernel data, and the loaded input features are the feature data of different positions in the feature map.

[0082] In the above working method,

[0083] The method of intra-group feature map parallelism and inter-group feature map parallelism is suitable for the first few layers of the convolutional neural network. In the first few layers of the convolutional neural network, the size of the feature map is larger, and the number of channels of the convolutional layer is smaller. By using all the computing cores to calculate different positions of the feature map at the same time, the processing efficiency is improved.

[0084] Specifically, when the output channel of the convolutional layer is less than or equal to the output channel parallelism n in the computing core, a method of intra-group feature map parallelism-inter-group feature map parallelism is adopted to obtain higher processing efficiency.

[0085] The method of intra-group output channel parallelism and inter-group output channel parallelism is suitable for the last few layers of the convolutional neural network. In the last few layers of the convolutional neural network, the size of the feature map is smaller, and the number of channels of the convolutional layer is more. By using all the computing cores to calculate different channels of the convolutional layer at the same time, the processing efficiency is improved.

[0086] Specifically, when the output channel of the convolutional layer is larger than the product of the output channel parallelism n within the computing core and the number of computing cores s, a method of intra-group output channel parallelism-inter-group output channel parallelism is adopted to obtain higher processing efficiency.

[0087] When the output channels of the convolutional layer are between n and n×s, it can be flexibly configured to be a parallel mode within the feature map within the group - parallel mode of output channels between groups or a parallel mode of output channels within the group - parallel mode of feature maps between groups, depending on the execution efficiency, to obtain higher processing efficiency.

[0088] Specifically, Figure 2 As shown, a router of the routing module includes data replicators U0~U s / s’ , data register R0~R s / s’-3 , data selector S0~S s / s’-1 ;

[0089] Among them, the input end of replicator U0 is connected to the output end of feature loader, and replicators U1~U s / s’-1 Connect a data loader output, a replicator U s / s’ The input end is connected to the output end of the weight loader;

[0090] The output of replicator U0 is connected to data registers R0 and R 4(i-1)-3 Input terminal;

[0091] ReplicatorU i The output terminals are connected to the data register R 4(i-1)-1 and R 4(i-1)-2 ;

[0092] ReplicatorU s / sThe output terminals are connected to the data register R s / s’-3 and R 4(i-1) Input terminal;

[0093] The first input terminal of the data selector S0 is connected to the output terminal of the register R1, and the second input terminal is connected to the output terminal of the register R2; the data selector S i-1 The first input terminal is connected to register R 4(i-1)-1 Output terminal, the second input terminal is connected to register R 4(i-1) Output terminal; i = 2, ..., s / s';

[0094] Registers R0 and R s / s’-3 The output terminal and data selector S0~S s / s’-1 As the output end of the routing module; wherein the output end of the register R0 and the data selector S1 is used as the input end of the first computing core; the data selector S s / s’-2 and register R 4s / s’-3 The output end of is used as the input end of the s / s' computing core; the data selector S i-1 and data selector S i+1 The output end of is used as the input end of the i-th computing core, i=2,…,s / s'-1.

[0095] When the router works in parallel mode within the group feature graph, the data selectors S0~S s / s’-1 Select the data of the second input terminal for output;

[0096] When the output channels in the router group are in parallel mode, data selectors S0~S s / s’-1 Select the data of the first input terminal for output.

[0097] By switching each router in the routing module, the required data under different working modes can be transmitted to the corresponding computing core to perform convolutional neural network acceleration processing.

[0098] Specifically, Figure 3 As shown, each computing core of the computing module includes two input terminals and one output terminal, including a preprocessing module, a convolution module, a pooling module, a first FIFO-B, a second FIFO-B, a third FIFO-B, a fourth FIFO-B, a FIFO-A, a first bypass, a second bypass and a third bypass;

[0099] The input ends of the first FIFO-B and the second FIFO-B are connected to the output ends of the corresponding redundant modules as the input ends of the computing core; the output end of the first FIFO-B is connected to the input ends of the preprocessing module and the first bypass respectively; the output end of the second FIFO-B is connected to the first input end of the convolution module;

[0100] The output ends of the preprocessing module and the first bypass are simultaneously connected to the input end of the third FIFO-B; the output end of the third FIFO-B is connected to the second input end of the convolution module; the second bypass is connected between the second input end and the output end of the convolution module;

[0101] The output end of the convolution module is connected to the input end of the fourth FIFO-B, and the output end of the fourth FIFO-B is connected to the input end of the pooling module and the third bypass;

[0102] The output ends of the pooling module and the third bypass are simultaneously connected to the input end of FIFO-A; the output end of FIFO-A is connected to the input end of the redundancy module as the output end of the computing core;

[0103] The preprocessing module is used to load input features and perform preprocessing of the convolutional neural network under the control of the instruction distribution control module;

[0104] The convolution module is used to load convolution kernel data and input features to perform calculations of convolution layers, fully connected layers or nonlinear activation layers under the control of the instruction distribution control module;

[0105] The pooling module is used to load the calculation result of the convolution module under the control of the instruction distribution control module to execute the calculation output of the pooling layer;

[0106] The first bypass, the second bypass and the third bypass are connected or disconnected under the control of the instruction distribution control module.

[0107] Specifically, under the control of the instruction distribution control module, the preprocessing module, the convolution module and the pooling module of the computing core are dynamically connected in a streaming structure; the convolution neural network is accelerated to process the input data according to the structure of the convolution neural network;

[0108] In the flow structure connection, including;

[0109] 1) The preprocessing module is connected to the convolution module, and the pooling module is bypassed through the third bypass; and the preprocessing module is configured as a preprocessing layer, and the convolution module is configured as a convolution layer or a convolution layer + a nonlinear activation layer;

[0110] 2) Connecting the convolution module to the pooling module, bypassing the preprocessing module through the first bypass; configuring the convolution module as a convolution layer or a convolution layer + a nonlinear activation layer, and configuring the pooling module as a pooling layer;

[0111] 3) The convolution module is retained, and the preprocessing module and the pooling module are bypassed through the first bypass and the third bypass, and the convolution module is configured as a convolution layer, a fully connected layer, a convolution layer + a non-linear activation layer, or a fully connected layer + a non-linear activation layer.

[0112] Since, in the convolutional neural network model, the calculation methods of the convolutional layer and the fully connected layer are basically the same, both can be abstracted as matrix-vector multiplication operations, and can be implemented by reusing the same set of hardware circuits in the FPGA; the nonlinear activation layer generally follows the convolutional layer or the fully connected layer, and the calculation process is simple, so it can be integrated with the convolutional layer or the fully connected layer. Therefore, in the convolution module of this embodiment, a hardware circuit that can realize the convolutional layer, the fully connected layer, the convolutional layer + the nonlinear activation layer, or the fully connected layer + the nonlinear activation layer is arranged, and under the control mode configuration and control of the instruction distribution module, the functions of the actual convolutional layer, the fully connected layer, the convolutional layer + the nonlinear activation layer, or the fully connected layer + the nonlinear activation layer are selected.

[0113] Since there are many types of calculation modes of the pooling layer, such as maximum pooling and average pooling, and the pooling window sizes and pooling step sizes of different layers are also different, it is not suitable to be integrated with the convolution module, otherwise it will affect the scalability of the accelerator. Therefore, a pooling module is separately set in the accelerator of this embodiment to realize the pooling layer function;

[0114] Since the convolutional neural network usually inputs images of large size (common sizes are 224×224, 416×416, 1024×1024, etc.), the preprocessing function has a large amount of computation and takes a long time to calculate in the CPU. It is suitable to perform hardware acceleration in the FPGA. However, the preprocessing function varies depending on the algorithm and application scenario. In order to improve the scalability of the system, a preprocessing module is separately set in the accelerator of this embodiment.

[0115] However, the post-processing function of the convolutional neural network has a small amount of computation and involves a large number of floating-point operations, so it is not suitable for implementation in FPGA hardware. This embodiment outputs the final output data of the accelerator to the DSP or other modules suitable for floating-point operations through the corresponding interface for corresponding processing.

[0116] In the computing core of the embodiment, the data transmission module is isolated from other modules in the system by using structures such as FIFO, so that the data transmission module can work in different clock domains, thereby improving the matching degree between data bandwidth and computing rate; a data buffer can be added in the data transmission module to provide the computing module with an instantaneous bandwidth higher than the physical bandwidth upper limit.

[0117] Wherein, both FIFO-A and FIFO-B are stream structure FIFOs. Modules are connected through stream structure FIFOs; after the control mode of each controlled module is configured by the instruction distribution module, the operation or blocking state of each module is driven by data; when the upper FIFO of the corresponding module is not empty and the lower FIFO is not full, the module enters the operation state; otherwise, it enters the blocking state.

[0118] By arranging FIFOs between modules, the data interfaces between modules inside the accelerator are all FIFO interfaces, and the control interface is a handshake protocol interface. The interface between the system and the outside depends on the on-chip bus protocol of the hardware platform. Common data transmission interfaces include AXI interfaces, and instruction transmission interfaces include AXI-Lite interfaces. In short, the interfaces of each module in the system are concise and unified, which provides convenience for adding, deleting modules, or adjusting the order between modules, thereby further improving the scalability of the system. In addition, through configuration and control, each computing module can be arranged upstream and downstream of the data flow as required, and the modules can also work in parallel in a "pipeline operation" manner, so that the system has higher overall performance.

[0119] This embodiment also discloses a specific solution to achieve this, such as Figure 4 The network structure of VGG-16 shown in the figure is executed under the configuration and control of the instruction distribution module to perform convolutional neural network calculation. The input image of this embodiment is set to RGB format, the size is 224×224, the step size of the convolution layer is 1, and the step size and window size of the pooling layer are 2.

[0120] In this embodiment, there are 8 computing cores (s=8), which are divided into two groups (s'=2). The input channel parallelism of the computing cores is m=32, and the output channel parallelism is n=32.

[0121] The specific processing process includes:

[0122] When calculating the preprocessing layer and the first convolutional layer, the number of output channels is less than n, and the feature map size is large (224). The working mode of intra-group feature map parallelism and inter-group feature map parallelism is adopted; that is, the 8 computing cores are divided into two groups, and each group adopts the intra-feature map parallelism. Since there are preprocessing modules and convolution modules in the pipeline structure of the computing core, the preprocessing layer and the first convolutional layer can be processed simultaneously.

[0123] When calculating the second convolution layer and the first pooling layer, the number of output channels is equal to n×s'. The working mode of intra-group feature map parallelism and inter-group output channel parallelism is adopted; the 8 computing cores are divided into two groups, and each group adopts the intra-feature map parallelism; the input feature data of the two groups are the same, but the convolution kernel data are different; because there are convolution modules and pooling modules in the pipeline structure of the computing core, the convolution layer and the pooling layer can be processed simultaneously.

[0124] When calculating the third convolutional layer, the number of output channels is equal to n×s / s'. The working mode of intra-group output channel parallelism and inter-group feature map parallelism is adopted; the 8 computing cores are divided into two groups, and each group adopts the output channel parallelism mode; the input feature data between the two groups are different, and the convolution kernel data is the same; because there is a convolution module in the pipeline structure of the computing core, the processing of the convolution layer can be realized.

[0125] When calculating each layer, the number of output channels is greater than or equal to n×s. The working mode of intra-group output channel parallelism and inter-group output channel parallelism is adopted; the 8 computing cores are divided into two groups, and each group is parallel along the output channel; the input feature data between the two groups is the same, and the convolution kernel data is different.

[0126] Through the above-mentioned dynamic reconstruction control operation, multi-core accelerated processing of convolutional neural networks is achieved.

[0127] In summary, the present invention uses dynamic reconstruction to achieve parallelism among multiple computing cores, improving the adaptability of the convolutional neural network accelerator to the large dynamic range of the number of convolutional neural network channels and feature map sizes. This further improves the actual computing efficiency of the high-parallelism convolutional neural network accelerator, thereby meeting the real-time requirements of deep algorithms with high computational complexity running on embedded devices.

[0128] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A dynamically reconfigurable convolutional neural network multi-core accelerator, characterized in that: It includes an instruction distribution module and a loading module, a routing module, a calculation module and an unloading module connected to and controlled by the instruction distribution module; The loading module loads input feature data and convolution kernel data from the outside; the routing module distributes the loaded data to the computing module for convolutional neural network acceleration processing; The calculation module outputs the result through the unloading module; The loading module includes multiple loaders, the routing module includes multiple routers, the computing module includes multiple computing cores, and the unloading module includes multiple unloaders; During the convolutional neural network processing, the instruction distribution module dynamically reconstructs the multi-way loaders, routers, computing cores and unloaders in groups, loads feature data and convolution kernel data in the computing core, and performs parallel convolutional neural network acceleration processing; The computing core includes a preprocessing module, a convolution module, a pooling module, a first FIFO-B, a second FIFO-B, a third FIFO-B, a fourth FIFO-B, a FIFO-A, a first bypass, a second bypass and a third bypass; The input ends of the first FIFO-B and the second FIFO-B are connected to the output ends of the corresponding redundant modules as the input ends of the computing core; the output end of the first FIFO-B is connected to the input ends of the preprocessing module and the first bypass respectively; the output end of the second FIFO-B is connected to the first input end of the convolution module; The output ends of the preprocessing module and the first bypass are simultaneously connected to the input end of the third FIFO-B; the output end of the third FIFO-B is connected to the second input end of the convolution module; the second bypass is connected between the second input end and the output end of the convolution module; The output end of the convolution module is connected to the input end of the fourth FIFO-B, and the output end of the fourth FIFO-B is connected to the input end of the pooling module and the third bypass; The output ends of the pooling module and the third bypass are simultaneously connected to the input end of FIFO-A; the output end of FIFO-A is connected to the input end of the redundancy module as the output end of the computing core; The preprocessing module is used to load input features and perform preprocessing of the convolutional neural network under the control of the instruction distribution control module; The convolution module is used to load convolution kernel data and input features to perform calculations of convolution layers, fully connected layers or nonlinear activation layers under the control of the instruction distribution control module; The pooling module is used to load the calculation result of the convolution module under the control of the instruction distribution control module to execute the calculation output of the pooling layer; The first bypass, the second bypass and the third bypass are connected or disconnected under the control of the instruction distribution control module.

2. The convolutional neural network multi-core accelerator according to claim 1, characterized in that: The number of computing cores is s and the number of routers is s ’ The number of loaders is s+s ’ The number of unloaders is s; By grouping, the computing cores are divided into s ’ groups; each group of computing cores is connected to a router; each router is connected to s / s ’ +1 loader connection; each compute core is connected to one unloader.

3. The convolutional neural network multi-core accelerator according to claim 2, characterized in that: Through group dynamic reconstruction, working in parallel within the feature map within the group and parallel within the feature map between groups, Under the allocation of each router, the convolution kernel data loaded by each computing core of all groups is the same convolution kernel data; the feature data loaded by each computing core is the feature data of different positions in the feature graph; The computing cores of all groups work together in parallel to perform accelerated processing of convolutional neural networks.

4. The convolutional neural network multi-core accelerator according to claim 2, characterized in that: Through group dynamic reconstruction, when working in the mode of parallel output channels within the group and parallel output channels between groups, Under the allocation of each router, the convolution kernel data loaded by each computing core of all groups is different convolution kernel data; the feature data loaded by each computing core is the feature data at the same position in the feature graph; The computing cores of all groups work together in parallel to perform accelerated processing of convolutional neural networks.

5. The convolutional neural network multi-core accelerator according to claim 2, characterized in that: Through group dynamic reconstruction, working in parallel within the feature map of the group and parallel output channels between groups, Under the allocation of each router, the convolution kernel data loaded by the computing cores in each group are the same, and the feature data loaded by the computing cores in each group are feature data at different positions in the feature map; while between computing core groups, the loaded feature data are the same, and the loaded convolution kernel data are different; the computing cores of all groups jointly and parallelly execute convolutional neural network acceleration processing.

6. The convolutional neural network multi-core accelerator according to claim 2, characterized in that: Through group dynamic reconstruction, working in parallel output channel within the group and parallel feature map between groups, Under the allocation of each router, the convolution kernel data loaded by the computing cores in each group are different convolution kernel data, and the feature data loaded by the computing cores in each group are feature data at the same position in the feature map; between computing core groups, the feature data loaded are feature data at different positions in the feature map, and the loaded convolution kernel data are the same convolution kernel data; the computing cores of all groups jointly and parallelly execute convolutional neural network acceleration processing.

7. The convolutional neural network multi-core accelerator according to claim 2, characterized in that: The router in the routing module includes data replicators U0-U s / s’ , data register R0~R s / s’-3 , data selector S0~S s / s’-1 ; Among them, the input end of replicator U0 is connected to the output end of feature loader, and replicators U1~U s / s’-1 Connect a data loader output, a replicator U s / s’ The input end is connected to the output end of the weight loader; The output of replicator U0 is connected to data registers R0 and R 4*(i-1)-3 Input terminal; ReplicatorU i The output terminals are connected to the data register R 4*(i-1)-1 and R 4*(i-1)-2 ; ReplicatorU s / s‘ The output terminals are connected to the data register R s / s’-3 and R 4*(i-1) Input terminal; The first input terminal of the data selector S0 is connected to the output terminal of the register R1, and the second input terminal is connected to the output terminal of the register R2; the data selector S i-1 The first input terminal is connected to register R 4*(i-1)-1 Output terminal, the second input terminal is connected to register R 4*(i-1) Output terminal; i = 2,…,s / s'; Registers R0 and R s / s’-3 The output terminal and data selector S0~S s / s’-1 As the output end of the routing module; wherein the output end of the register R0 and the data selector S1 is used as the input end of the first computing core; the data selector S s / s’-2 and register R 4s / s’-3 The output end of is used as the input end of the s / s' computing core; the data selector S i-1 and data selector S i+1 The output end of is used as the input end of the i-th computing core, i=2,…,s / s'-1.

8. The convolutional neural network multi-core accelerator according to claim 7, characterized in that: In the feature graph parallel mode within the computing core group, data selectors S0~S s / s’-1 Select the data of the second input terminal for output; In the parallel mode of the output channels in the computing core group, data selectors S0~S s / s’-1 Select the data of the first input terminal for output.

9. The convolutional neural network multi-core accelerator according to claim 1, characterized in that: Under the control of the instruction distribution control module, the preprocessing module, convolution module and pooling module of the computing core are dynamically connected in a streaming structure; the convolutional neural network is accelerated to process the input data according to the structure of the convolutional neural network; In the flow structure connection, including; 1) The preprocessing module is connected to the convolution module, and the pooling module is bypassed through the third bypass; And configure the preprocessing module as a preprocessing layer, and the convolution module as a convolution layer or a convolution layer + a nonlinear activation layer; 2) Connecting the convolution module to the pooling module, bypassing the preprocessing module through the first bypass; configuring the convolution module as a convolution layer or a convolution layer + a nonlinear activation layer, and configuring the pooling module as a pooling layer; 3) The convolution module is retained, and the preprocessing module and the pooling module are bypassed through the first bypass and the third bypass, and the convolution module is configured as a convolution layer, a fully connected layer, a convolution layer + a non-linear activation layer, or a fully connected layer + a non-linear activation layer.

Images