Neural network accelerator and acceleration method

By designing a convolutional operation module and an eltwise operation module in a neural network accelerator, the operation of the eltwise layer and the convolutional layer is supported, and the problem of not supporting eltwise layer acceleration in the existing technology is solved, reducing the load and bandwidth of the central processor demand, improve acceleration efficiency.

CN115049047BActive Publication Date: 2024-11-22ZHEJIANG XINMAI SILICON CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210768905.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-30
Publication Date
2024-11-22
Estimated Expiration
2042-06-30

AI Technical Summary

Technical Problem

Existing neural network accelerators do not support the acceleration of the eltwise layer and cannot effectively accelerate the residual network, resulting in an increase in the load of the central processor.

Method used

A neural network accelerator was designed, including a convolutional operation module and an eltwise operation module. By connecting the convolutional operation module and an eltwise operation module, it supports the operations of the eltwise layer and the convolutional layer, and directly performs eltwise operation based on the convolutional operation results. Reduce the number of segments and interrupts of a neural network.

Benefits of technology

It effectively reduces the load of the central processor, reduces frequent read and write operations to the memory, improves the efficiency of the multiplier array, and reduces the bandwidth requirement of neural network accelerators.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115049047B_ABST
    Figure CN115049047B_ABST
Patent Text Reader

Abstract

The present invention discloses a neural network accelerator and an acceleration method, wherein the accelerator includes a convolution operation module connected to a memory signal; an eltwise operation module connected to the convolution operation module and the memory signal respectively; the convolution operation module is used to receive a convolution operation instruction, and is also used to obtain convolution operation input data from the memory based on the convolution operation instruction, perform convolution operation on the convolution operation input data, generate convolution operation output data or data to be processed, and is also used to write the convolution operation output data into the memory, and send the data to be processed to the eltwise operation module; the eltwise operation module is used to receive an eltwise operation instruction, and is also used to obtain eltwise operation input data based on the eltwise operation instruction, perform eltwise operation on the eltwise operation input data, and write the obtained eltwise operation output data into the memory. The neural network accelerator provided by the present invention supports convolution, deconvolution and eltwise operation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a neural network acceleration technology. Background Art

[0002] Since the operations in deep learning networks are basically compute-intensive and memory-intensive, when they are deployed on edge devices (for example, surveillance cameras and smart door locks), in order to reduce costs and power consumption, the neural network is usually segmented into units of one convolutional layer, and the neural network is inferred by reading the input and weight data from the memory, thereby adapting to various parameter combinations in the neural network and balancing bandwidth and power consumption. Summary of the invention

[0003] In view of the shortcomings of the neural network accelerator provided by the prior art that does not support the acceleration of eltwise layers and is not suitable for accelerating residual networks, the present invention provides a neural network accelerator that supports eltwise layers and convolutional layers.

[0004] In order to solve the above technical problems, the present invention is solved by the following technical solutions:

[0005] A neural network accelerator is connected to an external memory signal, wherein the external memory may be a DRAM (dynamic random access memory), a SRAM (static random access memory), a ROM (read-only memory) or other existing publicly available memory, which is not limited in detail in the present invention;

[0006] The neural network accelerator comprises:

[0007] A convolution operation module, connected to the memory signal;

[0008] An eltwise operation module, connected to the convolution operation module and the memory signal respectively;

[0009] The convolution operation module is used to receive a convolution operation instruction, and is also used to obtain convolution operation input data from a memory based on the convolution operation instruction, perform a convolution operation on the convolution operation input data, generate convolution operation output data or data to be processed, and is also used to write the convolution operation output data into a memory, and send the data to be processed to an eltwise operation module;

[0010] The eltwise operation module is used to receive an eltwise operation instruction, and is also used to obtain eltwise operation input data based on the eltwise operation instruction, perform eltwise operation on the eltwise operation input data, and write the obtained eltwise operation output data into the memory;

[0011] Among them, the methods of obtaining eltwise operation input data include:

[0012] Reading corresponding data to be processed from the memory as eltwise operation input data;

[0013] Read the corresponding data to be processed from the memory, receive the data to be processed sent by the convolution operation module, and generate the eltwise operation input data.

[0014] As an implementable method, the convolution operation module includes:

[0015] A storage unit for caching convolution operation input data and intermediate calculation data;

[0016] A first operation unit, used for receiving and processing the data to be convolved, and obtaining corresponding first operation result data;

[0017] A first read-write unit is respectively connected to the storage unit, the first operation unit, the memory and the eltwise operation module by signal;

[0018] The first reading and writing unit:

[0019] Used to read the convolution operation input data from the memory and write the convolution operation input data into the storage unit;

[0020] Also used for reading the convolution operation input data from the memory as the data to be convolution operation, or reading the convolution operation input data or the intermediate calculation data read from the storage unit as the data to be convolution operation, and inputting the data to be convolution operation into the first operation unit;

[0021] Also used for receiving the first operation result data output by the first operation unit;

[0022] Also used for writing the first operation result data as intermediate calculation data into the storage unit, or writing it into the memory as convolution operation output data, or sending it to the eltwise operation module as data to be processed;

[0023] A first main control unit is signal-connected to the first read-write unit and the first operation unit respectively;

[0024] The first main control unit is used to receive a convolution operation instruction, and is also used to control the first read-write unit and the first operation unit to operate based on the convolution operation instruction.

[0025] Further:

[0026] The first operation unit includes a first buffer, a first calculation unit, a second buffer, a convolution post-processing unit and a third buffer which are connected in sequence, and the first main control unit is respectively connected to the first calculation unit and the convolution post-processing unit by signal, and is used to control the first calculation unit and the convolution post-processing unit to work;

[0027] The first buffer is connected to the first read-write unit signal, and the first read-write unit writes the read data to be convolved into the first buffer;

[0028] The first calculation unit is used to perform data rearrangement and multiplication and accumulation operations on the data to be convolved in the first buffer to obtain first calculation data;

[0029] The second buffer is used to store the first calculation data

[0030] The convolution post-processing unit is used to perform bias processing, activation processing and / or pooling processing on the first calculation data stored in the second buffer to obtain second calculation data;

[0031] The third buffer is used to store the second calculation data. The third buffer is connected to the first read-write unit signal. The first read-write unit uses the second calculation data read from the third buffer as the corresponding first operation result data.

[0032] Furthermore, the first computing unit includes:

[0033] Rearrange unit;

[0034] First register;

[0035] Second register;

[0036] Multiplier array;

[0037] The rearrangement unit is respectively connected to the first buffer, the first register and the second register signal, and is used to rearrange the data to be convolved stored in the first buffer to obtain corresponding rearranged data, and write the rearranged data into the first register or the second register;

[0038] The multiplier array is respectively connected to the second buffer, the first register and the second register signals, and is used to read the corresponding rearranged data from the first register or the second register, perform multiplication and accumulation calculations on the rearranged data, and write the obtained first calculation data into the second buffer.

[0039] As an implementable embodiment, the eltwise operation module includes:

[0040] A second operation unit is used to receive and process the data to be eltwise operated, and obtain corresponding eltwise operation output data;

[0041] A second read-write unit is respectively connected to the second operation unit, the memory and the convolution operation module signal;

[0042] The second reading and writing unit:

[0043] Used to obtain corresponding eltwise operation input data from the memory and / or the convolution operation module;

[0044] Also used for writing eltwise operation output data into the memory;

[0045] The second main control unit is respectively connected to the second read-write unit and the second operation unit by signal. The second main control unit is used to receive eltwise operation instructions and to control the operation of the second read-write unit and the second operation unit based on the eltwise operation instructions.

[0046] Further:

[0047] The second operation unit includes a fourth buffer, a second calculation unit and a fifth buffer connected in sequence; the fourth buffer is connected to the second read-write unit signal, and the second read-write unit writes the read data to be eltwise operated into the fourth buffer;

[0048] The second computing unit is used to perform multiplication or addition operations on the data to be eltwise operated in the fourth buffer to obtain corresponding eltwise operation output data;

[0049] The fifth buffer is used to store the eltwise operation output data. The fifth buffer is connected to the second read-write unit signal, and the second read-write unit will read the eltwise operation output data from the fifth buffer.

[0050] Further:

[0051] The second read-write unit includes a second read control unit and a second write control unit;

[0052] The second read control unit is respectively connected to the second main control unit, the memory, the convolution operation module and the fourth buffer signal;

[0053] The second write control unit is respectively connected to the second main control unit, the convolution operation module and the fifth buffer signal, and the second write control unit writes the eltwise operation output data into the memory through the convolution operation module.

[0054] As an implementable embodiment, it also includes:

[0055] The first interface module is respectively connected to the convolution operation module and the eltwise operation module by signal. The convolution operation module receives the convolution operation instruction through the first interface module, and the eltwise operation module receives the eltwise operation instruction through the first interface module.

[0056] As an implementable embodiment, it also includes:

[0057] The second interface module is respectively connected to the memory, the convolution operation module and the eltwise operation module signals.

[0058] The present invention also proposes a neural network acceleration method, based on any one of the above neural network accelerators, the method comprises the following steps:

[0059] Compile the target neural network, fuse the sequentially connected convolutional layers, activation layers and pooling layers to form a first fusion layer, fuse the sequentially connected convolutional layers and eltwise layers to form a second fusion layer, and use the single convolutional layer, the single eltwise layer, the first fusion layer and the second fusion layer as calculation layers;

[0060] Grouping the computing layers, and generating one or more acceleration tasks based on the grouping results;

[0061] Controlling the operation of the neural network accelerator based on the acceleration task is specifically as follows:

[0062] Generate corresponding convolution operation instructions and / or eltwise operation instructions based on the acceleration task;

[0063] The convolution operation instruction is sent to the convolution operation module of the neural network accelerator, and the convolution operation module obtains convolution operation input data from the memory based on the convolution operation instruction, implements the convolution operation corresponding to each computing layer in the acceleration task based on the convolution operation input data, generates convolution operation output data or data to be processed, and is also used to write the convolution operation output data into the memory, and send the data to be processed to the eltwise operation module;

[0064] The eltwise operation instruction is sent to the eltwise operation module of the neural network accelerator, and the eltwise operation module obtains eltwise operation input data from the memory and / or convolution operation module based on the eltwise operation instruction, implements the eltwise operation corresponding to each computing layer in the acceleration task based on the eltwise operation input data, and writes the obtained eltwise operation output data into the memory.

[0065] The present invention has significant technical effects due to the adoption of the above technical solution:

[0066] The present invention enables the neural network accelerator to support the operations of eltwise layers and convolutional layers through the design of an eltwise operation module. The design of signal connection between the eltwise operation module and the convolution operation module enables the eltwise operation module to perform eltwise operations directly based on the convolution operation results according to actual needs, which can effectively reduce the number of segments of the neural network and the number of interrupts, thereby reducing the load of the central processing unit.

[0067] The present invention uses a design in which a convolution operation module sends the data to be processed to an eltwise operation module, which can accelerate continuous convolution layers and eltwise layers once without the need to use an additional large-capacity SRAM.

[0068] The present invention designs the first calculation unit by first rearranging data and allowing the multiplier array to perform calculations based on the rearranged data, which can effectively improve the efficiency of the multiplier array. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0070] Figure 1 It is a schematic diagram of module connection of the neural network accelerator of the present invention;

[0071] Figure 2 yes Figure 1 Schematic diagram of module connection of convolution operation module 100;

[0072] Figure 3 is a schematic diagram of module connections of the first computing unit;

[0073] Figure 4 yes Figure 3A schematic diagram of the structure of the multiplier array 134;

[0074] Figure 5 yes Figure 1 Schematic diagram of module connection of eltwise operation module 200;

[0075] Figure 6 is a schematic diagram of the structure of the neural network accelerator of the present invention;

[0076] Figure 7 It is a schematic diagram of input data during data rearrangement;

[0077] Figure 8 It is a schematic diagram of output data output during data rearrangement;

[0078] Fig. 9 is a schematic diagram of rearranged data;

[0079] Fig.10 It is a schematic diagram of the data rearrangement process;

[0080] Fig.11 It is a schematic diagram of the convolution operation pipeline;

[0081] Fig.12 This is a schematic diagram of the eltwise operation pipeline;

[0082] Fig.13 This is a schematic diagram of the convolution + eltwise operation pipeline;

[0083] Fig.14 It is a structural diagram of the basic unit of the residual network. DETAILED DESCRIPTION

[0084] The present invention is further described in detail below in conjunction with embodiments. The following embodiments are for explanation of the present invention but the present invention is not limited to the following embodiments.

[0085] Embodiment 1: A neural network accelerator, such as Figure 1 As shown, it acts as a coprocessor, connected to external memory and central processing unit (CPU) signals respectively, to assist the central processing unit in accelerating the target neural network.

[0086] The neural network accelerator comprises:

[0087] A convolution operation module 100, connected to the memory signal;

[0088] An eltwise operation module 200 is respectively connected to the convolution operation module 100 and the memory signal;

[0089] A first interface module 300 is respectively connected to the convolution operation module 100, the eltwise operation module 200 and the central processing unit by signal, and the convolution operation module 100 and the eltwise operation module 200 perform data transmission with the central processing unit through the first interface module 300;

[0090] The second interface module 400 is respectively connected to the memory, the convolution operation module 100 and the eltwise operation module 200 by signal, and the convolution operation module 100 and the eltwise operation module 200 read and write the memory through the second interface module 400.

[0091] The convolution operation module 100:

[0092] Used to receive the convolution operation instruction sent by the central processing unit through the first interface module 300;

[0093] It is also used to obtain convolution operation input data from the memory through the second interface module 400 based on the convolution operation instruction, perform convolution operation on the convolution operation input data, and obtain corresponding convolution operation output data or data to be processed;

[0094] Also used for writing the obtained convolution operation output data into the memory through the second interface module 400;

[0095] It is also used to send the obtained data to be processed to the eltwise operation module 200;

[0096] The eltwise operation module 200:

[0097] Used to receive the eltwise operation instruction sent by the central processing unit through the first interface module 300;

[0098] It is also used to obtain eltwise operation input data from the memory and / or the convolution operation module 100 based on the eltwise operation instruction, perform eltwise operation on the eltwise operation input data, and write the obtained eltwise operation output data into the memory;

[0099] Among them, the methods of obtaining eltwise operation input data include:

[0100] The eltwise operation module 200 reads the corresponding data to be processed from the memory through the second interface module 400, and uses the obtained data to be processed as eltwise operation input data;

[0101] The eltwise operation module 200 receives the data to be processed sent by the convolution operation module 100, and also reads the corresponding data to be processed from the memory through the second interface module 400, and generates the eltwise operation input data based on the obtained data to be processed.

[0102] In the prior art, the convolutional layer is calculated by the neural network accelerator, and the other network layers (such as the eltwise layer and the fully connected layer) are calculated by the central processing unit. However, there are many eltwise layers in the residual network. When accelerating the residual network, the network needs to be segmented into many parts, and at least one interrupt is generated in each segment, which increases the processing overhead of the central processing unit.

[0103] In this embodiment, the design of the eltwise operation module 200 enables the neural network accelerator to support the operations of the eltwise layer and the convolution layer. The design of the signal connection between the eltwise operation module 200 and the convolution operation module 100 allows the eltwise operation module 200 to perform eltwise operations directly based on the convolution operation results according to actual needs, which can effectively reduce the number of segments of the neural network and the number of interrupts, thereby reducing the load on the central processing unit.

[0104] Further, refer to Figure 2 The convolution operation module 100 includes:

[0105] The storage unit 140 is used to cache the convolution operation input data and the intermediate calculation data. In this embodiment, the storage unit 140 uses a large-capacity on-chip SRAM (static random access memory);

[0106] A first operation unit 120, configured to receive and process the data to be convolved, and obtain corresponding first operation result data;

[0107] The first read / write unit 130 is respectively connected to the storage unit 140, the first operation unit 120, the memory and the eltwise operation module 200 by signals (for the neatness of the drawing, Figure 2 The connection between the first read-write unit 130 and the eltwise operation module 200 is not shown), wherein the first read-write unit 130 performs data transmission with the memory through the second interface module 400;

[0108] The first main control unit 110 is connected to the first read-write unit 130 and the first operation unit 120 by signals, and is also connected to an external central processing unit by signals through a first interface module 300;

[0109] The first main control unit 110 receives the convolution operation instruction sent by the central processing unit through the first interface module 300, and controls the first read-write unit 130 and the first operation unit 120 to work based on the convolution operation instruction.

[0110] The first reading and writing unit 130:

[0111] Used to read the convolution operation input data from the memory and write the convolution operation input data into the storage unit 140;

[0112] Reading convolution operation input data from the memory as data to be convolution operation, or reading convolution operation input data or intermediate calculation data from the storage unit 140 as data to be convolution operation, and inputting the data to be convolution operation into the first operation unit 120;

[0113] Also used to receive the first operation result data output by the first operation unit 120;

[0114] It is also used to write the first operation result data into the storage unit 140 as intermediate calculation data, or write it into the memory as convolution operation output data, or send it to the eltwise operation module 200 as data to be processed.

[0115] In the prior art, for the acceleration of multiple layers of continuous convolutional layers, the central processing unit generates acceleration tasks corresponding to the convolutional layers one by one, and the neural network accelerator reads the corresponding data from the memory and performs convolution operations based on the acceleration tasks to obtain the corresponding operation result data, and writes the operation result data into the memory to complete the operation of the corresponding convolutional layer. That is, for the acceleration of each convolutional layer, the operation of reading and writing the memory needs to be performed;

[0116] In actual applications, due to cost constraints and limited bandwidth on the end side, existing neural network accelerators have a large demand for bandwidth. When it performs inference in units of a convolutional layer, the instantaneous bandwidth is large, and it needs to compete for bandwidth with important IPs (intellectual property cores) such as the video encoder in the SOC (system on chip) system. Therefore, the existing solution has the problem of insufficient bandwidth, which leads to a decline in the performance of the neural network accelerator.

[0117] This embodiment can achieve one-time acceleration of multiple convolution layers through the design of the convolution operation module 100. One-time acceleration means that the multiple convolution layers are taken as an acceleration task. When the neural network accelerator performs the acceleration task, it only needs to read and write the memory once, that is, it only needs to read the convolution operation input data from the memory once, and only write the final convolution operation output data to the memory.

[0118] That is, the first operation unit 120 reads the corresponding intermediate calculation data from the storage unit 140 through the first read-write unit 130 to perform a convolution operation, and writes the obtained first operation result data into the storage unit 140 as new intermediate calculation data for use in the next convolution layer operation. There is no need to frequently read and write the memory, which effectively reduces the bandwidth requirements of the neural network accelerator.

[0119] Note that the convolution operation module 100 supports acceleration of convolution layers and deconvolution layers. When deconvolution is to be performed (deconvolution is converted into convolution to achieve this), a zero value needs to be inserted between real pixels in the input image.

[0120] Further:

[0121] The first operation unit 120 includes a first buffer, a first calculation unit, a second buffer, a convolution post-processing unit and a third buffer connected in sequence, and the first main control unit 110 is respectively connected to the first calculation unit and the convolution post-processing unit by signal, and is used to control the first calculation unit and the convolution post-processing unit to work;

[0122] The first buffer is signal-connected to the first read-write unit 130, and the first read-write unit 130 writes the read data to be convolved into the first buffer;

[0123] The first calculation unit is used to perform data rearrangement and multiplication and accumulation operations on the data to be convolved in the first buffer to obtain first calculation data;

[0124] The second buffer is used to store the first calculation data

[0125] The convolution post-processing unit is used to perform bias processing, activation processing and / or pooling processing on the first calculation data stored in the second buffer to obtain second calculation data;

[0126] The third buffer is used to store the second calculation data, and the third buffer is connected to the first read-write unit 130 by signal, and the first read-write unit 130 uses the second calculation data read from the third buffer as the corresponding first operation result data;

[0127] In this embodiment, the first buffer, the second buffer and the third buffer all use an on-chip SARM (static random access memory), which serves as a ping-pong buffer (buffer) for convolution pipeline calculation.

[0128] This embodiment can effectively improve the utilization rate of the multiplier performing the multiplication and accumulation operation by rearranging the data input by the first calculation unit and then performing the multiplication and accumulation operation on the rearranged data.

[0129] This embodiment can achieve one-time acceleration of the sequentially connected convolutional layer, activation layer and pooling layer by designing the convolution post-processing unit;

[0130] The convolution operation instruction contains the post-processing data of the corresponding convolution calculation layer, and the post-processing data includes bias data, activation data and pooling data;

[0131] The convolution calculation layer includes a convolution layer, a first fusion layer and a second fusion layer. The first fusion layer is obtained by layer fusion of sequentially connected convolution layers, activation layers and pooling layers. The second fusion layer is obtained by layer fusion of sequentially connected convolution layers and eltwise layers.

[0132] During the actual acceleration process, the first computing unit performs corresponding convolution operations and stores the calculation results in the second buffer. The convolution post-processing unit performs corresponding convolution post-processing on the data in the first buffer according to the configuration data, and stores the processing results in the third buffer. The first read-write module obtains the corresponding first operation result data based on the data in the third buffer.

[0133] Further,

[0134] Reference Figure 3 , the first computing unit includes:

[0135] Rearrangement unit 131;

[0136] First register 132;

[0137] The second register 133;

[0138] Multiplier array 134 (mac array);

[0139] The rearrangement unit 131 is respectively connected to the first buffer, the first register 132 and the second register 133 by signals, and the rearrangement unit 131 is used to rearrange the data to be convolved stored in the first buffer, obtain corresponding rearranged data, and write the rearranged data into the first register 132 or the second register 133;

[0140] The multiplier array 134 is connected to the second buffer, the first register 132 and the second register 133 signals respectively. The multiplier array 134 is used to read the corresponding rearranged data from the first register 132 or the second register 133, perform multiplication and accumulation calculations on the rearranged data, and write the obtained first calculation data into the second buffer.

[0141] In this embodiment, the first register 132 and the second register 133 perform a ping-pong operation. When the data to be convolved (such as an input image) is input into the multiplier array 134, the corresponding data is rearranged, and the obtained rearranged data is written into the first register 132 or the second register 133, and the rearrangement of the next row can be started. At the same time, the multiplier array 134 reads the corresponding rearranged data from the second register 133 or the first register 132 for convolution calculation, which can improve the utilization rate of the multiplier array 134.

[0142] In this embodiment, refer to Figure 4 :

[0143] The multiplier array 134 includes four calculation arrays, each of which corresponds to a different input channel;

[0144] The computing array includes several computing units (such as Figure 4 cu), each computing unit includes 3 multipliers;

[0145] In the actual calculation process, after the multiplication calculation, the calculation results of the cu at the same position are added together and written into the output buffer of the first calculation unit, that is, the second buffer;

[0146] In this embodiment, 384 multipliers are used. Each calculation will read in 4 input channels and calculate 16 output channels. The multiplier array 134 calculates 16 output pixels in the horizontal direction, 2 filter kernels in the vertical direction, and 4 channels in the channel direction. The weight data is reused to reduce power consumption.

[0147] Since there are many convolutional layers with different resolutions in a neural network, the first calculation module in this embodiment calculates 16 pixels horizontally. However, in the actual network, there are a large number of convolutional layers whose width is not a multiple of 16, so the data needs to be rearranged.

[0148] For example, if the convolution layer outputs 5x5, if the input data is not rearranged, it needs to be calculated 5 times, each time 5 output pixels are calculated, and the utilization rate of the multiplier is 25 / (16*5)=0.3125; but after rearrangement, multiple rows are spliced, the first 4 rows are spliced ​​into 4x5=20 output pixels, the first 16 output pixels are taken as a new row, and the remaining 4 output pixels are spliced ​​with the fifth row, thus converting it into a new block of 16 (width) x2 (height), and the utilization rate of the multiplier is 25 / 32=0.78125;

[0149] In the prior art, the CPU is often used to rearrange data before the neural network accelerator runs. The purpose of the rearrangement is to use matrix multiplication to perform convolution calculations, and each layer needs to be rearranged by the CPU before acceleration. This solution cannot run multiple convolution layers continuously, which increases the burden on the CPU.

[0150] In this embodiment, the rearrangement unit 131 is used to perform real-time rearrangement, and only the input data is rearranged without the participation of the CPU, in order to improve the utilization rate of the multiplier;

[0151] In this embodiment, the weight data is rearranged by the CPU when the neural network compiler compiles the neural network. However, when the neural network accelerator is running, the input data is rearranged by the rearrangement unit 131 without the participation of the CPU. Multiple layers of convolutional neural network layers can be run continuously, reducing the number of interruptions.

[0152] Further:

[0153] The first read-write unit 130 includes a first read control unit and a first write control unit;

[0154] The first read control unit is respectively connected to the first main control unit 110, the memory, the storage unit 140 and the first buffer signal

[0155] The first write control unit is respectively connected to the first main control unit 110, the memory, the storage unit 140, the eltwise operation module 200 and the third buffer signal.

[0156] In this embodiment, the neural network for image processing is accelerated, and the convolution operation input data read at this time includes the corresponding input image;

[0157] In this scenario, the first read control unit:

[0158] Used to calculate the width and height of the current input image block and its address in the memory;

[0159] It is also used to fold the convolution operation input data read from the memory (join different rows together and fold them into 16 pixels per row) and write them into the storage unit 140, so as to facilitate the reuse of the convolution operation input data by subsequent convolution calculations;

[0160] It is also used to read the input image currently undergoing the convolution operation from the storage unit 140, expand it into the real width and height, and write it into the first buffer;

[0161] It is also used to insert a zero value between real pixels in the input image when performing deconvolution (deconvolution is converted into convolution);

[0162] The first write control unit is used to calculate the width and height of the corresponding output image, and is also used to calculate the address of the output image in the memory or the address in the storage unit 140.

[0163] Further, refer to Figure 5 , the eltwise operation module 200 includes:

[0164] The second operation unit 220 is used to receive and process the data to be eltwise operated, and obtain the corresponding eltwise operation output data;

[0165] The second read-write unit 230 is signal-connected to the second operation unit 220 and the convolution operation module 100, respectively, wherein the second read-write unit 230 is signal-connected to the memory through the second interface module 400;

[0166] The second read-write unit 230 is used to obtain corresponding eltwise operation input data from the memory and / or the convolution operation module 100, and is also used to write eltwise operation output data into the memory. In this embodiment, the second read-write unit 230 writes the corresponding data into the memory through the first write control unit of the convolution operation module 100;

[0167] The second main control unit 210 is respectively connected to the first interface module 300, the second read-write unit 230 and the second operation unit 220 by signal, and is used to receive eltwise operation instructions through the first interface module 300, and is also used to control the operation of the second read-write unit 230 and the second operation unit 220 based on the eltwise operation instructions.

[0168] In this embodiment, the eltwise calculation instruction includes configuration data corresponding to the eltwise calculation layer, and the eltwise calculation layer includes the eltwise layer and the second fusion layer (convolution + eltwise), wherein the second fusion layer involves both convolution operation and eltwise calculation, so the second fusion layer belongs to both the convolution calculation layer and the eltwise calculation layer;

[0169] Eltwise operation requires two input data. In this embodiment, based on the configuration data of the eltwise calculation layer corresponding to the eltwise calculation instruction, the data to be eltwise calculated is obtained;

[0170] When the computation layer is a separate eltwise layer, both input data are read from the memory;

[0171] When the computing layer is the second fusion layer, one input data is the convolution result corresponding to the convolution layer. At this time, one input data is read from the memory, and the convolution operation output data sent by the convolution operation module 100 through the first read-write module is received to obtain another input data.

[0172] Further:

[0173] The second operation unit 220 includes a fourth buffer, a second calculation unit and a fifth buffer connected in sequence;

[0174] The fourth buffer is connected to the second read-write unit 230 by signal, and the second read-write unit 230 writes the read data to be eltwise operated into the fourth buffer;

[0175] The second computing unit is used to perform multiplication or addition operations on the data to be eltwise operated in the fourth buffer to obtain corresponding eltwise operation output data;

[0176] The fifth buffer is used to store the eltwise operation output data. The fifth buffer is connected to the second read-write unit 230 signal, and the second read-write unit 230 will read the eltwise operation output data from the fifth buffer.

[0177] Further:

[0178] The second read-write unit 230 includes a second read control unit and a second write control unit;

[0179] The second read control unit is respectively connected to the second main control unit 210, the second interface module 400, the first write control unit and the fourth buffer signal, wherein the second read control unit is used to read the corresponding to-be-processed data from the memory through the second interface module 400, and is also used to read the to-be-processed data written by the first write control unit;

[0180] The second write control unit is respectively connected to the second main control unit 210, the first write control unit and the fifth buffer signal, and the second write control unit writes the eltwise operation output data into the memory through the first write control unit.

[0181] The convolution operation process is read by block. In this embodiment, when the first read-write unit 130 writes the corresponding data into the memory, the width, height and memory address of the output data are calculated in units of blocks.

[0182] The separate eltwise operation is read by row. In this embodiment, the second read / write unit 230 calculates the width, height and memory address of the output data in units of rows.

[0183] The second fusion layer needs to perform convolution operation and then eltwise operation;

[0184] A person skilled in the art may write the obtained convolution layer calculation output block into a large-capacity SRAM for temporary storage after the convolution operation module 100 completely completes the convolution operation, and then start the eltwise operation after all channels are calculated, and the eltwise operation module 200 reads the convolution layer calculation result from the large-capacity SRAM to perform the eltwise operation. However, this solution requires the use of a large-capacity SRAM to temporarily store the convolution layer calculation output block;

[0185] In this embodiment, by designing the second read control unit and the second write control unit;

[0186] When the calculation layer is a separate eltwise layer, the second read control unit and the second write control unit perform corresponding calculations in units of rows, and when writing eltwise operation output data into the memory, the first write control unit is only used for data transmission;

[0187] When the computing layer is the second fusion layer, the second main control unit 210 modifies the read input data mode of the second read control unit to read by block, and when writing the eltwise operation output data into the memory, the second write control unit is only used for data transmission, and the first write control unit completes the calculation of the corresponding data in blocks;

[0188] The solution proposed in this embodiment does not require a large-capacity SRAM, and the convolution operation can mask the delay of the eltwise operation.

[0189] The following is a detailed introduction to the neural network accelerator proposed in this embodiment through a specific case;

[0190] Reference Figure 6 :

[0191] The central processing unit is cpu;

[0192] The first interface module 300 uses an apb (Advanced Peripheral Bus) interface, and the central processing unit cpu informs the neural network accelerator of the current acceleration task through the apb interface, that is, sends the corresponding convolution operation instruction to the convolution operation module 100 through the apb interface, and sends the corresponding eltwise operation instruction to the eltwise operation module 200;

[0193] The second interface module 400 uses an axi (Advanced eXtensible Interface) interface, which includes a read interface axi_rd and a write interface axi_wr, for transferring data with the memory;

[0194] For the convolution operation module 100:

[0195] The first main control unit 110 is cov_top, the first read control unit is cov_rd, the first calculation unit is pe, the convolution post-processing unit is Bias+relu+pooling, and the first write control unit is Ddr_wr. The first main control unit 110 controls the startup and interrupt generation of the four units, namely, the first read control unit, the first calculation unit, the convolution post-processing unit and the first write control unit;

[0196] The convolution post-processing unit Bias+relu+pooling is used for bias processing, activation processing, and pooling processing. The post-processing operation sequence can also be adjusted through the convolution post-processing data, and each post-processing function has a separate switch that can be turned on and off;

[0197] The storage unit 140 is bufferD, which is a large-capacity on-chip SRAM (static random access memory), mainly used to cache convolution operation input data, convolution operation output data, and corresponding weight data to achieve layer fusion and save bandwidth consumption; its storage of weight data and the use of weight data belong to the prior art in this field. The present invention optimizes the acquisition and reuse of input data and the storage and reuse of output data, so the use of weight data is not introduced in detail;

[0198] The first buffer is bufferA, the second buffer is bufferB, and the third buffer is bufferC. The above three buffers are all ping-pong buffers. In this embodiment, a small-capacity SRAM (static random access memory) is used, which is mainly used for pipeline operation. Here, the small capacity corresponds to the large capacity of the storage unit 140. Those skilled in the art can determine the capacity of each buffer and the storage unit 140 according to actual needs.

[0199] For the eltwise operation module 200:

[0200] The second main control unit 210 is elt_top, the second read control unit is elt_rd, the second calculation unit is elt_pe, and the second write control unit is elt_wr. The second main control unit 210 is used to control the startup of the second read control unit, the second calculation unit, and the second write control unit;

[0201] The second read control unit mainly calculates the width, height and address in the memory of the input data of the eltwise layer, and controls axi_rd;

[0202] The second calculation unit can calculate 16 output pixels at a time;

[0203] The second write control unit is mainly responsible for the width, height and memory address of the output data of the eltwise layer, as well as the control of axi_wr.

[0204] Data rearrangement:

[0205] For a convolution kernel of 3x3, a span of 1x1, and an input resolution of 7x7 (the input data is as follows Figure 7 As shown), the output resolution is 5x5 (output data output as shown Figure 8 As an example, the convolution of

[0206] In this embodiment, the first buffer adopts a single-port SRAM, so only one row of data in the block can be read in one clock cycle. The preparation time for data rearrangement is output channel / 2. If the current calculated output channel is 16, the current row input data stored in the register array needs to be reused 8 times, and there are 8 cycles to prepare the current row. At most 8 rows in the output can be assembled into a new row.

[0207] In this embodiment, the calculation unit uses 3 multipliers, so three rows of data need to be prepared to be input into the calculation unit;

[0208] line represents the rearranged data line, the width of the multiplier array 134 is a fixed value, which is 16 in this embodiment, and the number of pixels in the rearranged data line is also fixed to 16;

[0209] Cycle represents the clock frequency. One cycle can only read one line of input data, so multiple lines of input data need to be spliced;

[0210] Reference Fig. 9, based on the input data input, line0-line8 are obtained by rearrangement. Cycle0 indicates that line0, line1 and line2 are sent for the first time. Cycle1 indicates that line3, line4 and line5 are sent for the first time. Cycle2 indicates that line6, line7 and line8 are sent for the first time. After 3 cycles, the pixel values ​​(A0-A4, B0-B4, C0-C4 and D) colored in the output data output can be obtained.

[0211] As can be seen from the above, in this embodiment, 16 pixels are calculated in the horizontal direction in the calculation unit, so there are 16 pixels in one row of data after rearrangement. Fig.10 , 4 cycles are needed to get an output line, and the order of obtaining line0~line8 is to first obtain line0~line2 through 4 cycles, then obtain line3~line5 through 4 cycles, and then obtain line6~line8 through 4 cycles; when the pixels stored in the temporary buffer are the input pixels of the last two rows of the rearranged output data output, the temporarily stored pixels are directly taken out from the temporary buffer, and then a new row of input pixels is read, and then the rearrangement is continued directly afterwards.

[0212] Convolutional layer operations:

[0213] When performing a single convolution layer (for simplicity, we take one channel at a time as an example, the scheme is image reuse, multi-layer output (single channel in the input channel vs multiple convolution kernels), one layer of input channel image corresponds to multiple layers of output convolution kernels for convolution;

[0214] In this case, the number of convolution kernels is 24, and the output needs to save a maximum of 24 sets of convolution output images (i.e., bufferA). At the same time, the data accumulation operation after convolution of different input channel images needs to be completed until the accumulation is completed, and a corresponding column of convolution output data (maximum 24 groups) is generated and cached in bufferB. Finally, the obtained data is input into the memory, and the calculation method of the next column block (maximum 24 groups) is exactly the same until the calculation of all channel data is completed.

[0215] The calculation method is:

[0216] In units of blocks, the input data includes several data blocks, and the data blocks include several pieces of data. The channels corresponding to the pieces of data are different. The convolution calculation is performed on the target data block to obtain the convolution output corresponding to the target data block, and the next data block is used as the target data block. Repeat this step until all data blocks are calculated;

[0217] The specific processing steps for performing convolution calculation on the target data block and obtaining the convolution output corresponding to the target data block are as follows:

[0218] 1. Prepare data:

[0219] The first read control unit cov_rd reads the slice data corresponding to the first channel of the target data block from the memory, and reads the convolution kernel data (maximum 24) corresponding to the input channel data, and writes them into bufferA;

[0220] Write the input data that needs to be read multiple times into bufferD so that the next time the same data is read, it can be read directly from bufferD.

[0221] 2. Convolution calculation:

[0222] The first computing unit pe reads the cache data from bufferA and performs convolution calculation to generate corresponding convolution calculation results, and writes the convolution calculation results into bufferB.

[0223] 3. Accumulation operation:

[0224] According to steps 1 and 2, the convolution calculation results corresponding to the target slice data corresponding to each input channel are generated in turn, and the convolution calculation results are accumulated to generate the final convolution data Cov and written into Figure 1 in bufferB.

[0225] The convolution operation pipeline is as follows Fig.11 As shown, when the computing layer is a convolution layer or the first fusion layer (convolution layer + activation layer + pooling layer), convolution operation is performed based on the pipeline. When there are subsequent computing layers, the calculation result of the current computing layer is written into the storage unit 140 bufferD, and the operation of the next computing layer directly reads data from bufferD without reading the memory again. For example, in an acceleration task corresponding to multiple convolution layers, only the first convolution layer needs to read data from the memory when performing operations, and the last convolution layer writes the operation result to the memory to avoid frequent reading and writing of the memory.

[0226] Operation of eltwise layer:

[0227] The Eltwise layer is calculated in units of rows, which is different from the convolutional layer which is calculated in units of blocks;

[0228] According to the width of the input image and the internal buffer depth, we can calculate the maximum number of rows that can be calculated in one eltwise calculation. If the buffer depth is A and the width of the input image is W, the number of rows read in at one time is floor(A / W), starting from the first channel in the order of the channels until the last channel is calculated.

[0229] Reference Fig.12The eltwise operation pipeline shown in the figure, the specific steps of eltwise operation are:

[0230] 1. The second reading control unit elt_rd determines the actual number of rows to be read according to the actual input image height (because the actual number of rows may be less than the calculated number of rows), and then reads the input image from the memory through the axi bus and stores it in the fourth buffer bufferE.

[0231] 2. The second computing unit elt_pe reads input data from the fourth buffer bufferE and sends it to the internal computing unit (the second computing unit only includes a multiplier array, which is the same as the multiplier array 134 in the first computing unit pe, both of which include several computing units), and then writes it into the fifth buffer bufferF after calculation.

[0232] 3. The second write control unit elt_wr writes the final calculation result into the memory.

[0233] Operation of the second fusion layer (convolution + eltwise):

[0234] When running the second fusion layer, the pipeline diagram is as follows Fig.13 As shown in the figure, during the operation, the first write control unit ddr_wr and the second read control unit elt_rd are started at the same time, and the image block to be written into the memory is directly sent to the second read control unit elt_rd. In this way, the convolutional layer pipeline and the eltwise layer pipeline are connected, and the actual measurement shows that the speed can be increased by 5%:

[0235] The basic unit of residual network is Fig.14 As shown, it includes a first convolutional layer, a second convolutional layer and an eltwise layer connected in sequence, wherein the input data of the eltwise layer includes the input data of the corresponding residual network and the output data of the second convolutional layer. The neural network accelerator can fuse the second convolutional layer and the eltwise layer to form a second fusion layer, and the second convolutional layer is directly sent to the second read control unit elt_rd. Therefore, during the operation of the eltwise layer, only the input data of the corresponding residual network needs to be read from the memory, and finally the calculation result of the eltwise layer is written into the memory, which can save the write memory bandwidth of the second convolutional layer and the read bandwidth of the eltwise layer.

[0236] Embodiment 2, a neural network acceleration method, based on the neural network accelerator described in embodiment 1, comprises the following steps:

[0237] S100, compiling the target neural network, fusing the sequentially connected convolutional layers, activation layers and pooling layers to form a first fusion layer, fusing the sequentially connected convolutional layers and eltwise layers to form a second fusion layer, and using a single convolutional layer, a single eltwise layer, the first fusion layer and the second fusion layer as calculation layers;

[0238] Here, "separate" means that it does not belong to the first fusion layer or the second fusion layer;

[0239] In this embodiment, the computing layer includes a convolution computing layer and an eltwise computing layer;

[0240] The single convolution layer, the first fusion layer, and the second fusion layer belong to the convolution calculation layer;

[0241] The separate eltwise layer and the second fusion layer belong to the eltwise calculation layer;

[0242] The identification of the convolution calculation layer and the eltwise calculation layer is used to control the operation of the convolution operation module 100 and the eltwise operation module 200.

[0243] S200, grouping the computing layers, and generating one or more acceleration tasks based on the grouping results;

[0244] Those skilled in the art can set the grouping rules according to actual conditions. In this embodiment, the types that can be divided into the same group are:

[0245] Convolution layer + convolution layer;

[0246] Convolution layer + first fusion layer;

[0247] First fusion layer + convolution layer;

[0248] First fusion layer + first fusion layer;

[0249] Convolution layer + second fusion layer;

[0250] First fusion layer + second fusion layer;

[0251] Note that the above format is only used to indicate two computational layers connected in sequence. For example, the computational layers are a convolutional layer, a convolutional layer, a first fusion layer, and a second fusion layer connected in sequence, which can be divided into the same group.

[0252] S300, controlling the neural network accelerator to work based on the acceleration task;

[0253] Specifically:

[0254] S310, generating corresponding convolution operation instructions and / or eltwise operation instructions based on the acceleration task;

[0255] In this embodiment, convolution operation instructions and / or eltwise operation instructions are generated based on the categories of each computing layer corresponding to the acceleration task;

[0256] S320, sending the convolution operation instruction to the convolution operation module 100 of the neural network accelerator, and the convolution operation module 100 obtains the convolution operation input data from the memory based on the convolution operation instruction, implements the convolution operation corresponding to each computing layer in the acceleration task based on the convolution operation input data, generates convolution operation output data or data to be processed, and is also used to write the convolution operation output data into the memory, and send the data to be processed to the eltwise operation module 200;

[0257] During actual operation, operations are started on each computing layer in turn;

[0258] When there is no next calculation layer, it means that the operation is completed, and the convolution operation output data is written into the memory;

[0259] When the current calculation layer is the second fusion layer, the operation in the lower part is an eltwise operation, so the obtained operation result is used as the data to be processed, and the subsequent operation is completed by the eltwise operation module 200.

[0260] S330, sending the eltwise operation instruction to the eltwise operation module 200 of the neural network accelerator, and the eltwise operation module 200 obtains eltwise operation input data from the memory and / or the convolution operation module 100 based on the eltwise operation instruction, implements the eltwise operation corresponding to each computing layer in the acceleration task based on the eltwise operation input data, and writes the obtained eltwise operation output data into the memory.

[0261] Two input data are required in the eltwise operation process. When the calculation layer is a separate eltwise layer, the two required input data are directly read from the memory. When the calculation layer is the second fusion layer, one input data is the corresponding convolution result. At this time, the eltwise operation module 200 and the convolution operation module 100 cooperate with each other to directly read the calculation result obtained by the convolution operation module 100, and only the other input data needs to be read from the memory.

[0262] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0263] It should be noted that:

[0264] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0265] In addition, it should be noted that any equivalent or simple changes made according to the structure, features and principles described in the patent concept of the present invention are included in the protection scope of the patent of the present invention. Those skilled in the art of the present invention can make various modifications or supplements to the specific embodiments described or replace them in a similar manner, as long as they do not deviate from the structure of the present invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.

Claims

1. A neural network accelerator connected to an external memory signal, characterized in that: The neural network accelerator comprises: A convolution operation module, connected to the memory signal; An eltwise operation module, connected to the convolution operation module and the memory signal respectively; The convolution operation module is used to receive a convolution operation instruction, and is also used to obtain convolution operation input data from a memory based on the convolution operation instruction, perform a convolution operation on the convolution operation input data, generate convolution operation output data or data to be processed, and is also used to write the convolution operation output data into a memory, and send the data to be processed to an eltwise operation module; The eltwise operation module is used to receive an eltwise operation instruction, and is also used to obtain eltwise operation input data based on the eltwise operation instruction, perform eltwise operation on the eltwise operation input data, and write the obtained eltwise operation output data into the memory; Among them, the methods of obtaining eltwise operation input data include: Reading corresponding data to be processed from the memory as eltwise operation input data; Reading corresponding data to be processed from the memory, and receiving the data to be processed sent by the convolution operation module, to generate the eltwise operation input data; The eltwise operation module includes: A second operation unit is used to receive and process the data to be eltwise operated, and obtain the corresponding eltwise operation output data; A second read-write unit is respectively connected to the second operation unit, the memory and the convolution operation module signal; The second reading and writing unit: Used to obtain corresponding eltwise operation input data from the memory and / or the convolution operation module; Also used for writing eltwise operation output data into the memory; The second main control unit is respectively connected to the second read-write unit and the second operation unit by signal. The second main control unit is used to receive eltwise operation instructions and to control the operation of the second read-write unit and the second operation unit based on the eltwise operation instructions.

2. The neural network accelerator according to claim 1, characterized in that: The convolution operation module comprises: A storage unit for caching convolution operation input data and intermediate calculation data; A first operation unit, used for receiving and processing the data to be convolved, and obtaining corresponding first operation result data; A first read-write unit is respectively connected to the storage unit, the first operation unit, the memory and the eltwise operation module by signal; The first reading and writing unit: Used to read the convolution operation input data from the memory and write the convolution operation input data into the storage unit; Also used for reading the convolution operation input data from the memory as the data to be convolution operation, or reading the convolution operation input data or the intermediate calculation data read from the storage unit as the data to be convolution operation, and inputting the data to be convolution operation into the first operation unit; Also used for receiving the first operation result data output by the first operation unit; Also used for writing the first operation result data as intermediate calculation data into the storage unit, or writing it into the memory as convolution operation output data, or sending it to the eltwise operation module as data to be processed; A first main control unit is signal-connected to the first read-write unit and the first operation unit respectively; The first main control unit is used to receive a convolution operation instruction, and is also used to control the first read-write unit and the first operation unit to operate based on the convolution operation instruction.

3. The neural network accelerator according to claim 2, characterized in that: The first operation unit includes a first buffer, a first calculation unit, a second buffer, a convolution post-processing unit and a third buffer which are connected in sequence, and the first main control unit is respectively connected to the first calculation unit and the convolution post-processing unit by signal, and is used to control the first calculation unit and the convolution post-processing unit to work; The first buffer is connected to the first read-write unit signal, and the first read-write unit writes the read data to be convolved into the first buffer; The first calculation unit is used to perform data rearrangement and multiplication and accumulation operations on the data to be convolved in the first buffer to obtain first calculation data; The second buffer is used to store the first calculation data The convolution post-processing unit is used to perform bias processing, activation processing and / or pooling processing on the first calculation data stored in the second buffer to obtain second calculation data; The third buffer is used to store the second calculation data. The third buffer is connected to the first read-write unit signal. The first read-write unit uses the second calculation data read from the third buffer as the corresponding first operation result data.

4. The neural network accelerator according to claim 3, characterized in that: The first computing unit comprises: Rearrange unit; First register; Second register; Multiplier array; The rearrangement unit is respectively connected to the first buffer, the first register and the second register signal, and is used to rearrange the data to be convolved stored in the first buffer to obtain corresponding rearranged data, and write the rearranged data into the first register or the second register; The multiplier array is respectively connected to the second buffer, the first register and the second register signals, and is used to read the corresponding rearranged data from the first register or the second register, perform multiplication and accumulation calculations on the rearranged data, and write the obtained first calculation data into the second buffer.

5. The neural network accelerator according to claim 1, characterized in that: The second operation unit includes a fourth buffer, a second calculation unit and a fifth buffer connected in sequence; the fourth buffer is connected to the second read-write unit signal, and the second read-write unit writes the read data to be eltwise operated into the fourth buffer; The second computing unit is used to perform multiplication or addition operations on the data to be eltwise operated in the fourth buffer to obtain corresponding eltwise operation output data; The fifth buffer is used to store the eltwise operation output data. The fifth buffer is connected to the second read-write unit signal, and the second read-write unit will read the eltwise operation output data from the fifth buffer.

6. The neural network accelerator according to claim 5, characterized in that: The second read-write unit includes a second read control unit and a second write control unit; The second read control unit is respectively connected to the second main control unit, the memory, the convolution operation module and the fourth buffer signal; The second write control unit is respectively connected to the second main control unit, the convolution operation module and the fifth buffer signal, and the second write control unit writes the eltwise operation output data into the memory through the convolution operation module.

7. The neural network accelerator according to any one of claims 1 to 4, characterized in that: Also includes: The first interface module is respectively connected to the convolution operation module and the eltwise operation module by signal. The convolution operation module receives the convolution operation instruction through the first interface module, and the eltwise operation module receives the eltwise operation instruction through the first interface module.

8. The neural network accelerator according to any one of claims 1 to 4, characterized in that: Also includes: The second interface module is respectively connected to the memory, the convolution operation module and the eltwise operation module signals.

9. A neural network acceleration method, based on the neural network accelerator according to any one of claims 1 to 8, characterized in that: The following steps are involved: Compile the target neural network, fuse the sequentially connected convolutional layers, activation layers and pooling layers to form a first fusion layer, fuse the sequentially connected convolutional layers and eltwise layers to form a second fusion layer, and use the single convolutional layer, the single eltwise layer, the first fusion layer and the second fusion layer as calculation layers; Grouping the computing layers, and generating one or more acceleration tasks based on the grouping results; Controlling the operation of the neural network accelerator based on the acceleration task is specifically as follows: Generate corresponding convolution operation instructions and / or eltwise operation instructions based on the acceleration task; The convolution operation instruction is sent to the convolution operation module of the neural network accelerator, and the convolution operation module obtains convolution operation input data from the memory based on the convolution operation instruction, implements the convolution operation corresponding to each computing layer in the acceleration task based on the convolution operation input data, generates convolution operation output data or data to be processed, and is also used to write the convolution operation output data into the memory, and send the data to be processed to the eltwise operation module; The eltwise operation instruction is sent to the eltwise operation module of the neural network accelerator, and the eltwise operation module obtains eltwise operation input data from the memory and / or convolution operation module based on the eltwise operation instruction, implements the eltwise operation corresponding to each computing layer in the acceleration task based on the eltwise operation input data, and writes the obtained eltwise operation output data into the memory.