Low hardware overhead convolution computing structure and calculation method for lightweight neural networks

By designing a low-hardware-overhead convolutional computation structure, replacing multiplication with shift addition, and combining data partitioning and channel enhancement techniques, the problem that existing neural network accelerators cannot efficiently process depthwise separable convolutions and standard convolutions is solved, achieving both hardware resource savings and improved computational efficiency.

CN114997382BActive Publication Date: 2024-11-05SOUTHEAST UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210526577.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-16
Publication Date
2024-11-05
Estimated Expiration
2042-05-16

AI Technical Summary

Technical Problem

Existing neural network accelerators cannot efficiently process depthwise separable convolutions and standard convolutions, resulting in high hardware resource consumption and low utilization.

Method used

Design a low-hardware-overhead convolutional computation structure, including a preprocessing module, a computation module, and an accumulation module. It uses shift addition to replace multiplication and supports the computation of standard convolution and depthwise separable convolution through data partitioning and channel enhancement techniques.

Benefits of technology

It effectively reduces the area and power consumption of hardware circuits, while improving the utilization rate of computing modules.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114997382B_ABST
    Figure CN114997382B_ABST
Patent Text Reader

Abstract

The present invention discloses a low hardware overhead convolution calculation structure for lightweight neural networks, including a preprocessing module, a calculation module and an accumulation module. The calculation module is composed of a calculation array, a configurable addition tree and a common addition tree. The calculation array is composed of calculation slices, each of which is composed of calculation units. The calculation unit includes a shift operator and an adder. The calculation unit uses shift addition instead of multiplication calculation, which effectively reduces the area and power consumption of the hardware circuit while ensuring good accuracy. The configurable addition tree unit supports standard convolution and depth-separable convolution calculations, unifies the calculation hardware, and effectively reduces the area and power consumption of the convolution calculation hardware circuit. The present invention also discloses a calculation method for a low hardware overhead convolution calculation structure for lightweight neural networks. Through data blocking and channel enhancement technology, the calculation module is maximized in a working state, effectively improving the utilization rate of the calculation unit.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of neural network hardware acceleration, and in particular to a low hardware overhead convolution computing structure and a computing method for lightweight neural networks, belonging to the technical field of computing, reasoning, and technology. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology, more and more artificial intelligence technologies have penetrated into our lives and production. Among them, convolutional neural networks based on deep learning have been widely used in many fields such as autonomous driving, smart transportation, unmanned supermarkets, smart medical care, and smart factories, greatly improving our production and quality of life. However, advanced convolutional neural networks have a huge number of parameters and computational complexity, and cannot be deployed in embedded devices with weak computing performance, limited battery and storage, which poses a huge challenge. In order to meet the needs of practical applications, the network structure has developed towards a lightweight structure, and depthwise separable convolution is widely used instead of standard convolution. The standard convolution is decomposed into depthwise convolution and pointwise convolution, so that the network model has fewer parameters and computational complexity, while having an accuracy comparable to that of advanced network models. A neural network that uses depthwise separable convolution as the main convolutional layer is generally called a lightweight neural network. In order to avoid losing too much feature map information, some layers in the lightweight neural network still use standard convolution for feature extraction. Therefore, a lightweight network is a network model that includes standard convolution and depthwise separable convolution.

[0003] However, the convolution computing structure of the existing neural network accelerator cannot efficiently process depth-separable convolution and standard convolution at the same time. For example, the scheme of designing depth-convolution and point-convolution computing units separately (Kim S, Na S, Kong BY, et al. Real-time SSD Lite object detection on FPGA [J]. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 2021, 29 (6): 1192-1205.) has a large area and power consumption overhead in the hardware circuit, and there is a problem of low utilization due to idle computing hardware. Or a scheme of using a configurable addition tree compatible with standard convolution and depth-separable convolution (Yan, Shun, et al. "An FPGA-based Mobile Net Accelerator Considering Network Structure Characteristics." FPL2021.) is adopted. When processing standard convolution, the last configurable addition tree module of this scheme has the problem of idle adder. At the same time, the computing units inside the accelerator are all based on fixed-point number calculations, which require a large number of multipliers to participate, resulting in further hardware resource overhead. Therefore, according to the computational characteristics of depthwise separable convolution and standard convolution, it is of great significance to design a convolution computing structure that supports depthwise separable convolution and standard convolution and has low hardware resource overhead. Summary of the invention

[0004] Technical problem: The purpose of the invention is to address the problem that the convolution computing unit of the existing neural network accelerator cannot support depth-separable convolution calculations and has high hardware resource overhead, and to propose a low hardware overhead computing processing unit and computing method for lightweight neural networks.

[0005] Technical solution: In order to achieve the above-mentioned purpose, the present invention provides a low hardware overhead convolution calculation structure for lightweight neural networks, which includes a preprocessing module, a calculation module and an accumulation module. The preprocessing module includes a data splitting circuit and a data replication circuit, which are used to implement channel segmentation and channel enhancement processing of feature map data. The calculation module consists of a calculation array, a configurable addition tree module, and an ordinary addition tree module. The calculation array consists of m calculation slices (Calculate Slice, referred to as Slice), where m=n(k 2-1), each computing slice is composed of k×k computing units (abbreviated as CU), and each CU contains 2 shift arithmetic units and 1 adder. The function of the CU is to replace the multiplication calculation, and the implementation process is as follows: split the number of weight bits (excluding the 1-bit sign bit) into two parts, which are used as the shift bits of the 2 shift arithmetic units respectively; send the activation value into the 2 shift arithmetic units for shift operations respectively, and finally add the shifted results and combine the sign bit as the result of the activation value and weight calculation. The configurable adder tree unit consists of configurable adder tree blocks (abbreviated as Block), and each Block contains n k 2 -1 input ordinary adder trees, n / 2 data selectors, and 5n / 4 adders. The accumulation module contains an adder group, a partial sum buffer, and a state machine (abbreviated as FSM) control module.

[0006] To achieve the above object, the present invention provides a calculation method for a convolutional calculation structure with low hardware overhead for lightweight neural networks, which is characterized by including the following situations:

[0007] Optionally, for standard convolutional calculation, calculate the convolutional result of C in input channels and C out output channels. If C in ≥m, the input feature map needs to be block-processed by the preprocessing module, that is, the input channels are split into (rounded up) blocks, and m input channel data are sent to the calculation module each time. The k×k feature map convolution window of each input channel and the convolution kernel weight of the corresponding channel are used as the input of a Slice, and k×k×m calculation results are obtained. Then, the calculation results of each Slice are sent to the configurable adder tree for summation calculation. One shift register and one input of the adder tree in each Block of the configurable adder tree correspond to the output of a Slice. After the calculation, the partial sum results of output channels are obtained and sent to the accumulation module for temporary storage. And so on, the subsequent partial sum results will be continuously accumulated in the accumulation module, repeating (rounded up) times of the above operations to obtain output channel feature map results. Repeat times of all the above operations, and finally obtain the feature map results of C out output channels; if C in <m, the input feature map needs to be channel-enhanced by the preprocessing module. In particular, when Cin When C = 3, the input channels need to be expanded to 4 channels first, and the additional channel is filled with 0. At this time, C in = 4. The channel enhancement process will process the input feature map into (rounded down) groups of C in input channel data, and each group of data is the same. The k×k feature map convolution window of each input channel and the convolution kernel weights of the corresponding channel are used as the input of a Slice, and k×k×m calculation results are obtained. Then, the calculation results of each Slice are sent to the configurable adder tree for summation calculation to obtain the calculation results of the output channels. And so on, repeat (rounded up) times of the above operations. The output result of the configurable adder tree module is input to the ordinary adder tree module for summation calculation; finally, the feature map results of C out output channels are obtained.

[0008] Optionally, for depth convolution calculation, calculate the convolution result of C in input channels and C out output channels. At this time, C in = C out . If C in ≥ m, the input feature map needs to be block-processed through the preprocessing module, that is, the input channels are split into (rounded up) blocks, and m input channel data are sent to the calculation module each time. The k×k feature map convolution window of each input channel and the convolution kernel weights of the corresponding channel are used as the input of a Slice, and k×k×m calculation results are obtained. Then, the calculation results of each calculation slice are sent to the configurable adder tree for summation calculation. One shift register of each Block in the configurable adder tree and the input of an adder tree correspond to the output of a Slice. After the calculation, the calculation results of m output channels are obtained. And so on, repeat C out / m (rounded up) times of the above operations, and finally the feature map results of C out output channels are obtained; if C in < m, the input feature map needs to be processed by the channel enhancement process of the preprocessing module, that is, the input feature map is processed into m input channel data, and m - C in input channel data are filled with 0. The k×k feature map convolution window of each input channel and the convolution kernel weights of the corresponding channel are used as the input of a Slice, and k×k×m calculation results are obtained. Then, the calculation results of each calculation slice are sent to the configurable adder tree for summation calculation to obtain the calculation results of m output channels, and C out valid output channel feature map data are intercepted, and finally the feature map results of C out output channels are obtained.

[0009] Optionally, for point convolution calculation, calculate C in the convolution result of the input channels and C out output channels. If C in ≥ m, the input feature map needs to be block-processed through a preprocessing module, that is, the input channels are split into (ceiling) blocks, and m input channel data are sent to the calculation module each time. At the same time, k 2 -1 identical 1×1 feature map convolution windows and the k 2 -1 convolution kernel weights corresponding to the channels are used as the input for each Slice, and (k 2 -1)×m calculation results are obtained. Then, the calculation results of each Slice are sent to a configurable adder tree for summation calculation. The n inputs of the adder tree in each Block of the configurable adder tree respectively correspond to the outputs of n (k 2 -1) calculation results, and the shift registers have no inputs. After the calculation, the partial sum results of the output feature map for the output channels are obtained and sent to the accumulation module for temporary storage. And so on, the subsequent partial sum results will be continuously accumulated, repeating (ceiling) times the above operations to obtain the feature map results for the output channels. Repeat (ceiling) times all the above operations, and finally obtain the feature map results for C out output channels; if C in < m, the input feature map needs to be processed by channel enhancement, that is, the input feature map is processed into (floor) groups of C in input channel data, and each group of data is the same. The identical 1×1 feature map convolution windows and the convolution kernel weights corresponding to the channels are used to calculate (k 2 -1)×m calculation results. Then, the calculation results of each Slice are sent to a configurable adder tree for summation calculation to obtain the calculation results for the output channels. And so on, repeat (ceiling) times the above operations, and finally obtain the feature map results for C out output channels.

[0010] Beneficial effects: By adopting the above technical solutions, compared with the prior art, the present invention has the following effects:

[0011] (1) The present invention proposes a computing processing unit structure for lightweight neural networks. The computing unit uses shift addition instead of multiplication calculation, which effectively reduces the area and power consumption of the hardware circuit while ensuring good accuracy.

[0012] (2) The present invention proposes a computing processing unit structure for lightweight neural networks, which can support the calculation of standard convolution and depthwise separable convolution, and implement the two types of convolution calculations in the same hardware unit, effectively reducing the area and power consumption of the hardware circuit.

[0013] (3) The present invention proposes a calculation method for a computing processing unit structure for a lightweight neural network. Through data blocking and channel enhancement technology, the computing module is maximized to be in a working state, effectively improving the computing utilization of the computing module. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a calculation flow chart of the convolution calculation structure of the present invention.

[0015] Figure 2 It is a structural block diagram of the computing module in the convolution computing structure of the present invention.

[0016] Figure 3 It is a structural block diagram of the computing unit in the convolution computing structure of the present invention.

[0017] Figure 4 It is a structural block diagram of the accumulation module in the convolution calculation structure of the present invention.

[0018] Figure 5 It is a structural block diagram of a common addition tree in the convolution calculation structure of the present invention.

[0019] Figure 6 is a graph of the number of input and output channels of some convolutional layers of MobileNet SSD in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to more clearly explain the purpose, technical solution and technical effect of the present invention, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] The present invention provides a low hardware overhead convolution computing structure for lightweight neural networks. Figure 1 shown.

[0022] The present invention provides a low hardware overhead convolution computing structure for lightweight neural networks. Figure 2As shown. In the computing module, m = 32, k = 3, n = 8. That is, the computing array consists of 32 Slices, and each Slice consists of 3×3 computing units. The configurable adder tree consists of 8 Blocks, and each Block contains 4 8-input ordinary adder trees (adder trees 1.1 - 1.4), 2 data selectors, and 5 adders. Adder tree 2 is a 32-input ordinary adder tree, and the structure of the ordinary adder tree is as Figure 5 shown. The lightweight neural network selects MobileNet SSD. The standard convolution selects convolution layer 0 and convolution layer 21 of this network as examples, and the depthwise separable convolution selects convolution layer 10 of this network as an example. The number of input and output channels of the corresponding convolution layers in the network is as Figure 6 shown.

[0023] Case 1: Standard convolution calculation: For convolution layer 0, the number of input channels C in = 3, and the number of output channels C out = 32. Since C in < m, the input feature map needs to be processed for channel enhancement. First, the input feature map is filled with 4 channels. The preprocessing module processes the input feature map into 8 groups of 4-input-channel data, and each group of data is the same. After preprocessing, the 3×3 feature map convolution window of each input channel and the weights of the corresponding channel are used as the input of a Slice, and 3×3×32 calculation results are obtained. Then, the calculation results of each calculation slice are sent to the configurable adder tree for summation calculation to obtain the convolution results of 8 output channels. Repeat the above operation 4 times, and finally obtain the feature map results of 32 output channels; For convolution layer 21, the number of input channels C in = 128, and the number of output channels C out = 256. Since C in ≥ m, the input feature map needs to be processed in blocks. After preprocessing, the 3×3 feature map convolution window of each input channel and the weights of the corresponding channel are used as the input of a Slice, and 3×3×32 calculation results are obtained. Then, the calculation results of each calculation slice are sent to the configurable adder tree for summation calculation to obtain the partial sum results of 1 output channel, which are sent to the accumulation module for temporary storage. Repeat the above operation 4 times, and the subsequent partial sum results are continuously accumulated to obtain the feature map results of 1 output channel. The output results of the configurable adder tree module are input to the ordinary adder tree module for summation calculation. Repeat the above all operations 256 times, and finally obtain the feature map results of 256 output channels.

[0024] Case 2: Depthwise convolution calculation: For the depthwise convolution of convolution layer 10, the number of input channels C in = 512, and the number of output channels C out = 512. Since C in≥m, the input feature map needs to be processed in blocks. After preprocessing, the 3×3 feature map convolution window of each input channel and the weight of the corresponding channel are used as the input of a Slice, and 3×3×32 calculation results are calculated. Then the calculation results of each calculation slice are sent to the configurable addition tree for summation calculation to obtain the convolution results of 32 output channels. The above operation is repeated 16 times to obtain 512 output channel feature map results.

[0025] Case 3: Point convolution calculation: For the point convolution of convolution layer 10, the number of input channels C in =512, number of output channels C out =512. Satisfies C in ≥m, the input feature map needs to be processed in blocks. After preprocessing, the input channel is split into 16 blocks, and 32 input channel data are sent to the calculation module each time. 8 identical 1×1 feature map convolution windows and 8 convolution kernel weights of the corresponding channels are used as the input of each Slice, and the vacant parts are filled with 0 to calculate 8×32 calculation results. Then the calculation results of each calculation slice are sent to the configurable addition tree for summation calculation to obtain the partial sum results of 8 output channels, and sent to the accumulation module for temporary storage. Similarly, the subsequent partial sum results will be continuously accumulated. Repeat the above operation 16 times to obtain 8 output channel feature map results. Repeat all the above operations 64 times to finally obtain 512 output channel feature map results.

[0026] In summary, the present invention proposes a low hardware overhead convolution calculation structure and calculation method for lightweight neural networks. Through a hardware calculation unit that replaces multiplication operation with addition shift operation and a hardware structure that supports standard convolution and depthwise separable convolution, the calculation of standard convolution and depthwise separable convolution is unified in hardware, and the input channel blocking and input channel enhancement methods are used, which not only effectively reduces the area and power consumption of the convolution calculation hardware circuit, but also effectively improves the computational utilization of the convolution calculation module.

[0027] The above embodiments are only for illustrating the technical idea of ​​the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution according to the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A low hardware overhead convolution computing processing unit for lightweight neural networks, characterized in that: Including preprocessing module, calculation module and accumulation module; The preprocessing module is used to preprocess the input feature map; The computing module comprises a computing array and a configurable addition tree module; the computing array comprises a plurality of computing slices, each of which is composed of a plurality of computing units; each computing unit is composed of a shift operator and an adder, and the input of the adder is connected to the output of the shift operator; The configurable addition tree module is used to perform sum calculation on the calculation results of the calculation unit; The accumulation module is used to accumulate the calculation results of the configurable addition tree module and output the channel feature map result; The preprocessing module includes input channel segmentation and input channel enhancement hardware circuit; The configurable addition tree module is composed of a plurality of configurable addition tree blocks, each of which includes a plurality of shift registers and a plurality of addition trees, as well as a data selector and an adder; the input of a shift register and an addition tree corresponds to the output of a computing slice; The computing array includes m computing slices, m=n(k 2 -1); the computing slice is composed of k×k computing units; the configurable addition tree module includes k 2 - 1 configurable addition tree block; the configurable addition tree block includes n common addition trees, n / 2 data selectors, 5n / 4 adders and n shift registers; n is 4; the input of data selector 1 is connected to the output of addition tree 1.1 and the output of shift register 2; the input of data selector 2 is connected to the output of addition tree 1.4 and the output of shift register 3; the input of adder 1 is connected to the output of addition tree 1.1 and the output of shift register 1; the input of adder 2 is connected to the output of addition tree 1.2 and the output of data selector 1; the input of adder 3 is connected to the output of addition tree 1.3 and the output of data selector 2; the input of adder 4 is connected to the output of addition tree 1.4 and the output of shift register 4; the input of adder 5 is connected to the output of adder 2 and the output of adder 3; The accumulation module comprises an adder group, an accumulation buffer area, and a control circuit.

2. A calculation method of a low hardware overhead convolution calculation processing unit for lightweight neural networks based on the convolution calculation processing unit of claim 1, characterized in that: The calculation module also includes a common addition tree module; The steps of performing standard convolution calculation using the convolution calculation structure include: Calculation C in convolution results of C input channels and C out output channels; if C in ≥m, the input feature map is block - processed through the pre - processing module, that is, the input channels are split into blocks, and each time m = n(k 2 -1) input channel data is sent to the calculation module; the k×k feature map convolution window of each input channel and the convolution kernel weights of the corresponding channel are used as the input of a calculation slice, and k×k×m calculation results are obtained. Then, the calculation results of each calculation slice are sent to the configurable adder tree module for summation calculation. After each configurable adder tree block finishes the calculation, partial sum results of the output channels are obtained and sent to the accumulation module for temporary storage. And so on, the subsequent partial sum results will be continuously accumulated in the accumulation module, repeating times, and the feature map results of output channels are obtained; repeat times for all the above operations, and finally the feature map results of C out output channels are obtained; if C in <m, the input feature map needs to be processed for channel enhancement through the pre - processing module; the channel enhancement processing will process the input feature map into groups of C in input channel data, and each group of data is the same; the k×k feature map convolution window of each input channel and the convolution kernel weights of the corresponding channel are used as the input of a calculation slice, and k×k×m calculation results are obtained. Then, the calculation results of each calculation slice are sent to the configurable adder tree module for summation calculation, and calculation results of the output channels are obtained. And so on, repeating times, the output results of the configurable adder tree module are input to the ordinary adder tree module for summation calculation; finally, the feature map results of C out output channels are obtained.

3. A calculation method of a low hardware overhead convolution calculation processing unit for lightweight neural networks based on the convolution calculation processing unit of claim 1, characterized in that: The steps of performing deep convolution calculation using the convolution calculation structure include: Calculation C in The input channels and C out The convolution result of the output channels. At this time, C in = C out ; If C in ≥ m, the input feature map is processed in blocks through the preprocessing module, that is, the input channels are split into blocks, and each time m = n(k 2 - 1) input channel data are sent to the calculation module. The k×k feature map convolution window of each input channel and the convolution kernel weights of the corresponding channel are used as the input of a calculation slice, and k×k×m calculation results are obtained. Then, the calculation results of each calculation slice are sent to the configurable adder tree module for summation calculation. After each configurable adder tree block finishes the calculation, the calculation results of m output channels are obtained, and so on. Repeat C out / m times, and finally C out output channel feature map results are obtained; If C in < m, the input feature map is processed through the preprocessing module for channel enhancement, that is, the input feature map is processed into m input channel data, and m - C in input channel data are padded with 0; The k×k feature map convolution window of each input channel and the convolution kernel weights of the corresponding channel are used as the input of a calculation slice, and k×k×m calculation results are obtained. Then, the calculation results of each calculation slice are sent to the configurable adder tree module for summation calculation to obtain the calculation results of m output channels, and C out valid output channel feature map data are intercepted, and finally C out output channel feature map results are obtained.

4. A calculation method of a low hardware overhead convolution calculation processing unit for lightweight neural networks based on the convolution calculation processing unit of claim 1, characterized in that: The steps of performing point convolution calculation using the convolution calculation structure include: Calculation C in The convolution result of the input channel and C out For the output channel; if C in ≥m, the input feature map is processed in blocks through the preprocessing module, that is, the input channels are split into blocks, and each time m = n(k 2 -1) input channel data is sent to the calculation module; at the same time, k 2 -1 identical 1×1 feature map convolution windows and k 2 -1 convolution kernel weights of the corresponding channels are used as the input of each calculation slice, and (k 2 -1)×m calculation results are obtained. Then, the calculation results of each calculation slice are sent to the configurable adder tree module for summation calculation. The inputs of the n adder trees in each configurable adder tree block respectively correspond to the outputs of n (k 2 -1) calculation results, and the shift registers have no inputs. After the calculation, the partial sum results of the output feature map for each output channel are obtained and sent to the accumulation module for temporary storage. And so on, the subsequent partial sum results will be continuously accumulated, repeating times, and the feature map results of output channels are obtained; repeating times all the above operations, finally the feature map results of C out output channels are obtained; if C in <m, the input feature map is processed for channel enhancement through the preprocessing module, that is, the input feature map is processed into groups of C in input channel data, and each group of data is the same; identical 1×1 feature map convolution windows and the convolution kernel weights of the corresponding channels are used to calculate (k 2 -1)×m calculation results. Then, the calculation results of each calculation unit are sent to the configurable adder tree for summation calculation, and the calculation results of each output channel are obtained. And so on, repeating times, finally the feature map results of C out output channels are obtained.