Complementary Deep Neural Network Accelerator with Heterogeneous Convolutional Neural Network and Spiking Neural Network Core Architecture

Abstract

A complementary deep neural network accelerator with a heterogeneous convolutional neural network and a spiking neural network core architecture includes a spiking neural network processing module of an accumulator array that accumulates synaptic weights when spikes occur and generates neuron voltages; a multiplication and accumulator array convolutional neural network processing module that accumulates the product of the neural network input and the weights and generates neuron output values; a top RISC controller responsible for controlling the spiking neural network processing module and the convolutional neural network processing module, activation functions, and batch normalization processing; an attention module that performs pooling for each channel on the input, then performs convolution with pre-trained weights to generate an attention map; and a neural network operation allocator that divides the input into many tiles, calculates the spike frequency generated for each tile, predicts a neural network processing module with low energy consumption, and enables operations by transmitting the tiles to the neural network processing module. It has the effect of maintaining the accuracy of inference and learning of the deep neural network accelerator and increasing energy efficiency through the mutual complementarity of the spiking neural network and the convolutional neural network.

Images