A low-power convolutional pulse heterogeneous neuromorphic image recognition system

Abstract

The application discloses a low-power consumption convolution pulse heterogeneous neuromorphic image recognition system, comprising a neural network processor based on a full digital integrated circuit, including an event encoder, a convolution feature extraction circuit and an event-driven neural network inference engine connected in sequence; the event encoder is used for converting an input image into a pulse sequence event stream; the convolution feature extraction circuit is used for performing overlapped grid type scanning convolution on the pulse sequence event stream based on a convolution window, sequentially extracting local spatial features of each convolution window as a convolution response to form a sparse spatial convolution feature map; the event-driven neural network inference engine comprises a memory-independent event response neuron layer, a non-overlapping maximum aggregation layer, a synapse cross array, a memory-dependent event response neuron layer and an event decoder connected in sequence. The system provided by the application greatly reduces the delay, significantly improves the system-level parallelism and operation efficiency of neural computing, has higher precision and speed, and has lower power consumption.

Description

Technical Field

[0001] This invention belongs to the field of deep neural network technology, and in particular relates to a low-power convolutional pulse heterogeneous neuromorphic image recognition system. Background Technology

[0002] With the rapid development of artificial intelligence applications, deep neural network technology has achieved remarkable results in fields such as image classification, object detection, and speech recognition. Deep neural network structures can extract spatial hierarchical features from input data, possessing high recognition accuracy and generalization ability, and are therefore widely deployed in various smart terminals and server platforms. Among them, convolutional neural networks (CNNs) generally employ large-scale convolutional computations. In hardware implementation, the numerous multiplication operations in convolutional layers typically rely on high-speed, resource-intensive multipliers. Especially with high-resolution image input and multi-channel convolutional kernels, the number and overhead of multipliers increase rapidly, resulting in large computational unit area and high power consumption, becoming a major bottleneck limiting its deployment in edge computing and low-power devices. On the other hand, spiking neural networks (SNNs) are a type of neural network model inspired by biological nervous systems, possessing sparse activation and event-driven computational characteristics, capable of performing weighted accumulation operations without the need for high-frequency multipliers. SNNs use pulse signals for communication, and forward propagation can be completed at the hardware level through addition and logical comparisons, offering significant low-power advantages on neuromorphic chips. However, despite the inherent advantages of SNNs in low-power scenarios, they still have significant shortcomings in complex recognition tasks, particularly in their weak feature extraction capabilities, resulting in lower recognition accuracy. Traditional SNN networks rely on simplification mechanisms such as pulse frequency or time coding, which are far less effective than CNNs in extracting key information such as image edges and textures in the early feature stages. This makes it difficult for SNNs to construct deep discriminative features, leading to low accuracy and weak generalization ability in tasks such as image classification. Even with modeling using stacked multi-layer spiking neuron structures, their feature representation capabilities are still inferior to CNN networks of the same scale.

[0003] Currently, spiking neuromorphic processors (SNNs) face a series of key challenges, such as low energy efficiency, insufficient feature extraction capabilities, lack of information representation, and poor generalization ability. These issues hinder their practical deployment and widespread application, and also limit their deployment in computer systems. Although SNNs offer low power consumption advantages in certain tasks due to their sparse activation characteristics, in practical applications, the parallel processing of large-scale neurons and synaptic networks still results in high power consumption. Especially when processing complex tasks or high-dimensional data, although the sparse activation of existing spiking neurons helps reduce the power consumption of a single operation, the large-scale network connections and computational demands still lead to excessively high total power consumption. Compared to other low-power neural computing methods, SNNs have not achieved the expected low power consumption goals in many practical scenarios, limiting their application in embedded systems and mobile devices. Furthermore, existing spiking coding methods for SNNs (such as frequency coding and time coding) result in weak feature extraction capabilities when processing high-dimensional data. Due to the sparsity of neuronal activation and the spiking mechanism, SNNs often fail to fully capture the detailed features in complex inputs such as images and speech. Especially when processing fine-grained image information, SNNs struggle to extract key information such as edges and textures as efficiently and comprehensively as traditional CNNs, resulting in lower accuracy. Many existing SNN processors, such as SNPU and MorphIC, employ a single neuromorphic computing structure. Therefore, they generally suffer from weak feature extraction capabilities and limited information representation capabilities. Because SNNs use an event-driven mechanism, existing spiking neurons only trigger pulse signals when the membrane potential reaches a threshold. This makes it difficult to effectively capture multi-level, multi-scale spatial and semantic features in continuous, dense input data, leading to insufficient feature representation and weak discriminative power when facing complex tasks such as image recognition, thus affecting model accuracy. Furthermore, to improve SNN recognition accuracy, existing single neuromorphic processors often need to stack a large number of neurons and synaptic connections to increase network capacity, resulting in high hardware resource consumption, high chip area overhead, and increased overall system complexity. The overhead of register storage, event scheduling, and communication management brought by large-scale neuron arrays actually weakens the original advantages of SNNs in terms of low power consumption and high efficiency. Furthermore, the sparsity of existing spiking neuron activity cannot be maintained in deep stacked structures, making it difficult to simultaneously achieve response speed, power consumption control, and accuracy improvement. Summary of the Invention

[0004] To address the problems of high power consumption, simple structure, and poor hardware assimilation in existing neuromorphic image recognition systems, this invention proposes a low-power convolutional pulse heterogeneous neuromorphic image recognition system for low-power edge smart hardware applications. The system's overall architecture incorporates several innovative technologies, including a novel local feature extraction method based on pulse signal convolution, the design of memory-independent event response (M-ER) and memory-related event response (M-RER) neuron models, and a CNN–SNN heterogeneous neural network hierarchical structure. Furthermore, it utilizes semiconductor integrated circuits for a fully digital hardware implementation of the entire system, effectively overcoming the shortcomings of traditional neuromorphic hardware in terms of hardware recognition accuracy, power consumption, stability, and scalability.

[0005] To achieve the above objectives, the present invention provides a low-power convolutional pulse heterogeneous neuromorphic image recognition system, comprising: a neural network processor based on an all-digital integrated circuit; the neural network processor includes an event encoder, a convolutional feature extraction circuit, and an event-driven neural network inference engine connected in sequence.

[0006] The event encoder is used to convert the input image into a pulse sequence event stream; the height of the input image is... Width is The convolutional feature extraction circuit is used to perform overlapping grid-type scanning convolutions on the pulse sequence event stream with sliding steps S based on a convolutional window of size K×K, sequentially extracting the local spatial features of each convolutional window as the convolutional response to form a sparse spatial convolutional feature map; the height of the sparse spatial convolutional feature map is... Width is And 0 < S < K.

[0007] The event-driven neural network inference engine includes a memory-independent event response neuron layer, a non-overlapping maximum aggregation layer, a synaptic crossover array, a memory-related event response neuron layer, and an event decoder, which are connected in sequence.

[0008] The memory-independent event response neuron layer is used to perform non-cumulative event detection on each convolution response of the sparse spatial convolution feature map, generate one-to-one corresponding indicator pulse signals, and construct them into a memory-independent indicator variable feature map.

[0009] The non-overlapping maximum aggregation layer is used to select the maximum event activation in each non-overlapping indicator block on the indicator variable feature map based on the preset non-overlapping indicator block, and generate the maximum aggregation feature map by sequentially obtaining the maximum event activation features.

[0010] The synaptic crossover array is used to access each element of the maximum aggregated feature map and its corresponding pulse weight value in batches, perform weighted summation and propagation, and calculate a one-to-one weighted event potential increment for each neuron in the memory-related event response neuron layer.

[0011] The memory-related event response neuron layer is used to perform time-related membrane potential accumulation calculation and discharge operation on the weighted event potential increment.

[0012] The event decoder is used to statistically analyze and decode the output pulses of neurons in the memory-related event response neuron layer to generate a classification result for the input image.

[0013] Furthermore, the memory-independent event-response neuron layer comprises M memory-independent event-response neurons, and M ≤ This is used to perform non-cumulative event detection on each convolutional response of the sparse spatial convolutional feature map in batches by parallel reusing the M memory-independent event response neurons; each memory-independent event response neuron includes a first threshold register, a first threshold comparator, and a pulse indication logic circuit, used to perform non-cumulative event detection based on the following memoryless threshold model and generate an indication pulse signal:

[0014]

[0015] in, In time step t And the index of the top left corner of the convolution window is the coordinate ( , The convolutional response of ) α It is a proportionality constant; Neurons responding to events irrelevant to the current memory at time steps And the index is the coordinate ( , The instantaneous membrane potential; A configurable indicator pulse trigger threshold; Neurons responding to events irrelevant to the current memory at time steps And the index is the coordinate ( , The output indicator pulse signal; where,

[0016] The first threshold register is used to store a configurable indicator pulse trigger threshold;

[0017] The first threshold comparator is used to scale the received convolution response, generate the instantaneous membrane potential of the current memory-irrelevant event response neuron, and compare it with the indicator pulse trigger threshold.

[0018] The pulse indication logic circuit is used to receive the comparison result output by the first threshold comparator, and generate a digital pulse of unit width as the indication pulse signal when the instantaneous membrane potential is greater than or equal to the indication pulse trigger threshold.

[0019] In a preferred implementation, every four adjacent indicator pulse signals on the indicator variable feature map constitute a non-overlapping indicator block in a 2×2 two-dimensional space.

[0020] The non-overlapping maximum aggregation layer includes multiple four-input comparator trees; each four-input comparator tree is an OR gate logic circuit used to perform the following maximum value selection operation on each of the non-overlapping indicator blocks, generating a value at time step t with an output position index of (). u, v Maximum event activation feature :

[0021]

[0022] in, Neurons responding to events irrelevant to the current memory at time steps And the index is the coordinate ( , The output indicator pulse signal; and These are used to control the offset of the indicator pulse signal values ​​of the non-overlapping indicator block in the horizontal and vertical directions, respectively; output position index. The range of values ​​for is: ; Index of each output position Maximum event activation feature Composed of time steps The maximum aggregated feature map.

[0023] Furthermore, the synaptic cross array includes a row-column cross interconnect matrix composed of multiple synaptic computing units.

[0024] The row-column cross-interconnection matrix adopts a reconfigurable propagation structure with axonal row inputs and dendritic column outputs. The synaptic computing unit is located at the intersection of axons and dendrites and is used to realize weighted event propagation based on the following model to calculate the first [number] neuron in the memory-related event response neuron layer. At time step, one neuron Weighted event potential increment :

[0025]

[0026] in, For the position index is ( u, v And with the first Synaptic weights of dendritic connections; In time step t And the output position index is ( u, v The maximum event activation feature is defined as follows: R is the number of rows in the row-column cross-interconnection matrix, corresponding to the number of maximum event activation features output in each batch of the maximum aggregation feature map; C is the number of columns in the row-column cross-interconnection matrix, corresponding to the number of neurons in the memory-related event response neuron layer; the row direction of the row-column cross-interconnection matrix corresponds to the axonal propagation path of the non-overlapping maximum aggregation layer in the previous stage, and the column direction corresponds to the dendritic reception path of the memory-related event response neuron layer in the subsequent stage.

[0027] In a preferred implementation, the convolutional feature extraction circuit includes multiple sets of parallel convolutional units; each set of convolutional units is used to implement the following event convolution model, which is applied to features extracted at time steps. Weight accumulation calculation is performed on the pulse sequence event stream:

[0028]

[0029] in, In time step t The index of the top left corner of the convolution window is the coordinate ( , And the index coordinates inside the convolution window are ( m , n Pulse sequence events at position ) For the convolution kernel in ( m , n The convolution weights at position ); the index of the top-left corner of the convolution window. The range of values ​​for is: ; In time step t And the index of the top left corner of the convolution window is the coordinate ( , The convolution response of ) ; the convolution response of each index Composed of time steps t The sparse spatial convolution feature map.

[0030] Furthermore, the memory-related event response neuron layer includes N memory-related event response neurons; N is the number of identifiable image target categories; the circuit configuration of each memory-related event response neuron includes: a membrane potential register, an update adder, a difference calculator, and a leakage shifter.

[0031] The membrane potential register is used to store the cumulative membrane potential of the memory-related event response neuron in the previous time step;

[0032] The difference calculator is used to calculate the potential difference between the weighted event potential increment of the memory-related event response neuron at the current time step and its cumulative membrane potential at the previous time step.

[0033] The leakage shifter is used to perform an approximate division operation on the potential difference value based on a preset membrane time constant, thereby realizing leakage control of the cumulative membrane potential of the memory-related event response neuron and outputting potential decay margin.

[0034] The update adder is used to add the cumulative membrane potential of the memory-related event response neuron in the previous time step to the potential decay margin to generate the cumulative membrane potential of the memory-related event response neuron in the current time step.

[0035] The memory-related event response neuron further includes a second threshold comparator, a second threshold register, and a reset logic unit; the second threshold comparator is used to compare the cumulative membrane potential at the current time step with the neuron activation threshold stored in the second threshold register, and control whether the current memory-related event response neuron executes a firing event based on the comparison result; the reset logic unit is used to reset the cumulative membrane potential of the current memory-related event response neuron when a firing event occurs;

[0036] The membrane potential register is also used to update the data it stores: if the current memory-related event response neuron fires, the accumulated membrane potential after reset is stored; otherwise, the accumulated membrane potential output by the update adder at the current time step is stored.

[0037] Furthermore, the low-power convolutional pulse heterogeneous neuromorphic image recognition system also includes a control processing unit. The control processing unit is a RISC-V processor designed based on a pulse event computation extended instruction set architecture, used to hardware accelerate the membrane potential update and pulse firing logic of the memory-independent event-response neuron layer and the memory-related event-response neuron layer.

[0038] Compared with the prior art, the present invention has the following advantages and technical effects:

[0039] This invention proposes a low-power heterogeneous neuromorphic image recognition system. It combines a temporal event encoding mechanism, raster-scanning pulse convolution feature extraction, the design and physical circuit implementation of memory-independent and memory-related event-response neurons, and a RISC-V programmable controller and neural acceleration coprocessor to achieve high-precision pulse inference under extremely low resource and power consumption conditions. Specific advantages and technical effects include: First, after event encoding the input image, this invention proposes a pulse signal-based convolution feature extraction structure. It achieves local correlation extraction of spatial features through raster scanning of the convolution kernel, and combines memory-independent event-response neurons and non-overlapping maximum aggregation layers to achieve feature downsampling. Through event-driven sparse computation mechanisms and synaptic access compression strategies, it significantly reduces invalid operations and data transmission overhead, effectively improving feature representation capabilities under sparse pulse event input and significantly reducing the energy consumption of convolution weight access and computation. Second, this invention proposes a novel memory-independent event response (M-ER) neuron design. Employing simplified event response logic, it responds solely based on the current input pulse, eliminating the need for potential accumulation registers. This achieves low-power, low-latency pulse response characteristics, suitable for rapid filtering of high-throughput convolutional output signals. Under equivalent tasks, power consumption can be reduced to less than 1 / 5 of traditional SNN hardware. Third, this invention proposes a memory-related event response (M-RER) neuron model to enhance the model's inference capability. This model introduces a membrane potential register and a leakage shifter into the circuitry, achieving membrane potential integration across time steps and exhibiting short-term memory effects. Compared to traditional LIF models, it maintains higher inference accuracy and temporal feature sensitivity with lower power consumption. Fourth, this invention proposes a heterogeneous neural network hierarchical structure design. Structurally, it tightly couples the convolutional feature extraction layer (CNN part) with the event-driven inference layer (SNN part). By unifying the spatiotemporal format through feature mapping encoding, feature extraction and inference are seamlessly integrated at the hardware level, significantly reducing cross-layer communication latency and cache usage. Furthermore, this invention introduces a RISC-V processor as the on-chip control core, responsible for parameter configuration, task scheduling, power management, and data interaction control of the network model. This enables flexible programmability and multi-task support for system operation, facilitating dynamic reconfiguration under different neural network models or task scenarios. Moreover, the all-digital integrated circuit implementation uses digital logic circuits for all neuron units, synaptic cross-connection arrays, event buses, and decoders, avoiding the parameter drift, noise sensitivity, and process instability issues inherent in analog circuits. This provides better portability and scalable manufacturing advantages.

[0040] In summary, this invention constructs a convolutional pulse heterogeneous neuromorphic image recognition system that combines high precision, low power consumption, high energy efficiency, strong assimilation, and programmability by implementing event-driven convolutional feature extraction and pulse inference heterogeneously at the system level, and by combining RISC-V instruction set control with digital integrated circuit implementation. The entire system is implemented in hardware using fully digital integrated circuits, completing the entire process from event encoding, pulse convolutional feature extraction, M-ER neuron response, non-overlapping maximum aggregation, synaptic cross-array inference, LIF spiking neuron output to classification decoding at the circuit level. Compared with software algorithms, it achieves advantages such as low power consumption, small area, high speed, low cost for large-scale applications, and ease of integration at the edge, thus effectively solving the shortcomings of traditional neuromorphic systems in terms of accuracy, power consumption, stability, and scalability. This invention realizes a high-precision, high-energy-efficiency, highly assimilationable, low-power programmable neuromorphic computing system suitable for multiple application scenarios such as edge terminals, intelligent sensing, and low-power visual recognition. Attached Figure Description

[0041] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0042] Figure 1 A schematic diagram of the structure of an embodiment of the low-power convolutional pulse heterogeneous neuromorphic image recognition system provided by the present invention;

[0043] Figure 2 A schematic diagram of a circuit framework structure for a neural network processor provided in an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram illustrating one implementation process of block-based local augmentation convolution in an embodiment of the present invention;

[0045] Figure 4 A circuit structure diagram of a convolutional unit provided in an embodiment of the present invention;

[0046] Figure 5 A schematic diagram of the hardware architecture of the memory-independent event response neuron layer and the non-overlapping maximum aggregation layer, and the data flow structure between them, provided in an embodiment of the present invention.

[0047] Figure 6 This is a schematic diagram of an architecture of a synaptic cross array provided in an embodiment of the present invention;

[0048] Figure 7 This is a schematic diagram of a circuit structure for a synaptic computing unit PE provided in an embodiment of the present invention;

[0049] Figure 8A schematic diagram of a circuit structure for a memory-related event response neuron provided in an embodiment of the present invention;

[0050] Figure 9 This is a schematic diagram of another embodiment of the low-power convolutional pulse heterogeneous neuromorphic image recognition system provided by the present invention.

[0051] Figure 10 This is a chip layout of one embodiment of the neural network processor implemented based on a 40-nanometer all-digital integrated circuit provided by the present invention. Detailed Implementation

[0052] It should be noted that, without causing conflict, the embodiments and their apparent features mentioned in this application can be flexibly combined. To illustrate the content of this application in detail, the following will use accompanying drawings and specific embodiments.

[0053] It should be noted that the steps in the flowchart shown in the attached figure can be executed by a computer system, specifically through a series of computer-executable instructions. Furthermore, although the flowchart illustrates the logical execution order of the steps, in some cases, the execution order of these steps may differ from the order shown or described.

[0054] Figure 1This is a schematic diagram of a low-power convolutional pulse heterogeneous neuromorphic image recognition system provided in an embodiment of the present invention. In specific implementation, the low-power convolutional pulse heterogeneous neuromorphic image recognition system provided in this embodiment receives an image input to be recognized (e.g., handwritten digits). After passing through an event encoder, it performs preliminary local spatial feature extraction through a pulse-based multi-channel parallel convolutional layer. This layer contains multiple parallel convolutional operations, aiming to efficiently capture local features of the image from different dimensions or scales, thereby providing a more comprehensive and discriminative feature representation for subsequent processing. Subsequently, a memory-independent event response neuron layer (M-ER layer) is introduced. This layer efficiently encodes continuous spatial feature maps into pulse signals, with the core purpose of optimizing computational efficiency and laying the foundation for subsequent pulse domain processing. The pulse-coded features output by the M-ER layer are then downsampled by a non-overlapping maximum aggregation layer (MaxP layer) to effectively reduce data dimensionality, reduce redundant information, and enhance the translation invariance and robustness of the features. Subsequently, the multidimensional feature map is flattened into a one-dimensional vector and fed as input to a linear layer composed of synaptic crossover arrays. Here, the features are weighted and combined to provide input membrane potentials for the neurons in the output layer. In this embodiment, the network output layer is a memory-related event response neuron layer (M-RER layer), which completes the final decision or classification output of the image. As a key component of the neuromorphic computing model, the event-driven nature of the memory-related event response neuron results in significant ultra-low power consumption and high energy efficiency in hardware implementation. Therefore, this embodiment of the invention integrates a hybrid convolutional pulse heterogeneous neural network model architecture of "event encoding → multi-channel parallel convolution → M-ER neuron → maximum aggregation MaxP → flattening → linear layer → M-RER neuron," which effectively reduces the overall energy consumption during inference while maintaining high recognition accuracy, making it particularly suitable for power-sensitive embedded systems and edge computing devices. Meanwhile, by combining the unique characteristics of Memoryless Event-Response Neuron (M-ER) and Memory-Related Event-Response Neuron (M-RER) with their pulse coding mechanism, an innovative solution is provided for developing more efficient and cost-effective dedicated hardware accelerators, comprehensively improving the overall recognition rate and energy efficiency of the system, and further optimizing the hardware implementation scheme to enhance the flexibility of the hardware system.

[0055] Figure 2This is a schematic diagram of a circuit framework structure for a neural network processor provided in an embodiment of the present invention. The low-power convolutional pulse heterogeneous neuromorphic image recognition system includes: a neural network processor 10 based on all-digital integrated circuits. The neural network processor includes an event encoder 100, a convolutional feature extraction circuit 200, and an event-driven neural network inference engine 300 connected in sequence.

[0056] The event encoder 100 is used to convert the input image into a pulse sequence event stream; the height of the input image is... Width is .

[0057] The convolutional feature extraction circuit 200 is used to perform overlapping grid-like scanning convolution on the pulse sequence event stream with sliding steps S based on a convolutional window of size K×K, sequentially extracting the local spatial features of each convolutional window as the convolutional response to form a sparse spatial convolutional feature map; the height of the sparse spatial convolutional feature map is... Width is And 0 < S < K. Preferably, the convolution feature extraction circuit 200 sets the sliding step S=1 for the convolution window and the convolution window size K=5, so that when performing convolution operation on the pulse sequence event stream of the input image, a raster scan convolution is performed based on the 5×5 convolution window with a step S=1, thereby maximizing the spatial overlap between adjacent convolution windows, so as to capture continuous local region features during feature extraction, enhance the spatial correlation between adjacent features, effectively avoid abrupt changes between local feature responses of different convolution windows, and achieve smooth extraction and stable encoding of image details and edge features.

[0058] Preferably, when performing overlapping raster-style scanning convolution on the pulse sequence event stream based on a K×K convolution window with a sliding step S, it is preferable to merge four adjacent convolution windows scanned in a raster manner into a single convolution block, and then perform raster-style scanning convolution on the entire image using this (K+S)×(K+S) convolution block as the unit, forming a two-level raster-style scanning convolution structure with blocks as the basic unit, simply referred to as Block-Raster Convolution. The convolution effect based on (K+S)×(K+S) blocks is consistent with the convolution effect of directly using a K×K convolution window with a sliding step S, but the hardware implementation will result in different circuit systems and different hardware efficiencies.

[0059] Figure 3 This is a schematic diagram illustrating one implementation process of block-based local enhancement convolution in an embodiment of the present invention. Figure 3The illustrated embodiment first performs local convolution on four convolutional window regions (Cell1, Cell2, Cell3, Cell4) arranged in a locally adjacent grid pattern (or Z-row arrangement) to form convolutional blocks. Then, it performs global convolution in the order of grid scanning. This hierarchical combination of local feature blocks and global relational features is equivalent to introducing a mesoscale feature fusion layer. This clearly captures the directionality and structural continuity of the local neighborhood, as well as the spatial consistency and abrupt boundary between local features, enhancing the robust recognition capability for object edges and texture directions. Simultaneously, it allows for computational reuse of local convolutional block results, reducing redundant computation. This enables the adoption of a hierarchical modular pipeline structure in hardware mapping, significantly reducing repeated convolution operations, making it more suitable for on-chip reuse and cache reuse, and reducing on-chip power consumption and access bandwidth.

[0060] Furthermore, such as Figure 2 As shown, the event-driven neural network inference engine 300 includes a memory-independent event response neuron layer 301, a non-overlapping maximum aggregation layer 302, a synaptic crossover array 303, a memory-related event response neuron layer 304, and an event decoder 305, all connected in sequence. The memory-independent event response neuron layer is also referred to as the M-ER layer, the non-overlapping maximum aggregation layer as the MaxP layer, and the memory-related event response neuron layer as the M-RER layer. The memory-independent event response neuron layer 301 performs non-cumulative event detection on each convolutional response of the sparse spatial convolutional feature map, generating corresponding indicator pulse signals and constructing them into a memory-independent indicator variable feature map. The non-overlapping maximum aggregation layer 302 selects the maximum event activation within each non-overlapping indicator block on the indicator variable feature map based on preset non-overlapping indicator blocks, and generates a maximum aggregation feature map from the sequentially obtained maximum event activation features. The synaptic cross-connection array 303 is used to access each element of the maximum aggregated feature map and its corresponding pulse weight value in batches, perform weighted summation and propagation, and calculate a one-to-one weighted event potential increment for each neuron in the memory-related event response neuron layer 304. The memory-related event response neuron layer 304 is used to perform time-related membrane potential accumulation calculation and firing operation on the weighted event potential increment. The event decoder 305 is used to statistically analyze and decode the output pulses of the neurons in the memory-related event response neuron layer 304 to generate the classification result of the input image.

[0061] In a specific implementation, the event encoder 100 converts each pixel value I(i, j) of the input image into a pulse sequence event stream, with each pixel corresponding to an independent event pulse code. In one possible implementation, the event encoder 100 includes a random number generator 101, a comparator 102, a trigger array 103, and a convolutional unit extractor 104. Specifically, the trigger array 103 is a linear feedback shift register array with a multi-channel parallel array structure, and each pixel is assigned a corresponding T... r A linear feedback shift register for bits; in each clock cycle, random number generator 101 generates a set of pseudo-random numbers r. t ∈[0, 2 Tr -1]; Comparator 102 performs the following judgment, and when the currently generated pseudo-random number is less than the pixel grayscale intensity I(i, j), comparator 102 outputs an event pulse signal "1", otherwise it outputs "0". A pulse sequence event stream is generated at each time step t, and transmitted via the bus. Enter the convolutional feature extraction circuit 200.

[0062]

[0063] Among them, the random number generator 101 generates a set of pseudo-random numbers r t The generation constraint relation used is: That is, when the time window (number of time steps) T r When sufficiently large, the average event rate approaches the grayscale intensity of the input pixel.

[0064] In a preferred implementation, such as Figure 2 As shown, the convolutional feature extraction circuit 200 includes multiple sets of parallel convolutional units 201; each set of convolutional units 201 is used to implement the following event convolution model, which is applied to features extracted at time steps. Weight accumulation calculation is performed on the pulse sequence event stream:

[0065]

[0066] in, In time step t The index of the top left corner of the convolution window is the coordinate ( , And the index coordinates inside the convolution window are ( m , n Pulse sequence events at position ) For the convolution kernel in ( m , n The convolution weights at position ); the index of the top-left corner of the convolution window. The range of values ​​for is: ; In time step t And the index of the top left corner of the convolution window is the coordinate ( , The convolution response of ) ; the convolution response of each index Composed of time steps t The sparse spatial convolution feature map.

[0067] Multi-channel convolution enables the simultaneous capture of feature patterns at different scales, directions, and frequencies by weighted combinations of multiple convolutional kernels on the same input event stream. For the input event stream within the same convolutional window, the convolutional operation of each channel learns the convolutional response pattern of a class of features after calculation according to the above formula, resulting in a multi-view pulse feature combination in the temporal dimension of the entire front-end sparse spatial convolutional feature map. Furthermore, multi-channel convolution can distribute the energy of the input event across multiple feature maps, thereby activating fewer neurons in each channel. This sparsity characteristic brings direct benefits to hardware implementation: reduced pulse transmission number, reduced energy consumption of the synaptic cross-array 303, and reduced pulse firing density of the memory-related event response neuron layer 304 as an inference layer, thus achieving lower power consumption and higher energy efficiency. In addition, since each convolutional channel can independently control the weight loading, accumulation, and pulse firing logic, it is easy to map to the configurable parallel processing units or custom extended instruction sets in the RISC-V processor, realizing flexible task scheduling and channel reuse, laying the foundation for software and hardware co-optimization.

[0068] Figure 4 This is a circuit structure diagram of a convolutional unit provided in an embodiment of the present invention. Specifically, through... Figure 4 The hardware circuit implementation of the convolutional unit shown in this embodiment of the invention provides a structure of multiple parallel convolutional units 201 that can realize multi-channel parallel convolution functionality. Corresponding to the block-based grid scanning convolution method and the subsequent 2×2 block-based maximum aggregation operation, each convolutional unit 201 integrates a circuit module that can realize parallel computation of four convolutional windows.

[0069] In specific implementation, such as Figure 2 As shown, each group of convolutional units includes a first weight register array 2011, an input selector array 2012, a first adder tree circuit 2013, and a convolution control logic module 2014.

[0070] The first weight register array 2011 is used to store and output convolution weight values, providing the corresponding convolution weight values ​​for the convolution calculation of each pulse sequence event. That is, it stores the convolution weight values ​​of each convolution kernel k(m, n).

[0071] The input selector array 2012 is composed of AND gate circuits, used to perform a logical AND operation between each bit of the convolution-weight value and the current pulse sequence event, and then perform a bit-by-bit addition operation on the result to achieve a dot-matrix weighted operation between the current pulse sequence event and the convolution weight value at the corresponding position. Because Since it's already a pulse sequence event, the input selector array 2012 can use AND gates to implement a "two-to-one" function, based on the input pulse sequence event. The value is selected as either the convolution weight value or "0". AND gates, compared to traditional "two-to-one" selectors, can achieve faster processing speeds with less hardware resources.

[0072] The first adder tree circuit 2013 is used to aggregate and sum the multiple weighted values ​​output by the input selector array to generate the convolution response of each convolution window at each time step. The first adder tree circuit 2013 mainly implements the multiply-accumulate operation in the convolution process.

[0073] The convolution control logic module 2014 is used to coordinate the timing and data flow of each group of convolutional units 201, realizing weight loading, convolution window sliding, pulse timing control, and addition tree startup. Since this embodiment of the invention uses a raster-scan convolution operation on the input pulse sequence event stream, the convolution control logic module 2014 is needed to implement raster-scan scheduling. As an event-driven neuromorphic system, the input signals of each group of convolutional units 201 in this embodiment are time-sparse pulse event streams. Raster-scan convolution allows the neuromorphic system to execute in an "event-as-it-arrives" mode, achieving low-latency and low-power response. Furthermore, the raster-scan method has the characteristics of point-by-point, ordered, and overlapping sliding, resulting in high overlap between convolutional windows (adjacent windows share K). (1 row or column of pixels) to ensure the continuity of spatial features and local smoothness.

[0074] Meanwhile, since the subsequent neural network layers respond to input events instantly in the time dimension, the input signals of the memory-independent event-response neuron layer 301, which does not rely on the accumulation of historical potentials, should be as spatially continuous as possible but temporally sparse as possible. This allows for immediate triggering of impulse events without the need for caching historical frames or complex integration registers. Raster scanning facilitates event-driven, time-independent real-time processing, with each convolutional response output generated immediately at the current scan moment and fed into the M-ER neuron. Furthermore, the raster scanning method allows the system to employ a line buffer structure in hardware, enabling batch-based local sequence transmission computation, thus preparing data for the computational processing of the memory-independent event-response neuron layer 301.

[0075] In a preferred implementation, the memory-independent event response neuron layer 301 includes M memory-independent event response neurons, and M ≤ This is used to perform non-cumulative event detection on each convolutional response of the sparse spatial convolutional feature map in batches by parallel reusing the M memory-independent event response neurons; each memory-independent event response neuron includes a first threshold register 3011, a first threshold comparator 3012, and a pulse indication logic circuit 3013, used to perform non-cumulative event detection based on the following memoryless threshold model and generate an indication pulse signal:

[0076]

[0077] in, In time step t And the index of the top left corner of the convolution window is the coordinate ( , The convolutional response of ) α It is a proportionality constant; Neurons responding to events irrelevant to the current memory at time steps And the index is the coordinate ( , The instantaneous membrane potential; A configurable indicator pulse trigger threshold; Neurons responding to events irrelevant to the current memory at time steps And the index is the coordinate ( , The output indicator pulse signal; the height of the sparse spatial convolution feature map is Width is And 0 < S < K. Wherein, at each time step t:

[0078] The first threshold register 3011 is used to store a configurable indicator pulse trigger threshold. .

[0079] The first threshold comparator 3012 is used to compare the received convolution response Perform scaling (using a scaling constant α as a scaling factor) to generate the transient membrane potential of neurons responding to current memory-irrelevant events. and compare it with the indicated pulse trigger threshold. Compare them.

[0080] The pulse indication logic circuit 3013 is used to receive the comparison result output by the first threshold comparator 3012, and to determine the pulse trigger threshold when the instantaneous membrane potential is greater than or equal to the indication pulse trigger threshold. At that time, a digital pulse of unit width is generated as the indication pulse signal. .

[0081] Preferably, every four adjacent indicator pulse signals on the indicator variable feature map constitute a non-overlapping indicator block in a 2×2 two-dimensional space. For example... Figure 5 The diagram shown is a schematic representation of the hardware architecture of the memory-independent event response neuron layer and the non-overlapping maximum aggregation layer provided in an embodiment of the present invention, as well as the data flow structure between the two.

[0082] The non-overlapping maximum aggregation layer 302 includes multiple four-input comparator trees 3021. Each four-input comparator tree 3021 is an OR gate logic circuit, used to perform the following maximum value selection operation on each of the non-overlapping indicator blocks, generating a value at time step t with an output position index of (...). u, v Maximum event activation feature :

[0083]

[0084] in, Neurons responding to events irrelevant to the current memory at time steps And the index is the coordinate ( , The output indicator pulse signal; and These are used to control the offset of the indicator pulse signal values ​​of the non-overlapping indicator block in the horizontal and vertical directions, respectively; output position index. The range of values ​​for is: ; Index of each output position Maximum event activation feature Composed of time steps The maximum aggregated feature map. When the indicator pulse signal value offset... and When the value range is all {0,1}, the maximum value is activated for the four indicator pulse signals in each 2×2 non-overlapping indicator block. Furthermore, this embodiment further optimizes the hardware circuit implementation, employing methods such as... Figure 5 The OR gate logic circuit shown implements a four-input comparator tree 3021 to save hardware resources and improve computational efficiency. The size of the maximum aggregate feature map obtained after the maximum event activation will be reduced by a factor of 2 relative to the size of the indicator variable feature map.

[0085] Furthermore, the synaptic cross array 303 includes a row-column cross interconnect matrix composed of multiple synaptic computing units (PEs). Figure 6 The diagram shown is a schematic diagram of an architecture of a synaptic cross array provided in an embodiment of the present invention.

[0086] The row-column cross-interconnection matrix adopts a reconfigurable propagation structure with axon row input and dendrite column output. The synaptic computation unit (PE) is located at the intersection of axon and dendrite and is used to implement weighted event propagation based on the following model to calculate the first 104-th neuron in the memory-related event response neuron layer 304. At time step, one neuron Weighted event potential increment :

[0087]

[0088] in, For the position index is ( u, v And with the first Synaptic weights of dendritic connections; In time step t And the output position index is ( u, v The maximum event activation feature is defined as follows: R is the number of rows in the row-column cross-interconnection matrix, corresponding to the number of maximum event activation features output in each batch of the maximum aggregation feature map; C is the number of columns in the row-column cross-interconnection matrix, corresponding to the number of neurons in the memory-related event response neuron layer; the row direction of the row-column cross-interconnection matrix corresponds to the axonal propagation path of the non-overlapping maximum aggregation layer 302 in the preceding stage, and the column direction corresponds to the dendritic receiving path of the memory-related event response neuron layer in the following stage. Therefore, the rows correspond to the axons in the event propagation direction, used to receive pulse input generated by the maximum aggregation feature map from the preceding neural layer; the columns correspond to the dendrites in the event propagation direction, used to transmit weighted potentials to the following memory-related event response neurons (M-RER neurons). The synaptic computing unit (PE) is set between the axon and the dendrite, and the number of PEs on the dendrites in the column direction is the number of maximum event activation features input in parallel in a single batch. In specific implementation, such as Figure 6 As shown, the maximum event activation feature in the upper row of the maximum aggregated feature map can be preferably used as the multiple maximum event activation features that are synchronously input in a single batch.

[0089] In particular, such as Figure 5 As shown, the number of M-ER neurons in the current-level memory-independent event response neuron layer 301 is M, and the value of M is equal to the number of single-row convolutional responses of the sparse spatial convolutional feature map. By reusing the M memory-independent event response neurons in batches and in parallel, performing non-cumulative event detection on each row of M convolutional responses of the sparse spatial convolutional feature map, the size of the generated indicator variable feature map remains the same as the size of the sparse spatial convolutional feature map, but can be represented as... ; H oAfter several batches, the data processing of the entire image is completed. After the indicator variable feature map passes through the non-overlapping maximum aggregation layer 302, due to the use of 2×2 two-dimensional non-overlapping indicator blocks for maximum aggregation, the size of the output maximum aggregated feature map can be further compressed to half of its original size, i.e. At this time, also in order to reuse the synaptic computing units PE in the synaptic cross array 303 and reduce hardware resource consumption and power consumption, this embodiment will use them in batches. The largest aggregate feature map of the size is input into the synaptic cross array 303 in batches in "rows" for calculation.

[0090] The synaptic cross-array 303 provided in this embodiment of the invention can solve the problems of low data reuse, high bandwidth pressure, and high energy consumption for storage access in traditional neural network hardware implementations. Since the non-overlapping maximum aggregation layer 302 and the memory-related event response neuron layer 304 are fully connected, each memoryless M-ER neuron needs to access all elements in the non-overlapping maximum aggregation layer 302 for superposition judgment. In traditional cross-arrays, each calculation requires re-accessing the stored input features and broadcasting them to the entire array, resulting in repetitive data access and high bandwidth consumption.

[0091] In embodiments of the present invention, such as Figure 6 As shown, a pipelined input feature transfer structure for the synaptic cross array is proposed. Structurally, the (axon) input end is a time-sequential streaming data interface, and each synaptic computation unit (PE) has multiply-accumulate and accumulate register capabilities; the (dendritic) output end is directly connected to the M-RER neuron potential register array. The input feature vector (i.e., the elements in the maximum aggregated feature map) is injected row by row into the synaptic cross array 303; each row of elements is activated using time-division multiplexing; each input element only needs to be accessed once to propagate sequentially along the row direction (axon direction) of the array; each column (dendritic) receives the weighted result of all input elements sequentially at different clock cycles, and the last PE unit on the dendrite can obtain the local point increment of each row of input elements, completing the H... o After each batch, the input membrane potentials of each M-RER neuron at the current time step t are obtained.

[0092] In a preferred implementation, the synaptic computing unit PE includes a second weight register array P1, a multiplication logic unit P2, and a second addition tree circuit P3. Figure 7 This is a schematic diagram of a circuit structure of a synaptic computing unit PE provided in an embodiment of the present invention.

[0093] The second weight register array P1 is used to store the respective synaptic weight values. .

[0094] The multiplication logic unit P2 is composed of a multiplexer (MUX) and is used to calculate the synaptic weight values ​​at each of the intersection points. With the maximum event activation feature Weighted dot product operations between them.

[0095] The second addition tree circuit P3 is used to sum all weighted results from the same dendritic direction, and sequentially generate the weighted event potential increments accumulated by each dendritic receiving path, which correspond to the input potentials of each neuron in the memory-related event response neuron layer 304.

[0096] The above synaptic cross array design achieves multiple advantages, including single access, global reuse, and pipelined parallelism. Through inter-row shifting and propagation mechanisms, each input feature element... Reading from memory only once and then sequentially utilized by all synaptic computation units (PEs), the rows of PEs form a computational pipeline in time, greatly reducing input bus access bandwidth and memory access energy consumption. For PEs on the same dendrite, the result of the previous input is accumulated in the current PE as the current input propagates, improving throughput. In terms of scalability, the array size of the synaptic crossover array can be flexibly configured; the number of different neurons in the 304 memory-related event response neuron layers only affects the number of array columns, without changing the overall structure. Furthermore, since data is weighted within the PEs, it employs a near-memory computation method, such as... Figure 7 As shown, this avoids the large-scale data movement of traditional MAC (multiply-accumulate) arrays, reduces on-chip interconnect power consumption, and, combined with time-division multiplexing mechanism, reduces the number of on-chip memory or register accesses, automatically reducing power consumption during sparse input.

[0097] Preferably, in the low-power convolutional pulse heterogeneous neuromorphic image recognition system, the memory-related event-response neuron layer 304 includes N memory-related event-response neurons (M-RER neurons); N is the number of recognizable image target categories; in specific implementation, since the dendrite direction of each column is connected to each neuron in the neuron layer 304, the number of columns C of the row-column cross-connection matrix should be equal to the number N of M-RER neurons in the memory-related event-response neuron layer 304, that is, N=C, both of which are the total number of target categories that the system can recognize.

[0098] The circuitry of each memory-related event-response neuron includes: a membrane potential register 3041, an update adder 3042, a difference calculator 3043, and a leakage shifter 3044. For example... Figure 8 The diagram shown is a schematic diagram of a circuit structure of a memory-related event response neuron provided in an embodiment of the present invention.

[0099] The membrane potential register 3041 is used to store the cumulative membrane potential of the memory-related event response neuron in the previous time step. ;

[0100] The difference calculator 3043 is used to calculate the weighted event potential increment of the memory-related event response neuron at the current time step. Compared with the accumulated membrane potential in the previous time step Potential difference between V.

[0101] The leakage shifter 3044 is used to adjust the potential difference value based on a preset membrane time constant. V performs an approximate division operation to control the leakage of the cumulative membrane potential of the memory-related event response neurons, and outputs a potential decay margin V. l In this embodiment, the membrane time constant (τ) m This is used to simulate the response speed of biological neurons based on an RC circuit model. Specifically, 1 / τ m The τ determines the proportion of potential decay at the previous moment. m The larger the value, the higher the old potential at the previous time step (t-1). The higher the weight (stronger the memory), the higher the τ. m The smaller the value, the more new inputs. The impact is greater (the response is faster). Therefore, the membrane time constant corresponds to the width of the time window in event-driven networks, controlling the balance between the length of a neuron's memory and its response speed.

[0102] The update adder 3042 is used to add the accumulated membrane potential of the memory-related event response neuron at the previous time step to the potential decay margin V. l The potentials of the memory-related event response neurons are superimposed to generate the cumulative membrane potentials at the current time step t. .

[0103] Furthermore, the memory-related event response neuron also includes a second threshold comparator 3045, a second threshold register 3046, and a reset logic unit 3047.

[0104] The second threshold comparator 3045 is used to compare the cumulative membrane potential at the current time step with the neuron activation threshold stored in the second threshold register 3046, and control whether the current memory-related event response neuron executes a firing event based on the comparison result; the reset logic unit 3047 is used to reset the cumulative membrane potential of the current memory-related event response neuron when a firing event occurs.

[0105] The membrane potential register 3041 is also used to update the data it stores: if the current memory-related event response neuron executes a firing event, the accumulated membrane potential after reset is stored; otherwise, the accumulated membrane potential output by the update adder 3042 at the current time step is stored.

[0106] In practice, For the first n A memory-related event response neuron at time step The cumulative membrane potential; Indicates the first A memory-related event response neuron at time step The weighted event potential increment; For the first The cumulative membrane potential of the memory-related event-responding neurons in the previous time step; τ m is the membrane time constant, used to control the rate of membrane potential decay in neurons responding to memory-related events. Modules communicate via a clock-synchronized event interface, using a time step t as the global control cycle to maintain event timing consistency.

[0107] For an image dataset with 10 target categories, such as the MNIST handwritten digit dataset or the CIFAR-10 image dataset, the memory-related event response neuron layer 304 contains N=10 M-RER neurons. Each M-RER neuron stores its accumulated membrane potential at the current time step and its neuronal activation threshold V. M-RER_th Compare and control within V M-RER_th The output pulse is timed. It should be noted that in this embodiment, at each time step, the cumulative membrane potential of the current time step and the current batch of input features is compared with the activation threshold V of the M-RER neuron. M-RER_th Instead of accumulating all the input information at all time steps before comparing it with the activation threshold, or accumulating all the image features of a frame within the next encoding time step before comparing it with the activation threshold, traditional spiking neural network (SNN) implementations either accumulate all the input potentials at all time steps before comparing them with the activation threshold, or accumulate all the pixel or convolutional unit outputs of a frame within an encoding time step before triggering the threshold comparison. This mode requires waiting for the entire pulse coding cycle to end before determining whether to discharge, resulting in serious time delay and power consumption problems.

[0108] This invention proposes a time-step dynamic threshold comparison mechanism that immediately determines whether a discharge (output event pulse) is triggered within each time step. This mechanism offers significant innovative technical benefits, including: significantly reduced inference latency, enabling event response within a single time step; reduced invalid computations and memory accesses by preemptively closing invalid multiply-accumulate operations and memory access paths, thus reducing dynamic power consumption; enhanced event-driven characteristics by incorporating a time-step judgment and triggering mechanism more closely resembling that of biological neurons; and improved hardware pipeline parallelism and reduced register usage and access frequency. Furthermore, this activation mechanism effectively improves the sensitivity to weak signals. For short-duration, strong input pulse sequences, it can immediately trigger a response without being masked by the averaging effect of subsequent time steps, thereby facilitating the detection of instantaneous significant events and improving the detection of small targets and the extraction of low-contrast features.

[0109] Furthermore, due to the implementation of M-RER neurons in digital circuits, the cumulative membrane potential of M-RER... Using a fixed-point number with a finite bit width for representation, the membrane potential decays rapidly when the input is continuously negative or there is no input, potentially leading to underflow. That is, when the membrane potential decreases to its lower limit (e.g., 0 or a negative number), the digital register may overflow, resulting in an erroneous maximum value. After excessive decay, when subsequent input signals are superimposed, the membrane potential that should have been triggered fails to reach the threshold, causing "missed activation" and resulting in misjudgment of activation. Furthermore, a membrane potential that remains close to zero for a long time weakens the memory effect of M-RER neurons. Therefore, to solve the above problems, in this embodiment of the invention, as... Figure 8 As shown, the memory-related event response neuron also integrates a lower threshold comparison circuit 3048 to prevent false activation due to downward overflow of membrane potential.

[0110] In specific implementation, the lower threshold comparison circuit 3048 mainly includes a lower threshold comparator and a multiplexer. The lower threshold comparator is used to detect the membrane potential of the previous time step. Is it less than the set threshold? If true, a selection signal LowFlag is generated to select the input signal of the multiplexer. When LowFlag=1, the selected path switches to baseline compensation mode and the lower threshold is applied. When LowFlag=0, the selected pathway is the standard membrane potential update mode. In the baseline compensation mode, the lower threshold... With weighted event potential increment Superimposed generation of the first A memory-related event response neuron at time step The cumulative membrane potential, i.e. This simple circuit setup ensures that even if no pulse is triggered for a long time, the M-RER neuron maintains a certain baseline potential and does not completely fall asleep, improving the temporal continuity and stability of the neural network. The comparators and multiplexers inside the M-RER neuron are basic logic units, requiring no additional storage or multiplication devices, resulting in low latency and low power consumption, making it suitable for large-scale neuron array deployments.

[0111] Furthermore, in a preferred embodiment, the event decoder 305 includes: a pulse counter array, a maximum value comparison logic circuit, a tag selector array, and a classification tag register file.

[0112] The pulse counter array includes multiple event counting units corresponding one-to-one with the memory-related event response neurons. Each event counting unit includes at least an additivity register, a pulse detection logic circuit, and a clearing controller. The pulse counter array is used to count the cumulative pulse count of each memory-related event response neuron across all time steps. The maximum value comparison logic circuit is a priority encoder with a multi-level comparator tree structure. Each level of the comparator's input is connected to the corresponding output of the pulse counter array to determine the index signal of the neuron with the largest pulse count. The classification label register file contains N register units, each storing a category label signal obtained from offline training. The label selector array includes a multiplexer group, which includes a selection control terminal and multiple data input terminals. The selection control terminal receives the index signal of the neuron with the largest pulse count; each data input terminal receives a corresponding category label signal from the classification label register; the label selector array selects and outputs the corresponding category label signal based on the index signal of the neuron with the largest pulse count, thus completing the decoding from pulse events to classification results.

[0113] Figure 9 This is a schematic diagram of another embodiment of the low-power convolutional pulse heterogeneous neuromorphic image recognition system provided by the present invention. In this embodiment, the low-power convolutional pulse heterogeneous neuromorphic image recognition system further includes a control processing unit 20. The control processing unit 20 is a RISC-V processor designed based on a pulse event computing extended instruction set architecture, used to hardware accelerate the membrane potential update and pulse firing logic of the memory-independent event response neuron layer 301 and the memory-related event response neuron layer 304.

[0114] In practical implementation, the RISC-V processor refers to a processor designed using the RISC-V instruction set architecture. The RISC-V instruction set can be functionally classified into a basic integer instruction set (e.g., RV32I / RV64I / RV128I) and an extended instruction set. The extended instruction set adds instructions to the basic integer instruction set to meet specific application requirements. It is important to note that although the RISC-V instruction set architecture provides standard definitions for various instruction formats, the RISC-V instruction set is merely an interface standard between hardware circuits and software programs; the RISC-V processor is a specific hardware circuit system implemented according to this standard. Therefore, based on this RISC-V instruction set architecture, RISC-V processors with completely different hardware circuit structures and varying operating effects and efficiencies can be designed.

[0115] In this embodiment, the internal structure of the RISC-V processor, which serves as the control processing unit 20, is as follows: Figure 9 As shown. Specifically, the RISC-V processor sequentially includes an instruction fetch unit 21, an instruction buffer unit 22, a decode unit 23, an issue unit 24, a physical register file 25, an execution unit 26, and a reorder cache 27, comprising a seven-stage pipelined architecture. In this embodiment, the RISC-V control processing unit adopts a seven-stage pipelined architecture to achieve high instruction throughput and low control latency processing performance.

[0116] Specifically, the instruction fetch unit 21 is responsible for reading the next instruction to be executed from the instruction cache according to the program counter (PC); the instruction buffer unit 22 is used to temporarily store the fetched instruction stream and maintain the continuity of the instruction queue to prevent rate mismatch between the instruction fetch and decoding stages; the decoding unit 23 performs opcode parsing, register address extraction, and extended instruction identification for instructions, providing support for subsequent operand preparation and control signal generation; the issue unit 24 allocates instructions to integer, memory access, floating-point, or extended neural network instruction issue queues according to the instruction type and execution unit status, realizing multi-channel parallel issuance; the physical register file 25 is responsible for storing operands and intermediate results, and avoids data-related conflicts through the register renaming mechanism; the execution unit 26 completes arithmetic and logical operations, address calculation, and the generation of dedicated signals (control signals) for extended instructions, and transmits the results to the write-back or peripheral interface; the reorder cache 27 is used to ensure the consistency of instruction order under out-of-order execution, and to perform execution exception detection and commit management. Overall, this seven-stage pipeline significantly improves the real-time scheduling capability and data path utilization of the control processing unit 20 in the neural network system through refined task division and structural parallelism.

[0117] Furthermore, the RISC-V control processing unit 20, based on the aforementioned seven-stage pipeline architecture, incorporates architectural-level extensions for the event-driven computing characteristics of the neural network processor. Specifically, a parallel extended instruction issuance queue is added to the issue unit 24. This queue is independent of the regular integer and memory access instruction issuance channels and is used to schedule and issue extended instructions for neural network acceleration. These extended instructions include operation control signals such as event encoding triggering, weight loading, synaptic array mapping, membrane potential updating, and pulse synchronization. Correspondingly, the execution unit 26 integrates a dedicated extended instruction execution unit corresponding one-to-one with this extended instruction set. This execution unit can generate specific neural network control signals based on the decoding results and send them to the event-driven neural network processor through the on-chip high-speed interconnect interface, thereby achieving precise scheduling and asynchronous collaborative control of the neural network subsystem. This design achieves instruction-level hardware and software collaboration, enabling the main control processor to directly drive the neural network engine through extended instructions without complex software interrupts or peripheral access operations, significantly reducing control latency and significantly improving the system-level parallelism and overall computational efficiency of neural computing.

[0118] Figure 10 This is a magnified microscopic image of a chip from an embodiment of the neural network processor implemented using a 40nm all-digital integrated circuit, provided by the present invention. Specifically, this embodiment further utilizes 40nm CMOS (Complementary Metal-Oxide-Semiconductor) digital circuit technology to achieve superior performance of the convolutional pulse heterogeneous neuromorphic image recognition system in the field of neural network inference.

[0119] This embodiment first conducts offline training of the neural network on the NVIDIA GeForce RTX 4060 platform, using supervised learning to optimize the model parameters and ultimately obtain the trained network weight parameters. After obtaining the trained network weight parameters, a hardware design platform is built for chip IC design. Specifically, this includes: building a simulation verification platform using VCS and Verdi tools for accurate timing simulation and functional verification of the designed logic functions; building a back-end design platform and using EDA tools such as DC and Innovus to perform synthesis and placement and routing of the chip design to obtain detailed power consumption and area data. Finally, based on the data collected by the IC evaluation platform, relevant performance indicators are calculated to comprehensively evaluate the actual operating performance of the neuromorphic processor.

[0120] As shown in Table 1, the experimental data demonstrates that this design achieves a maximum operating frequency of 357MHz and an extremely low total power consumption of 20.80mW while maintaining an accuracy of 91.11% on the MNIST handwritten digit recognition dataset. Its single-operation energy consumption is only 0.395 pJ / SOP, and the chip area is only 0.225 mm². Regarding power consumption, the 20.80mW total power consumption consists of 11.09mW of static power consumption and 9.71mW of dynamic power consumption, the latter further subdivided into 1.36mW of switching power consumption and 8.35mW of internal power consumption.

[0121] Table 1 Performance of Neural Network Processors Based on 40nm CMOS Digital Process

[0122]

[0123] As shown in Table 2, compared with other existing neuromorphic processors, the hardware architecture proposed in this embodiment of the neural network processor demonstrates significant advantages in several key performance indicators. In terms of inference accuracy, the basic architecture achieves 91.11% accuracy under 40nm process technology, comparable to existing mainstream solutions. By adding an M-RER layer, the accuracy can be further improved to 96.34%, exceeding all compared technologies and ensuring high-accuracy inference results. Regarding energy efficiency, the power consumption of this embodiment is only 0.02W, with inference energy consumption as low as approximately 0.4pJ / SOP. Compared to the watt-level power consumption of existing solutions, this represents an order-of-magnitude reduction, significantly extending device battery life and significantly improving inference speed and system throughput. Although there are slight differences in normalized area compared to some older process solutions, considering the comprehensive improvement in accuracy, power consumption, and frequency, the area efficiency of this invention remains competitive. Furthermore, it achieves a perfect balance between high performance and low power consumption under 40nm process technology, demonstrating good process adaptability and cost advantages.

[0124] Table 2 Performance comparison with existing neuromorphic processors

[0125]

[0126] Therefore, the embodiments of the present invention provide an innovative hardware acceleration solution for artificial intelligence inference that combines high precision, ultra-low power consumption, high frequency and area optimization, and has broad application prospects and huge economic value.

[0127] It should be noted that the above content is merely a preferred embodiment of this application and does not imply that the scope of protection of this application is limited to this. Any equivalent modifications or alternatives made by those skilled in the art within the technical framework disclosed in this application, based on existing technical knowledge, should fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the appended claims.

Claims

1. A low-power convolutional pulse heterogeneous neuromorphic image recognition system, characterized in that, include: Neural network processor based on all-digital integrated circuits; The neural network processor includes an event encoder, a convolutional feature extraction circuit, and an event-driven neural network inference engine connected in sequence. The event encoder is used to convert the input image into a pulse sequence event stream; the height of the input image is... Width is ; The convolutional feature extraction circuit is used to perform overlapping grid-like scanning convolutions on the pulse sequence event stream with sliding steps S based on a convolutional window of size K×K, sequentially extracting the local spatial features of each convolutional window as the convolutional response to form a sparse spatial convolutional feature map; the height of the sparse spatial convolutional feature map is... Width is And 0 < S < K; The event-driven neural network inference engine includes a memory-independent event response neuron layer, a non-overlapping maximum aggregation layer, a synaptic crossover array, a memory-related event response neuron layer, and an event decoder, which are connected in sequence. The memory-independent event response neuron layer is used to perform non-cumulative event detection on each convolution response of the sparse spatial convolution feature map, generate one-to-one corresponding indicator pulse signals and construct them into a memory-independent indicator variable feature map. The non-overlapping maximum aggregation layer is used to select the maximum event activation in each non-overlapping indicator block on the indicator variable feature map based on the preset non-overlapping indicator block, and generate the maximum aggregation feature map by sequentially obtaining the maximum event activation features. The synaptic crossover array is used to access each element of the maximum aggregated feature map and its corresponding pulse weight value in batches, perform weighted summation and propagation, and calculate a one-to-one weighted event potential increment for each neuron of the memory-related event response neuron layer. The memory-related event response neuron layer is used to perform time-related membrane potential accumulation calculation and firing operation on the weighted event potential increment. The event decoder is used to statistically analyze and decode the output pulses of neurons in the memory-related event response neuron layer to generate a classification result for the input image.

2. The low-power convolutional pulse heterogeneous neuromorphic image recognition system according to claim 1, characterized in that, The memory-independent event-response neuron layer comprises M memory-independent event-response neurons, and M ≤ This is used to perform non-cumulative event detection on each convolutional response of the sparse spatial convolutional feature map in batches by parallel reusing the M memory-independent event response neurons; each memory-independent event response neuron includes a first threshold register, a first threshold comparator, and a pulse indication logic circuit, used to perform non-cumulative event detection based on the following memoryless threshold model and generate an indication pulse signal: in, In time step t And the index of the top left corner of the convolution window is the coordinate ( , The convolutional response of ). α It is a proportionality constant; Neurons responding to events irrelevant to the current memory at time steps And the index is the coordinate ( , The instantaneous membrane potential; A configurable indicator pulse trigger threshold; Neurons responding to events irrelevant to the current memory at time steps And the index is the coordinate ( , The output indicator pulse signal; where, The first threshold register is used to store a configurable indicator pulse trigger threshold; The first threshold comparator is used to scale the received convolution response, generate the instantaneous membrane potential of the current memory-irrelevant event response neuron, and compare it with the indicator pulse trigger threshold. The pulse indication logic circuit is used to receive the comparison result output by the first threshold comparator, and generate a digital pulse of unit width as the indication pulse signal when the instantaneous membrane potential is greater than or equal to the indication pulse trigger threshold.

3. The low-power convolutional pulse heterogeneous neuromorphic image recognition system according to claim 1, characterized in that, Every four adjacent indicator pulse signals on the indicator variable feature map constitute a non-overlapping indicator block in a 2×2 two-dimensional space; The non-overlapping maximum aggregation layer includes multiple four-input comparator trees; each four-input comparator tree is an OR gate logic circuit used to perform the following maximum value selection operation on each of the non-overlapping indicator blocks, generating a maximum value at time step. t And the output position index is ( u, v Maximum event activation feature : in, Neurons responding to events irrelevant to the current memory at time steps And the index is the coordinate ( , The output indicator pulse signal; and These are used to control the offset of the indicator pulse signal values ​​of the non-overlapping indicator block in the horizontal and vertical directions, respectively; output position index. The range of values ​​for is: ; Index of each output position Maximum event activation feature Composed of time steps The maximum aggregated feature map.

4. The low-power convolutional pulse heterogeneous neuromorphic image recognition system according to claim 1, characterized in that, The synaptic cross array includes a row and column cross interconnection matrix composed of multiple synaptic computing units; The row-column cross-interconnection matrix adopts a reconfigurable propagation structure with axonal row inputs and dendritic column outputs. The synaptic computing unit is located at the intersection of axons and dendrites and is used to realize weighted event propagation based on the following model to calculate the first [number] neuron in the memory-related event response neuron layer. At time step, one neuron Weighted event potential increment : in, For the position index is ( u, v And with the first Synaptic weights of dendritic connections; In time step t And the output position index is ( u, v The maximum event activation feature is defined as follows: R is the number of rows in the row-column cross-interconnection matrix, corresponding to the number of maximum event activation features output in each batch of the maximum aggregation feature map; C is the number of columns in the row-column cross-interconnection matrix, corresponding to the number of neurons in the memory-related event response neuron layer; the row direction of the row-column cross-interconnection matrix corresponds to the axonal propagation path of the non-overlapping maximum aggregation layer of the preceding stage, and the column direction corresponds to the dendritic reception path of the memory-related event response neuron layer of the following stage.

5. The low-power convolutional pulse heterogeneous neuromorphic image recognition system according to claim 1, characterized in that, The convolutional feature extraction circuit includes multiple sets of parallel convolutional units; each set of convolutional units is used to implement the following event convolution model, which is applied to features extracted at time steps. Weight accumulation calculation is performed on the pulse sequence event stream: in, In time step t The index of the top left corner of the convolution window is the coordinate ( , And the index coordinates inside the convolution window are ( m , n Pulse sequence events at position ) For the convolution kernel in ( m , n The convolution weights at position ); the index of the top-left corner of the convolution window. The range of values ​​for is: ; In time step t And the index of the top left corner of the convolution window is the coordinate ( , The convolution response of ) ; the convolution response of each index Composed of time steps t The sparse spatial convolution feature map.

6. The low-power convolutional pulse heterogeneous neuromorphic image recognition system according to claim 5, characterized in that, Each group of convolutional units includes a first weight register array, an input selector array, a first adder tree circuit, and a convolution control logic module; The first weight register array is used to store and output convolution weight values, providing convolution weight values ​​at corresponding positions for the convolution calculation of each pulse sequence event; The input selector array is composed of AND gate circuits, which are used to perform a logical AND operation between each bit of the convolution weight value and the current pulse sequence event, and perform a bit-by-bit addition operation on the result to realize the dot-matrix weighted operation between the current pulse sequence event and the convolution weight value at the corresponding position. The first adder tree circuit is used to aggregate and sum the multiple weighted values ​​output by the input selector array to generate the convolution response of each convolution window at each time step. The convolution control logic module is used to coordinate the timing and data flow of each group of convolution units, and to realize weight loading, convolution window sliding, pulse timing control and addition tree startup.

7. The low-power convolutional pulse heterogeneous neuromorphic image recognition system according to claim 4, characterized in that, The synaptic computing unit includes a second weight register array, a multiplication logic unit, and a second addition tree circuit; The second weight register array is used to store the weight values ​​of each synapse; The multiplication logic unit is composed of a multiplexer and is used to perform a dot-multiplication weighted operation between the synaptic weight values ​​at each of the intersection points and the maximum event activation feature. The second additive tree circuit is used to sum all weighted results from the same dendritic direction, and sequentially generate the weighted event potential increments accumulated by each dendritic receiving path, which correspond to the input potentials of each neuron in the memory-related event response neuron layer.

8. The low-power convolutional pulse heterogeneous neuromorphic image recognition system according to claim 1, characterized in that, The memory-related event response neuron layer includes N memory-related event response neurons; N is the number of identifiable image target categories; the circuitry of each memory-related event response neuron includes: a membrane potential register, an update adder, a difference calculator, and a leakage shifter; The membrane potential register is used to store the cumulative membrane potential of the memory-related event response neuron in the previous time step; The difference calculator is used to calculate the potential difference between the weighted event potential increment of the memory-related event response neuron at the current time step and its cumulative membrane potential at the previous time step. The leakage shifter is used to perform an approximate division operation on the potential difference value based on a preset membrane time constant, thereby realizing leakage control of the cumulative membrane potential of the memory-related event response neuron and outputting potential decay margin. The update adder is used to add the cumulative membrane potential of the memory-related event response neuron in the previous time step to the potential decay margin to generate the cumulative membrane potential of the memory-related event response neuron in the current time step. The memory-related event response neuron further includes a second threshold comparator, a second threshold register, and a reset logic unit; the second threshold comparator is used to compare the cumulative membrane potential at the current time step with the neuron activation threshold stored in the second threshold register, and control whether the current memory-related event response neuron executes a firing event based on the comparison result; the reset logic unit is used to reset the cumulative membrane potential of the current memory-related event response neuron when a firing event occurs; The membrane potential register is also used to update the data it stores: if the current memory-related event response neuron fires, the accumulated membrane potential after reset is stored; otherwise, the accumulated membrane potential output by the update adder at the current time step is stored.

9. The low-power convolutional pulse heterogeneous neuromorphic image recognition system according to claim 8, characterized in that, The event decoder includes: a pulse counter array, a maximum value comparison logic circuit, a tag selector array, and a classification tag register file; The pulse counter array includes: a plurality of event counting units corresponding one-to-one with the memory-related event response neurons, each of the event counting units including at least an additivity register, a pulse detection logic circuit and a clearing controller; the pulse counter array is used to count the cumulative number of pulses of each memory-related event response neuron in all time steps; The maximum value comparison logic circuit is a priority encoder with a multi-level comparator tree structure. Its input terminals at each level are connected to the corresponding output terminals of the pulse counter array to determine the index signal of the maximum pulse count. The classification label register file contains N register units, each of which stores a category label signal obtained from offline training; The tag selector array includes a multiplexer group, which includes a selection control terminal and multiple data input terminals. The selection control terminal is used to receive the index signal of the pulse with the largest pulse count. Each of the data input terminals is used to receive the category tag signal from the classification tag register. The tag selector array is used to select the corresponding category tag signal according to the index signal of the pulse with the largest pulse count and output it, thus completing the decoding from pulse event to classification result.

10. The low-power convolutional pulse heterogeneous neuromorphic image recognition system according to any one of claims 1 to 9, characterized in that, The system also includes a control processing unit; the control processing unit is a RISC-V processor designed based on the pulse event computing extended instruction set architecture, used to hardware accelerate the membrane potential update and pulse firing logic of the memory-independent event response neuron layer and the memory-related event response neuron layer.

Images