A pulse neural network handwritten digit recognition system based on FPGA
By constructing an FPGA-based spiking neural network handwritten digit recognition system, the problems of parameter inconsistency and verification during SNN model inference from software to hardware were solved, achieving end-to-end code closed-loop verification and ensuring the stability and reliability of hardware deployment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING UNIV OF TECH
- Filing Date
- 2026-01-18
- Publication Date
- 2026-06-23
AI Technical Summary
In existing technologies, when training SNN models from the software side to inference on FPGA hardware circuits, there are problems such as inconsistent parameter formats, inconsistent interface timing, and difficulty in reproducing verification, and there is a lack of end-to-end code-level closed-loop verification process.
We adopt an integrated software-hardware collaborative process of "model training - parameter fixed-point quantization - test input generation - hardware inference and simulation verification". We construct a two-layer fully connected and LIF neural network through PyTorch and snnTorch to realize parameter fixed-point quantization and export, generate input files that can be reproduced in simulation, and perform inference and simulation verification on the FPGA to ensure the consistency of hardware and software.
An end-to-end code closed loop from training to hardware inference was achieved, ensuring the repeatability and regressibility of parameter initialization and simulation verification. Through multi-sample regression testing and fault localization, the stability and reliability of SNN hardware deployment were improved.
Smart Images
Figure CN122263987A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to an FPGA-based pulse neural network inference digital circuit implementation for handwritten digit recognition, belonging to the field of integrated circuits. Background Technology
[0002] Spiking Neural Networks (SNNs), as a third-generation artificial neural network model that is closer to the working mechanism of biological nervous systems, use discrete spike events as information carriers and complete information transmission and computation through dynamic behaviors such as the integration, leakage and threshold triggering of neuronal membrane potential in the time domain.
[0003] In fields such as pattern recognition, traditional deep artificial neural network (ANN) computational models have achieved significant results, but they are highly dependent on continuous value computation and synchronous clock driving. In SNN models, information is encoded as peak arrival times and rates, which have advantages such as event-driven, sparse computation and potential low power consumption, promoting research on brain-like computing paradigms and neuromorphic realizations.
[0004] The efficient operation of SNNs relies on event-driven communication and parallel computational organization of neurons / synapses, demonstrating a natural advantage in hardware implementation by combining with neuromorphic hardware and dedicated accelerators. While general-purpose processors like CPUs / GPUs excel at tensor computations such as parameter training, they suffer from significant idle computation and energy waste during SNN inference. Field-programmable gate arrays (FPGAs), with their programmability, parallel processing capabilities, and low latency, can be used for SNN verification and acceleration architecture implementation. The division of labor between these two technologies in building the network makes them a crucial platform for realizing hardware acceleration of SNNs.
[0005] However, when migrating SNN models trained in software to FPGA hardware circuits for inference, the training framework often uses floating-point weights / biases and continuous-value inputs, while the hardware typically uses low-bit-width fixed-point representations and requires reproducible input pulse sequences and strictly aligned timing schedules. Without a closed-loop code chain from training, quantization derivation, input generation to hardware simulation, problems such as inconsistent parameter formats, inconsistent interface timing, and difficulty in reproducing verification can easily arise. Therefore, it is urgent to construct an end-to-end software-hardware co-implementation process to complete code-level closed-loop verification and provide stable interfaces and repeatable test benchmarks for subsequent hardware deployment. Summary of the Invention
[0006] This invention provides a handwritten digit recognition system based on an FPGA-based spiking neural network, employing an integrated software-hardware collaborative process encompassing "model training—parameter fixed-point quantization—test input generation—hardware inference and simulation verification".
[0007] This invention is achieved through the following technical solution:
[0008] The FPGA-based spiking neural network handwritten digit recognition system consists of: a CPU / GPU software training module, a parameter fixed-point quantization and export module, a test input generation module, an FPGA-based model prediction module, and a simulation verification module.
[0009] On the software side, the CPU / GPU training module constructs a two-layer fully connected and two-level Leaky Integrate Fire (LIF) neural network based on PyTorch and snnTorch. It performs forward propagation within a preset time step, using time-dimension peak counts as the classification criterion. Backpropagation is performed using the cross-entropy loss function and updating the network weights and bias parameters. After training, the network parameter file is saved. The parameter fixed-point quantization and export module reads the trained weights and biases, scales and rounds them according to a signed fixed-point format, and clamps them within a 16-bit range. It outputs a .coe file compatible with FPGA initialization and a .mem file for RTL simulation. It can also segment and organize large-scale weight matrices according to the hardware-side ROM bit width. The test input generation module selects samples from the MNIST test set, quantizes grayscale pixels in the same fixed-point format, and generates corresponding .mem (optional .coe) files to achieve simulation regression testing.
[0010] In terms of hardware, the FPGA-side model prediction module includes a top-level module TOP, a first fully connected computation module FC1, a parallel-to-serial conversion module PAR2SER, a second fully connected computation module FC2, and the next-layer LIF neuron computation modules LIF_FC1 / LIF_FC2. The top-level module TOP is responsible for image loading and caching, random pulse coding, time step / playback scheduling, inference computation, and pulse counting. FC1 / FC2 completes the fully connected weighted accumulation and LIF distribution. PAR2SER is used for parallel-to-serial conversion between layers to ensure the consistency of input and output between the two layers. LIF_FC1 / LIF_FC2 is responsible for simulating the computational behavior of snnTorch spiking neural network neurons (membrane potential leakage, accumulation, threshold triggering, and reset). The simulation verification module tb_test reads the test input .mem file and drives the TOP interface to complete the simulation process of "triggering - loading - classification - output comparison and logging", realizing code-level closed-loop verification before board deployment.
[0011] The mapping function between the fixed-point parameters and the floating-point tensor to the vertex is derived and written into a hexadecimal string in two's complement form, avoiding complex floating-point operations in hardware and ensuring consistency between Vivado / Verilog reading.
[0012] The FPGA random pulse encoding uses a linear feedback shift register (LFSR) to generate pseudo-random numbers, which are compared with fixed-point pixel grayscale values. If the value is less than the fixed-point value, an input pulse is generated. This discards binary values and addresses the low precision issue caused by the input. By rewinding over multiple time steps, the average pulse firing rate approaches the original grayscale intensity, thus achieving encoding from continuous grayscale to a pulse sequence.
[0013] Compared with existing technologies, this invention provides an end-to-end code closed loop from snnTorch training to FPGA inference, ensuring consistency between training parameters and RTL inference structure; through dual-format export of .coe / .mem, parameter initialization and simulation verification are repeatable and regressive; through the unified scheduling of the "load-playback-interval-completion" cycle and inter-layer parallel-serial conversion of the TOP internal state machine, the two-level fully connected layers and LIF, ROM IP cores and LIF updates are strictly aligned, facilitating timing verification and parameter tuning; by generating labeled test input generation scripts and a multi-image simulation test platform, stable regression testing and fault location are achieved before board deployment. Attached Figure Description
[0014] Figure 1 This is a structural block diagram of the handwritten digit recognition system of the present invention.
[0015] Figure 2 This is the circuit design diagram of the LIF model of this invention.
[0016] Figure 3 This is a timing diagram of the top-level (TOP) state machine of this invention. Detailed Implementation
[0017] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0018] A spiking neural network handwritten digit recognition system based on FPGA is disclosed. The software training module employs PyTorch and snnTorch to construct an architecture consisting of a pixel input layer (784-dimensional), a hidden layer (500-dimensional), and a digit output layer (10-dimensional). The network comprises two fully connected layers and two levels of LIF neurons. During training, the MNIST dataset and the PyTorch-integrated DataLoader are used to batch load data. Forward propagation (fc1→LIF→fc2→LIF) is performed at each time step to obtain the spike counts for each digit category. Backpropagation calculates the cross-entropy loss and approximates the gradient using the arctan derivative, updating the weights / biases of the fully connected layers with Adam. After several epochs of training, the network parameters are saved to a .pth file for subsequent fixed-point and data segmentation storage. It should be noted that hyperparameters such as the threshold and beta leakage coefficient do not require training during the software training phase and can be set according to the hardware task accuracy and distribution sparsity.
[0019] To map the trained floating-point weights / biases to a synthesizable fixed-point circuit on the FPGA, the parameter fixed-point quantization and derivation module scales, rounds, and clamps the parameters within the signed range using a quantization function of scale=2^n, ultimately outputting a torch tensor. The parameters are exported as .coe and .mem files, common formats for Vivado IP cores and Verilog, and can be appropriately segmented considering the FPGA ROM size and the dimensions of network neurons.
[0020] To ensure reproducible simulations, the test input generation module script provides MNIST test input samples, flattens their grayscale pixels (0~1) into a 784-pixel long vector, and quantizes them using the same fixed-point format as the parameters. It outputs a .mem file (optionally also outputting a .coe file) containing one hexadecimal pixel value per line, which can be directly read and loaded by the simulation platform using `$readmemh`, avoiding test inconsistencies caused by input randomness.
[0021] The hardware inference module is implemented with TOP as the top-level module. Internally, it uses a 784-depth image buffer to store the grayscale pixels of the input image and employs a finite state machine (FSM) to control the flow: S_IDLE (wait for trigger), S_LOAD (load pixels), S_PLAY (play back pixels and generate input pulses), S_GAP (inter-round control and time step accumulation), and S_DONE (execute argmax to latch the classification result and return to idle). This ensures a consistent rhythm for the "loading-playback-counting-decision" process. In the S_PLAY phase, TOP uses a 12-bit LFSR to generate random numbers and compares them with the current grayscale pixel. When the random number is lower than the pixel value, it outputs the pulse for that frame, thus mapping continuous grayscale values to a random pulse sequence. Each playback cycle traverses 784 pixels. After playback ends, it enters S_GAP to control the frame and increments the time step count. Only after completing the preset number of playback cycles can it enter S_DONE to wait for the input of the next image. Throughout the inference process, TOP dynamically accumulates the output layer spikes. In the S_DONE stage, it performs maximum selection on the ten results, latches them to the prediction result output CLASS_OUT, sets the prediction result output enable CLASS_VALID, and then returns to S_IDLE. This counting-based decision method can utilize multi-timestep pulse statistics to suppress random encoding fluctuations and improve classification stability.
[0022] The first fully connected computation module FC1 receives serial input spikes and valid signals from TOP, counts the 784 inputs within the playback cycle, and uses this as an address to read parallel weights and bias vectors from ROM / .mem. FC1 instantiates 500 LIF_FC1 units through the generate structure. Each unit independently selects the corresponding weight slice and bias, completes current accumulation within the playback cycle, and triggers membrane potential update and threshold decision at the end of the cycle, outputting a 500-bit parallel spike sequence and an output valid signal (obtained by OR reduction of the neuron output enable vector to avoid single-path lockout and explicitly mark "output has been generated in this round").
[0023] During the playback cycle, when the input spike is valid, LIF_FC1 adds the corresponding weight as the current contribution to the accumulator ACC_TEMP. At the end of the playback cycle, it triggers the calculation enable EN_CAL, decays the previous membrane potential VOL_TEMP by the leakage coefficient beta, and sums it with ACC_TEMP and the bias BIA. After ReLU, it obtains the new membrane potential mem_new. When mem_new exceeds the threshold th, it outputs a spike and resets the membrane potential to 0; otherwise, it remains at mem_new, and ACC_TEMP is cleared to enter the next playback cycle. LIF_FC2 uses the same structure but can set different thresholds and leakage coefficients to match the output layer's sparsity. The module provides example parameters that show good performance in this case: th1=1280, beta1=3072, th2=2048, beta2=2867.
[0024] The parallel-to-serial module PAR2SER is used for inter-layer interface conversion. When SPK1_OUT_VALID is enabled on the FC1 layer output, PAR2SER latches a 500-bit parallel spike vector into the internal buffer TRANS_BUFFER, and outputs the least significant bit spike in a right-shift manner over the next 500 cycles. This module enables serial transmission of parallel vectors, reduces inter-layer routing complexity, ensures consistency between the input and output of the two layers, and better unifies the working mechanisms of the two layers. Therefore, for ease of explanation, only the first layer is shown.
[0025] The simulation verification module tb_test is used for closed-loop verification of circuit function simulation. It generates a 100MHz clock and reset sequence, uses $readmemh to read multiple test input images and sequentially drives TOP to complete the loading; after the images are loaded, it waits for the prediction result output to enable CLASS_VALID to go high, outputs the predicted category and true value, and prints various spike counts, while recording key intermediate signals for fault location; after each image is completed, a light reset is performed to enter the next image, realizing multi-sample regression testing; at the same time, mon_4in1.log records information such as FC1 input validity, FC1 output sparsity (popcount), parallel and serial outputs, and FC2 output vectors, and sets TIMEOUT_CYCLES timeout protection for fault location.
[0026] The simulation platform described above enables end-to-end inference link verification without relying on development board peripherals. It also allows for iterative parameter tuning of parameters such as threshold, leakage coefficient, number of replay rounds, number of quantization decimal places, and weight segmentation method, thereby achieving closed-loop verification before board deployment.
[0027] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.
Claims
1. A handwritten digit recognition system based on FPGA and spiking neural network, characterized in that: include: The system includes a CPU / GPU software training module, a parameter fixed-point quantization and export module, a test input generation module, an FPGA-side model prediction module, and a simulation verification module. The CPU / GPU-side software training module constructs a two-layer fully connected and two-level leaky integral-distribution LIF neural network based on PyTorch and snnTorch. It performs forward propagation within a preset time step, using time-dimensional peak counts as the classification criterion. Backpropagation is performed using the cross-entropy loss function and updating the network weights and bias parameters. After training, the network parameter file is saved. The parameter fixed-point quantization and export module reads the trained weights and biases, scales and rounds them according to a signed fixed-point format, and clamps them within a 16-bit range. It outputs a .coe file compatible with FPGA initialization and a .mem file for RTL simulation. It can also segment and organize large-scale weight matrices according to the hardware-side ROM bit width. The test input generation module selects samples from the MNIST test set, quantizes grayscale pixels in the same fixed-point format, and generates corresponding .mem files to achieve simulation regression testing. The FPGA-side model prediction module includes a top-level module TOP, a first fully connected computation module FC1, a parallel-to-serial conversion module PAR2SER, a second fully connected computation module FC2, and the next-layer LIF neuron computation modules LIF_FC1 / LIF_FC2. The top-level module TOP is responsible for image loading and caching, random pulse coding, time step / playback scheduling, inference computation, and pulse counting. FC1 / FC2 completes fully connected weighted accumulation and LIF distribution. PAR2SER is used for parallel-to-serial conversion between layers to ensure the consistency of input and output between the two layers. LIF_FC1 / LIF_FC2 is responsible for simulating the computation behavior of snnTorch's spiking neural network neurons. The simulation verification module tb_test reads the test input .mem file and drives the TOP interface to complete the simulation process of triggering, loading, classification, output comparison, and log recording, realizing code-level closed-loop verification before board deployment.
2. The FPGA-based spiking neural network handwritten digit recognition system according to claim 1, characterized in that: The computational behavior of the spiking neural network neurons includes: leakage, accumulation, threshold triggering, and reset of membrane potential.
3. The FPGA-based spiking neural network handwritten digit recognition system according to claim 1, characterized in that: The mapping function between the fixed-point parameters and the floating-point tensor to the vertex is derived and written into a hexadecimal string in two's complement form, avoiding complex floating-point operations in hardware and ensuring consistency between Vivado / Verilog reading.
4. The FPGA-based spiking neural network handwritten digit recognition system according to claim 1, characterized in that: The FPGA random pulse encoding uses a linear feedback shift register (LFSR) to generate pseudo-random numbers, which are compared with fixed-point pixel grayscale values. If the value is less than the latter, an input pulse is generated. The binary values are discarded to solve the problem of low precision caused by the input. The pulse average firing rate is brought close to the original grayscale intensity by playing back at multiple time steps, thus realizing the encoding from continuous grayscale to pulse sequence.
5. The FPGA-based spiking neural network handwritten digit recognition system according to claim 1, characterized in that: The CPU / GPU-side software training module uses PyTorch and snnTorch to construct an architecture consisting of a pixel input layer, hidden layers, and a digit output layer, which comprises two fully connected layers and two levels of LIF neurons. During training, the MNIST dataset and the DataLoader integrated within PyTorch are used to batch load data. Forward propagation is performed at each time step to obtain the peak counts for each digit category. Backpropagation calculates the cross-entropy loss and approximates the gradient using the arctan derivative, and updates the weights / biases of the fully connected layers using Adam. After completing several epochs of training, the network parameters are saved to a .pth file for subsequent fixed-point and data segmentation storage.
6. The FPGA-based spiking neural network handwritten digit recognition system according to claim 1, characterized in that: To ensure reproducibility of the simulation, the test input generation module script provides MNIST test input samples, flattens their grayscale pixels into a 784-pixel long vector, and quantizes them using the same fixed-point format as the parameters; it outputs a .mem file with one hexadecimal pixel value per line, which is directly read and loaded by the simulation platform via $readmemh, avoiding test inconsistencies caused by input randomness.
7. The FPGA-based spiking neural network handwritten digit recognition system according to claim 1, characterized in that: The top-level module TOP internally sets up a 784-depth image buffer to store the grayscale pixels of the input image and uses a finite state machine (FSM) to control the flow: S_IDLE waits for trigger, S_LOAD loads pixels, S_PLAY plays back pixels and generates input pulses, S_GAP controls the shooting between rounds and accumulates time steps, and S_DONE executes argmax to latch the classification results and returns to idle, ensuring that the timing of loading, playback, counting, and decision is consistent. In the S_PLAY phase, TOP uses a 12-bit LFSR to generate random numbers and compares them with the current grayscale pixels. When the random number is lower than the pixel value, the current pulse is output, thus mapping the continuous grayscale to a random pulse sequence. Each playback cycle traverses 784 pixels. After playback ends, it enters S_GAP control and increments the time step count. Only after completing the preset number of playback cycles can it enter S_DONE to wait for the input shown in the image below. During the entire inference process, TOP dynamically accumulates the output layer spikes. In the S_DONE stage, it performs maximum value selection on the ten results, latches them to the prediction result output CLASS_OUT, sets the prediction result output enable CLASS_VALID, and then returns to S_IDLE.
8. The FPGA-based spiking neural network handwritten digit recognition system according to claim 1, characterized in that: The first fully connected computing module FC1 receives the serial input spikes and valid signals from TOP, counts the 784 inputs within the playback cycle, and uses this as an address to read the parallel weights and bias vectors from ROM / .mem. FC1 instantiates 500 LIF_FC1 units through the generate structure. Each unit independently selects the corresponding weight slice and bias, completes current accumulation within the playback cycle, and triggers membrane potential update and threshold decision at the end of the cycle, outputting a 500-bit parallel spike sequence and an output valid signal.
9. The FPGA-based spiking neural network handwritten digit recognition system according to claim 8, characterized in that: When the input spike is valid during the playback cycle, the LIF_FC1 unit adds the corresponding weight as the current contribution of the current beat to the accumulator ACC_TEMP. At the end of the playback cycle, the calculation enable EN_CAL is triggered, the membrane potential VOL_TEMP of the previous round is decayed according to the leakage coefficient beta, and summed with ACC_TEMP and the bias BIA. The new membrane potential mem_new is obtained by ReLU. When mem_new exceeds the threshold th, a spike is output and the membrane potential is reset to 0. Otherwise, it remains at mem_new, and ACC_TEMP is cleared to enter the next playback round.