A still image to eventized image conversion circuit
By designing a conversion circuit from static images to event-driven images and using hardware circuitry to simulate micro-motion patterns and convolution operators, the problem of high-efficiency and low-cost conversion in existing technologies is solved, achieving high-quality event-driven image generation suitable for edge computer vision applications.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- UNIV OF ELECTRONICS SCI & TECH OF CHINA
- Filing Date
- 2026-03-17
- Publication Date
- 2026-05-29
AI Technical Summary
Existing static image to event-driven image conversion techniques are difficult to achieve high-quality, high-efficiency, and low-cost conversion in edge computing, which limits the large-scale deployment of SNNs in real-time edge vision tasks.
A circuit for converting static images to event-driven images was designed, including a light intensity extraction circuit, a dynamic threshold calculation circuit, a micro-motion feature extraction circuit, a feature buffering circuit, and an event release circuit. The hardware circuit directly simulates eight micro-motion modes and calculates the spatial derivative of the image through a 3×3 convolution operator to achieve efficient conversion.
It achieves high-quality event-driven image generation, reduces hardware costs, minimizes artifact noise, improves conversion speed and data quality, and is suitable for resource-constrained edge devices.
Smart Images
Figure CN122115193A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of neuromorphic computing and machine vision technology, specifically to a circuit for converting static images to event-driven images, which is particularly suitable for providing real-time, high-speed event-driven data input for spiking neural networks (SNNs). Background Technology
[0002] Spiking neural networks (SNNs), as the third generation of neural networks, are inspired by the biological brain and exhibit excellent energy efficiency on dedicated neuromorphic hardware due to their sparse, event-driven characteristics. However, existing neuromorphic processors are primarily designed for asynchronous event streams, while mainstream imaging devices still acquire frame-based RGB static images, resulting in a significant heterogeneous data format mismatch. Current solutions to this problem and their limitations are mainly reflected in the following three aspects:
[0003] Hardware alternatives (Dynamic Visual Sensors (DVS): This approach acquires event streams from the source by employing novel sensors such as DVS. While it solves the format matching problem, the scarcity of relevant large-scale, high-quality datasets makes network training difficult. Furthermore, DVS is susceptible to ambient light interference (such as flickering lights or LCD refreshes), generating significant artifact noise, and its high hardware cost makes it difficult to meet the cost-sensitive requirements of edge computing.
[0004] Encoding conversion scheme (Poisson coding equal probability mapping): Image intensity is mapped to pulse frequency through a Poisson coding circuit. Its drawbacks are: to ensure the accuracy of information representation, a long simulation time step is usually required, which significantly increases inference latency; and the random sampling process loses the fine spatial features of the image, resulting in low information conversion efficiency.
[0005] The computational transformation scheme (ANN-SNN hybrid architecture) utilizes convolutional units of traditional artificial neural networks (ANNs) for preprocessing, combined with LIF neurons to achieve feature extraction and transformation. This approach involves high-density floating-point operations, resulting in significant computational overhead and hardware resource consumption, directly contradicting the original intention of edge computing to achieve low latency and lightweight operation.
[0006] In summary, existing event image generation technologies struggle to balance data quality, conversion efficiency, and hardware costs, hindering the large-scale deployment of SNNs in real-time edge vision tasks. A high-quality, high-efficiency, and low-cost edge event-based image generation solution is still needed. Summary of the Invention
[0007] To address the aforementioned problems and shortcomings, and to resolve the difficulty of balancing data quality, conversion efficiency, and hardware cost in existing event image generation technologies, this invention provides a conversion circuit for converting static images to event-based images. This circuit achieves high-quality, high-efficiency, and low-cost conversion from static images to event-based images, and is particularly suitable for event-based images at the edge.
[0008] A circuit for converting static images to event-driven images includes: a light intensity extraction circuit, a dynamic threshold calculation circuit, a micro-motion feature extraction circuit, a feature buffering circuit, and an event triggering circuit.
[0009] The light intensity extraction circuit receives pixel-by-pixel RGB image data and a valid synchronization control signal from an external source at its input terminal, and broadcasts the light intensity calculation result to the dynamic threshold calculation circuit and the micro-motion feature extraction circuit at its output terminal. The light intensity extraction circuit calculates the light intensity value of each pixel in the input static image, and uses a comparator to calculate the maximum value (V=max(R,G,B)) of the three channels (R,G,B) in real time, outputting this maximum value as the single-channel light intensity value.
[0010] The dynamic threshold calculation circuit receives light intensity data streams from the light intensity extraction circuit and row and column synchronization signals of the image at its input terminal, and outputs the dynamic threshold calculation result to the event firing circuit at its output terminal; it is used to adaptively extract the global contrast features of the current frame and generate the dynamic threshold calculation result - the pulse firing threshold.
[0011] Furthermore, the dynamic threshold calculation circuit includes a comparator, a multiplier, a subtractor, a maximum value tracking register, and a minimum value tracking register. During the input of light intensity values for each frame's pixels, the circuit structure composed of the comparator, the maximum value tracking register, and the minimum value tracking register extracts the maximum and minimum light intensity values of the pixels in that frame in real time and stores them in their respective tracking registers. At the end of the frame, the subtractor outputs the difference between the values stored in the maximum value tracking register and the minimum value tracking register. This difference is input into the multiplier and multiplied by a fixed scaling factor (sensitivity parameter). The output of the multiplier is the dynamic threshold specific to that frame.
[0012] The micro-motion feature extraction circuit includes two sub-circuits: a pixel rearrangement circuit and a differential convolution circuit.
[0013] The pixel rearrangement circuit (which has the same structure as the rearrangement circuit in the industry's convolution calculation circuit) is used to rearrange the input light intensity value data stream and output the rearranged data stream to the differential convolution circuit for micro-motion calculation simulation.
[0014] The differential convolution circuit is used to perform 8-channel convolution calculations on the data stream rearranged by the pixel rearrangement circuit. The convolution kernels used are eight specially designed 3×3 convolution kernels:
[0015]
[0016]
[0017] By performing convolution calculations on the frame image using these 8 3×3 convolution kernels, the simulation of 8 micro-motion patterns is achieved, generating a micro-motion feature pattern map with 8 channels and a size of H×W.
[0018] The output of the differential convolution circuit serves as the output of the micro-motion feature extraction circuit, outputting the data stream of the micro-motion feature pattern map to the feature buffer circuit.
[0019] The feature buffer circuit is used to buffer the micro-motion feature pattern map to wait for the dynamic threshold calculation to be completed, and the buffer depth is greater than or equal to (H-1)×W. The input of the feature buffer circuit receives the data stream of the micro-motion feature pattern map calculated by the micro-motion feature extraction circuit, and the output of the feature buffer circuit outputs the data stream to the event dispatch circuit after the dynamic threshold calculation is completed.
[0020] The event emission circuit performs pulse emission judgment on the data stream of the input 8-channel micro-motion feature pattern map based on the dynamic threshold input by the dynamic threshold calculation circuit, and converts the 8-channel micro-motion feature pattern map calculated by the above micro-motion feature extraction circuit into a pulse event map with 8 time steps and 2 channels (corresponding to positive polarity channel and negative polarity channel respectively).
[0021] Furthermore, the specific conversion processing rules are as follows: For a pixel (x,y) in the i-th channel of the micro-motion feature pattern image that is greater than the dynamic threshold, it is converted into a "1" pulse event generated at the corresponding coordinate (x,y) in the positive polarity channel at the i-th time step; similarly, pixels with a value less than the opposite of the dynamic threshold are converted into a "1" pulse event generated at the corresponding coordinate in the negative polarity channel; other pixels with an absolute value less than or equal to the dynamic threshold are converted into a "0" pulse event generated at the corresponding coordinate.
[0022] The output of the event issuing circuit corresponds to the output of the overall conversion circuit, and outputs the data stream composed of the calculated pulse event graph as the conversion result from static image to event-based image.
[0023] Furthermore, the aforementioned static image to event-driven image conversion circuit is used in edge computer vision scenarios.
[0024] The reason why this invention can achieve the technical effects of "high data quality, real-time conversion, and low cost" that are impossible to achieve with existing technologies is that:
[0025] 1. This invention simulates eight micro-motion modes using pure hardware circuit algorithms to convert static images into event-driven images, directly reusing existing low-cost, mature RGB image sensor pipelines. This not only reduces the cost of acquiring event stream data by several orders of magnitude, but also fundamentally eliminates the high-frequency artifact noise caused by LCD screen refresh or fluorescent lamp flicker in physical DVS sensors, providing a high-quality data source for SNNs.
[0026] 2. Compared to existing Poisson coding and equal-probability transformation methods, which require extremely long time steps to fit light intensity through statistical frequencies and whose random sampling easily destroys image edge contours, this invention utilizes a 3×3 convolution operator to directly calculate the spatial derivatives of the image in eight directions. The spatial difference is naturally mapped to eight extremely short time steps, capturing and preserving the fine contour features of the target object.
[0027] 3. Compared to existing hybrid schemes that use ANN convolutional layers to extract features, which involve a large number of dense floating-point multiply-accumulate (MAC) operations and have extremely high power consumption, the circuit structure of this invention, except for the dynamic threshold calculation circuit, achieves "zero multiplier" feature extraction. Light intensity conversion uses only comparators, differential convolution degenerates into addition and shift operations with only +1, -1, and 0 sparse weights, and the dynamic threshold calculation circuit uses only one multiplier. The architecture consumes fewer logic gate resources, making it more suitable for the integration needs of resource-constrained edge devices. Attached Figure Description
[0028] Figure 1 This is a schematic diagram of the overall architecture of the circuit for converting static images to event-driven images according to the present invention.
[0029] Figure 2 This is a structural diagram of the dynamic threshold calculation circuit in the embodiment;
[0030] Figure 3 This is a diagram of the pixel rearrangement circuit structure.
[0031] Figure 4 This is a diagram of the differential convolution circuit structure;
[0032] Figure 5 The circuit structure diagram is provided for the event;
[0033] Figure 6 A screenshot showing the effect of converting a static image into an event-driven image.
[0034] Figure 7 This is a circuit waveform diagram displayed on Verdi. Detailed Implementation
[0035] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments, taking the conversion of a static ImageNet image (224×224 pixels, 3 channels per pixel, 8 bits per channel) into an event-based image as an example.
[0036] The overall architecture of the static image to event-driven image conversion circuit designed in this embodiment is as follows: Figure 1 As shown. The top-level module takes three 8-bit data channels (img_r, img_g, img_b) and transmission status signals: data valid signal, row scan signal, and column scan signal as input. The top-level module outputs two channels of event graph data (event_p, event_n) with eight time steps and transmission status signals with the same function as the input.
[0037] In this embodiment, the light intensity extraction circuit is implemented using cascaded hardware comparators; the dynamic threshold calculation circuit is implemented using a DSP.
[0038] The structural diagram of the dynamic threshold calculation circuit in this embodiment is as follows: Figure 2 As shown. The pixel rearrangement circuit structure designed in this embodiment is as follows. Figure 3 As shown. Under the control of the timing controller, three SRAM line buffers (each 8 bits wide and 2^24 deep) store the input light intensity data of the three most recent lines in a rotating manner. Simultaneously, under the control of the timing controller, the data from the two SRAM line buffers containing the previous one and two lines is retrieved and sent to the first two lines of a 3×3 horizontal shift register array. The light intensity data input in real-time from the external module (corresponding to the current line data) is sent to the third line of the register array and simultaneously written to another SRAM line buffer. The data in the shift register matrix is the rearranged data.
[0039] The differential convolution circuit structure designed in this embodiment is as follows: Figure 4 As shown, the data from the shift register is broadcast to eight parallel-running differential convolution channels.
[0040] Since each convolutional kernel contains only one "1" and one "-1", this embodiment uses two 4-bit registers to store the CSR encoding of the dimensionality-reduced convolutional kernels (weight_p and weight_n correspond to the addresses of "+1" and "-1" respectively, and 9 addresses require 4 bits for encoding). For example, for the first convolutional kernel, its encoding is: weight_p=3, weight_n=8. The two multiplexers (MUX) in the differential convolution channel select the minuend and subtrahend from the 9 registers to the two ends of the built-in subtractor, and output the 8-bit subtraction result—corresponding to the convolution calculation result. The output of the differential convolution circuit concatenates the data bits of the differential convolution channel into a 64-bit micro-motion feature pattern data stream.
[0041] The feature cache circuit designed in this embodiment is implemented by calling the synchronous FIFO IP. The called FIFO has a width of 64 bits and a depth of 65536 (greater than 224×223).
[0042] The event dispatching circuit structure designed in this embodiment is as follows: Figure 5 As shown, the 64-bit micro-motion feature pattern data stream is re-split into eight 8-bit feature channel data streams, which flow into eight event emission units respectively. Additionally, a dynamic threshold is broadcast into the eight event emission units. Each event emission unit contains two comparators that compare the feature data with the dynamic threshold and its inverse. The comparator output for the dynamic threshold input is a positive pulse; the comparator output for the inverse of the dynamic threshold input is a negative pulse, thus achieving pulse event emission for two channels per time step. The positive pulse bits of the eight event emission units are concatenated to form an 8-bit positive pulse data stream event_p; the negative pulse bits are concatenated to form an 8-bit negative pulse data stream event_n, serving as the top-level output data in this embodiment.
[0043] The following experiments test the data quality, conversion rate, and hardware resource consumption of this embodiment:
[0044] The experiment imports the RTL design of this embodiment into the DUT layer of the Synopsys VCS simulation platform, setting its operating frequency to 100MHz. ImageNet images are imported into memory in matrix form, reduced to 1D, and row and column scan flags are inserted to form an image data stream. The image data stream is imported into the VCS testbench layer in text form. The testbench logic is as follows: 1) The image data stream is used as stimulus input to the DUT; 2) The DUT output results are exported to the testbench in text form; 3) The simulation stops after the DUT outputs its results; 4) The waveform file of the entire simulation process is exported.
[0045] Data quality testing: The event-based image generated by processing the simulation output results of this embodiment using a Python script is shown below. Figure 6 As shown in the figure, red pixels represent pulse events in the positive polarity channel; blue pixels represent pulse events in the negative polarity channel; and black pixels represent no pulse events. It is evident that the edge contours of the event-generated image converted in this embodiment are well preserved, indicating high data quality. When event-generated images converted from ImageNet static images using this invention are input into an SNN-ResNet-18 network for training and inference, an accuracy of 59.28% is achieved, exceeding the 43.74% accuracy of traditional conversion methods.
[0046] Conversion rate test: Import the simulation waveform file of this embodiment into Synopsys Verdi for viewing. The operating status of the input / output ports of this embodiment is as follows: Figure 7 As shown in the simulation waveform, this embodiment starts outputting the event-driven converted image of the first static image 0.504 milliseconds after the input of the first static image; the runtime for outputting 4 event-driven images in this embodiment is 2.016 milliseconds. Therefore, the processing latency of this embodiment is 0.504 milliseconds, and the throughput is 1984 frames / second. This conversion rate is applicable in most edge application scenarios and does not constitute a performance bottleneck.
[0047] Hardware resource overhead test: The RTL design of this embodiment was imported into Xilinx Vivado 2024.2, the FPGA model was selected as XC7K325TFBG676-1, and the synthesis and implementation programs were run. The resource usage is shown in Table 1.
[0048] Table 1:
[0049] Resource types Resource usage Lookup Table (LUT) 801 （0.39%） Trigger (FF) 449 （0.11%） Digital Signal Processing Unit (DSP) 1 （0.12%） Storage usage (BRAM) 117.5 （30%）
[0050] As shown in Table 1, compared with the traditional conversion scheme from static image to event-driven image, the conversion circuit proposed in this invention occupies significantly fewer resources, which is beneficial for deployment at resource-constrained edge devices.
[0051] As can be seen from the analysis of the above embodiments, the present invention extracts the light intensity of the input static image through a light intensity extraction circuit; extracts the global contrast features of the image through a dynamic threshold calculation circuit; generates a micro-motion feature pattern map by simulating eight micro-motion modes using a specially designed 3×3 convolution operator; caches the micro-motion feature pattern map through a feature caching circuit; and finally converts the micro-motion feature pattern map into an event-driven image through an event-based circuit. Compared with traditional conversion schemes, the static image to event-driven image conversion circuit proposed in this invention generates image data of better quality, has a higher conversion rate, can achieve real-time processing, and has lower resource overhead, making it particularly suitable for edge computer vision applications requiring high accuracy, high real-time performance, and low overhead.
Claims
1. A circuit for converting a static image to an event-driven image, characterized in that: It includes a light intensity extraction circuit, a dynamic threshold calculation circuit, a micro-motion feature extraction circuit, a feature buffering circuit, and an event issuance circuit; The light intensity extraction circuit receives pixel-by-pixel RGB image data and effective synchronization control signals from external input at its input end, and broadcasts the light intensity calculation results to the dynamic threshold calculation circuit and the micro-motion feature extraction circuit at its output end. The light intensity extraction circuit is used to calculate the light intensity value of each pixel in the input static image. The maximum value V=max(R,G,B) among the three channels R, G, and B is calculated in real time by the comparator, and the maximum value V is used as the single-channel light intensity value output. The dynamic threshold calculation circuit receives light intensity data stream from the light intensity extraction circuit and row and column synchronization signals of the image at its input terminal, and outputs the dynamic threshold calculation result to the event release circuit at its output terminal; it is used to adaptively extract the global contrast features of the current frame and generate the dynamic threshold calculation result. The micro-motion feature extraction circuit includes a pixel rearrangement circuit and a differential convolution circuit; The pixel rearrangement circuit is used to rearrange the input light intensity value data stream and output the rearranged data stream to the differential convolution circuit for micro-motion calculation simulation. The differential convolution circuit is used to perform 8-channel convolution calculations on the data stream rearranged by the pixel rearrangement circuit. The convolution kernels used are eight specially designed 3×3 convolution kernels: ； ； By performing convolution calculations on the frame image using these 8 3×3 convolution kernels, the simulation of 8 micro-motion patterns is achieved, generating a micro-motion feature pattern map with 8 channels. The size of the micro-motion feature pattern map is H×W. The output of the differential convolution circuit is used as the output of the micro-motion feature extraction circuit, and the data stream of the micro-motion feature pattern map is output to the feature buffer circuit. The feature caching circuit is used to cache the micro-motion feature pattern map to wait for the dynamic threshold calculation to be completed, and the cache depth is greater than or equal to (H-1)×W; the input of the feature caching circuit receives the data stream of the micro-motion feature pattern map calculated by the micro-motion feature extraction circuit, and the output of the feature caching circuit outputs the data stream to the event issuing circuit after the dynamic threshold calculation is completed. The event emission circuit performs pulse emission judgment on the data stream of the input 8-channel micro-motion feature pattern map based on the dynamic threshold input by the dynamic threshold calculation circuit. The micro-motion feature pattern map of the 8 channels calculated by the micro-motion feature extraction circuit is converted into a pulse event map of 8 time steps and 2 channels, with the 2 channels corresponding to the positive polarity channel and the negative polarity channel, respectively.
2. The still image to event-based image conversion circuit as described in claim 1, characterized in that: The dynamic threshold calculation circuit includes a comparator, a multiplier, a subtractor, a maximum value tracking register, and a minimum value tracking register. During the input of light intensity values for each frame of pixels, the circuit structure composed of the comparator, the maximum value tracking register, and the minimum value tracking register extracts the maximum and minimum light intensity values of the pixels in that frame in real time and stores them in the corresponding maximum value tracking register and minimum value tracking register, respectively. At the end of the frame, the subtractor outputs the difference between the values stored in the maximum value tracking register and the minimum value tracking register. This difference is input into the multiplier and multiplied by a fixed scaling factor. The output of the multiplier is the dynamic threshold specific to that frame.
3. The still image to event-driven image conversion circuit as described in claim 1, characterized in that: The specific conversion and processing rule in the event issuance circuit is as follows: For a pixel (x,y) in the i-th channel of the micro-motion feature pattern image that is greater than the dynamic threshold, it is converted into a 1-pulse event generated by the corresponding coordinate (x,y) in the positive polarity channel of the i-th time step; Similarly, pixels with a value less than the opposite of the dynamic threshold are converted into a 1-pulse event generated at the corresponding coordinate on the negative polarity channel; Pixels whose absolute value is less than or equal to the dynamic threshold are converted to 0 pulse events generated at the corresponding coordinates.
4. The still image to event-based image conversion circuit as described in any one of claims 1-3, characterized in that: For edge computer vision scenarios.