A method and device for generating output based on an event stream of an FPGA frame image sensor

Abstract

This invention relates to the fields of image sensor data processing and event vision technology, specifically a method and apparatus for generating and outputting event streams based on an FPGA-based frame-type image sensor. The method first receives the pixel-by-pixel raw data stream output from the frame-type image sensor, recovers the pixel coordinates and sampling time, and then performs brightness characterization mapping to obtain brightness characterization values. Next, it reads the brightness characterization value saved at the time of the last event trigger for that pixel as a reference state value and calculates the difference. This difference is compared with a preset threshold to generate events of corresponding polarity. When the difference crosses multiple thresholds, multiple events are generated and the reference state value is updated synchronously. Then, a timestamp is assigned to each generated event. When no event is triggered, the reference state value remains unchanged; when triggered, only the reference state value of the corresponding pixel is updated. Finally, the event stream is output. This invention converts frame-type image sensors into event stream outputs in real time, offering advantages such as low latency, high throughput, and strong compatibility.

Description

Technical Field

[0001] This invention relates to the field of image sensor data processing and event vision technology, and in particular to a method and apparatus for generating and outputting event streams based on FPGA frame-type image sensors. Background Technology

[0002] Existing machine vision and imaging systems widely employ frame-based image sensors to acquire scene information. These sensors typically output complete two-dimensional image data at a fixed frame rate and in a fixed scanning order, which is then stored, transmitted, and processed by subsequent systems. For time-sensitive scenarios such as object detection, motion perception, edge tracking, and fast pose estimation, the full-frame output method leads to problems such as large data redundancy, high processing latency, and significant bandwidth and power consumption pressures. These issues are particularly pronounced in scenarios involving high-speed motion, wide dynamic range changes, and real-time edge response.

[0003] Event cameras output data by independently controlling pixels and asynchronously triggering events based on brightness changes. They offer advantages such as high temporal resolution, high dynamic range, and sparse output, making them valuable for applications in high-speed vision, robotics, unmanned platforms, and neuromorphic computing. Traditional event vision systems often employ dedicated event sensor chips, where pixel units directly handle logarithmic response, threshold comparison, and event triggering at the front end. While these approaches are mechanistically similar to biological vision, they typically require specialized pixel structures, dedicated readout links, and compatible interfaces, resulting in complex device design, long development cycles, and limited compatibility with existing frame-based image sensor systems.

[0004] To fully utilize existing frame-based image sensors and mature readout chains, previous research has proposed methods to convert ordinary video or raw frame data into event streams. These methods include those based on the difference between adjacent frames, those based on logarithmic domain brightness change thresholds, and those that approximate the camera output behavior of real events using simulation models. However, most existing conversion methods are primarily geared towards offline software processing, dataset generation, or algorithm verification. They typically run on CPU or GPU platforms, performing batch processing or post-buffering of input frames, which is insufficient to meet the requirements of embedded devices and edge hardware for deterministic low latency and real-time continuous output.

[0005] On the other hand, some simple frame difference methods typically only compare the corresponding pixel values ​​of the current frame with those of the previous frame, or compare the absolute differences between adjacent sample values, lacking a hardware representation of the "state memory-threshold trigger-reference reset" mechanism of the event camera. Such methods often struggle to stably characterize the features of real event flows when brightness changes accumulate across multiple frames, there are high-speed brightness transitions, threshold mismatches, thermal noise, dark current disturbances, and background activity noise. They are also not conducive to efficient integration with subsequent neuromorphic interfaces.

[0006] Furthermore, several specific technical challenges exist in the engineering implementation of converting frame-based data into event streams in real time: First, frame-based sensors typically read out data by scanning rows and columns, and the raw data essentially only carries the scanning time sequence rather than the event occurrence time. How to obtain a more refined sub-frame-level time expression in the back-end digital logic is an important issue in ensuring the quality of event timing. Second, when the brightness changes significantly, a single sampling may span multiple threshold steps. If only a single event is output, dynamic information will be lost. Third, non-ideal factors such as inconsistent responses between different pixels, threshold deviations, hot pixels, and link bandwidth limitations can lead to an increase in pseudo-events or distortion in event distribution, requiring programmable suppression and calibration capabilities.

[0007] Therefore, there is a need for an event stream generation and output scheme that is compatible with existing frame-type image sensors and readout circuits, can be implemented in real time on reconfigurable digital hardware such as FPGAs, and can also take into account state memory, multi-event output, subframe time estimation and noise suppression capabilities, so as to reduce system modification costs and improve real-time performance and engineering deployability. Summary of the Invention

[0008] The purpose of this invention is to provide a method and apparatus for generating and outputting event streams based on FPGA frame-type image sensors, which mainly solves the problems of large software conversion delay, insufficient real-time performance, and the inability of simple frame difference methods to reflect the event camera state memory and relative brightness triggering characteristics in the prior art.

[0009] To achieve the above objectives, the technical solution adopted by this invention is to provide a frame-based image sensor event stream generation and output method based on FPGA, characterized in that the method is executed in real time by fully pipelined digital logic within the FPGA programmable logic, including the following steps:

[0010] Step S100: Receive the pixel-by-pixel raw data stream output by the frame image sensor, and recover the pixel coordinates and sampling time based on the frame synchronization signal, line synchronization signal, pixel clock or equivalent timing information.

[0011] Step S200: Perform brightness characterization mapping on the pixel-by-pixel raw data stream to obtain the brightness characterization value of the current pixel;

[0012] Step S300: Read the brightness characterization value saved when the pixel was last triggered as a reference state value, and calculate the difference between the current brightness characterization value and the reference state value;

[0013] Step S400: Compare the difference with a preset positive threshold and a preset negative threshold; both the preset positive threshold and the preset negative threshold are positive numbers; generate a positive polarity event when the difference is greater than or equal to the preset positive threshold, and generate a negative polarity event when the difference is less than or equal to the opposite of the preset negative threshold.

[0014] Step S500: When the absolute value of the difference spans multiple threshold steps, multiple positive or negative events are generated in a single sampling process, and for each event generated, the reference state value is incremented or decremented by one threshold step.

[0015] Step S600: Assign an event timestamp to the generated positive or negative event;

[0016] Step S700: When the difference does not meet the event triggering condition, the reference state value is not updated; when the difference meets the event triggering condition, only the reference state value of the corresponding pixel is updated.

[0017] Step S800: Encode the generated positive or negative event and output the event stream.

[0018] Further, in step S200, the brightness representation mapping adopts any one of lookup table mapping, piecewise linear mapping, logarithmic approximation mapping or a combination thereof, to convert the pixel value into a logarithmic domain brightness value, a pseudo-logarithmic domain brightness value or a relative brightness representation value.

[0019] Further, in step S600, the event timestamp is determined based on one or more of the following: intra-frame scan position, pixel clock, row-column timing, and pipeline delay compensation.

[0020] Further, in step S600, the event timestamp is determined by linear interpolation of the threshold crossing time based on the brightness characterization value at the previous moment and the brightness characterization value at the current moment.

[0021] Further, in step S500, when the difference is greater than or equal to the preset positive threshold, the integer part of the difference divided by the preset positive threshold is taken to generate a corresponding number of positive polarity events, and the reference state value is updated by increasing the preset positive threshold sequentially; when the difference is less than or equal to the opposite of the preset negative threshold, the integer part of the absolute value of the difference divided by the preset negative threshold is taken to generate a corresponding number of negative polarity events, and the reference state value is updated by decreasing the preset negative threshold sequentially.

[0022] Furthermore, it also includes a noise suppression step, which includes at least one of the following: setting a refractory period for each pixel and suppressing repeated event output during the refractory period; masking specified pixel coordinates using a hot pixel table or an abnormal pixel table; determining the validity of isolated events based on spatiotemporal neighborhood consistency; suppressing sudden background events based on event density gating, event rate limiting, or FIFO level feedback; and compensating for or suppressing threshold drift, leakage noise, and insufficient bandwidth under low light conditions.

[0023] Furthermore, it also includes a threshold calibration step, which uses a pixel-related threshold calibration table, a threshold offset table, or an illumination-related parameter table to independently configure the positive and negative polarity thresholds of different pixels in order to compensate for threshold mismatch, inconsistent pixel response, or threshold offset caused by illumination changes.

[0024] Furthermore, while outputting the event stream, the original frame data, downsampled frame data, grayscale images, or statistical information are retained and output to form a frame / event dual-stream output mode.

[0025] The present invention also discloses an FPGA-based frame-type image sensor event stream generation device to implement the above method. The device is characterized by comprising a data receiving and coordinate recovery module, a brightness mapping module, a reference state storage module, a difference comparison and multi-event generation module, a timestamp generation module, a noise suppression and threshold calibration module, and an event encoding and FIFO buffer module connected in a pipeline manner.

[0026] The data receiving and coordinate recovery module receives the pixel-by-pixel raw data stream output by the frame image sensor and recovers the pixel coordinates and sampling time based on the time information.

[0027] The brightness mapping module is connected to the data receiving and coordinate recovery module, and performs brightness characterization mapping on the pixel-by-pixel raw data stream to obtain the brightness characterization value of the current pixel.

[0028] The reference state storage module is used to store the reference state value corresponding to each pixel; the reference state storage module is implemented by FPGA on-chip memory, on-chip cache, block cache or a combination thereof; the reference state storage module completes the reading and writing back of the reference state value in one or more pipeline cycles;

[0029] The difference comparison and multi-event generation module is connected to the brightness mapping module and the reference state storage module. It reads the reference state value corresponding to the current pixel, calculates the difference between the current brightness characterization value and the reference state value, and compares the difference with the preset positive threshold and the preset negative threshold. When the difference is greater than or equal to the preset positive threshold, a positive polarity event is generated; when the difference is less than or equal to the opposite of the preset negative threshold, a negative polarity event is generated. When the absolute value of the difference spans multiple threshold steps, multiple positive or negative polarity events are generated in a single sampling process.

[0030] The timestamp generation module is connected to the difference comparison and multi-event generation module, and assigns event timestamps to the positive or negative polarity events.

[0031] The noise suppression and threshold calibration module is connected to the timestamp generation module, performs noise suppression on the generated events, and achieves threshold calibration by providing the pixel-related preset positive threshold and preset negative threshold or threshold offset parameter to the difference comparison and multi-event generation module.

[0032] The event encoding and FIFO caching module is connected to the noise suppression and threshold calibration module. It encodes the processed positive or negative events, caches them, and outputs the event stream.

[0033] The device also includes an on-chip configuration and status register for operating parameters and status information; the device is also configured with an event stream output interface for outputting encoded event data, which does not participate in event generation and judgment.

[0034] Furthermore, the event stream output interface adopts any one of the following: AXI-Stream interface, AER format interface, or LVDS event output interface.

[0035] In view of the above technical features, the present invention, a method and apparatus for generating and outputting event streams based on FPGA frame-type image sensors, has the following significant advantages compared with the prior art:

[0036] 1. This invention has strong compatibility: it does not require modification of the front-end pixel array and readout circuit, is compatible with existing frame-type image sensors, and has low system modification costs.

[0037] 2. This invention has high real-time performance: It implements full pipeline hardware processing based on FPGA, which has deterministic low latency and high throughput real-time performance, making it suitable for edge deployment.

[0038] 3. The mechanism of this invention is closer to that of a real event camera: It adopts state-triggered reference management, which updates the reference state value only when an event occurs. It can express the brightness changes accumulated across multiple frames, which is closer to the "memory-trigger-reset" mechanism of real event pixels.

[0039] 4. Enhanced dynamic expression capability of the present invention: By using logarithmic domain mapping, sub-frame timestamp interpolation and multi-event generation mechanism, the ability to express high-speed motion and wide dynamic range changes is improved.

[0040] 5. The invention has strong robustness: Through pixel-related threshold calibration and noise suppression mechanisms, it improves robustness under low light, complex lighting and real hardware conditions.

[0041] 6. The present invention has good interface compatibility: it is compatible with common digital event stream interfaces such as AXI-Stream, AER and LVDS, which facilitates the interface with on-chip event processing modules, neuromorphic processors or visual accelerators.

[0042] 7. This invention offers optional dual-stream output: optional frame-event dual-stream output, which facilitates system debugging, dataset acquisition, and collaborative work between traditional vision algorithms and event vision algorithms. Attached Figure Description

[0043] Figure 1 This is a flowchart of a preferred embodiment of the method for generating and outputting event streams based on FPGA frame-type image sensor according to the present invention;

[0044] Figure 2 This is a detailed flowchart of step S100 in a preferred embodiment of the FPGA-based frame image sensor event stream generation and output method of the present invention.

[0045] Figure 3 This is a schematic diagram showing the detailed process of step S200 in a preferred embodiment of the FPGA frame-based image sensor event stream generation and output method of the present invention.

[0046] Figure 4 This is a schematic diagram of the state storage and update process involved in steps S300 and S700 in a preferred embodiment of the FPGA frame-based image sensor event stream generation and output method of the present invention.

[0047] Figure 5 This is a flowchart illustrating the difference comparison and multi-event generation involved in steps S400 and S500 of a preferred embodiment of the FPGA frame-based image sensor event stream generation and output method of the present invention.

[0048] Figure 6 This is a schematic diagram of the sub-frame-level timestamp determination process involved in step S600 in a preferred embodiment of the FPGA-based frame-type image sensor event stream generation and output method of the present invention.

[0049] Figure 7 This is a schematic diagram of the system structure of a preferred embodiment of the FPGA-based frame-type image sensor event stream generation and output device of the present invention;

[0050] Figure 8 This is a schematic diagram of the noise suppression and threshold calibration module in a preferred embodiment of the FPGA-based frame image sensor event stream generation and output device of the present invention;

[0051] Figure 9 This is a schematic diagram of the storage format of the event encoding and FIFO buffer module in a preferred embodiment of the FPGA-based frame-type image sensor event stream generation and output device of the present invention;

[0052] Figure 10This is a timing diagram of the interface between the event encoding and FIFO buffer modules in a preferred embodiment of the FPGA-based frame-type image sensor event stream generation and output device of the present invention.

[0053] Figure 11 This is a data path diagram of a preferred embodiment of the FPGA-based frame image sensor event stream generation and output device of the present invention.

[0054] In the diagram: 10-Data reception and coordinate recovery module, 20-Brightness mapping module, 30-Reference state storage module, 40-Difference comparison and multi-event generation module, 50-Timestamp generation module, 60-Noise suppression and threshold calibration module, 70-Event encoding and FIFO buffer module, 80-Event stream output interface, 90-On-chip configuration and status register, 100-Frame-type image sensor;

[0055] 61-Refractory Period Submodule, 62-Hot Pixel Masking Submodule, 63-Spatiotemporal Consistency Filtering Submodule, 64-Event Density Gating Submodule, 65-Threshold Calibration Submodule, 66-Pixel Correlation Threshold Calibration Table. Detailed Implementation

[0056] The present invention will be further described below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0057] Example 1: A method for generating and outputting event streams for a frame-based image sensor based on FPGA.

[0058] Please see Figures 1 to 6 This invention discloses a method for generating and outputting event streams from a frame-based image sensor using an FPGA. As shown in the figure, a preferred embodiment is executed by fully pipelined digital logic within the FPGA and includes the following steps:

[0059] Step S100: Data reception and coordinate recovery.

[0060] The system receives the pixel-by-pixel raw data stream output by the frame image sensor 100, as well as the frame synchronization signal, line synchronization signal, pixel clock or equivalent timing information corresponding to the pixel-by-pixel raw data stream, and recovers the pixel coordinates and sampling time from the pixel-by-pixel raw data stream based on this information.

[0061] For details, please refer to [link / reference]. Figure 2Taking a 640×512 resolution, 14-bit wide frame-type image sensor 100 as an example, the FPGA receives data from the pixel data bus and, based on synchronization signals such as frame_valid, line_valid, pixel_valid, and pixel_clk, recovers the row number y and column number x of the current pixel, i.e., the pixel coordinates (x, y), through row and column counters. Simultaneously, the system time counter records the sampling time t_s. For multi-channel parallel output mode, the global pixel coordinates can also be recovered through channel identifiers, channel alignment buffers, and splicing logic. Optionally, if the system has a fixed pipeline delay Δt_pipe, the sampling time can be compensated to obtain the corrected sampling time t_s' = t_s + Δt_pipe.

[0062] Step S200, brightness characterization mapping.

[0063] A luminance characterization mapping is performed on the pixel-by-pixel raw data stream to obtain the luminance characterization value of the current pixel.

[0064] For details, please refer to [link / reference]. Figure 3 In this embodiment, a logarithmic domain mapping is used: L_cur = log(I + ε). Here, I is the original pixel value, and ε is a bias term to prevent undefined cases from occurring during low-grayscale input. The logarithmic domain mapping transforms the event triggering criterion from absolute brightness difference to relative brightness change, which better aligns with the mechanism by which event vision triggers events based on logarithmic brightness changes.

[0065] The luminance representation mapping can employ any of the following methods: lookup table (LUT) mapping, piecewise linear mapping, logarithmic approximation mapping, or a combination thereof, to convert linear pixel values ​​into logarithmic domain luminance values, pseudo-logarithmic domain luminance values, or relative luminance representation values. In this embodiment, a lookup table is used to map 14-bit original pixel values ​​to 16-bit luminance representation values.

[0066] Step S300: Read the reference state value and calculate the difference.

[0067] For each pixel, the brightness representation value saved at the time of the last triggered event is read as the reference state value L_ref. The reference state value is stored in the FPGA's on-chip memory (such as BRAM or URAM), with one storage unit corresponding to each pixel. The difference between the current brightness representation value and the reference state value is calculated as: ΔL = L_cur - L_ref. Figure 4 The process of reading the reference status value is demonstrated.

[0068] Step S400, threshold comparison and event generation.

[0069] Please see Figure 5, the difference is compared with a preset positive threshold C_on and a preset negative threshold C_off, where both C_on and C_off are positive numbers. When the difference ΔL is greater than or equal to C_on (i.e., ΔL≥C_on), a positive-polarity event is generated. When the difference ΔL is less than or equal to -C_off (i.e., ΔL≤-C_off), a negative-polarity event is generated. When the difference ΔL is between -C_off and C_on (i.e., -C_off<ΔL<C_on), no event is generated. The specific values of the preset positive threshold C_on and the preset negative threshold C_off can be the same or different. In this embodiment, their absolute values are the same.

[0070] In this embodiment, during the threshold comparison and event generation process, noise suppression and threshold calibration steps are also optionally inserted. Among them, the noise suppression steps include at least one of the following:

[0071] · Refractory period: Set a refractory period counter for each pixel. Among the short-interval events repeatedly generated by the same pixel within the refractory period time window, only the first event is retained, and the rest are suppressed.

[0072] · Hot pixel masking: Store the coordinates of the calibrated hot pixels in the hot pixel table or the abnormal pixel table. If the current event comes from the hot pixel coordinates, directly mask it or reduce its output priority.

[0073] · Spatiotemporal consistency screening: Maintain a neighborhood event window on the chip, and judge whether the current event has sufficient supporting events in the spatial neighborhood and the temporal neighborhood; suppress isolated events.

[0074] · Event density gating: Suppress burst background events based on event density gating, event rate limiting, or FIFO water level feedback. When the FIFO water level is too high or the local event density increases abnormally, suppress invalid emergency events through current limiting, hierarchical priority, or arbitration mechanisms.

[0075] · Compensate for or suppress threshold drift, leakage noise, and insufficient bandwidth under low-light conditions.

[0076] For the threshold calibration step, the positive-polarity threshold and the negative-polarity threshold of different pixels are independently configured through a pixel-related threshold calibration table, a threshold offset table, or a light-related parameter table to compensate for threshold mismatch, pixel response inconsistency, or threshold offset caused by light changes.

[0077] Step S500, multi-event generation.

[0078] When the absolute value of the difference spans multiple threshold steps, multiple positive-polarity events or negative-polarity events are generated in a single sampling process. At the same time, for each event generated, the reference state value corresponding to the pixel is incremented or decremented by a threshold step.

[0079] Please see Figure 5 Specifically, when ΔL ≥ C_on, the integer part N = floor(ΔL / C_on) of ΔL divided by C_on is taken to generate N positive polarity events. For each event generated, the reference state value is incremented by C_on. When ΔL ≤ -C_off, the integer part M = floor(|ΔL| / C_off) of |ΔL| divided by C_off is taken to generate M negative polarity events. For each event generated, the reference state value is decremented by C_off.

[0080] This mechanism allows high-speed brightness transitions to be expressed in detail as multiple consecutive events, rather than a single event.

[0081] Step S600: Assign event timestamps.

[0082] Assign an event timestamp t_evt to the generated positive or negative polarity event. See also Figure 6 This embodiment provides two timestamp determination methods. Method one is the basic scan time method, which determines the intra-frame scan time based on the row and column order of the current pixel in the current frame and the pixel clock. Specifically, it calculates t_scan = T_frame + y·T_row + x·T_pix based on the pixel coordinates (x,y), row period T_row, pixel period T_pix, and frame start time T_frame. t_scan is used as the event timestamp. Method two is the linear interpolation method, which performs linear interpolation on the threshold crossing time based on the previous brightness representation value L_prev and the current brightness representation value L_cur. Assuming that the previous brightness representation value L_prev is obtained at the previous sampling time t_prev, and the current brightness representation value L_cur is obtained at the current sampling time t_cur, and the threshold C is crossed between the two, then linear interpolation is used to estimate the threshold crossing time: t_evt = t_prev + [(L_ref + C - L_prev) / (L_cur - L_prev)]·(t_cur - t_prev). If multiple thresholds are crossed, the event time is calculated for each threshold crossing point, forming a set of event timestamps arranged in chronological order. In other embodiments, the event timestamps may also be determined based on one or more of the following methods: intra-frame scan position, pixel clock, row and column timing, and pipeline delay compensation.

[0083] Step S700, Status update control.

[0084] When the difference does not meet the event trigger condition (i.e., -C_off < ΔL < C_on), the reference state value is kept unchanged. When the difference meets the event trigger condition (i.e., ΔL ≥ C_on or ΔL ≤ -C_off), only the reference state value of the corresponding pixel is updated. The updated reference state value is the original reference state value plus or minus several times the threshold step. Specifically, the step is updated once for each generated event. Figure 4 shows the writing process of the reference state value. This "state-triggered reference management" mechanism enables slow changes in brightness but still triggers events after accumulating across multiple frames.

[0085] Step S800, event encoding and output.

[0086] Encode the generated positive or negative polarity events and output the event stream. In this embodiment, the event encoding format is 64-bit data, including a timestamp field, a polarity field, a vertical coordinate Y field, a horizontal coordinate X field, and a reserved / extended field. The order of each field in the event encoding can be configured according to the event stream interface protocol.

[0087] This embodiment also includes an optional dual-stream output mode: while outputting the event stream, the original frame data, downsampled frame data, grayscale image, or statistical information is retained and output to form a frame and event dual-stream output mode, which is convenient for system debugging, dataset collection, and the collaborative work of traditional vision algorithms and event vision algorithms.

[0088] Embodiment 2: An event stream generation device for a frame-type image sensor based on FPGA.

[0089] Please refer to Figures 7 to 11 , the present invention discloses an event stream generation device for a frame-type image sensor based on FPGA, which is used to implement the method described in Embodiment 1. As shown in the figure, a preferred embodiment thereof includes a data reception and coordinate recovery module 10, a brightness mapping module 20, a reference state storage module 30, a difference comparison and multi-event generation module 40, a timestamp generation module 50, a noise suppression and threshold calibration module 60, and an event encoding and FIFO cache module 70 connected in a pipeline manner. Refer to Figure 11 , these modules work in parallel within the same pixel clock cycle, and are segmented by registers between levels to achieve continuous pipelined processing pixel by pixel. After the input pixel stream passes through a fixed pipeline delay, the corresponding event stream is output. In addition, this embodiment also includes an event stream output interface 80 and an on-chip configuration and status register 90.

[0090] The data receiving and coordinate recovery module 10 receives the pixel-by-pixel raw data stream output from the frame-type image sensor and recovers the pixel coordinates and sampling time based on time information (including frame synchronization signal, line synchronization signal, pixel clock, or equivalent timing information). Specifically, this module includes a row counter and a column counter, which update the row and column coordinates respectively when frame_valid, line_valid, and pixel_valid are valid. Simultaneously, a system time counter records the sampling time. For multi-channel parallel input, this module also includes channel alignment buffers and splicing logic. Optionally, if the system has a fixed pipeline delay, this module can compensate for the sampling time to obtain a corrected sampling time.

[0091] The luminance mapping module 20 is connected to the data receiving and coordinate recovery module 10. It performs luminance characterization mapping on the pixel-by-pixel raw data stream to obtain the luminance characterization value of the current pixel. In this embodiment, the luminance mapping module 20 is implemented using a lookup table (LUT) to map the 14-bit raw pixel value to a 16-bit logarithmic domain luminance characterization value. Specifically, logarithmic domain mapping is used: L_cur = log(I + ε), where I is the raw pixel value and ε is a bias term to prevent undefined cases when there is low grayscale input. Logarithmic domain mapping changes the event triggering basis from absolute luminance difference to relative luminance change, which is more in line with the mechanism of event vision triggering events by logarithmic luminance change. In mixed low-light and strong-light scenes, this method is beneficial for maintaining uniform relative sensitivity. In an alternative implementation, the luminance mapping module 20 can also be implemented using a piecewise linear mapping circuit or a logarithmic approximation circuit to convert linear pixel values ​​into pseudo-logarithmic domain luminance values ​​or relative luminance characterization values.

[0092] The reference state storage module 30 stores the reference state value corresponding to each pixel. The reference state value is the brightness characterization value saved when the pixel last triggered an event. In this embodiment, the reference state storage module 30 is implemented by an on-chip dual-port BRAM or dual-port URAM of the FPGA. The first port of the dual-port BRAM or dual-port URAM is used to read the reference state value of the current pixel, and the second port is used to write back the updated reference state value after the event is triggered. Reading and writing back are completed within the same pixel clock cycle, avoiding pipeline interruptions and achieving uninterrupted processing at the pixel pipeline level. In other embodiments, the dual-port BRAM or dual-port URAM can be replaced by other on-chip caches, block caches, local storage, or combinations thereof. Correspondingly, the reading and writing back of the reference state value can also be completed within multiple pipeline cycles. Each pixel corresponds to one storage unit, and the storage bit width is consistent with the bit width of the brightness characterization value, which is 16 bits in this embodiment.

[0093] When the resolution or reference state bit width exceeds the on-chip storage capacity, external storage resources directly connected to the FPGA logic can be used as an extended cache. In this case, the external storage resources are only used for backup storage of the reference state values ​​and do not participate in event triggering judgment.

[0094] The difference comparison and multi-event generation module 40 is connected to the brightness mapping module 20 and the reference state storage module 30. This module reads the reference state value corresponding to the current pixel, calculates the difference between the current brightness representation value and the reference state value, and compares the difference with a preset positive threshold and a preset negative threshold. Both the preset positive threshold and the preset negative threshold are positive values. Specifically, when the difference is greater than or equal to the preset positive threshold, a positive event is generated. When the difference is less than or equal to the opposite of the preset negative threshold, a negative event is generated. When the difference is between the opposites of the preset positive threshold and the preset negative threshold, no event is generated. The specific values ​​of the preset positive threshold and the preset negative threshold can be the same or different. In this embodiment, their absolute values ​​are the same.

[0095] When the absolute value of the difference spans multiple threshold steps, multiple positive or negative polarity events are generated in a single sampling process. Simultaneously, for each event generated, the reference state value is synchronously incremented or decremented by one threshold step. Specifically, when the difference is greater than or equal to the positive polarity threshold, the integer part of the difference divided by the positive polarity threshold is used to generate the corresponding number of positive polarity events. When the difference is less than or equal to the negative polarity threshold, the absolute value of the difference is divided by the integer part of the negative polarity threshold, generating the corresponding number of negative polarity events. This multi-event generation mechanism allows high-speed brightness transitions to be precisely represented as multiple consecutive events, rather than a single event.

[0096] The timestamp generation module 50 is connected to the difference comparison and multi-event generation module 40 to assign event timestamps for positive or negative events. In this embodiment, the timestamp generation module 50 supports two operating modes. The first mode is the intra-frame scan time mode. Specifically, the intra-frame scan time is determined based on the row and column order of the current pixel in the current frame and the pixel clock. That is, the intra-frame scan time is calculated based on the pixel coordinates (x, y), row period T_row, pixel period T_pix, and frame start time T_frame: t_scan = T_frame + y·T_row + x·T_pix, and t_scan is used as the event timestamp. The second mode is the linear interpolation mode, which performs linear interpolation on the threshold crossing time based on the brightness characterization value of the previous moment and the brightness characterization value of the current moment. When the brightness characterization value of the previous moment and the brightness characterization value of the current moment cross the threshold, linear interpolation is used to estimate the threshold crossing time. In the linear interpolation mode, if there are multiple threshold crossings, the event time is calculated for each threshold crossing point, forming a set of event timestamps arranged in chronological order. The two modes can be selected via on-chip configuration and status register 90.

[0097] The noise suppression and threshold calibration module 60 is connected to the timestamp generation module 50. It performs noise suppression on the generated events and achieves threshold calibration by providing pixel-related preset positive and negative thresholds or threshold offset parameters to the difference comparison and multi-event generation module 40. (See also...) Figure 8 In this embodiment, the noise suppression and threshold calibration module 60 includes a refractory period submodule 61, a hot pixel masking submodule 62, a spatiotemporal consistency filtering submodule 63, and an event density gating submodule 64. Each submodule corresponds to a noise suppression function. In other embodiments, at least one of the above submodules is included. Specifically, the refractory period submodule 61 sets a refractory period counter for each pixel. Among short-interval events that are repeatedly generated by the same pixel within the refractory period time window, only the first event is retained, and the rest are suppressed. The hot pixel masking submodule 62 stores the calibrated hot pixel coordinates into a hot pixel table or an abnormal pixel table. If the current event comes from hot pixel coordinates, its output priority is directly masked or reduced. The spatiotemporal consistency filtering submodule 63 maintains a neighborhood event window on-chip and determines whether the current event has enough supporting events in the spatial and temporal neighborhoods; isolated events are suppressed. The event density gating submodule 64 suppresses sudden background events based on event density gating, event rate limiting, or FIFO level feedback. When the FIFO water level is too high or the density of local events increases abnormally, ineffective sudden events are suppressed through flow restriction, hierarchical priority, or arbitration mechanisms.

[0098] exist Figure 8In the noise suppression and threshold calibration module 60, there is also a threshold calibration submodule 65, which is used to perform threshold calibration. The threshold calibration submodule 65 is implemented through a pixel-related threshold calibration table 66, a threshold offset table or an illumination-related parameter table, to independently configure the positive and negative thresholds of different pixels in order to compensate for threshold mismatch, inconsistent pixel response or threshold offset caused by illumination changes.

[0099] Please see Figure 9 The event encoding and FIFO buffer module 70 is connected to the noise suppression and threshold calibration module 60 to encode the processed positive or negative events and output the event stream after buffering.

[0100] In this embodiment, the event encoding format is 64-bit data, including a timestamp field, a polarity field, a y-coordinate field, an x-coordinate field, and a reserved field. The encoded event data is written to the event FIFO buffer, awaiting output. The event FIFO buffer is used to match the event generation rate with the output bandwidth. When the FIFO level is too high, the event generation rate can be suppressed through an event density gating mechanism.

[0101] The event stream output interface 80 is used to output the encoded event data and is connected to the event encoding and FIFO buffer module 70. The event stream output interface 80 uses any one of the following: AXI-Stream interface, AER format interface, or LVDS event output interface. This interface is only used to transmit generated event data and does not participate in event generation judgment, reference state update, noise suppression, or event encoding processing. The data packet output by the event stream output interface 80 contains complete information such as event coordinates, polarity, and timestamp. This embodiment also includes a dual-stream output module, which, while outputting the event stream, retains and outputs the original frame data, downsampled frame data, grayscale image, or statistical information to form a frame and event dual-stream output mode, facilitating system debugging, dataset acquisition, and collaborative work between traditional vision algorithms and event vision algorithms. Figure 10 The timing signals on the interface are shown.

[0102] This device also includes an on-chip configuration and status register 90, used to store threshold tables, mapping parameters, operating status, and FIFO level information. Specifically, the on-chip configuration and status register 90 stores, but is not limited to: preset positive and negative thresholds, brightness mapping lookup table parameters, pixel-related threshold calibration table, hot pixel table, operating status flags of each module, current FIFO level, and event count statistics. These registers can be read and written via an external bus interface to enable runtime parameter adjustment and status monitoring.

[0103] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for generating and outputting event streams from a frame-based image sensor using FPGA, characterized in that, The method is executed in real time by fully pipelined digital logic within the FPGA programmable logic, and includes the following steps: Step S100: Receive the pixel-by-pixel raw data stream output by the frame image sensor, and recover the pixel coordinates and sampling time based on the frame synchronization signal, line synchronization signal, pixel clock or equivalent timing information. Step S200: Perform brightness characterization mapping on the pixel-by-pixel raw data stream to obtain the brightness characterization value of the current pixel; Step S300: Read the brightness characterization value saved when the pixel was last triggered as a reference state value, and calculate the difference between the current brightness characterization value and the reference state value; Step S400: Compare the difference with a preset positive threshold and a preset negative threshold; Both the preset positive threshold and the preset negative threshold are positive numbers; A positive polarity event is generated when the difference is greater than or equal to the preset positive threshold, and a negative polarity event is generated when the difference is less than or equal to the opposite of the preset negative threshold. Step S500: When the absolute value of the difference spans multiple threshold steps, multiple positive or negative events are generated in a single sampling process, and for each event generated, the reference state value is incremented or decremented by one threshold step. Step S600: Assign an event timestamp to the generated positive or negative event; Step S700: When the difference does not meet the event triggering condition, the reference state value is not updated; when the difference meets the event triggering condition, only the reference state value of the corresponding pixel is updated. Step S800: Encode the generated positive or negative event and output the event stream.

2. The method according to claim 1, characterized in that, In step S200, the brightness characterization mapping employs any one of lookup table mapping, piecewise linear mapping, logarithmic approximation mapping, or a combination thereof to convert pixel values ​​into logarithmic domain brightness values, pseudo-logarithmic domain brightness values, or relative brightness characterization values.

3. The method according to claim 1, characterized in that, In step S600, the event timestamp is determined based on one or more of the following: intra-frame scan position, pixel clock, row-column timing, and pipeline delay compensation.

4. The method according to claim 1, characterized in that, In step S600, the event timestamp is determined by linear interpolation of the threshold crossing time based on the brightness characterization value at the previous moment and the brightness characterization value at the current moment.

5. The method according to claim 1, characterized in that, In step S500, when the difference is greater than or equal to the preset positive threshold, the integer part of the difference divided by the preset positive threshold is taken to generate a corresponding number of positive polarity events, and the reference state value is updated sequentially according to the preset positive threshold. When the difference is less than or equal to the opposite of the preset negative threshold, the absolute value of the difference is divided by the integer part of the preset negative threshold to generate a corresponding number of negative polarity events, and the reference state value is updated by decreasing the preset negative threshold one by one.

6. The method according to claim 1, characterized in that, It also includes a noise suppression step, which includes at least one of the following: setting a refractory period for each pixel and suppressing repeated event output during the refractory period; Masking of specified pixel coordinates using hot pixel tables or abnormal pixel tables; determining the validity of isolated events based on spatiotemporal neighborhood consistency; suppressing sudden background events based on event density gating, event rate limiting, or FIFO level feedback; Compensate for or suppress threshold drift, leakage noise, and insufficient bandwidth under low-light conditions.

7. The method according to claim 1, characterized in that, It also includes a threshold calibration step, which uses a pixel-related threshold calibration table, a threshold offset table, or an illumination-related parameter table to independently configure the positive and negative polarity thresholds of different pixels in order to compensate for threshold mismatch, inconsistent pixel response, or threshold offset caused by illumination changes.

8. The method according to claim 1, characterized in that, While outputting the event stream, it retains and outputs the original frame data, downsampled frame data, grayscale images, or statistical information to form a frame / event dual-stream output mode.

9. An FPGA-based frame-based image sensor event stream generation device, implementing the method as described in any one of claims 1 to 8, characterized in that, It includes a data receiving and coordinate recovery module, a brightness mapping module, a reference state storage module, a difference comparison and multi-event generation module, a timestamp generation module, a noise suppression and threshold calibration module, and an event encoding and FIFO buffer module connected in a pipeline manner; The data receiving and coordinate recovery module receives the pixel-by-pixel raw data stream output by the frame image sensor and recovers the pixel coordinates and sampling time based on the time information. The brightness mapping module is connected to the data receiving and coordinate recovery module, and performs brightness characterization mapping on the pixel-by-pixel raw data stream to obtain the brightness characterization value of the current pixel. The reference state storage module is used to store the reference state value corresponding to each pixel; The reference state storage module is implemented by an FPGA on-chip memory, on-chip cache, block cache, or a combination thereof; the reference state storage module completes the reading and writing back of the reference state value within one or more pipeline cycles; The difference comparison and multi-event generation module is connected to the brightness mapping module and the reference state storage module. It reads the reference state value corresponding to the current pixel, calculates the difference between the current brightness characterization value and the reference state value, and compares the difference with the preset positive threshold and the preset negative threshold. A positive polarity event is generated when the difference is greater than or equal to the preset positive threshold, and a negative polarity event is generated when the difference is less than or equal to the opposite of the preset negative threshold. When the absolute value of the difference spans multiple threshold steps, multiple positive or negative events are generated in a single sampling process. The timestamp generation module is connected to the difference comparison and multi-event generation module, and assigns event timestamps to the positive or negative polarity events. The noise suppression and threshold calibration module is connected to the timestamp generation module, performs noise suppression on the generated events, and achieves threshold calibration by providing the pixel-related preset positive threshold and preset negative threshold or threshold offset parameter to the difference comparison and multi-event generation module. The event encoding and FIFO caching module is connected to the noise suppression and threshold calibration module. It encodes the processed positive or negative events, caches them, and outputs the event stream. The device also includes an on-chip configuration and status register for operating parameters and status information; the device is also configured with an event stream output interface for outputting encoded event data, which does not participate in event generation and judgment.

10. The apparatus according to claim 9, characterized in that, The event stream output interface adopts any one of the following: AXI-Stream interface, AER format interface, or LVDS event output interface.

Images