A fusion event and rgb vision system and method

Abstract

The application belongs to the technical field of high-speed vision, and discloses a vision system and method fusing events and RGB. The application obtains spatially aligned RGB frame data and event stream data by using an optical acquisition subsystem; time-aligns the RGB frame data and the event stream data by using a heterogeneous logic control subsystem, and generates a heterogeneous data packet with a unified timestamp; receives the heterogeneous data packet by using a fusion operation subsystem, and performs fusion calculation to obtain vision processing information. The application can improve the spatiotemporal alignment accuracy of the imaging scheme of the fusion event camera and the RGB camera.

Description

Technical Field

[0001] This invention belongs to the field of high-speed vision technology, and more specifically, relates to a vision system and method that integrates events and RGB. Background Technology

[0002] Visual information is the primary medium for machines to perceive and understand the physical world. With the rapid development of technologies such as industrial automation, autonomous driving, and high-speed fluid analysis, the requirements for visual sensors to capture information in the temporal dimension have reached unprecedented levels. In these scenarios, dynamic processes change rapidly, and their timescales often far exceed the perception limits of the human eye and even traditional imaging systems. Existing high-speed vision processing solutions are mainly divided into two categories. The first category is high-speed cameras based on traditional frame sensors. Although these cameras can provide high-quality color images, they must use extremely short exposure times and extremely high frame rates when capturing high-speed moving targets, resulting in massive data throughput and putting enormous pressure on data transmission bandwidth and backend storage. In addition, frame cameras repeatedly record a large amount of redundant background data when capturing local motion scenes, resulting in a waste of computing resources. The second category is dynamic vision sensors based on event sensors, which respond to changes in light intensity and output asynchronous event streams, possessing microsecond-level temporal resolution and extremely high data sparsity. However, the images output by event cameras lack texture and color details, making it difficult to meet the requirements of backend processing.

[0003] To balance high temporal resolution and high-quality texture information, existing research attempts to fuse event cameras and RGB cameras. However, existing fusion schemes still face technical bottlenecks: in terms of spatiotemporal alignment, simple binocular side-by-side arrangement suffers from parallax, resulting in inaccurate pixel alignment during shooting; while existing soft synchronization schemes have low accuracy and cannot cope with high-speed changes at the microsecond level, easily leading to data misalignment.

[0004] Furthermore, existing converged systems suffer from limited functionality and rigid architecture. Existing heterogeneous converged systems are typically designed for single tasks, with their hardware logic and algorithm pipelines being fixed, making it difficult to cope with the diverse needs of real-world deployment scenarios. Summary of the Invention

[0005] This invention provides a visual system and method for fusing event and RGB data, thereby addressing the problem of low spatiotemporal alignment accuracy in existing imaging schemes that fuse event cameras and RGB cameras.

[0006] This invention provides a visual system that integrates events and RGB, comprising: an optical acquisition subsystem, a heterogeneous logic control subsystem, and a fusion computing subsystem; The optical acquisition subsystem includes an imaging lens, a beam splitter, and a dual-mode sensing module. The beam splitter is positioned on the outgoing light path of the imaging lens. The dual-mode sensing module includes an RGB sensor and an event sensor, which are respectively positioned on the two outgoing light paths of the beam splitter and located on the same equivalent focal plane. The optical center of the imaging lens coincides with the incident optical axis of the beam splitter, and the two outgoing optical axes of the beam splitter coincide with the optical axes of the RGB sensor and the event sensor, respectively, making the RGB sensor and the event sensor share a common optical center. The optical acquisition subsystem is used to obtain spatially aligned RGB frame data and event stream data. The heterogeneous logic control subsystem is used to time-align RGB frame data and event stream data to generate heterogeneous data packets with a unified timestamp. The fusion computing subsystem is used to receive heterogeneous data packets and perform fusion calculations to obtain visual processing information.

[0007] Preferably, the beam-splitting unit includes: a beam-splitting prism and a relay lens group; The beam splitter is used to separate the incident beam into a transmitted beam and a reflected beam. The relay lens group is used to correct optical path difference and aberration, so that the light beam emitted from the imaging lens and passing through its optical center is split by the beam splitter and forms a strictly consistent imaging field of view on the RGB sensor and the event sensor.

[0008] Preferably, the heterogeneous logic control subsystem includes: a sensor driving module, a synchronization control module, and an interface encapsulation module; The sensor driving module is used to convert the electrical signal output by the dual-mode sensing module into a digital signal; The synchronization control module is used to monitor the frame synchronization signal output by the RGB sensor in real time, and when a frame start pulse is detected, it immediately sends an external trigger pulse to the event sensor, so that the event sensor inserts a synchronization event packet with a specific mark into the event stream data; The interface encapsulation module is used to receive RGB frame data and event stream data after synchronization processing, mix and package them to obtain the heterogeneous data packet, and send the heterogeneous data packet to the fusion computing subsystem.

[0009] Preferably, the sensor driving module includes: a physical signal receiving layer, a protocol parsing and splitting layer, an image frame unpacking unit, and an event stream decoding unit; The physical signal receiving layer is used to receive the two electrical signals output by the dual-mode sensing module through two independent signal channels, and convert them into RGB frame data and event stream data. The protocol parsing and offloading layer is used to logically isolate and offload heterogeneous data based on the data protocol, transmitting RGB frame data to the image frame unpacking unit and transmitting event stream data to the event stream decoding unit. The image frame unpacking unit is used to perform data verification and pixel matrix mapping, extract the frame valid signal and the line valid signal, and convert them into a video stream format commonly used inside the FPGA. The event stream decoding unit is used to perform asynchronous caching and protocol packet parsing, extract the X coordinate, Y coordinate and polarity bit of each event, and convert them into a standardized parallel data bus output.

[0010] Preferably, the synchronization control module is constructed based on a finite state machine; the finite state machine is initialized to an idle state; once the synchronization control module captures the frame start pulse output by the RGB sensor, the finite state machine enters the signal capture state, records the local timestamp of the current moment, and automatically enters the trigger generation state, sending the external trigger pulse to the event sensor through a physical connection; after generating the external trigger pulse, the finite state machine immediately enters the dead-zone locking state, blocking the input of the frame synchronization signal for a set time.

[0011] Preferably, the fusion computing subsystem has multiple different visual algorithm models pre-installed; the fusion computing subsystem is used to load the visual algorithm model corresponding to the target visual task according to system configuration or instruction information.

[0012] Preferably, the fusion operation subsystem directly reads data from the heterogeneous logic control subsystem and writes it into system memory to establish a circular buffer; the fusion operation subsystem reads data from the circular buffer, separates RGB frame data and event stream data, and restores the temporal correspondence between the two based on the synchronization event packets inserted in the event stream data to construct a heterogeneous fusion frame object; the fusion operation subsystem uses the heterogeneous fusion frame object as input and performs fusion calculations based on the loaded visual algorithm model.

[0013] Preferably, the visual system that integrates events and RGB further includes: an interaction subsystem; The interaction subsystem is used to receive configuration instructions corresponding to the target visual task sent by the operation terminal, parse the configuration instructions to obtain instruction information, and send the instruction information to the fusion computing subsystem.

[0014] Preferably, the interactive subsystem is further configured to display the visual processing information locally, and to compile the visual processing information into network data packets and send them to a remote server.

[0015] On the other hand, the present invention provides a visual method for fusing events and RGB, implemented using the aforementioned visual system for fusing events and RGB, the visual method for fusing events and RGB comprising the following steps: Spatially aligned RGB frame data and event stream data are obtained using an optical acquisition subsystem; The heterogeneous logic control subsystem is used to time-align RGB frame data and event stream data to generate heterogeneous data packets with a unified timestamp. The fusion computing subsystem receives heterogeneous data packets and performs fusion calculations to obtain visual processing information.

[0016] One or more technical solutions provided in this invention have at least the following technical effects or advantages: (1) In the optical acquisition subsystem of this invention, the RGB sensor and the event sensor are located on the same equivalent focal plane, which can solve the problems of inconsistent focal planes and spatial alignment errors, and effectively ensure the imaging quality; and the optical center of the imaging lens in this invention coincides with the incident optical axis of the beam splitting unit, and the two outgoing optical axes of the beam splitting unit coincide with the optical axes of the RGB sensor and the event sensor, respectively, so that the RGB sensor and the event sensor share the same optical center ("shared optical center" means that the two sensors share the same optical center of the imaging lens through the beam splitting unit, that is, the light they receive comes from the same entrance pupil center. After optical design, the two The equivalent optical center of the path sensor coincides with the optical center of the imaging lens, thus ensuring a completely consistent field of view without parallax. This "common optical center" design eliminates the parallax blind spot that is unavoidable in close-range shooting by traditional binocular side-by-side cameras from the physical source. Combining the "common optical center" and the "same equivalent focal plane," this invention ensures that both sensors are simultaneously clear in dynamic scenes and achieves pixel-level spatial alignment, thereby effectively improving the spatial alignment accuracy of the imaging scheme fused with the event camera and the RGB camera. Combined with the use of a heterogeneous logic control subsystem for time alignment, this invention achieves high-precision spatiotemporal alignment overall. Furthermore, compared to the traditional scheme that uses operating system timestamps for soft synchronization, the heterogeneous logic control subsystem in this invention, triggered by hardware signals, can control the time alignment error between the RGB frame and the event stream to the microsecond level.

[0017] (2) The fusion computing subsystem of the present invention has a variety of different visual algorithm models pre-installed. According to the system configuration or instruction information, the visual algorithm model corresponding to the target visual task can be loaded. Therefore, the present invention also has reconfigurable computing capabilities, and can dynamically deploy different algorithm models according to the application scenario. It can flexibly switch between multiple tasks such as imaging, detection, and recognition to meet the diverse needs in the actual deployment scenario. Attached Figure Description

[0018] Figure 1This is a schematic diagram of the overall framework of a visual system that integrates events and RGB according to an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the framework of a heterogeneous logic control subsystem in a visual system that integrates events and RGB, as provided in an embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of the framework of a fusion operation subsystem in a visual system that fuses events and RGB provided in an embodiment of the present invention.

[0021] Figure 4 This is a schematic diagram of the optical devices and hardware equipment corresponding to a visual system that fuses events and RGB provided in an embodiment of the present invention.

[0022] Figure 5 This is a flowchart illustrating a visual method for fusing events and RGB provided in an embodiment of the present invention.

[0023] Among them, 100-optical acquisition subsystem, 200-heterogeneous logic control subsystem, 300-fusion computing subsystem, and 400-interaction subsystem; 110 - Imaging lens, 120 - Beam splitter unit, 130 - Dual-mode sensor module; 131 - RGB sensor, 132 - Event sensor; 210 - Synchronization control module, 220 - Sensor driver module, 230 - Interface packaging module; 310 - Direct Memory Access Control Module, 320 - Heterogeneous Data Decoding Module, 330 - Visual Task Deployment Port, 340 - Computation and Processing Unit; 410 - Network transmission module, 420 - Local display module, 430 - Instruction parsing module. Detailed Implementation

[0024] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0025] Example 1: Example 1 provides a visual system that integrates events and RGB, see [link to example]. Figure 1 , Figure 2 and Figure 3 It mainly includes: optical acquisition subsystem 100, heterogeneous logic control subsystem 200 and fusion computing subsystem 300.

[0026] The optical acquisition subsystem 100 includes an imaging lens 110, a beam splitting unit 120, and a dual-mode sensing module 130. The beam splitting unit 120 is disposed on the outgoing light path of the imaging lens 110. The dual-mode sensing module 130 includes an RGB sensor 131 (i.e., an RGB image sensor) and an event sensor 132 (i.e., an event vision sensor). The two sensors are respectively disposed on the two outgoing light paths of the beam splitting unit 120 and are located on the same equivalent focal plane. The optical center of the imaging lens 110 coincides with the incident optical axis of the beam splitting unit 120, and the two outgoing optical axes of the beam splitting unit 120 coincide with the optical axes of the RGB sensor 131 and the event sensor 132, respectively, so that the RGB sensor 131 and the event sensor 132 share a common optical center.

[0027] The optical acquisition subsystem 100 is used to acquire spatially aligned RGB frame data and event stream data. The heterogeneous logic control subsystem 200 is used to time-align the RGB frame data and event stream data to generate heterogeneous data packets with a unified timestamp. The fusion calculation subsystem 300 is used to receive the heterogeneous data packets and perform fusion calculations to obtain visual processing information.

[0028] In Embodiment 1, the RGB sensor 131 and the event sensor 132 in the optical acquisition subsystem 100 are located on the same equivalent focal plane, which can solve the problems of inconsistent focal planes and spatial alignment differences, effectively ensuring image quality. In addition, the optical center of the imaging lens 110 in Embodiment 1 coincides with the incident optical axis of the beam splitting unit 120, and the two outgoing optical axes of the beam splitting unit 120 coincide with the optical axes of the RGB sensor 131 and the event sensor 132, respectively, so that the RGB sensor 131 and the event sensor 132 share a common optical center. This "common optical center" design can eliminate the parallax blind spot that is unavoidable when shooting at close range by traditional binocular side-by-side cameras from the physical source. Embodiment 1, combining "common optical center" and "same equivalent focal plane", can ensure that the two sensors are clear at the same time in dynamic scenes and achieve pixel-level spatial alignment, thereby effectively improving the spatial alignment accuracy of the imaging scheme of the fusion event camera and RGB camera. Embodiment 1 uses the heterogeneous logic control subsystem 200 for time alignment, which can achieve high-precision spatiotemporal alignment overall.

[0029] Specifically, the optical acquisition subsystem 100 is positioned at the front end of the high-speed scene to be observed, and projects optical information within the same field of view onto two independent imaging focal planes simultaneously through physical beam splitting.

[0030] The beam splitting unit 120 includes a beam splitting prism and a relay lens group. The beam splitting prism separates the incident beam into a transmitted beam and a reflected beam. The relay lens group corrects optical path difference and aberrations, ensuring that the beam emitted from the imaging lens 110 and passing through its optical center, after being split by the beam splitting prism, forms a strictly consistent imaging field of view on the RGB sensor 131 and the event sensor 132. That is, Embodiment 1 utilizes the beam splitting unit 120 to ensure that the field of view of the two beams on the imaging plane is completely consistent, effectively guaranteeing imaging quality and avoiding aberration interference caused by beam splitting. For example, the RGB sensor 131 receives the transmitted beam and converts it into a frame synchronization electrical signal containing color information, and the event sensor 132 receives the reflected beam and converts it into an asynchronous event electrical signal responding to changes in light intensity.

[0031] The RGB sensor 131 and the event sensor 132 are located on the same equivalent focal plane and acquire data synchronously. In Embodiment 1, the RGB sensor 131 and the event sensor 132 are fused to balance high temporal resolution and high-quality texture information. The RGB sensor 131 can output image frame data in 24-bit data format to the heterogeneous logic control subsystem 200 through a dedicated signal line; the event sensor 132 can output an asynchronous event stream composed of event packets to the heterogeneous logic control subsystem 200 through a dedicated signal line in 32-bit data format.

[0032] Furthermore, the imaging lens 110 can be designed as a replaceable structure to adapt to a wider range of fusion imaging scenarios, and the lens interface is compatible with the CS interface standard protocol. The front end of the imaging lens 110 can also integrate a filter interface for loading filters, thereby allowing filters of specific wavelengths to be loaded according to the application scenario, enabling the system to operate at night or under specific industrial light sources.

[0033] The beam splitter can be a wideband unpolarized beam splitter cube; the beam splitter has an average transmittance of not less than 99.5% with a thickness of 10 mm in the spectral range of 400 nm to 1000 nm; the cemented surface of the beam splitter is coated with a precisely designed dielectric beam splitting film, which can transmit and reflect incident light in an energy ratio of 0.5:0.5 in the wide spectral range of 400 nm to 1000 nm.

[0034] The RGB sensor 131 and the event sensor 132 can be mounted on two independent carrier plates made of materials with low thermal expansion coefficients, respectively. Physical isolation reduces interference between the sensors, thereby ensuring that the system can maintain pixel-level spatial alignment stability under vibration or temperature change environments.

[0035] A black light-absorbing coating with a high light absorption rate can also be coated on the inner wall of the optical acquisition subsystem 100, and a light-blocking plate can be set around the beam splitter to eliminate diffuse reflection interference inside the system.

[0036] In summary, the optical acquisition subsystem 100 in Embodiment 1 adopts a precise optical beam splitting architecture, which can solve the parallax problem existing in traditional binocular vision systems and ensure strict spatial alignment of multimodal data.

[0037] In order to accurately align the two types of heterogeneous data (RGB frame data and event stream data) on the timeline, Embodiment 1 uses the heterogeneous logic control subsystem 200 as the main control unit to execute a synchronization strategy based on hardware signal handshake.

[0038] See Figure 1 , Figure 4 To coordinate with the heterogeneous logic control subsystem 200, the RGB image sensor 131, in addition to the signal line outputting image data, also has a frame synchronization signal line. Using the exposure timing of the RGB sensor 131 as the main clock reference, the RGB sensor 131 operates periodically according to a preset frame rate. Whenever the RGB sensor 131 begins the exposure integration of a new frame, it outputs a VSYNC (Vertical synchronization) signal to the heterogeneous logic control subsystem 200 through the frame synchronization signal line. After the logic control subsystem 200 detects the falling edge of the frame synchronization signal (i.e., the VSYNC signal), it generates an external trigger pulse (e.g., a pulse with a pulse width of 10 microseconds) and sends it to the external trigger input interface of the event sensor 132. The internal logic of the event sensor 132 is configured to a slave mode. When it detects the trigger pulse, it will not interrupt the current event acquisition process, but will force the insertion of a special synchronization marker event into the current asynchronous event data stream. This synchronization marker event has a special address code that is different from ordinary light intensity events and carries a precise timestamp of the trigger time.

[0039] Overall, the heterogeneous logic control subsystem 200 is connected to two sensors in the optical acquisition subsystem 100, respectively, to parse the raw data output by the sensors, use a hardware triggering mechanism to synchronize the heterogeneous data at the microsecond level, and encapsulate the synchronized data into a heterogeneous data packet.

[0040] Specifically, the heterogeneous logic control subsystem 200 includes: a sensor driving module 220, a synchronization control module 210, and an interface encapsulation module 230.

[0041] The sensor driving module 220 is used to convert the electrical signal output by the dual-mode sensing module into a digital signal. Specifically, it can receive electrical signals in parallel through the MIPI CSI interface.

[0042] The sensor driving module 220 includes a physical signal receiving layer, a protocol parsing and splitting layer, an image frame unpacking unit, and an event stream decoding unit. The physical signal receiving layer receives two electrical signals output from the dual-mode sensing module via two independent signal channels, and converts them into RGB frame data and event stream data. The protocol parsing and splitting layer performs logical isolation and splitting of heterogeneous data based on a data protocol, transmitting RGB frame data to the image frame unpacking unit and event stream data to the event stream decoding unit. The image frame unpacking unit performs data verification and pixel matrix mapping, extracts frame valid signals and line valid signals, and converts them into a common video stream format used within a programmable logic device (FPGA). The event stream decoding unit performs asynchronous buffering and protocol packet parsing, extracts the X-coordinate, Y-coordinate, and polarity bit of each event, and converts them into a standardized parallel data bus output.

[0043] The sensor driver module 220 serves as a data input interface, connected in parallel to the front-end dual-mode sensing module via a configured high-speed MIPI CSI interface. It receives an asynchronous event stream composed of image frame data output from the RGB sensor 131 via differential signal lines and event packets output from the event sensor 132 via differential signal lines. Because the data protocols output by the RGB sensor 131 and the event sensor 132 are different, the sensor driver module 220 internally divides the data into independent virtual channels to deserialize and decode the two signals respectively. For RGB data, the sensor driver module 220 performs pixel clock recovery and line / field signal separation; for event data, it performs handshake protocol parsing to ensure that data is not lost or blocked during high event rates (e.g., hundreds of millions of events per second).

[0044] Specifically, the sensor driver module 220 integrates physical layer interface logic and protocol layer parsing logic conforming to the MIPI Alliance protocol standard. At the physical signal reception level, this module is configured to connect the RGB sensor 131 and the event sensor 132 via two independent high-speed signal channels. To address the high bandwidth requirements of the RGB sensor 131, the module can enable a 4-channel reception mode with a single-channel rate configured at 1.5Gbps to meet the throughput requirements of high-resolution images; to address the bursty data characteristics of the event sensor 132, the module can enable a 2-channel reception mode. When the dual-mode sensing module is working, the sensor driving module 220 first performs clock recovery and serial-to-parallel conversion on the input low-voltage differential signal, restoring the high-speed serial bit stream into parallel byte stream data. At the protocol parsing and splitting level, the sensor driving module 220 uses the virtual channel identifier in the protocol header to achieve logical isolation and precise splitting of heterogeneous data. For the RGB data stream, the module identifies packets carrying image data and enters the image frame unpacking unit. The image frame unpacking unit removes the overhead bits at the beginning and end of the data packets, performs cyclic redundancy check to ensure data integrity, and then maps the original byte stream into a pixel matrix, and simultaneously extracts the frame valid signal and line valid signal, converting them into a video stream format common to the FPGA. The event data stream enters the event stream decoding unit. Due to the non-fixed length and asynchronous burst characteristics of event data, this unit has a built-in shallow asynchronous FIFO buffer to smooth instantaneous data peaks. The decoding logic is responsible for parsing the protocol packets, extracting the X coordinate, Y coordinate, and polarity bit of each event, and converting them into a standardized 64-bit parallel data bus output.

[0045] In addition, the sensor driving module 220 can also have built-in link status monitoring logic. Once a physical layer signal loss or verification error is detected, the logic immediately sets an error flag in the system status register and automatically triggers a soft reset process to attempt to re-establish the link, thereby preventing data link deadlock caused by electromagnetic interference.

[0046] Through the above design, the sensor driver module 220 achieves hardware-level heterogeneous data isolation and parallel throughput, realizes logical isolation of data streams of different natures under the same physical interface standard, avoids mutual blocking, and the built-in verification and automatic reset mechanism greatly enhances the system's anti-interference ability in complex electromagnetic environments and ensures the stability of long-term operation.

[0047] The synchronization control module 210 (i.e., hardware synchronization control module) is used to monitor the frame synchronization signal output by the RGB sensor 131 in real time, and when the frame start pulse is detected, it immediately sends an external trigger pulse (i.e., synchronization trigger command) to the event sensor 132, so that the event sensor 132 inserts a synchronization event packet (Time Mark) with a specific mark into the event stream data, thereby establishing a unified time reference for the two data at the hardware level.

[0048] Specifically, the operation mode of the synchronization control module 210 is based on the master-slave triggering mechanism. The synchronization control module 210 is built on a highly reliable finite state machine (FSM) inside the FPGA logic. The state machine includes four core states: IDLE, CAPTURE, TRIGGER, and LOCK-OUT.

[0049] The synchronization control module 210 internally houses a high-precision counter and monitors the frame synchronization signal from the RGB sensor 131 in real time. During operation, the finite state machine is initialized to an idle state. The RGB sensor 131, acting as the timing master, outputs the falling edge of the VSYNC signal whenever a new frame exposure begins. Once the synchronization control module 210 captures the frame start pulse output by the RGB sensor 131 within a clock cycle, the finite state machine enters a signal capture state, records the local timestamp of the current moment, and automatically enters a trigger generation state, sending the external trigger pulse to the event sensor 132 via a physical connection. In response to this pulse, the event sensor 132 forcibly inserts a synchronization event packet with a specific header marker into the current continuous event stream. This entire process is completed within the FPGA's hardware logic gates, bypassing any interrupt delays from the software operating system.

[0050] That is, the working logic of the synchronization control module 210 is as follows: capture the frame synchronization falling edge or rising edge signal output by the RGB sensor 131 before the exposure of a frame image begins; generate a trigger pulse with a microsecond delay in the internal logic of the FPGA; inject the trigger pulse into the external trigger pin of the event sensor 132 or write it into a specific register through the configuration interface, so that a synchronization marker event with polarity characteristics is generated in the event data stream; in subsequent processing, the synchronization marker event is used as the time zero point of the RGB image of the frame.

[0051] To prevent false triggering caused by unstable short pulses output by the RGB sensor 131, the finite state machine immediately enters a dead-zone locked state after generating the external trigger pulse. In this state, the module starts a countdown timer with a duration slightly shorter than one frame of RGB exposure time. During this period, any external VSYNC signal input is blocked to ensure that each frame of image corresponds to only one unique synchronization marker, thus eliminating multiple triggering or false triggering.

[0052] Furthermore, considering the potential electromagnetic interference in industrial environments, the synchronization control module 210 can first sample the frame synchronization signal (VSYNC signal) of the RGB sensor 131 through a digital filter composed of a multi-stage shift register. Only when the signal level remains consistent over N consecutive clock cycles will it be determined as a valid state transition signal, thereby effectively filtering out interference.

[0053] The interface encapsulation module 230 is used to receive RGB frame data and event stream data after synchronization processing, mix and package them to obtain the heterogeneous data packet, and send the heterogeneous data packet to the fusion computing subsystem 300.

[0054] The interface encapsulation module 230 is a high-speed interface encapsulation module responsible for data packaging and transmission. It receives raw data from the sensor driving module 220 and timestamp information from the synchronization control module 210. The interface encapsulation module 230 adopts a hybrid data packet structure, logically binding a frame of RGB image data with all event data corresponding to its exposure time period and stamping it with a unified FPGA system timestamp. Subsequently, the interface encapsulation module 230 uses a DMA (Direct Memory Access) controller and the PCIe bus protocol to directly write these encapsulated heterogeneous data packets into the memory address space of the fusion computing subsystem 300 in a "zero-copy" manner.

[0055] Specifically, the interface encapsulation module 230 has a built-in protocol arbitration and packet encapsulation engine. When a data stream is received from the sensor driver module 220, the engine first generates a 64-bit custom header. This header contains the following key fields: Magic Word, a feature code used to verify data stream synchronization (e.g., 0xAA55); Data Type, identifying the payload type (0x01 represents an RGB frame, 0x02 represents an event stream); Timestamp, a 64-bit high-precision FPGA local timestamp latched by the synchronization control module 210; and Payload Length, indicating the byte length of subsequent valid data. Subsequently, the engine fills in the actual image data or event data immediately after the header to form a complete transmission frame. The encapsulated transmission frame is then fed into a depth-configurable dual-port RAM for ping-pong buffering. The DMA controller reads the buffer data and segments it into transaction layer packets that meet the maximum PCIe load. The DMA controller directly reads the descriptor list in system memory and writes the packets directly into non-contiguous physical pages of memory via the PCIe TX channel in burst transfer mode, without any CPU involvement in data transfer. The module integrates flow control logic to monitor the PCIe bus's credit status in real time. When the receive buffer is detected to be nearing full, the interface encapsulation module 230 automatically sends a backpressure signal to the sensor driver module to pause data reading and prevent data truncation or out-of-order delivery due to FIFO overflow.

[0056] The heterogeneous logic control subsystem 200 is primarily built upon high-performance FPGA devices, aiming to achieve parallel acquisition of multimodal data, precise time alignment, and high-speed transmission. Each module works collaboratively to ensure that heterogeneous data streams undergo strict physical layer synchronization before entering the fused computing subsystem.

[0057] In summary, compared to traditional solutions that utilize operating system timestamps for soft synchronization, the heterogeneous logic control subsystem 200 in Example 1 controls the time alignment error between the RGB frames and the event stream to the microsecond level through hardware signal triggering, providing an absolutely reliable time reference for subsequent high-speed motion deblurring and trajectory reconstruction algorithms. Furthermore, the pure hardware pipeline processing based on FPGA ensures extremely low latency from data output from the sensor to transmission to the bus, avoiding the computational power consumption and waiting time caused by CPU polling. The dedicated interface encapsulation module 230 effectively solves the transmission congestion problem caused by the explosive growth of event data in high-speed motion scenarios, ensuring the real-time performance and stability of the system.

[0058] Specifically, the fusion operation subsystem 300 directly reads data from the heterogeneous logic control subsystem 200 and writes it into system memory to establish a circular buffer; the fusion operation subsystem 300 reads data from the circular buffer, separates RGB frame data and event stream data, and restores the timing correspondence between the two based on the synchronization event packets inserted in the event stream data to construct a heterogeneous fusion frame object; the fusion operation subsystem 300 uses the heterogeneous fusion frame object as input to perform fusion calculations.

[0059] In summary, Example 1 can achieve pixel-level spatial alignment and microsecond-level hardware synchronization.

[0060] Example 2: See Figure 1 Based on Embodiment 1, multiple different visual algorithm models are pre-configured within the fusion computing subsystem 300. The fusion computing subsystem 300 is used to load the visual algorithm model corresponding to the target visual task according to system configuration or instruction information. That is, the fusion computing subsystem 300 in Embodiment 2 is a "reconfigurable" fusion computing subsystem. In this case, the fusion computing subsystem 300 uses the heterogeneous fusion frame object as input and performs fusion calculations based on the loaded visual algorithm model.

[0061] Furthermore, Embodiment 2 may also include an interaction subsystem 400. The interaction subsystem 400 is used to receive configuration instructions corresponding to the target visual task sent by the operating terminal, parse the configuration instructions to obtain instruction information, and send the instruction information to the fusion computing subsystem 300.

[0062] The interactive subsystem 400 can also be used to display the visual processing information locally, and to compile the visual processing information into network data packets and send them to a remote server.

[0063] The following provides a further explanation of the "reconfigurable" fusion computing subsystem 300.

[0064] The fusion computing subsystem 300 is connected to the heterogeneous logic control subsystem 200 via a high-speed bus, receives the heterogeneous data packets, and performs fusion computing tasks by loading different visual algorithm models and combining the high temporal resolution of the event stream with the texture information of the color image to generate visual task processing results.

[0065] For details, see Figure 3 , Figure 4The fusion computing subsystem 300 includes: a direct memory access control module 310, a heterogeneous data decoding module 320, a vision task deployment port 330, and a computing processing unit 340. The fusion computing subsystem 300 can be built on a high-performance embedded system-on-chip (SoC) platform, enabling dynamic scheduling of computing resources and flexible switching of vision tasks. The modules in the fusion computing subsystem 300 collaborate efficiently through a shared memory mechanism, ensuring pipelined low-latency processing from data reception to result output.

[0066] The direct memory access control module 310 is used to establish a circular buffer and write heterogeneous data packets sent by the heterogeneous logic control subsystem 200 into the system memory at high speed through direct memory access technology, without CPU intervention.

[0067] The Direct Memory Access Control (DMI) module 310 acts as a bridge between hardware and software. During system initialization, the DMI module 310 driver pre-allocates a large, physically contiguous memory region in the SoC's kernel space, constructing a globally shared circular buffer. The DMI module 310, through the PCIe bus's DMA controller, directly writes high-speed heterogeneous data packets from the heterogeneous logic control subsystem 200 into this circular buffer in a "zero-copy" manner. The DMI module 310 maintains the buffer's write and read pointers in real time. The write pointer is updated in real time by the front-end heterogeneous logic control subsystem 200 via the PCIe bus. Whenever the FPGA completes the writing of a heterogeneous data packet, it updates the SoC-side write pointer offset through a PCIe space register write operation, indicating the end of the latest data storage. The read pointer is maintained by the back-end heterogeneous data decoding module; after the algorithm completes the reading and unpacking of a segment of data, the software updates the read pointer position. Meanwhile, the Direct Memory Access Control (DMI) module 310 incorporates overflow prevention logic. When the write speed exceeds the processing speed, causing the buffer to nearly fill up, the module proactively sends a flow control signal to the front end to ensure that the system memory does not experience illegal access or data overwriting under the impact of high-bandwidth data flows at Gbps levels. The DMI module 310 uses a memory mapping mechanism to directly map this physically contiguous kernel space to the virtual address range in user space. This means that subsequent visual algorithm models running at the application layer can directly access the original data in this physical memory through pointers, completely eliminating the CPU cycles and memory bandwidth overhead required in traditional drivers to copy data from kernel mode to user space.

[0068] The heterogeneous data decoding module 320 is used to read heterogeneous data packets from memory, separate RGB frame data and event stream data, and restore the timing correspondence between the two based on the synchronization event packets.

[0069] The heterogeneous data decoding module 320 runs as a high-priority background process. To ensure the real-time and deterministic nature of data processing, this process is bound to an independent physical core within the SoC to avoid jitter caused by thread switching. The heterogeneous data decoding module 320 reads the header metadata of the shared memory region in real time, compares the offsets of the write and read pointers, and immediately initiates the unpacking process upon detecting a valid data increment. Based on the Magic Word and Data Type fields in the aforementioned custom packet header, the heterogeneous data decoding module 320 quickly separates the RGB frame data and event stream data. When a data packet crosses the end and start addresses of physical memory, the module automatically performs segmented reading and concatenation, shielding the upper-layer application from the discontinuity of the underlying physical addresses. Furthermore, the heterogeneous data decoding module 320 executes time-stamp-based reassembly logic. It retrieves synchronization marker events in the data stream. When the first synchronization marker event is detected, it begins caching all subsequent ordinary event data in a temporary linear container until the next synchronization marker event is detected. It then extracts all asynchronous event data between two consecutive synchronization markers and logically binds them to their corresponding RGB frame data to construct a standard heterogeneous fused frame object. The object is a structure containing pointers to RGB pixel data, pointers to event vectors, and precise start and end timestamps for the frame. This process transforms the originally discrete, unstructured event stream into time-slice data that is strictly aligned with the image frame, providing a standardized input format for backend algorithms.

[0070] Through the above mechanism, this invention transforms the originally discrete, unstructured bitstream, and potentially physically discontinuous bitstream into logically strictly aligned and structurally highly standardized time-slice data. This allows backend deep learning algorithms to perform inference without needing to concern themselves with the underlying hardware synchronization details or bus protocols; they only need to retrieve standard objects from the queue. This significantly reduces the complexity of algorithm development and ensures precise correspondence of heterogeneous data in the microsecond-level time dimension.

[0071] The visual task deployment port 330 is configured as an algorithm model loading interface, which is used to load visual task algorithm models, including high-speed video reconstruction, object detection, trajectory tracking or action recognition, according to instruction information (i.e. user instructions) or system configuration, so as to realize the reconfigurability of the operation.

[0072] The vision task deployment port 330 provides a standardized API (Application Programming Interface) and runtime environment, shielding the differences in underlying heterogeneous computing resources and providing a unified algorithm loading channel. It supports dynamically loading different deep learning model files, such as TensorRT engine files or ONNX models. The system pre-installs various algorithm libraries for specific scenarios and performs hot-switching of models according to instructions. This mechanism allows a single hardware terminal to completely switch its functional role through software commands. To achieve hot-switching of models without interrupting the current data stream, the vision task deployment port 330 employs a double-buffered loading mechanism. When the system receives a switching command, the vision task deployment port 330 preloads the new model weights and network structure in the background video memory. After loading and verification are complete, the complete functional role switch can be instantly completed by simply modifying the inference engine pointer within one clock cycle.

[0073] The computing unit 340 is used to run the algorithm model loaded by the visual task deployment port, perform fusion calculation with RGB frame data and event stream data as input, output the corresponding image video stream or structured data result, and write the real-time calculation result into the video memory.

[0074] The computing unit 340 is responsible for scheduling the computing resources of the multi-core CPU, graphics processing unit (GPU), or neural network processor (NPU) within the SoC to perform hardware-accelerated inference on the loaded algorithm model. The computing unit 340 receives the heterogeneous fused frame object as input and finally outputs the video stream or structured detection data generated by the computing task. It can also directly write the results to the video memory for subsequent display or transmission.

[0075] In summary, the fusion computing subsystem 300 in Embodiment 2 overcomes the drawbacks of the fixed functions of traditional high-speed cameras through the above design. It realizes dynamic hot switching of algorithm models through the vision task deployment port 330, which greatly improves the reusability and economic value of the equipment. The application of circular buffer and zero-copy technology eliminates unnecessary data transfer between user mode and kernel mode, enabling the system to easily handle heterogeneous data throughput of hundreds of megabytes per second and significantly reducing CPU load. The strict data reorganization based on synchronization marker events ensures that each set of data input to the neural network model is absolutely matched in physical time, thereby ensuring that the fusion algorithm will not experience spatiotemporal misalignment when processing high-speed moving objects.

[0076] The interactive subsystem 400 will be further described below.

[0077] The interactive subsystem 400 serves as the sole port for information interaction between the entire device and the external environment. Its main responsibilities include visually presenting the fusion processing results, executing remote data transmission, and parsing user configuration commands. Each module within the interactive subsystem 400 is tightly coupled to the front-end fusion computing subsystem 300 via a system bus, achieving closed-loop control from data output to human-computer interaction.

[0078] The interactive subsystem 400 may specifically include: a network transmission module 410, a local display module 420, and an instruction parsing module 430.

[0079] The network transmission module 410 supports a UDP / IP protocol stack and a 100G fiber optic interface, and is used to compile the calculation results generated by the fusion computing subsystem 300 into network data packets and send them to a remote server.

[0080] The network transmission module 410 is responsible for remote data distribution and supports the standard UDP / IP protocol stack. For the high-bandwidth data stream generated by high-speed imaging, the network transmission module 410 sends the processed video stream or structured detection results to a remote server or cloud platform through the onboard 100G fiber optic interface or gigabit Ethernet port, supporting concurrent streaming access from multiple clients.

[0081] The local display module 420 is used to receive the calculation results, generate display information and corresponding horizontal and vertical scanning signals, and drive the display to display them in real time through the HDMI interface.

[0082] The local display module 420 is designed specifically for manual monitoring, and its input is directly connected to the video memory frame buffer of the processing unit 340. The local display module 420 can overlay key system operating indicators, such as the current event trigger rate, RGB frame rate, chip temperature, and algorithm confidence information, onto the image information and output them through the HDMI physical interface to directly drive the display, ensuring that operators can observe the algorithm's operating effect.

[0083] The instruction parsing module 430 is used to receive the computational processing task configuration instruction sent by the operation terminal, parse it, and distribute it to the fusion computing subsystem 300.

[0084] The instruction parsing module 430 resides in the background and listens for control messages from the operating terminal in real time. If an algorithm task switching instruction is received, the instruction parsing module 430 forwards it to the fusion computing subsystem 300, triggering the model hot loading process of the vision task deployment port 330.

[0085] Overall, in Embodiment 2, the optical acquisition subsystem 100 is oriented towards the target area. The optical acquisition subsystem 100 physically separates the incident light within the field of view into two beams, which are then projected onto the RGB sensor 131 and the event sensor 132, located on the same equivalent focal plane, respectively, to acquire pixel-level spatially aligned RGB frame data and event stream data. The heterogeneous logic control subsystem 200 is located after the optical acquisition subsystem 100 and provides sensor synchronization timing for the optical acquisition subsystem 100, parsing the sensor outputs of the optical acquisition subsystem 100. The system receives raw data and uses hardware synchronization logic to perform microsecond-level time alignment between color image frames and asynchronous event streams, generating heterogeneous data packets with unified timestamps. The fusion computing subsystem 300 receives these heterogeneous data packets and can load different visual algorithm models. It combines the high temporal resolution of the event stream with the texture information of the color image to perform a fusion computing task, generating visual task processing results (i.e., visual processing information). The interaction subsystem 400 can send the visual task processing results to a display or a server for storage, and can also parse configuration commands issued by the operating terminal to feed back to the fusion computing subsystem 300. The operating terminal, display, and server are all peripheral devices.

[0086] The schematic diagram of the optical devices and hardware equipment corresponding to the fusion event and RGB visual system provided in Example 2 is shown below. Figure 4 As shown, the hardware mainly includes an FPGA and a SoC. Optical components are connected to the FPGA via data and synchronization signal lines. The FPGA has a first memory storing a program that controls its behavior logic. It connects to the SoC via a high-speed data transmission interface using a PCIe interface. The SoC integrates a CPU, GPU, and NPU, as well as a second memory storing its program. The SoC communicates with operating devices and servers via an Ethernet interface and is connected to a display via a video signal line. The optical acquisition subsystem 100 is deployed on the optical components. When the program storing the FPGA's behavior logic in the first memory is executed, the FPGA circuitry completes the operation of the heterogeneous logic control subsystem 200. The SoC computing platform uses its internal CPU, GPU, and NPU to execute the program stored in the second memory, enabling the operation of the fused computing subsystem 300 and the interactive subsystem 400.

[0087] In summary, Example 2 not only achieves pixel-level spatial alignment and microsecond-level hardware synchronization, but also has reconfigurable computing capabilities, enabling dynamic deployment of different algorithm models according to application scenarios and flexible switching between various tasks such as imaging, detection, and recognition.

[0088] Example 3: Example 3 provides a visual method for fusing events and RGB, implemented using a visual system for fusing events and RGB as described in Example 1 or Example 2. See [link to example]. Figure 5 The visual method for fusing events and RGB provided in Example 3 mainly includes the following steps: Spatially aligned RGB frame data and event stream data are obtained using an optical acquisition subsystem; The heterogeneous logic control subsystem is used to time-align RGB frame data and event stream data to generate heterogeneous data packets with a unified timestamp. The fusion computing subsystem receives heterogeneous data packets and performs fusion calculations to obtain visual processing information.

[0089] In addition, Embodiment 3 can also receive configuration instructions corresponding to the target visual task sent by the operation terminal based on the interaction subsystem, parse the configuration instructions to obtain instruction information, and send the instruction information to the fusion computing subsystem; the fusion computing subsystem loads or switches the corresponding visual algorithm model based on the instruction information, and then performs the corresponding fusion calculation.

[0090] Since the steps in the fusion event and RGB vision method provided in Embodiment 3 correspond to the functions of each subsystem in the fusion event and RGB vision system provided in Embodiment 1 or Embodiment 2, they can be understood by referring to the description of Embodiment 1 or Embodiment 2, and will not be repeated here.

[0091] In summary, this invention can improve the spatiotemporal alignment accuracy of the imaging scheme of fusion event camera and RGB camera, and can also flexibly adapt to a variety of visual tasks. It also has good real-time performance and stability, and can achieve high-speed real-time detection and processing of targets with low latency and high dynamic range.

[0092] Finally, it should be noted that the above specific embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to examples, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A visual system that integrates events and RGB, characterized in that, include: Optical acquisition subsystem, heterogeneous logic control subsystem, and fusion computing subsystem; The optical acquisition subsystem includes an imaging lens, a beam splitter, and a dual-mode sensing module. The beam splitter is positioned on the outgoing light path of the imaging lens. The dual-mode sensing module includes an RGB sensor and an event sensor, which are respectively positioned on the two outgoing light paths of the beam splitter and located on the same equivalent focal plane. The optical center of the imaging lens coincides with the incident optical axis of the beam splitter, and the two outgoing optical axes of the beam splitter coincide with the optical axes of the RGB sensor and the event sensor, respectively, making the RGB sensor and the event sensor share a common optical center. The optical acquisition subsystem is used to obtain spatially aligned RGB frame data and event stream data. The heterogeneous logic control subsystem is used to time-align RGB frame data and event stream data to generate heterogeneous data packets with a unified timestamp. The fusion computing subsystem is used to receive heterogeneous data packets and perform fusion calculations to obtain visual processing information.

2. The visual system for fusing events and RGB according to claim 1, characterized in that, The beam-splitting unit includes: a beam-splitting prism and a relay lens group; The beam splitter is used to separate the incident beam into a transmitted beam and a reflected beam. The relay lens group is used to correct optical path difference and aberration, so that the light beam emitted from the imaging lens and passing through its optical center is split by the beam splitter and forms a strictly consistent imaging field of view on the RGB sensor and the event sensor.

3. The visual system for fusing events and RGB according to claim 1, characterized in that, The heterogeneous logic control subsystem includes: a sensor driving module, a synchronization control module, and an interface encapsulation module; The sensor driving module is used to convert the electrical signal output by the dual-mode sensing module into a digital signal; The synchronization control module is used to monitor the frame synchronization signal output by the RGB sensor in real time, and when a frame start pulse is detected, it immediately sends an external trigger pulse to the event sensor, so that the event sensor inserts a synchronization event packet with a specific mark into the event stream data; The interface encapsulation module is used to receive RGB frame data and event stream data after synchronization processing, mix and package them to obtain the heterogeneous data packet, and send the heterogeneous data packet to the fusion computing subsystem.

4. The visual system for fusing events and RGB according to claim 3, characterized in that, The sensor driving module includes: a physical signal receiving layer, a protocol parsing and splitting layer, an image frame unpacking unit, and an event stream decoding unit; The physical signal receiving layer is used to receive the two electrical signals output by the dual-mode sensing module through two independent signal channels, and convert them into RGB frame data and event stream data. The protocol parsing and offloading layer is used to logically isolate and offload heterogeneous data based on the data protocol, transmitting RGB frame data to the image frame unpacking unit and transmitting event stream data to the event stream decoding unit. The image frame unpacking unit is used to perform data verification and pixel matrix mapping, extract the frame valid signal and the line valid signal, and convert them into a video stream format commonly used inside the FPGA. The event stream decoding unit is used to perform asynchronous caching and protocol packet parsing, extract the X coordinate, Y coordinate and polarity bit of each event, and convert them into a standardized parallel data bus output.

5. The visual system for fusing events and RGB according to claim 3, characterized in that, The synchronization control module is constructed based on a finite state machine. The finite state machine is initialized to an idle state. Once the synchronization control module captures the frame start pulse output by the RGB sensor, the finite state machine enters the signal capture state, records the local timestamp of the current moment, and automatically enters the trigger generation state, sending the external trigger pulse to the event sensor through a physical connection. After generating the external trigger pulse, the finite state machine immediately enters the dead-zone locking state, blocking the input of the frame synchronization signal for a set time.

6. The visual system for fusing events and RGB according to claim 1, characterized in that, The fusion computing subsystem has multiple pre-set visual algorithm models; the fusion computing subsystem is used to load the visual algorithm model corresponding to the target visual task according to system configuration or instruction information.

7. The visual system for fusing events and RGB according to claim 6, characterized in that, The fusion computing subsystem directly reads data from the heterogeneous logic control subsystem and writes it into system memory to establish a circular buffer. The fusion computing subsystem reads data from the circular buffer, separates RGB frame data from event stream data, and restores the temporal correspondence between the two based on the synchronization event packets inserted in the event stream data to construct a heterogeneous fusion frame object. The fusion computing subsystem uses the heterogeneous fusion frame object as input and performs fusion calculations based on the loaded visual algorithm model.

8. The visual system for fusing events and RGB according to claim 6, characterized in that, Also includes: Interactive subsystem; The interaction subsystem is used to receive configuration instructions corresponding to the target visual task sent by the operation terminal, parse the configuration instructions to obtain instruction information, and send the instruction information to the fusion computing subsystem.

9. The visual system for fusing events and RGB according to claim 8, characterized in that, The interactive subsystem is also used to display the visual processing information locally, and to compile the visual processing information into network data packets and send them to a remote server.

10. A visual method for fusing events and RGB, characterized in that, The visual method for fusing events and RGB, implemented using any one of claims 1 to 9, comprises the following steps: Spatially aligned RGB frame data and event stream data are obtained using an optical acquisition subsystem; The heterogeneous logic control subsystem is used to time-align RGB frame data and event stream data to generate heterogeneous data packets with a unified timestamp. The fusion computing subsystem receives heterogeneous data packets and performs fusion calculations to obtain visual processing information.

Images