A method, apparatus and medium for event camera image and frame camera image fusion

Abstract

The application discloses a kind of event camera image and frame camera image fusion method, device and medium, the present method is triggered event camera and frame camera synchronous acquisition signal;Under the synchronous acquisition signal, event stream is collected by event camera and frame camera image is collected by frame camera;According to the time stamp of synchronous acquisition signal, segment event stream is carried out, and segmented event stream is obtained;Wherein, segmented event stream is time-aligned with frame camera image;Event time image method is used to process segmented event stream, and event camera image is obtained;Wherein, event time image method is used to retain the time information and polarity information of segmented event stream;Based on the event camera image and the frame camera image, feature fusion is carried out, and fusion feature map is obtained.The application obtains event camera image by event time image method, then event camera image and frame camera image are fused, and the effect of event camera image and frame camera image fusion can be improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus and medium for fusing event camera images and frame camera images. Background Technology

[0002] Frame cameras and event cameras are two different image acquisition techniques. Frame cameras acquire image frames continuously at fixed time intervals, providing high resolution and rich image information, suitable for a variety of computer vision tasks. However, they cannot capture instantaneous, rapid changes and motion details, limiting their analysis of high-speed moving objects. Event cameras acquire data asynchronously by detecting pixel-level changes, offering advantages such as high temporal resolution, low latency, and high dynamic range. They can capture details in fast-moving and high dynamic range scenes and perform exceptionally well in low-light conditions.

[0003] However, existing technologies do not perform well in processing event streams captured by frame cameras. Commonly used event representation methods include event images, event binarization, and time surfaces. Among these, event images and event binarization ignore the temporal information of events. Time surfaces only retain the time of the latest event for each pixel, discarding a large amount of event information. Therefore, the fusion effect between event camera images and frame camera images is poor. Summary of the Invention

[0004] This invention aims to at least solve one of the technical problems existing in the prior art. To this end, this invention proposes a method, apparatus, and medium for fusing event camera images and frame camera images, which can improve the fusion effect of event camera images and frame camera images.

[0005] In a first aspect, embodiments of the present invention provide a method for fusing event camera images and frame camera images, the method comprising:

[0006] The event camera and the frame camera synchronize their acquisition signals;

[0007] Under the synchronous acquisition signal, the event camera acquires the event stream and the frame camera acquires the frame camera images;

[0008] The event stream is segmented according to the timestamp of the synchronous acquisition signal to obtain a segmented event stream; wherein the segmented event stream is time-aligned with the frame camera image;

[0009] The segmented event stream is processed using a constructed event temporal image method to obtain event camera images; wherein, the event temporal image method is used to preserve the temporal and polarity information of the segmented event stream;

[0010] Feature fusion is performed on the event camera image and the frame camera image to obtain a fused feature map.

[0011] Compared with the prior art, the first aspect of the present invention has the following beneficial effects:

[0012] This method triggers the synchronous acquisition signals of the event camera and the frame camera, and acquires images under the synchronous acquisition signal, so that the event camera and the frame camera are synchronized in time. This can ensure that the time error between the timestamp of the event stream and the actual imaging time of the frame image is minimized. By processing the segmented event stream through the event time image method, the time information and polarity information of the segmented event stream can be preserved, reducing the loss of event information. Then, feature fusion based on the event camera image and the frame camera image can improve the fusion effect of the event camera image and the frame camera image, and achieve stable imaging effect in low light, high dynamic range or fast motion scenes, thereby improving scene understanding and image processing capabilities.

[0013] According to some embodiments of the present invention, the synchronous acquisition signals of the trigger event camera and the frame camera include:

[0014] A first chip is used to design the trigger signal, and a second chip is used to control the phase shift of the trigger signal to generate a synchronous acquisition signal for the event camera and the frame camera.

[0015] According to some embodiments of the present invention, the trigger signal is phase-shifted in the following manner:

[0016] Obtain the start exposure time and the middle exposure time of the frame camera;

[0017] Based on the start exposure time and the midpoint of the exposure, the required phase shift length of the trigger signal is calculated as follows:

[0018] Δt=t1-t0

[0019] Where t0 represents the start exposure time of the frame camera, t1 represents the middle time of the exposure, and Δt represents the length of phase shift required for the trigger signal.

[0020] According to some embodiments of the present invention, the event time image method is constructed in the following manner:

[0021]

[0022]

[0023]

[0024] in, Indicates each event based on t i Classified to the The time period, N e The number of events is represented by t, and the timestamp is represented by t. i Let B represent the timestamp of the i-th event, B represent the number of time periods, a represent the cumulative sum of the polarities of events within a single pixel over a time period, (x,y) represent the pixel coordinates, σ(a) represent the activation function, δ(·) represent the Dirac function, G represent the transformation that converts the event into a tensor representation, and p∈[-1,1] represent the polarity of the event.

[0025] According to some embodiments of the present invention, the step of performing feature fusion based on the event camera image and the frame camera image to obtain a fused feature map includes:

[0026] The event camera image and the frame camera image are registered using an image registration method to obtain the registered event camera image and frame camera image;

[0027] A feature extraction network is used to extract features from the registered event camera image and the frame camera image to obtain a first feature map of the registered event camera image and a second feature map of the registered frame camera image.

[0028] The first feature map and the second feature map are transmitted to the constructed attention-based fusion module for feature fusion to obtain a fused feature map.

[0029] According to some embodiments of the present invention, the attention-based fusion module is constructed in the following manner:

[0030]

[0031] F c =concatenate(f 1×1 (F s ),f 1×1 (F l ))

[0032]

[0033] Where F′ represents the fused feature map, F s Let F represent the first feature map. l Denotes the second feature map, M(F) s ,F l ) represents an attention graph, concatenate represents concatenation, and f 1×1 This represents a 1×1 convolution operation. This represents a 3×3 dilated convolution operation. F represents element-wise multiplication. c This represents the feature map after concatenation.

[0034] According to some embodiments of the present invention, after processing the segmented event stream using a constructed event temporal image method, the method for fusing the event camera image and the frame camera image further includes:

[0035] The intrinsic and extrinsic parameters of the event camera and the frame camera are estimated using the checkerboard and Kalibr calibration methods to obtain the spatial transformation matrix;

[0036] The frame camera images are calibrated to obtain labels;

[0037] The spatial transformation matrix is ​​used to transfer the labels onto the event camera image, thereby providing labels for the event camera image.

[0038] Secondly, embodiments of the present invention also provide an apparatus for fusing event camera images and frame camera images, the apparatus comprising:

[0039] The signal triggering unit is used to trigger the synchronous acquisition signals of the event camera and the frame camera;

[0040] The video processing unit is used to acquire event streams through the event camera and frame camera images through the frame camera under the synchronous acquisition signal.

[0041] The CPU core unit is configured to segment the event stream according to the timestamp of the synchronization trigger signal to obtain a segmented event stream; wherein the segmented event stream is time-aligned with the frame camera image; and to process the segmented event stream using a constructed event time image method to obtain an event camera image; wherein the event time image method is configured to retain the time information and polarity information of the segmented event stream; and to perform feature fusion based on the event camera image and the frame camera image to obtain a fused feature map.

[0042] Compared with the prior art, the second aspect of the present invention has the following beneficial effects:

[0043] This device triggers the synchronous acquisition signals of the event camera and frame camera through a signal triggering unit, and acquires images under the synchronous acquisition signal through a video processing unit, so that the event camera and frame camera are synchronized in time, which can ensure that the time stamp of the event stream and the actual imaging time of the frame image are minimized in time. The CPU core unit processes the segmented event stream using the constructed event time image method, which can retain the time information and polarity information of the segmented event stream, reduce the loss of event information, and then perform feature fusion based on the event camera image and the frame camera image, which can improve the fusion effect of the event camera image and the frame camera image, and achieve stable imaging effect in low light, high dynamic range or fast motion scenes, thereby improving scene understanding and image processing capabilities.

[0044] According to some embodiments of the present invention, the apparatus for fusing event camera images and frame camera images further includes a power management unit, a dataset interaction unit, and an interface expansion unit connected to the CPU core unit, wherein:

[0045] The power management unit is used to supply power to the CPU core unit, the event camera, and the frame camera.

[0046] The dataset interaction unit is used by the CPU core unit for wired and wireless communication.

[0047] The interface expansion unit is used for data transmission and data storage of the CPU core unit.

[0048] Thirdly, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform a method for fusing event camera images and frame camera images as described above.

[0049] It is understood that the beneficial effects of the third aspect compared with the related technologies are the same as the beneficial effects of the first aspect compared with the related technologies. Please refer to the relevant description in the first aspect above, which will not be repeated here. Attached Figure Description

[0050] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0051] Figure 1 This is a flowchart of a method for fusing event camera images and frame camera images according to an embodiment of the present invention;

[0052] Figure 2 This is a schematic diagram of the phase shift of the trigger signal according to an embodiment of the present invention;

[0053] Figure 3 This is an example diagram of an event camera image according to an embodiment of the present invention;

[0054] Figure 4 This is a schematic diagram of an attention-based fusion module according to an embodiment of the present invention;

[0055] Figure 5 This is a structural diagram of an apparatus for fusing event camera images and frame camera images according to an embodiment of the present invention;

[0056] Figure 6 This is a structural diagram of the event camera and frame camera synchronously acquiring data according to an embodiment of the present invention. Detailed Implementation

[0057] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0058] In the description of this invention, the use of terms such as "first," "second," etc., is for the purpose of distinguishing technical features only and should not be construed as indicating or implying relative importance, or implicitly indicating the number of technical features indicated, or implicitly indicating the order of the technical features indicated.

[0059] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, etc., are based on the orientation or positional relationship shown in the drawings and are only for the convenience of describing this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0060] In the description of this invention, it should be noted that, unless otherwise explicitly defined, terms such as "setting," "installation," and "connection" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0061] Existing technologies commonly use event representation methods such as event images, event binarization, and temporal surfaces. However, event images and event binarization methods ignore the temporal information of events. Temporal surfaces only retain the time of the latest event for each pixel, discarding a large amount of event information. Therefore, the fusion of event camera images and frame camera images results in poor quality.

[0062] To address the aforementioned issues, this invention triggers the synchronous acquisition signals of the event camera and the frame camera, acquiring images under these signals. This ensures temporal synchronization between the event camera and the frame camera, minimizing the temporal error between the timestamp of the event stream and the actual imaging time of the frame image. Furthermore, by processing the segmented event stream using an event-time image method, the temporal and polarity information of the segmented event streams is preserved, reducing event information loss. Finally, the event camera images and frame camera images are fused, improving the fusion effect and enabling stable imaging in low-light, high dynamic range, or fast-moving scenes, thus enhancing scene understanding and image processing capabilities.

[0063] Reference Figure 1This invention provides a method for fusing event camera images and frame camera images. This method includes, but is not limited to, steps S100 to S500, wherein:

[0064] Step S100: Trigger the synchronous acquisition signals of the event camera and the frame camera;

[0065] Step S200: Under the synchronous acquisition signal, the event stream is acquired by the event camera and the frame camera is acquired by the frame camera;

[0066] Step S300: Segment the event stream according to the timestamp of the synchronous acquisition signal to obtain a segmented event stream; wherein the segmented event stream is time-aligned with the frame camera image.

[0067] Step S400: The segmented event stream is processed using the constructed event time-image method to obtain event camera images; wherein, the event time-image method is used to preserve the time information and polarity information of the segmented event stream;

[0068] Step S500: Perform feature fusion based on event camera images and frame camera images to obtain a fused feature map.

[0069] In this embodiment, to minimize the time error between the timestamp of the event stream and the actual imaging time of the frame image, the event camera and the frame camera are synchronized by triggering a synchronous acquisition signal. Under the synchronous acquisition signal, the event camera acquires the event stream and the frame camera acquires the frame camera image. To reduce the loss of event information, the event stream is segmented according to the timestamp of the synchronous acquisition signal to obtain a segmented event stream. The segmented event stream is processed using a constructed event time image method to obtain the event camera image. To improve scene understanding and image processing capabilities, the event camera image and the frame camera image are fused to obtain a fused feature map.

[0070] In some embodiments, triggering the synchronous acquisition of signals by the event camera and the frame camera includes:

[0071] The first chip is used to design the trigger signal, and the second chip is used to control the phase shift of the trigger signal to generate synchronous acquisition signals for the event camera and the frame camera.

[0072] In this embodiment, by designing a synchronous acquisition signal, the event camera and the frame camera are synchronized in time, which can ensure that the time error between the timestamp of the event stream and the actual imaging time of the frame image is minimized.

[0073] In some embodiments, the trigger signal is phase-shifted in the following manner:

[0074] Obtain the start and middle times of the exposure from the frame camera;

[0075] Based on the start exposure time and the midpoint of the exposure, the required phase shift length of the trigger signal is calculated as follows:

[0076] Δt=t1-t0

[0077] Where t0 represents the start exposure time of the frame camera, t1 represents the middle time of the exposure, and Δt represents the length of phase shift required for the trigger signal.

[0078] In this embodiment, since the event camera operates at the microsecond level and the frame camera requires an exposure time for imaging, the trigger signal of the event camera is aligned with the exposure midpoint of the frame camera by phase shifting. This minimizes the time error between the timestamp of the output event stream and the actual imaging time of the frame image, thus synchronizing the event camera and the frame camera in time.

[0079] In some embodiments, the event time image method is constructed as follows:

[0080]

[0081]

[0082]

[0083] in, Indicates each event based on t i Classified to the The time period, N e The number of events is represented by t, and the timestamp is represented by t. i Let B represent the timestamp of the i-th event, B represent the number of time periods, a represent the cumulative sum of the polarities of events within a single pixel over a time period, (x,y) represent the pixel coordinates, σ(a) represent the activation function, δ(·) represent the Dirac function, G represent the transformation that converts the event into a tensor representation, and p∈[-1,1] represent the polarity of the event.

[0084] In this embodiment, because the event image and event binarization methods ignore the temporal information of the events, and the time surface only retains the time of the latest event for each pixel, a large amount of event information is discarded. The method of constructing an event time image not only retains the temporal and polarity information of the events, but also reduces noise to a certain extent.

[0085] In some embodiments, feature fusion is performed based on event camera images and frame camera images to obtain a fused feature map, including:

[0086] The event camera image and the frame camera image are registered using an image registration method to obtain the registered event camera image and frame camera image;

[0087] A feature extraction network is used to extract features from the registered event camera image and frame camera image to obtain the first feature map of the registered event camera image and the second feature map of the registered frame camera image.

[0088] The first and second feature maps are transmitted to the constructed attention-based fusion module for feature fusion to obtain a fused feature map.

[0089] In this embodiment, since the frame camera image contains rich texture and color features and the event camera image contains motion contour information of the target object, the feature map of the frame camera image can be used as supplementary information to enhance the feature map of the event camera image, thereby improving the robustness and accuracy of target detection.

[0090] In some embodiments, the attention-based fusion module is constructed as follows:

[0091]

[0092] F c =concatenate(f 1×1 (F s ),f 1×1 (F l ))

[0093]

[0094] Where F′ represents the fused feature map, F s Let F represent the first feature map. l Denotes the second feature map, M(F) s ,F l ) represents an attention graph, concatenate represents concatenation, and f 1×1 This represents a 1×1 convolution operation. This represents a 3×3 dilated convolution operation. F represents element-wise multiplication. c This represents the feature map after concatenation.

[0095] In this embodiment, the attention-based fusion module constructed in the above manner can accelerate model convergence, improve model performance, enhance model stability, and effectively extract contextual information.

[0096] In this embodiment, after processing the segmented event stream using the constructed event temporal image method, the method for fusing event camera images and frame camera images further includes:

[0097] The intrinsic and extrinsic parameters of the event camera and the frame camera are estimated using the checkerboard and Kalibr calibration methods to obtain the spatial transformation matrix;

[0098] The frame camera images are calibrated to obtain labels;

[0099] A spatial transformation matrix is ​​used to transfer labels onto event camera images, thus providing labels for the event camera images.

[0100] In this embodiment, it is necessary to calibrate the intrinsic and extrinsic parameter matrices of the event camera and the frame camera, and the relative positions between the cameras and the camera lenses cannot be changed after calibration.

[0101] To facilitate understanding by those skilled in the art, a set of preferred embodiments is provided below:

[0102] This embodiment uses an IMU chip to design a trigger timing signal. The microcontroller controls the phase shift of the trigger signal to generate a synchronous acquisition signal for the camera. Based on the synchronous acquisition signal, the event camera begins acquiring the dynamic event stream of the scene, and the RGB frame camera begins acquiring RGB image information and sending it to the edge computing host. The event camera synchronously acquires the event stream, adds a synchronous trigger timestamp to the event stream, records the trigger time, and sends it to the edge computing host. The edge computing host obtains the RGB image and performs calculations on the event stream to obtain an event frame image containing scene contour information. A multimodal visual fusion module (i.e., an attention-based fusion module) is used to fuse the event frame image and the RGB image. This invention can achieve synchronous extraction and image fusion of multimodal visual information in dynamic scenes, realizing robust imaging in complex scenes such as high speed and low light. Specifically, it includes the following steps:

[0103] 1. The trigger timing signal is designed using an IMU chip, and the camera synchronous acquisition signal is generated based on the phase shift of the trigger signal controlled by the microcontroller. For example... Figure 2 As shown, the start exposure time of the frame camera is t0, the middle exposure time is t1, and the end exposure time is t2. The required phase shift length for the trigger signal is Δt = t1 - t0. Because the event camera operates at the microsecond level, while the frame camera requires an exposure time for imaging, this embodiment aligns the trigger signal of the event camera with the middle exposure time of the frame camera. This minimizes the time error between the timestamp of the output event stream and the actual imaging time of the frame image, ensuring that the event camera and the frame camera are synchronized in time.

[0104] 2. Using an event / frame camera synchronous acquisition device, RGB images (i.e., frame camera images) and event streams of the same scene are acquired respectively. Each asynchronous event in the event stream is recorded as e = (x, y, t, p), where (x, y) represents the pixel coordinates, t represents the timestamp, and p ∈ [-1, 1] represents the polarity of the event; p = 1 indicates that the brightness of the pixel increases, and p = -1 indicates that the brightness of the pixel decreases. Given a time period Δt, this embodiment can obtain a set of asynchronous events. Where Ne This represents the number of events. In this embodiment, Δt is divided into a set of B time periods along the time dimension to preserve the time information in the event stream. Then, within each segmented time period, this embodiment accumulates the polarity p of all events and uses an activation function to map it to the range of 0-255. This calculation method is called the event time-image method in this embodiment, and can be expressed as follows:

[0105]

[0106]

[0107]

[0108] in, Indicates each event based on t i Classified to the The time period, N e The number of events is represented by t, and the timestamp is represented by t. i Let B represent the timestamp of the i-th event, B represent the number of time segments, a represent the cumulative sum of the polarities of events within a single pixel over a time segment, (x,y) represent the pixel coordinates, σ(a) represent the sigmoid function, which maps variables to the range [0,1]. Multiplying by 255 maps variables to the range [0,255] to ensure that pixel values ​​fall within the range [0,255]. δ(·) represents the Dirac function, and G represents the transformation that converts events into tensor representations. Therefore, a three-dimensional tensor, called the event-temporal image, can be obtained, where the first and second dimensions correspond to the height and width of the frame representation, and the third dimension represents the number of channels, i.e., the number of B time segments. The proposed event-temporal image comprehensively considers the temporal information and polarity of events. Figure 3 This embodiment demonstrates that by setting B to 3, the motion trends of vehicles, pedestrians, and two-wheeled vehicles can be observed using RGB channel information. Furthermore, this representation helps reduce noise interference to some extent, as the cumulative positive value of occasional noise events tends towards 0. After applying the activation function σ(a), the pixel values ​​become closer to gray. However, the range of pixels containing more positive events is from 128 to 255, while the range of pixels containing more negative events is from 0 to 127.

[0109] The advantages of this event time-image calculation method are as follows: Existing commonly used event representation methods include event images, event binarization, and time surfaces. Among them, event images and event binarization methods ignore the temporal information of the event. Time surfaces only retain the time of the latest event for each pixel, discarding a large amount of event information. The event time-image method in this embodiment not only retains the temporal and polarity information of the event, but also reduces noise to a certain extent.

[0110] 3. The obtained event time image and RGB image are fused to achieve downstream tasks such as classification, detection, and tracking based on multimodal visual information. The specific fusion method is as follows:

[0111] Reference Figure 4 First, image registration methods from computer vision are used to register the event time image and the RGB image. The registered event time image I... s and RGB image I l The data are first fed into a feature extraction network, and feature maps F are extracted from each network. s and F l The resulting feature maps are then fed into an attention-based fusion module. Since RGB images contain rich texture and color features, and event-time images contain motion contour information of the target object, RGB image feature maps can be used as supplementary information to enhance the feature maps of event-time images, improving the robustness and accuracy of object detection. First, feature map F... s and F l After channel reduction using a 1×1 convolutional layer, the feature map is obtained. Next, the integrated feature maps will be... The input is fed into three 3×3 dilated convolutional layers to extract contextual information, and the resulting feature map is then fed into a 1×1 convolutional layer to obtain the spatial attention map M(F). s ,F l )∈R 1×H×W The attention map is then compared with F. l Perform element-wise multiplication and add the resulting feature map to F. s The fused feature map F′ is obtained. Batch normalization (BN) and SILU activation function are applied after the convolutional layer to accelerate convergence, improve model performance, and enhance its stability. The attention-based fusion module can be expressed as follows:

[0112]

[0113] F c =concatenate(f 1×1 (F s ),f 1×1 (F l ))

[0114]

[0115] Where F′ represents the fused feature map, F s Let F represent the first feature map. l Denotes the second feature map, M(F) s ,F l) represents an attention graph, concatenate represents concatenation, and f 1×1 This represents a 1×1 convolution operation. This represents a 3×3 dilated convolution operation. F represents element-wise multiplication. c This represents the concatenated feature map. In this embodiment, dilated convolution is used to increase the receptive field because research shows that dilated convolution is more effective at extracting contextual information than ordinary convolution.

[0116] The fused feature map is then fed into subsequent modules for processing to achieve downstream tasks such as classification, detection, and tracking based on multimodal visual information.

[0117] Because it is necessary to calibrate the intrinsic and extrinsic parameter matrices of the event camera and the frame camera, and the relative positions between the cameras and the camera lenses cannot be changed after calibration, this embodiment also uses frame camera images to calibrate the images, and then the specific steps for transferring the labels to the event camera images include:

[0118] First, the event stream needs to be segmented at the middle of the frame exposure time using a hardware time synchronization scheme, and the segmented event stream needs to be reconstructed using the event time image method to obtain the event frame image.

[0119] Secondly, the spatial transformation matrix was calculated using the checkerboard pattern and the Kalibr toolbox to estimate the camera's intrinsic and extrinsic parameters.

[0120] Finally, the RGB images (i.e., frame camera images) are calibrated, and the obtained labels are transformed onto the event camera images using a spatial transformation matrix, thereby providing labels for the event stream data.

[0121] In this embodiment, the purpose of fusing frame cameras and event cameras is to leverage their respective advantages and characteristics. First, frame cameras provide high-resolution and color-rich image information, while event cameras offer high temporal resolution and sensitivity to rapid changes. Fusion allows the system to simultaneously possess the advantages of high resolution and high temporal resolution, achieving more comprehensive and accurate scene perception and analysis. Second, event cameras trigger events only when pixel-level changes occur. Compared to the continuous image acquisition of frame cameras, this significantly reduces data volume and processing burden, improving processing efficiency and real-time performance. Finally, event cameras have advantages in analyzing fast-moving and high dynamic range scenes; fusing them with frame cameras can provide more accurate and complete analysis results in dynamic scenes.

[0122] Reference Figure 5 This invention also provides an apparatus for fusing event camera images and frame camera images. This apparatus includes a signal triggering unit, a video processing unit, and a CPU core unit, wherein:

[0123] The signal triggering unit is used to trigger the synchronous acquisition signals of the event camera and the frame camera;

[0124] The video processing unit is used to acquire event streams through the event camera and frame camera images through the frame camera under synchronous acquisition signals.

[0125] The CPU core unit is used to segment the event stream according to the timestamp of the synchronization trigger signal to obtain segmented event streams; wherein the segmented event streams are time-aligned with the frame camera images; and the segmented event streams are processed using a constructed event time image method to obtain event camera images; wherein the event time image method is used to preserve the time information and polarity information of the segmented event streams; and the event camera images and frame camera images are fused to obtain fused images.

[0126] In this embodiment, the event camera and frame camera are synchronized by a signal triggering unit, and the video processing unit acquires images under the synchronized acquisition signal, so that the event camera and frame camera are synchronized in time, which can ensure that the time stamp of the event stream and the actual imaging time of the frame image are minimized in time. The segmented event stream is processed by the CPU core unit using the constructed event time image method, which can retain the time information and polarity information of the segmented event stream and reduce the loss of event information. Then, the event camera image and frame camera image are fused, which can improve the fusion effect of the event camera image and frame camera image, and achieve stable imaging effect in low light, high dynamic range or fast motion scenes, thereby improving scene understanding and image processing capabilities.

[0127] In some embodiments, the apparatus for fusing event camera images and frame camera images further includes a power management unit, a dataset interaction unit, and an interface expansion unit connected to the CPU core unit, wherein:

[0128] The power management unit is used to power the CPU core unit, event camera, and frame camera.

[0129] Data set interaction unit, used by the CPU core unit for wired and wireless communication;

[0130] The interface expansion unit is used for data transmission and data storage of the CPU core unit.

[0131] In this embodiment, taking into account both practicality and safety, a small portable power supply is used, and the edge computing host can use the NVIDIA NX2 core board.

[0132] To facilitate understanding by those skilled in the art, a set of preferred embodiments is provided below:

[0133] This embodiment includes a fixed platform on which the event camera, frame camera, power supply, and edge computing host are uniformly fixed. (Refer to...) Figure 6 The system includes two cameras: an event camera and a frame camera. The PCBA consists of assembled circuit boards. The imaging device's housing (top and bottom covers) is made of aluminum alloy. The lenses are mounted as close as possible to each other at the same vertical height to minimize parallax between the event and frame cameras. Considering both practicality and security, a small, portable power supply is used, and the edge computing host can utilize an NVIDIA NX2 core board.

[0134] This embodiment specifically includes a CPU core unit (i.e., an edge computing host), a power management unit, a data set interaction unit, an interface expansion unit, a video processing unit, a frame camera, and an event camera. The CPU core unit is primarily used for processing event streams and RGB images, and performing downstream tasks; the power management unit provides power to the CPU and camera; the data interaction unit handles wired and wireless communication; the interface expansion unit is used for data transmission and storage; and the video processing unit is connected to the frame camera and the event camera.

[0135] This embodiment includes an event camera, a frame camera, an edge computing host, and a multimodal visual fusion module. The specific operations are as follows:

[0136] Because event cameras are sensitive to dynamic information, they can capture details in fast-moving and high dynamic range scenes, generating asynchronous event streams. Event cameras are used to collect dynamic information from the scene, resulting in asynchronous event streams, which are then sent to the edge computing host.

[0137] A frame camera can capture high-resolution and rich-color image information of a scene. In this embodiment, a frame camera is used to capture RGB images of the scene (i.e., frame camera images) and send them to the edge computing host.

[0138] The edge computing host segments the asynchronous event stream according to the timestamp of the synchronization trigger signal, thereby aligning it with the RGB image in time, and processes the asynchronous event stream to obtain the event time image (i.e., the event camera image).

[0139] The multimodal visual fusion module is used to fuse time-aligned event time images and RGB images to output the result image.

[0140] It's important to note that RGB frame cameras are traditional color cameras that capture static information of a scene by continuously acquiring complete image frames at fixed time intervals. They provide high resolution and rich color information, making them suitable for most computer vision tasks, such as image classification, object detection, and image recognition. Because of the continuous acquisition of image frames, RGB cameras can capture details of static scenes, but they may exhibit blurring or overexposure for fast-moving objects or scenes with high dynamic range.

[0141] Event cameras, as a novel type of bio-vision inspired sensor, mimic the way the biological retina perceives changes in illumination during their imaging process. Each pixel independently and asynchronously detects logarithmic changes in brightness. When the cumulative brightness change of a pixel reaches a certain threshold, an event is output. Event cameras possess unique asynchronous response characteristics, enabling them to exhibit superior scene structure and visual information capture capabilities even in extreme lighting conditions. First, the output of an event camera depends only on the relative change in brightness, unaffected by the absolute value of brightness. Second, thanks to its logarithmic intensity response, it has a wide dynamic range, reliably capturing dynamic information even in extremely low or high illumination conditions. Finally, the event stream has no minimum time unit, unlike traditional cameras that output data at fixed intervals, thus exhibiting low latency.

[0142] This embodiment combines the advantages and characteristics of both event cameras and frame cameras. It utilizes frame cameras to capture high-resolution texture details and rich color information in static scenes, and utilizes event cameras to capture motion and contour information in dynamic scenes, thereby improving the system's perception capabilities for fast-moving and high dynamic range scenes.

[0143] The purpose of this embodiment is to combine the advantages and characteristics of event cameras and frame cameras to achieve reliable imaging results in various complex visual scenes, including dynamic and low-light conditions. Frame cameras are used to image objects and backgrounds under static and normal lighting conditions, obtaining high-resolution texture details and color information. Then, the high temporal resolution and high dynamic range of event cameras are utilized to image objects in excessively bright or dark environments and on rapidly moving objects. Event frame images are obtained by processing the event stream using event temporal imaging methods. Subsequently, relevant computer vision algorithms are used to process and fuse the data from the two modalities.

[0144] In this embodiment, by configuring two different imaging devices—an event camera and a frame camera—and fully utilizing their respective characteristics and advantages, along with a visual fusion algorithm, the imaging system achieves versatility and adaptability. By fusing the event camera and the frame camera, the imaging system of this embodiment can achieve reliable imaging results in various extreme visual scenarios. Whether in nighttime, bright light, high-speed motion, foggy weather, or sudden changes in brightness, the system can provide stable and clear image information. The complementary characteristics and advantages of the two cameras enable the system to comprehensively capture and present multiple visual features in a scene, thereby improving the perception and understanding of complex scenes.

[0145] This invention also provides a computer-readable storage medium storing computer-executable instructions. These instructions, when executed by one or more control processors, cause the processors to perform a method for fusing event camera images and frame camera images as described in the above-described method embodiments. For example, they can execute the above-described method. Figure 1 The functions of steps S100 to S500 in the method.

[0146] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0147] The above is a detailed description of the preferred embodiments of this application. However, the embodiments of this application are not limited to the above-described implementation methods. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the embodiments of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of the embodiments of this application.

Claims

1. A method for fusing event camera images and frame camera images, characterized in that, The method for fusing event camera images and frame camera images includes: The synchronous acquisition signals of the event camera and the frame camera are triggered, including: A first chip is used to design the trigger signal, and a second chip is used to control the phase shift of the trigger signal to generate a synchronous acquisition signal for the event camera and the frame camera. The trigger signal is phase-shifted in the following way to align the trigger signal of the event camera with the exposure midpoint of the frame camera: Obtain the start exposure time and the middle exposure time of the frame camera; Based on the start exposure time and the midpoint of the exposure, the required phase shift length of the trigger signal is calculated as follows: in, Indicates the start time of the frame camera's exposure. Indicates the midpoint of the exposure. Indicates the length of phase shift required for the trigger signal; Under the synchronous acquisition signal, the event camera acquires the event stream and the frame camera acquires the frame camera images; The event stream is segmented according to the timestamp of the synchronous acquisition signal to obtain a segmented event stream; wherein the segmented event stream is time-aligned with the frame camera image; The segmented event stream is processed using a constructed event temporal image method to obtain event camera images; wherein, the event temporal image method is used to preserve the temporal and polarity information of the segmented event stream; Feature fusion is performed on the event camera image and the frame camera image to obtain a fused feature map.

2. The method for fusing event camera images and frame camera images according to claim 1, characterized in that, The event time image method is constructed as follows: in, Indicate each event according to Classified to the The time period Indicates the number of events. Represents a timestamp. Indicates the first The timestamp of each event Indicates the number of time periods. This represents the cumulative sum of the polarities of events occurring within a single pixel over a period of time. Represents pixel coordinates, This represents the activation function. Represents the Dirac function, This represents the conversion of an event into a tensor representation. Indicates the polarity of an event.

3. The method for fusing event camera images and frame camera images according to claim 1, characterized in that, The step of fusing features based on the event camera image and the frame camera image to obtain a fused feature map includes: The event camera image and the frame camera image are registered using an image registration method to obtain the registered event camera image and frame camera image; A feature extraction network is used to extract features from the registered event camera image and the frame camera image to obtain a first feature map of the registered event camera image and a second feature map of the registered frame camera image. The first feature map and the second feature map are transmitted to the constructed attention-based fusion module for feature fusion to obtain a fused feature map.

4. The method for fusing event camera images and frame camera images according to claim 3, characterized in that, The attention-based fusion module is constructed as follows: in, Represents the fused feature map. Represents the first feature map. This represents the second feature map. Represents an attention map. Indicates series connection. express Convolution operation, express The dilated convolution operation, This indicates element-wise multiplication. This represents the feature map after concatenation.

5. The method for fusing event camera images and frame camera images according to claim 1, characterized in that, After processing the segmented event stream using the constructed event-temporal image method, the method for fusing event camera images and frame camera images further includes: The intrinsic and extrinsic parameters of the event camera and the frame camera are estimated using the checkerboard and Kalibr calibration methods to obtain the spatial transformation matrix; The frame camera images are calibrated to obtain labels; The spatial transformation matrix is ​​used to transfer the labels onto the event camera image, thereby providing labels for the event camera image.

6. An apparatus for fusing event camera images and frame camera images, characterized in that, The apparatus for fusing event camera images and frame camera images includes: A signal triggering unit, used to trigger the synchronous acquisition signals of the event camera and the frame camera, includes: A first chip is used to design the trigger signal, and a second chip is used to control the phase shift of the trigger signal to generate a synchronous acquisition signal for the event camera and the frame camera. The trigger signal is phase-shifted in the following way to align the trigger signal of the event camera with the exposure midpoint of the frame camera: Obtain the start exposure time and the middle exposure time of the frame camera; Based on the start exposure time and the midpoint of the exposure, the required phase shift length of the trigger signal is calculated as follows: in, Indicates the start time of the frame camera's exposure. Indicates the midpoint of the exposure. Indicates the length of phase shift required for the trigger signal; The video processing unit is used to acquire event streams through the event camera and frame camera images through the frame camera under the synchronous acquisition signal. The CPU core unit is used to segment the event stream according to the timestamp of the synchronous acquisition signal to obtain a segmented event stream; wherein the segmented event stream is time-aligned with the frame camera image; and to process the segmented event stream using a constructed event time image method to obtain an event camera image; wherein the event time image method is used to retain the time information and polarity information of the segmented event stream; and to perform feature fusion based on the event camera image and the frame camera image to obtain a fused feature map.

7. The apparatus for fusing event camera images and frame camera images according to claim 6, characterized in that, The device for fusing event camera images and frame camera images further includes a power management unit, a dataset interaction unit, and an interface expansion unit connected to the CPU core unit, wherein: The power management unit is used to supply power to the CPU core unit, the event camera, and the frame camera. The dataset interaction unit is used by the CPU core unit for wired and wireless communication. The interface expansion unit is used for data transmission and data storage of the CPU core unit.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the method of fusing event camera images and frame camera images as described in any one of claims 1 to 4.

Images