System and method for removing image reflection flare based on event camera

Through an event camera-based system, the problem of difficult to remove reflected flares in traditional camera images is solved, and high-quality image removal effect is achieved.

CN119583980AActive Publication Date: 2025-03-07AEROSPACE SCI & IND GRP INTELLIGENT TECH RES INST CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411566494.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-11-05
Publication Date
2025-03-07
Estimated Expiration
2044-11-05

AI Technical Summary

Technical Problem

The prior art is difficult to effectively remove reflective flares in traditional camera images, especially in complex backgrounds.

Method used

The image reflection flare removal system based on the event camera is adopted, and the backlight conditions are provided through the light source. The dual-mode camera is used to collect image frames and event streams. The data processing unit denoised and reconstructed the event streams to obtain event pseudo-frames, and fused and restored the image frames and event pseudo-frames to realize the removal of image frame reflection flares.

Benefits of technology

Effectively remove reflected flares in traditional camera images, restore image details in flare areas, improve image quality, and improve adaptability to flare modes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119583980A_ABST
    Figure CN119583980A_ABST
Patent Text Reader

Abstract

The invention provides an event camera-based image reflection flare removal system and method, and the system comprises a light source which is used for providing a backlight shooting condition; the dual-mode camera has an image frame mode and an event mode and is used for collecting image frames and event streams under a backlight condition; the data processing unit is used for receiving the image frame and the event stream, and denoising and reconstructing the event stream to obtain an event pseudo frame; and the fusion module is also used for fusing and restoring the image frame and the event pseudo frame to obtain a restored flare-removed image and the estimated flare so as to remove the reflection flare of the image frame. The technical problem that the reflection flare in the traditional camera image cannot be removed in the prior art can be solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a system and method for removing image reflection flare based on an event camera. Background Art

[0002] When a light source enters the field of view of a traditional camera, the light is reflected multiple times within the lens. As the reflected light accumulates during the camera exposure time, bright spots or "ghosting effects" appear in the image, which is commonly referred to as reflection flare. Current methods for removing image flare mainly rely on feature detection techniques to identify the flare area and reconstruct it. However, these methods are often limited to specific flare patterns and have difficulty detecting and removing flare in complex backgrounds.

[0003] An event camera is a sensor designed based on bionic vision that can asynchronously trigger event signals based on the brightness changes of each pixel. When the logarithmic change in brightness exceeds a certain threshold, an event is triggered. Under backlit conditions at night, a stationary event camera can capture the illumination fluctuations of the light source while avoiding the appearance of reflection flares. Because reflection flares involve energy attenuation caused by repeated reflections, the intensity fluctuations in these flare areas are often too small to trigger events. Therefore, event cameras have good application potential for removing reflection flares from images of traditional cameras.

[0004] Currently, there is a lack of effective methods for removing image reflection flares based on event cameras in both scientific research and application fields. Summary of the invention

[0005] The present invention provides a system and method for removing image reflection flare based on an event camera, which can solve the technical problem in the prior art that the reflection flare in the traditional camera image cannot be removed.

[0006] According to one aspect of the present invention, a system for removing image reflection flare based on an event camera is provided, the system comprising:

[0007] Light source, used to provide conditions for backlit shooting;

[0008] Dual-mode camera with image frame mode and event mode for capturing image frames and event streams in backlit conditions;

[0009] The data processing unit is used to receive image frames and event streams, and to denoise and reconstruct the event streams to obtain event pseudo frames; it is also used to fuse and restore the image frames with the event pseudo frames to obtain a restored image with flare removed and an estimated flare, so as to remove the reflected flare from the image frames.

[0010] Preferably, the data processing unit includes a preprocessing module, an event-guided fusion module and a multi-stage restoration module;

[0011] The preprocessing module is used to perform denoising on the event stream to obtain a denoised event stream; and is also used to perform time dimension compression on the denoised event stream to obtain an event pseudo frame;

[0012] The event-guided fusion module is used to fuse the image frame with the event pseudo frame to obtain a spliced ​​feature map;

[0013] The multi-stage restoration module is used to restore the spliced ​​feature map to obtain a restored image with flare removed and an estimated flare.

[0014] Preferably, the event-guided fusion module includes three convolutional layers, an activation layer, an operator module and a splicing submodule;

[0015] The first convolutional layer is used to convolve the image frame to obtain the image frame feature map;

[0016] The second convolutional layer is used to convolve the event pseudo frame rotated 180° to obtain the event feature map;

[0017] The activation layer is used to activate the event feature map to obtain the activated event feature map;

[0018] The operation submodule is used to perform a bitwise multiplication operation on the image frame feature map and the activated event feature map to obtain a multiplied feature map;

[0019] The third convolutional layer is used to convolve the multiplied feature map to obtain a fused feature map;

[0020] The splicing submodule is used to splice the image frame and the fused feature map to obtain a spliced ​​feature map.

[0021] Preferably, the multi-stage restoration module includes an encoder sub-network and a decoder sub-network;

[0022] The encoder sub-network is used to encode the concatenated feature map to obtain an encoded feature map;

[0023] The decoder subnetwork is used to decode the encoded feature map to obtain a restored flare-removed image and an estimated flare.

[0024] Preferably, the event stream includes coordinate information of the triggered pixel, time information and polarity information of the event.

[0025] According to another aspect of the present invention, a method for removing image reflection flare based on an event camera is provided, the method comprising:

[0026] When the illumination is less than the preset brightness, turn on the light source to provide conditions for backlit shooting;

[0027] The dual-mode camera captures image frames and event streams under backlight conditions;

[0028] The data processing unit receives the image frame and the event stream, and performs denoising and reconstruction on the event stream to obtain an event pseudo frame;

[0029] The data processing unit fuses and restores the image frame and the event pseudo frame to obtain a restored image with flare removed and an estimated flare, so as to remove the reflected flare from the image frame.

[0030] Preferably, the preset brightness is 400 lx.

[0031] Preferably, the data processing unit receives the image frame and the event stream, and denoises and reconstructs the event stream to obtain an event pseudo frame; the data processing unit fuses and restores the image frame and the event pseudo frame to obtain a restored image after flare removal and an estimated flare, including:

[0032] The preprocessing module in the data processing unit receives the image frame and the event stream, and uses the spatiotemporal correlation discrimination algorithm to denoise the event stream to obtain the denoised event stream; and uses the time dimension compression algorithm to reconstruct the denoised event stream to obtain an event pseudo frame;

[0033] The event-guided fusion module in the data processing unit fuses the image frame with the event pseudo frame to obtain a spliced ​​feature map;

[0034] The multi-stage restoration module in the data processing unit restores the spliced ​​feature maps to obtain a restored image with flare removed and an estimated flare.

[0035] Preferably, a spatiotemporal correlation discrimination algorithm is used to denoise the event stream, and the denoised event stream includes:

[0036] Count the number of events corresponding to each pixel in the event stream within a fixed period; when the number of events exceeds a preset threshold, delete the event corresponding to the current pixel to obtain an event stream with thermal noise eliminated;

[0037] On the basis of obtaining the event stream for eliminating thermal noise, when the time interval between adjacent events of the same pixel is less than a preset time threshold, the latter event is deleted to obtain the event stream for eliminating refractory period noise;

[0038] On the basis of obtaining the event stream with refractory period noise eliminated, when the spatial interval between adjacent events is greater than a preset interval threshold, the adjacent events are deleted to remove the background noise, thereby obtaining the denoised event stream.

[0039] Preferably, the denoised event stream is reconstructed using a time dimension compression algorithm to obtain an event pseudo frame, which includes: normalizing the number of denoised events on each pixel in the denoised event stream to obtain the event pseudo frame.

[0040] Preferably, the event-guided fusion module in the data processing unit fuses the image frame with the event pseudo frame to obtain a spliced ​​feature map including:

[0041] The first convolutional layer in the event-guided fusion module is used to convolve the image frame to obtain the image frame feature map;

[0042] The second convolutional layer in the event-guided fusion module is used to convolve the event pseudo-frame rotated 180° to obtain the event feature map;

[0043] The activation layer in the event-guided fusion module activates the event feature map to obtain an activated event feature map;

[0044] The operation submodule in the event-guided fusion module performs a bitwise multiplication operation on the image frame feature map and the activated event feature map to obtain a multiplied feature map;

[0045] The third convolutional layer in the event-guided fusion module convolves the multiplied feature map to obtain a fused feature map;

[0046] The splicing submodule in the event-guided fusion module splices the image frame and the fused feature map to obtain a spliced ​​feature map.

[0047] Preferably, the multi-stage restoration module in the data processing unit restores the spliced ​​feature map to obtain a restored image with flare removed and an estimated flare, including:

[0048] The encoder sub-network in the multi-stage restoration module is used to encode the concatenated feature map to obtain an encoded feature map;

[0049] The decoder subnetwork in the multi-stage restoration module is used to decode the encoded feature map to obtain the restored flare-removed image and the estimated flare.

[0050] Preferably, the event stream includes coordinate information of the triggered pixel, time information and polarity information of the event.

[0051] By applying the technical solution of the present invention, an image frame and an event stream are received by a data processing unit, and the event stream is denoised and reconstructed to obtain an event pseudo frame; the image frame and the event pseudo frame are fused and restored to obtain a restored image with flare removed and an estimated flare, so as to remove the reflected flare of the image frame. The present invention can remove the reflected flare in traditional camera images, restore the image details of the flare area, improve the image quality, and improve the adaptability to the flare mode. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The included drawings are used to provide a further understanding of the embodiments of the present invention, which constitute a part of the specification, are used to illustrate the embodiments of the present invention, and together with the text description, explain the principles of the present invention. Obviously, the drawings in the following description are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0053] Figure 1 A schematic diagram of the structure of a system for removing image reflection flare based on an event camera according to an embodiment of the present invention is shown;

[0054] Figure 2 Shows Figure 1 Schematic diagram of the structure of the event-guided fusion module and the multi-stage reduction module;

[0055] Figure 3 A flow chart of a method for removing image reflection flare based on an event camera according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0056] It should be noted that, in the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other. The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0058] Unless otherwise specifically stated, the relative arrangement of the parts and steps described in these embodiments, numerical expressions and numerical values ​​do not limit the scope of the present invention. At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to the actual proportional relationship. The technology, method and equipment known to ordinary technicians in the relevant field may not be discussed in detail, but in appropriate cases, the technology, method and equipment should be regarded as a part of the authorization specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once a certain item is defined in an accompanying drawing, it does not need to be further discussed in subsequent drawings.

[0059] like Figure 1 As shown, the present invention provides a system for removing image reflection flare based on an event camera, the system comprising:

[0060] Light source, used to provide conditions for backlit shooting;

[0061] Dual-mode camera with image frame mode and event mode for capturing image frames and event streams in backlit conditions;

[0062] The data processing unit is used to receive image frames and event streams, and to denoise and reconstruct the event streams to obtain event pseudo frames; it is also used to fuse and restore the image frames with the event pseudo frames to obtain a restored image with flare removed and an estimated flare, so as to remove the reflected flare from the image frames.

[0063] The present invention receives image frames and event streams through a data processing unit, and performs denoising and reconstruction on the event streams to obtain event pseudo frames; the image frames and the event pseudo frames are fused and restored to obtain restored images with flare removed and estimated flare, so as to remove the reflected flare from the image frames. The present invention can remove the reflected flare in traditional camera images, restore the image details in the flare area, improve the image quality, and improve the adaptability to the flare mode.

[0064] According to an embodiment of the present invention, the data processing unit includes a preprocessing module, an event-guided fusion module and a multi-stage restoration module;

[0065] The preprocessing module is used to perform denoising on the event stream to obtain a denoised event stream; and is also used to perform time dimension compression on the denoised event stream to obtain an event pseudo frame;

[0066] The event-guided fusion module is used to fuse the image frame with the event pseudo frame to obtain a spliced ​​feature map;

[0067] The multi-stage restoration module is used to restore the spliced ​​feature map to obtain a restored image with flare removed and an estimated flare.

[0068] According to an embodiment of the present invention, Figure 2 As shown, the event-guided fusion module includes three convolutional layers, an activation layer, an operator module and a splicing submodule;

[0069] The first convolutional layer is used to convolve the image frame to obtain the image frame feature map;

[0070] The second convolutional layer is used to convolve the event pseudo frame rotated 180° to obtain the event feature map;

[0071] The activation layer is used to activate the event feature map to obtain the activated event feature map; wherein the activation layer can be a sigmoid activation layer;

[0072] The operation submodule is used to perform a bitwise multiplication operation on the image frame feature map and the activated event feature map to obtain a multiplied feature map;

[0073] The third convolutional layer is used to convolve the multiplied feature map to obtain a fused feature map;

[0074] The splicing submodule is used to splice the image frame and the fused feature map to obtain a spliced ​​feature map.

[0075] According to an embodiment of the present invention, Figure 2 As shown, the multi-stage restoration module includes an encoder sub-network and a decoder sub-network;

[0076] The encoder sub-network is used to encode the concatenated feature map to obtain an encoded feature map;

[0077] The decoder subnetwork is used to decode the encoded feature map to obtain a restored flare-removed image and an estimated flare.

[0078] According to an embodiment of the present invention, the event stream includes coordinate information of the triggered pixel, time information and polarity information of the event.

[0079] In this embodiment, the first two stages of the multi-stage restoration module capture extensive contextual information through their wide receptive fields, and the last stage directly operates on the receptive field of the original input image resolution, preserving the fine texture details of the output image.

[0080] like Figure 3 As shown, the present invention also provides a method for removing image reflection flare based on an event camera, the method comprising:

[0081] When the illumination is less than the preset brightness, turn on the light source to provide conditions for backlit shooting;

[0082] The dual-mode camera captures image frames and event streams under backlight conditions;

[0083] The data processing unit receives the image frame and the event stream, and performs denoising and reconstruction on the event stream to obtain an event pseudo frame;

[0084] The data processing unit fuses and restores the image frame and the event pseudo frame to obtain a restored image with flare removed and an estimated flare, so as to remove the reflected flare from the image frame.

[0085] According to an embodiment of the present invention, the preset brightness is 400 lx.

[0086] According to an embodiment of the present invention, a data processing unit receives an image frame and an event stream, and denoises and reconstructs the event stream to obtain an event pseudo frame; the data processing unit fuses and restores the image frame and the event pseudo frame to obtain a restored image after flare removal and an estimated flare, including:

[0087] The preprocessing module in the data processing unit receives the image frame and the event stream, and uses the spatiotemporal correlation discrimination algorithm to denoise the event stream to obtain the denoised event stream; and uses the time dimension compression algorithm to reconstruct the denoised event stream to obtain an event pseudo frame;

[0088] The event-guided fusion module in the data processing unit fuses the image frame with the event pseudo frame to obtain a spliced ​​feature map;

[0089] The multi-stage restoration module in the data processing unit restores the spliced ​​feature maps to obtain a restored image with flare removed and an estimated flare.

[0090] According to an embodiment of the present invention, a spatiotemporal correlation discrimination algorithm is used to perform denoising on an event stream, and the denoised event stream includes:

[0091] Count the number of events corresponding to each pixel in the event stream within a fixed period; when the number of events exceeds a preset threshold, delete the event corresponding to the current pixel to obtain an event stream with thermal noise eliminated;

[0092] On the basis of obtaining the event stream for eliminating thermal noise, when the time interval between adjacent events of the same pixel is less than a preset time threshold, the latter event is deleted to obtain the event stream for eliminating refractory period noise;

[0093] On the basis of obtaining the event stream with refractory period noise eliminated, when the spatial interval between adjacent events is greater than a preset interval threshold, the adjacent events are deleted to remove background noise and obtain a denoised event stream, wherein the spatial interval size can be set to 1 pixel.

[0094] According to an embodiment of the present invention, a time dimension compression algorithm is used to reconstruct the denoised event stream to obtain an event pseudo frame, which includes: normalizing the number of denoised events on each pixel in the denoised event stream to obtain the event pseudo frame.

[0095] According to an embodiment of the present invention, the event-guided fusion module in the data processing unit fuses the image frame with the event pseudo frame to obtain a spliced ​​feature map including:

[0096] The first convolutional layer in the event-guided fusion module is used to convolve the image frame to obtain the image frame feature map;

[0097] The second convolutional layer in the event-guided fusion module is used to convolve the event pseudo-frame rotated 180° to obtain the event feature map;

[0098] The activation layer in the event-guided fusion module activates the event feature map to obtain an activated event feature map;

[0099] The operation submodule in the event-guided fusion module performs a bitwise multiplication operation on the image frame feature map and the activated event feature map to obtain a multiplied feature map;

[0100] The third convolutional layer in the event-guided fusion module convolves the multiplied feature map to obtain a fused feature map;

[0101] The splicing submodule in the event-guided fusion module splices the image frame and the fused feature map to obtain a spliced ​​feature map.

[0102] According to an embodiment of the present invention, the multi-stage restoration module in the data processing unit restores the spliced ​​feature map to obtain a restored image after flare removal and an estimated flare, including:

[0103] The encoder sub-network in the multi-stage restoration module is used to encode the concatenated feature map to obtain an encoded feature map;

[0104] The decoder subnetwork in the multi-stage restoration module is used to decode the encoded feature map to obtain the restored flare-removed image and the estimated flare.

[0105] According to an embodiment of the present invention, the event stream includes coordinate information of the triggered pixel, time information and polarity information of the event.

[0106] In order to further understand the present invention, the method for removing image reflection flare based on an event camera of the present invention is described in detail below, which specifically includes the following steps:

[0107] S1. In darker indoor or outdoor conditions at night (i.e., illumination less than 400lx), configure the light source and turn on the light source switch;

[0108] S2. Install the dual-mode event camera, fix the dual-mode event camera on a tripod, adjust the dual-mode event camera lens so that the light source appears within the camera field of view, and record the dual-mode dynamic vision sensor image frame and event stream data;

[0109] S3. Remove the noise of the original event stream through the spatiotemporal correlation discrimination algorithm; reconstruct the event pseudo frame through the time dimension compression algorithm; obtain the restored image frame by rotating the event pseudo frame 180° and fusing it with the image frame to remove the reflection flare of the image frame.

[0110] In summary, the present invention provides a system and method for removing image reflection flare based on an event camera, wherein an image frame and an event stream are received by a data processing unit, and the event stream is denoised and reconstructed to obtain an event pseudo frame; the image frame and the event pseudo frame are fused and restored to obtain a restored image with flare removed and an estimated flare, thereby removing the image frame reflection flare. The present invention can remove reflection flare in traditional camera images, restore image details in the flare area, improve image quality, and improve adaptability to flare patterns.

[0111] For ease of description, spatially relative terms such as "above", "above", "on the upper surface of", "above", etc. may be used here to describe the spatial positional relationship between a device or feature and other devices or features as shown in the figure. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figure. For example, if the device in the accompanying drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "below other devices or structures". Thus, the exemplary term "above" can include both "above" and "below". The device can also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatially relative descriptions used here are interpreted accordingly.

[0112] In addition, it should be noted that the use of terms such as "first" and "second" to limit components is only for the convenience of distinguishing the corresponding components. If not otherwise stated, the above terms have no special meaning and therefore cannot be understood as limiting the scope of protection of the present invention.

[0113] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A system for removing image reflection flare based on an event camera, characterized in that: The system comprises: Light source, used to provide conditions for backlit shooting; Dual-mode camera with image frame mode and event mode for capturing image frames and event streams in backlit conditions; The data processing unit is used to receive image frames and event streams, and to denoise and reconstruct the event streams to obtain event pseudo frames; it is also used to fuse and restore the image frames with the event pseudo frames to obtain a restored image with flare removed and an estimated flare, so as to remove the reflected flare from the image frames.

2. The system according to claim 1, characterized in that The data processing unit includes a preprocessing module, an event-guided fusion module and a multi-stage restoration module; The preprocessing module is used to perform denoising on the event stream to obtain a denoised event stream; and is also used to perform time dimension compression on the denoised event stream to obtain an event pseudo frame; The event-guided fusion module is used to fuse the image frame with the event pseudo frame to obtain a spliced ​​feature map; The multi-stage restoration module is used to restore the spliced ​​feature map to obtain a restored image with flare removed and an estimated flare.

3. The system according to claim 1, characterized in that The event-guided fusion module includes three convolutional layers, an activation layer, an operator module, and a splicing submodule; The first convolutional layer is used to convolve the image frame to obtain the image frame feature map; The second convolutional layer is used to convolve the event pseudo frame rotated 180° to obtain the event feature map; The activation layer is used to activate the event feature map to obtain the activated event feature map; The operation submodule is used to perform a bitwise multiplication operation on the image frame feature map and the activated event feature map to obtain a multiplied feature map; The third convolutional layer is used to convolve the multiplied feature map to obtain a fused feature map; The splicing submodule is used to splice the image frame and the fused feature map to obtain a spliced ​​feature map.

4. The system according to claim 1, characterized in that The multi-stage restoration module includes an encoder sub-network and a decoder sub-network; The encoder sub-network is used to encode the concatenated feature map to obtain an encoded feature map; The decoder subnetwork is used to decode the encoded feature map to obtain a restored flare-removed image and an estimated flare.

5. The system according to claim 1, characterized in that The event stream includes coordinate information of the triggered pixel, time information and polarity information of the event.

6. A method for removing image reflection flare based on an event camera, characterized in that: The method comprises: When the illumination is less than the preset brightness, turn on the light source to provide conditions for backlit shooting; The dual-mode camera captures image frames and event streams under backlight conditions; The data processing unit receives the image frame and the event stream, and performs denoising and reconstruction on the event stream to obtain an event pseudo frame; The data processing unit fuses and restores the image frame and the event pseudo frame to obtain a restored image with flare removed and an estimated flare, so as to remove the reflected flare from the image frame.

7. The method according to claim 6, characterized in that The preset brightness is 400 lx.

8. The method according to claim 6, characterized in that The data processing unit receives the image frame and the event stream, and performs denoising and reconstruction on the event stream to obtain an event pseudo frame; The data processing unit fuses and restores the image frame and the event pseudo frame to obtain a restored image with flare removed and an estimated flare including: The preprocessing module in the data processing unit receives the image frame and the event stream, and uses the spatiotemporal correlation discrimination algorithm to denoise the event stream to obtain the denoised event stream; and uses the time dimension compression algorithm to reconstruct the denoised event stream to obtain an event pseudo frame; The event-guided fusion module in the data processing unit fuses the image frame with the event pseudo frame to obtain a spliced ​​feature map; The multi-stage restoration module in the data processing unit restores the spliced ​​feature maps to obtain a restored image with flare removed and an estimated flare.

9. The method according to claim 6, characterized in that The event stream is denoised using the spatiotemporal correlation discrimination algorithm. The denoised event stream includes: Count the number of events corresponding to each pixel in the event stream within a fixed period; when the number of events exceeds a preset threshold, delete the event corresponding to the current pixel to obtain an event stream with thermal noise eliminated; On the basis of obtaining the event stream for eliminating thermal noise, when the time interval between adjacent events of the same pixel is less than a preset time threshold, the latter event is deleted to obtain the event stream for eliminating refractory period noise; On the basis of obtaining the event stream with refractory period noise eliminated, when the spatial interval between adjacent events is greater than a preset interval threshold, the adjacent events are deleted to remove the background noise, thereby obtaining the denoised event stream.

10. The method according to claim 6, characterized in that The denoised event stream is reconstructed using a time dimension compression algorithm to obtain an event pseudo frame, including: normalizing the number of denoised events on each pixel in the denoised event stream to obtain the event pseudo frame.

11. The method according to claim 6, characterized in that The event-guided fusion module in the data processing unit fuses the image frame with the event pseudo frame to obtain a spliced ​​feature map including: The first convolutional layer in the event-guided fusion module is used to convolve the image frame to obtain the image frame feature map; The second convolutional layer in the event-guided fusion module is used to convolve the event pseudo-frame rotated 180° to obtain the event feature map; The activation layer in the event-guided fusion module activates the event feature map to obtain an activated event feature map; The operation submodule in the event-guided fusion module performs a bitwise multiplication operation on the image frame feature map and the activated event feature map to obtain a multiplied feature map; The third convolutional layer in the event-guided fusion module convolves the multiplied feature map to obtain a fused feature map; The splicing submodule in the event-guided fusion module splices the image frame and the fused feature map to obtain a spliced ​​feature map.

12. The method according to claim 6, characterized in that The multi-stage restoration module in the data processing unit restores the spliced ​​feature map to obtain a restored image with flare removed and an estimated flare including: The encoder sub-network in the multi-stage restoration module is used to encode the concatenated feature map to obtain an encoded feature map; The decoder subnetwork in the multi-stage restoration module is used to decode the encoded feature map to obtain the restored flare-removed image and the estimated flare.

13. The method according to claim 6, characterized in that The event stream includes coordinate information of the triggered pixel, time information and polarity information of the event.

Citation Information

Patent Citations

  • Dynamic visual event stream noise reduction method based on thermal pixel and enhanced spatial-temporal correlation

    CN115442544A

  • Image enhancement method, device and system

    CN115836318A

  • Non-uniform illumination image enhancement method based on event stream and color image fusion

    CN115908176A

  • Image deblurring method based on event guidance

    CN117726549A

  • Automatic compensation of lens flare

    US20170070689A1