Method and device for generating clear enhanced image by fusing images and events under low illumination

Abstract

The present invention relates to the technical fields of computer vision and image processing, and more particularly to a method and apparatus for fusing images and events to generate a clear and enhanced image under low illumination conditions. The method comprises: acquiring image data and event data under low illumination conditions; extracting multi-resolution image features from the image data and multi-resolution event features from the event data; cross-modally selecting new image features and new event features for the multi-resolution image features and multi-resolution event features at the same resolution scale; cross-scale fusing the new image features and new event features of different resolutions to obtain new comprehensive image features and new comprehensive event features, cross-modally fusing the comprehensive image features and comprehensive event features to obtain fused comprehensive features, and generating a clear and enhanced image based on the fused comprehensive features. This method solves the problems of blurred images from traditional cameras under low illumination conditions and low resolution and stroboscopic effects of event cameras in related technologies.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to a method and device for fusing images and events to generate a clear enhanced image under low illumination. Background Art

[0002] Acquiring high-quality images in low-light conditions is a long-standing challenge. Images captured in low-light conditions often suffer from various types of degradation, including poor visibility, noise, and color inaccuracies. Traditional RGB (Red, Green, Blue) cameras—which capture red, green, and blue color information to produce true-color images—struggle to capture sufficient detail in low-light environments, resulting in blurry images, low contrast, and loss of detail. To overcome these limitations, researchers have explored various image enhancement techniques to improve the visual quality of low-light images. Early methods, such as histogram equalization, adjusted the contrast of the image to improve overall visual quality. However, these methods rely solely on the histogram information of the input image and fail to fully consider the semantic information of the image content, resulting in limited enhancement results. Deep learning-based low-light image enhancement methods have made significant progress in improving image quality in low-light conditions. For example, convolutional neural networks are used for end-to-end mapping to directly convert low-light images into enhanced images. However, due to the low dynamic range and high latency of traditional cameras, these methods are prone to information loss in extreme low illumination and blur in motion scenes.

[0003] By asynchronously detecting brightness changes at each pixel, event cameras are able to capture a wider range of brightness, thereby preserving more details in low-light scenes. Since each pixel responds to changes independently and immediately, the events generated by event cameras have low latency and can effectively capture fast-moving targets. Therefore, the use of events can effectively alleviate the limitations of current low-light enhancement methods based on image frames. However, in low-light conditions, low-light enhancement methods that integrate event cameras also face some challenges. On the one hand, under night conditions, the output of event cameras is easily interfered by global stroboscopic light, which generates a lot of noise. On the other hand, in extremely dark conditions, the same voltage change requires more response time, which will produce tailing noise. In addition, the resolution of event cameras is generally lower than that of traditional cameras, which affects the generation of high-resolution images. Summary of the Invention

[0004] The present invention provides a method and device for fusing images and events to generate a clear enhanced image under low illumination, so as to solve the problems in the related art of blurred images of traditional cameras, low resolution of event cameras and stroboscopic phenomena.

[0005] The first aspect of the present invention provides a method for fusing images and events under low illumination to generate a clear enhanced image, comprising the following steps: acquiring image data and event data under low illumination conditions; extracting multi-resolution image features from the image data and multi-resolution event features from the event data, and for the multi-resolution image features and the multi-resolution event features, cross-modally selecting new image features and new event features at the same resolution scale; cross-scale fusing the new image features and new event features of different resolutions to obtain new comprehensive image features and new comprehensive event features, cross-modally fusing the comprehensive image features and comprehensive event features to obtain fused comprehensive features, and generating a clearly enhanced image of the image data based on the fused comprehensive features.

[0006] Optionally, extracting multi-resolution image features from image data and multi-resolution event features from event data includes: inputting the image data and event data into a multi-scale encoder, and the multi-scale encoder outputting multi-resolution image features and multi-resolution event features, wherein the multi-scale encoder includes an image encoder and an event encoder; the image encoder and the event encoder are provided with a processing path at a resolution layer, and the processing path includes a convolution layer and a convolutional neural network Swin-CNN module enhanced by a shifted window; the image encoder downsamples the image data to obtain multi-resolution image features, and the event encoder upsamples the event data to obtain multi-resolution event features.

[0007] Optionally, for multi-resolution image features and multi-resolution event features, new image features and new event features are selected cross-modally at the same resolution scale, including: inputting the multi-resolution image features and multi-resolution event features into a cross-modal feature selection module, and the cross-modal feature selection module outputs new image features and new event features, wherein the cross-modal feature selection module includes a down-projection layer and an up-projection layer, and the up-projection layer maps the features after down-projection by the down-projection layer back to the corresponding modal space.

[0008] Optionally, the lower projection layer includes a connection layer, a linear layer, a normalization layer, and a fusion layer, wherein the image features and event features are connected using the connection layer; the linear layer and the normalization layer perform feature dimensionality reduction on the connected features to obtain multi-source features, generate image prompts and event prompts based on the multi-source features, and obtain image source embeddings and event source embeddings; generate image source features based on image prompts, image source embeddings, and image features, and generate event source features based on event prompts, event source embeddings, and event features; fuse image source features and event source features to obtain fused features, generate new image features based on the fused features, image features, and image source embeddings, and generate new event features based on the fused features, event features, and event source embeddings.

[0009] Optionally, new image features and new event features of different resolutions are cross-scale fused respectively to obtain new comprehensive image features and new comprehensive event features, including: inputting new image features of different resolutions and new event features of different resolutions into a cross-scale feature fusion module, and the cross-scale feature fusion module outputs comprehensive image features and comprehensive event features, wherein the cross-scale feature fusion module includes a convolution layer and a Transformer block; at each resolution scale, local information and global information are extracted through the convolution layer and the Transformer block, features at different resolution scales are aligned through sampling operations, and features at different resolution scales are fused through the convolution layer.

[0010] Optionally, cross-modal fusion is performed on the comprehensive image features and the comprehensive event features to obtain fused comprehensive features, and a clearly enhanced image of the image data is generated based on the fused comprehensive features, including: inputting the comprehensive image features and the comprehensive event features into a cross-modal fusion decoder, and the cross-modal fusion decoder outputs a clearly enhanced image, wherein the cross-modal fusion decoder includes a convolution layer, an activation function and a Swin-CNN; the convolution layer and the activation function preprocess the comprehensive image features and the comprehensive event features, and the preprocessed features are input into a first processing channel and a second processing channel; the first processing channel obtains enhanced features through FFT transformation, Gaussian low-pass filter, FFC block and inverse FFT transformation, the second processing channel is processed through Swin-CNN block and convolution layer, and the features processed by the second processing channel are fused with the enhanced features to obtain fused comprehensive features; the fused comprehensive features are processed through Swin-CNN and convolution layer to obtain a clearly enhanced image of the image data.

[0011] Optionally, a loss function is constructed and used to guide the training of the model, wherein the loss function includes frequency loss, reconstruction loss and total loss. The frequency loss is determined by calculating the norm in the frequency domain, the reconstruction loss is determined by combining the pixel-level error and the structural similarity index, and the frequency loss and reconstruction loss are weighted to determine the total loss.

[0012] The second aspect of the present invention provides an apparatus for fusing images and events under low illumination to generate a clear enhanced image, comprising: an acquisition module for acquiring image data and event data under low illumination conditions; an extraction module for extracting multi-resolution image features in the image data and multi-resolution event features in the event data, and for multi-resolution image features and multi-resolution event features, cross-modally selecting new image features and new event features at the same resolution scale; a fusion module for performing cross-scale fusing of new image features and new event features of different resolutions to obtain new comprehensive image features and new comprehensive event features, performing cross-modal fusing of the comprehensive image features and comprehensive event features to obtain fused comprehensive features, and generating a clearly enhanced image of the image data based on the fused comprehensive features.

[0013] A third aspect of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein the processor executes the program to implement a method for fusing images and events under low illumination to generate a clear enhanced image as in the above-mentioned embodiment.

[0014] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the method for generating a clear enhanced image by fusing images and events under low illumination as described in the above embodiment.

[0015] Thus, the present invention includes the following beneficial effects:

[0016] The embodiment of the present invention obtains image data and event data under low illumination conditions; extracts multi-resolution image features from the image data and multi-resolution event features from the event data; cross-modally selects new image features and new event features for the multi-resolution image features and multi-resolution event features at the same resolution scale; cross-scale fuses the new image features and new event features of different resolutions to obtain new comprehensive image features and new comprehensive event features; cross-modally fuses the comprehensive image features and comprehensive event features to obtain fused comprehensive features; generates a clear and enhanced image of the image data based on the fused comprehensive features; utilizes high-resolution blurred images and low-resolution event fusion to effectively remove stroboscopic noise and noise under extreme low illumination conditions, and generates a high-resolution, clear, and enhanced image. Thus, the problems of blurred images of traditional cameras, low resolution of event cameras, and stroboscopic phenomena in related technologies are solved.

[0017] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0019] Figure 1 A schematic flow chart of a method for generating a clear enhanced image by fusing images and events under low illumination according to an embodiment of the present invention;

[0020] Figure 2 An overall block diagram of a method and technical solution for generating a clear enhanced image by fusing images and events under low illumination provided by one embodiment of the present invention;

[0021] Figure 3 A schematic flow chart of a method for generating a clear enhanced image by fusing images and events under low illumination according to an embodiment of the present invention;

[0022] Figure 4 A structural diagram of a Swin-CNN multi-scale encoder according to an embodiment of the present invention;

[0023] Figure 5 A diagram showing the structure of a Swin-CNN according to an embodiment of the present invention;

[0024] Figure 6 A diagram showing the structure of SwinT according to an embodiment of the present invention;

[0025] Figure 7 A diagram showing a feature dimensionality reduction layer structure according to an embodiment of the present invention;

[0026] Figure 8 A diagram showing a fusion layer structure according to an embodiment of the present invention;

[0027] Figure 9 A structural diagram of a cross-scale feature fusion module based on a sparse transformer block according to an embodiment of the present invention;

[0028] Figure 10 A block diagram of SparSTB and SparCTB according to one embodiment of the present invention;

[0029] Figure 11 This is a structural diagram of a cross-modal fusion decoder based on frequency domain filtering according to one embodiment of the present invention;

[0030] Figure 12 A block diagram of an apparatus for fusing images and events to generate a clear enhanced image under low illumination provided by an embodiment of the present invention;

[0031] Figure 13 A schematic structural diagram of an electronic device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present invention, but are not to be construed as limiting the present invention.

[0033] The following describes, with reference to the accompanying drawings, a method and apparatus for fusing images and events to generate a clear and enhanced image under low illumination, according to an embodiment of the present invention. Addressing the aforementioned problems in the related art mentioned in the background art, such as blurry images from conventional cameras and low-resolution and flickering event cameras, the present invention provides a method for fusing images and events to generate a clear and enhanced image under low illumination. The method comprises acquiring image data and event data under low illumination conditions; extracting multi-resolution image features from the image data and multi-resolution event features from the event data; selecting new image features and new event features across modalities at the same resolution scale for the multi-resolution image features and multi-resolution event features; cross-scale fusion of the new image features and new event features at different resolutions to obtain new comprehensive image features and new comprehensive event features; cross-modally fusion of the comprehensive image features and comprehensive event features to obtain fused comprehensive features; and generating a clear and enhanced image based on the fused comprehensive features. By fusing the high-resolution blurred image with the low-resolution event, a clear image with normal brightness and high resolution is generated. This method solves the problems in the related art, such as blurry images from conventional cameras and low-resolution and flickering event cameras.

[0034] Specifically, Figure 1 The present invention provides a flowchart of a method for fusing images and events to generate a clear enhanced image under low illumination.

[0035] like Figure 1 As shown, the method for generating a clear enhanced image by fusing images and events under low illumination includes the following steps:

[0036] In step S101 , image data and event data are acquired under low illumination conditions.

[0037] Among them, low-light conditions refer to environmental conditions with very weak ambient light; image data refers to the information captured by traditional RGB cameras, including visual information such as color and brightness; event data refers to the information captured by event cameras, which are bio-inspired visual sensors that work by asynchronously detecting brightness changes of each pixel.

[0038] In step S102, multi-resolution image features in the image data and multi-resolution event features in the event data are extracted. For the multi-resolution image features and multi-resolution event features, new image features and new event features are selected across modalities at the same resolution scale.

[0039] Among them, multi-resolution image features refer to feature information at different scales extracted from image data, including high-, medium-, and low-resolution image features; multi-resolution event features refer to features extracted at different time resolutions based on the information output by the event camera, including high-, medium-, and low-resolution event features.

[0040] It can be understood that the embodiments of the present invention extract multi-resolution image features and multi-resolution event features from image data and event data obtained under low-light conditions, and select new image features and new event features across modalities at the same resolution scale. The specific selection method will be described in detail below and will not be repeated here.

[0041] In an embodiment of the present invention, multi-resolution image features in image data and multi-resolution event features in event data are extracted, including: inputting image data and event data into a multi-scale encoder, and the multi-scale encoder outputting multi-resolution image features and multi-resolution event features, wherein the multi-scale encoder includes an image encoder and an event encoder; the image encoder and the event encoder are provided with a processing path at a resolution layer, and the processing path includes a convolution layer and a convolutional neural network enhanced with a shifted window (Swin-CNN) module; the image encoder downsamples the image data to obtain multi-resolution image features, and the event encoder upsamples the event data to obtain multi-resolution event features.

[0042] Among them, Swin-CNN is a convolutional neural network enhanced with shifted windows. The Swin-CNN module contains convolutional layers and SwinT (Shifted Window Multi-head Self-Attention) modules. The SwinT module further enhances feature representation capabilities through components such as multi-layer perceptrons, normalization layers, window multi-head self-attention, and shifted window multi-head self-attention.

[0043] It can be understood that the embodiment of the present invention uses a multi-scale encoder to obtain multi-resolution image features and multi-resolution event features based on image data and event data. Specifically, the multi-scale encoder includes an image encoder and an event encoder. The image encoder receives a high-resolution traditional image as input and generates multi-resolution image features through a downsampling operation; the event encoder accepts low-resolution event data input and obtains multi-resolution event features through an upsampling operation. The downsampled image and the upsampled event can achieve cross-scale alignment of high-resolution images and low-resolution events, while ensuring that features of different scales are correctly aligned, avoiding scale conversion, and making subsequent feature selection more direct.

[0044] In an embodiment of the present invention, for multi-resolution image features and multi-resolution event features, new image features and new event features are selected cross-modally at the same resolution scale, including: inputting the multi-resolution image features and multi-resolution event features into a cross-modal feature selection module, and the cross-modal feature selection module outputs new image features and new event features, wherein the cross-modal feature selection module includes a lower projection layer and an upper projection layer, and the upper projection layer maps the features after down-projection by the lower projection layer back to the corresponding modal space.

[0045] The lower projection layer and the upper projection layer will be described in detail below and will not be repeated here.

[0046] It can be understood that the embodiment of the present invention outputs new image features and new event features through a cross-modal feature selection module. This cross-modal feature selection module includes a lower projection layer and an upper projection layer. The upper projection layer maps the features after down-projection by the lower projection layer back to the corresponding modal space, thereby forming improved new image features and new event features. Through cross-modal feature selection, the most beneficial features can be extracted and fused from the two modalities to generate more accurate and detailed new image features and new event features.

[0047] In an embodiment of the present invention, the lower projection layer includes a connection layer, a linear layer, a normalization layer, and a fusion layer, wherein the image features and event features are connected using the connection layer; the linear layer and the normalization layer perform feature dimensionality reduction on the connected features to obtain multi-source features, generate image prompts and event prompts based on the multi-source features, and obtain image source embedding and event source embedding; generate image source features based on the image prompts, image source embedding and image features, and generate event source features based on the event prompts, event source embedding and event features; fuse the image source features and event source features to obtain fusion features, generate new image features based on the fusion features, image features and image source embedding, and generate new event features based on the fusion features, event features and event source embedding.

[0048] Among them, image source embedding is the basic representation extracted from the input high-resolution blurred image, which contains information such as the basic structure, color, and texture of the image; event source embedding is the basic representation extracted from the input low-resolution event data, which reflects the time series information of pixel brightness changes.

[0049] It can be understood that in the embodiment of the present invention, in the lower projection layer of the cross-modal feature selection module, the connection layer is used to connect the image features and the event features together to obtain the connected features, and then the linear layer and the normalization layer are used to perform feature dimensionality reduction on the connected features to obtain multi-source features. Based on the multi-source features, image prompts and event prompts are generated, and these prompts are used together with the image source embedding and the event source embedding to generate refined image source features and event source features, and the image source features and the event source features are fused to obtain fused features. Finally, new image features are generated based on the fused features, image features and image source embedding, and new event features are generated based on the fused features, event features and event source embedding.

[0050] In step S103, new image features and new event features of different resolutions are cross-scale fused to obtain new comprehensive image features and new comprehensive event features, and the comprehensive image features and comprehensive event features are cross-modally fused to obtain fused comprehensive features. An image with clear enhancement of the image data is generated based on the fused comprehensive features.

[0051] Among them, cross-scale fusion refers to the integration of features at different resolution levels to obtain a more comprehensive information representation; cross-modal fusion refers to the fusion of data from different modalities, such as images and events, with the aim of combining their respective advantages to generate more accurate and rich feature representations.

[0052] It can be understood that the embodiments of the present invention obtain optimized new comprehensive image features and new comprehensive event features by cross-scale fusing new image features and new event features of different resolutions, aiming to integrate multi-level information to capture more detailed local details and global structures; cross-modally fuse the comprehensive image features and comprehensive event features, and further refine the fused comprehensive features by utilizing their respective advantages; finally, generate an image with clear enhanced image data based on the fused comprehensive features, that is, an image with higher resolution and better visual quality.

[0053] In an embodiment of the present invention, new image features and new event features of different resolutions are respectively cross-scale fused to obtain new comprehensive image features and new comprehensive event features, including: inputting new image features of different resolutions and new event features of different resolutions into a cross-scale feature fusion module, and the cross-scale feature fusion module outputs comprehensive image features and comprehensive event features, wherein the cross-scale feature fusion module includes a convolutional layer and a Transformer block; at each resolution scale, local information and global information are extracted through the convolutional layer and the Transformer block, features at different resolution scales are aligned through sampling operations, and features at different resolution scales are fused through the convolutional layer.

[0054] Among them, the Transformer block is a sparse Transformer block, including a spatial window-based sparse Transformer block (SparSTB) and a channel window-based sparse Transformer block (SparCTB). Specifically, the spatial window-based Transformer block (STB) is a self-attention mechanism that divides the input image into multiple local windows and performs self-attention calculations within each window, thereby reducing computational complexity and improving processing efficiency. The channel-based Transformer block (CTB) is an important variant of the Transformer in the field of computer vision. In particular, it improves computational efficiency by focusing on the attention mechanism of the channel dimension while maintaining good modeling capabilities for the spatial structure of the image. SparSTB and SparCTB are spatial window-based sparse Transformer blocks and channel window-based sparse Transformer blocks designed based on STB and CTB.

[0055] It can be understood that the embodiment of the present invention inputs new image features and new event features of different resolutions into the cross-scale feature fusion module. In the cross-scale feature fusion module, the features of each resolution level are processed through the convolution layer and the Transformer block to extract local information and global information. Sampling operations are used to ensure that features at different resolution scales can be correctly aligned. The features at different resolution scales are fused through the convolution layer. Finally, the cross-scale feature fusion module outputs comprehensive image features and comprehensive event features, providing a basis for subsequent cross-modal fusion steps.

[0056] In an embodiment of the present invention, cross-modal fusion is performed on comprehensive image features and comprehensive event features to obtain fused comprehensive features, and a clearly enhanced image of the image data is generated based on the fused comprehensive features, including: inputting the comprehensive image features and comprehensive event features into a cross-modal fusion decoder, and the cross-modal fusion decoder outputting the clearly enhanced image, wherein the cross-modal fusion decoder includes a convolution layer, an activation function and a Swin-CNN; the convolution layer and the activation function preprocess the comprehensive image features and the comprehensive event features, and the preprocessed features are respectively input into a first processing channel and a second processing channel; the first processing channel obtains enhanced features through FFT (Fast Fourier Transform), a Gaussian low-pass filter, an FFC block (Fast Fourier Convolution Block) and an inverse FFT transform, the second processing channel is processed through a Swin-CNN block and a convolution layer, and the features processed by the second processing channel are fused with the enhanced features to obtain a fused comprehensive feature; the fused comprehensive features are processed through Swin-CNN and a convolution layer to obtain a clearly enhanced image of the image data.

[0057] Among them, the activation function can be a LeakyReLU (Leaky Rectified Linear Unit) function; the FFT transform is used to convert the time domain signal into a frequency domain signal for frequency domain filtering; the inverse FFT transform is the inverse process of the fast Fourier transform, which converts the processed frequency domain signal back to the time domain or space domain; the Gaussian low-pass filter is used to filter out high-frequency noise and retain low-frequency information, which helps to smooth image features; the FFC block uses the convolution operation implemented by the fast Fourier transform to perform efficient feature selection in the frequency domain.

[0058] It can be understood that the embodiment of the present invention inputs the comprehensive image features and comprehensive event features into the cross-modal fusion decoder. In the cross-modal fusion decoder, the comprehensive image features and comprehensive event features are preprocessed by convolution layers and activation functions, and then input into two processing channels: the first processing channel processes features through fast Fourier transform FFT, Gaussian low-pass filter, FFC block and inverse FFT transform to remove noise and highlight low-frequency information, thereby obtaining enhanced features; the second processing channel processes features through Swin-CNN blocks and convolution layers, and combines the features obtained from the second processing channel with the enhanced features of the first processing channel to form a fused comprehensive feature. This fused comprehensive feature is again processed by Swin-CNN and convolution layers to finally generate a clarity-enhanced image data, which effectively removes noise, retains important details, and effectively combines the advantages of frequency domain and spatial domain processing to achieve high-quality low-light image enhancement.

[0059] In an embodiment of the present invention, a loss function is constructed and used to guide the training of the model, wherein the loss function includes frequency loss, reconstruction loss and total loss. The frequency loss is determined by calculating the norm in the frequency domain, the reconstruction loss is determined by combining the pixel-level error and the structural similarity index, and the frequency loss and reconstruction loss are weighted to determine the total loss.

[0060] Specifically, the loss function is ,in is the final network output, is the corresponding true value of the main view, =0.1 , is the frequency loss, , represents Fourier transform; is the reconstruction loss, which consists of two parts: pixel error and structural consistency constraint. ,in, To enhance the results, is the true value, is the structural similarity index between the two, using Norm to calculate the enhanced results and the true value The absolute pixel error between To compare the similarity of images.

[0061] It can be understood that the embodiment of the present invention first determines the frequency loss by calculating the norm in the frequency domain, reduces the high-frequency noise in the output image, combines the pixel-level error and the structural similarity index, and obtains the reconstruction loss. Finally, by assigning appropriate weights to the frequency loss and the reconstruction loss, the total loss is calculated. The total loss supervision model is used to make the output image highly similar to the real image in both the spatial and frequency domains, thereby effectively guiding model training and improving the quality of image enhancement under low illumination conditions.

[0062] According to an embodiment of the present invention, a method for fusing images and events to generate a clear enhanced image under low illumination is proposed. The method comprises the following steps: obtaining image data and event data under low illumination conditions; extracting multi-resolution image features from the image data and multi-resolution event features from the event data; cross-modally selecting new image features and new event features for the multi-resolution image features and the multi-resolution event features at the same resolution scale; cross-scale fusion of the new image features and new event features of different resolutions to obtain new comprehensive image features and new comprehensive event features, cross-modally fusion of the comprehensive image features and the comprehensive event features to obtain fused comprehensive features, generating a clearly enhanced image of the image data based on the fused comprehensive features, and utilizing the high-resolution blurred image and low-resolution event fusion to effectively remove noise under stroboscopic and extreme low illumination conditions, thereby generating a high-resolution, clear and enhanced image.

[0063] The following further describes a method for generating a clear enhanced image by fusing images and events under low illumination through a specific embodiment.

[0064] Figure 2 This is a general block diagram of a method and technical solution for generating a clear enhanced image by fusing images and events under low illumination provided by this embodiment. Figure 3 This is a flow chart of a method for fusing images and events to generate a clear enhanced image under low illumination provided by this embodiment. Figure 3 As shown, the specific implementation plan of this embodiment is as follows:

[0065] Step S201: Build Swin-CNN multi-scale encoder

[0066] The Swin-CNN multi-scale image encoder based on downsampling and the Swin-CNN multi-scale event encoder based on upsampling are as follows: Figure 4 shown.

[0067] First, build a low-resolution event expression, low-resolution event It consists of two parts: time-dependent representation frame and full-time event representation frame , time dependency means that the frame contains Central Event information within time, full-time event means the frame contains the entire exposure time Event information within.

[0068]

[0069] in Stands for pixel-by-pixel overlay.

[0070] Low-resolution events combine event information at different temporal resolutions, which not only obtains clear events with low latency, but can also be used to supplement the information loss and remove blur of traditional cameras; it can also obtain the overall event over the entire exposure time, providing overall information for subsequently reducing the impact of flicker and noise in event cameras under extreme low-light conditions.

[0071] Input a high-resolution image and downsample it. Each scale of image and event is processed through its own convolutional layer and Swin-CNN to generate high-, medium-, and low-resolution image features. Similarly, input a low-resolution event and upsample it to generate high-, medium-, and low-resolution event features.

[0072] Downsampled images and upsampled events enable cross-scale alignment of high-resolution images and low-resolution events. This ensures that features at different scales are correctly aligned, avoiding scale conversion and making subsequent feature selection more straightforward. Features at different scales have different emphases: higher-resolution features are beneficial for extracting shallow features, while lower-resolution features are beneficial for extracting deeper features. Multi-scale feature extraction comprehensively extracts information from both images and events.

[0073] The Swin-CNN module is the core of this process. Figure 5 As shown, it contains convolutional layers and SwinT modules to ensure the effective extraction of local and global features. The SwinT module is as follows Figure 6 The SwinT module further enhances the model's feature representation capabilities through components such as MLP (Multilayer Perceptron), layer normalization (LayerNorm), W-MSA (Window-based Multi-head Self-Attention), and SW-MSA (Shifted Window Multi-head Self-Attention).

[0074] Step S202: Building a prompt-driven cross-modal feature selection module

[0075] In low-light conditions, images provide color and texture information, while events provide high-dynamic detail. Through cross-modal feature selection, the most beneficial features from both modalities can be extracted and fused to produce a more accurate and detailed representation. Operating at the same scale ensures that features from different modalities are aligned in spatial resolution, allowing the system to more accurately identify and localize objects in scenes under challenging conditions such as low light.

[0076] Feature selection is divided into four steps: multi-source feature generation, cue generation, cue-driven fusion, and mutual information regularization.

[0077] First, generate multi-source features. Input image features and event characteristics ,use Perform feature dimensionality reduction to obtain processed multi-source features

[0078]

[0079] in Represents the connection of features, Including linear layers and normalization layers, such as Figure 7 shown.

[0080] Then generate the prompt,

[0081]

[0082] in represents global average pooling, represents the activation function, Represents a cross-modal adapter, Indicates the gating network corresponding to the adapter, prompt Prompted by image and event prompts composition.

[0083] Gated Network It is defined as,

[0084]

[0085] in and represents the activation function, Indicates that only the front values ​​( ), and is a learnable parameter, It is the multi-source feature of input.

[0086] Cross-modal adapters It is defined as,

[0087]

[0088]

[0089] Cross-modal adapters enable interaction between multimodal information and unify multiple modalities into a single framework through partial weight sharing. Their main goal is to enhance the interaction between different modalities while reducing additional adjustable parameters. Represents image features, Represents event characteristics, Represents the scaling factor. Unified lower projection layer Is a shared layer used to project input features to a bottleneck layer; nonlinear activation function Used to introduce nonlinearity and enhance the expressiveness of the model; modality-specific up-projection layer These layers are specific to each modality and are used to map the down-projected features back to their respective modality space. In the multimodal case, an additional cross-modal up-projection layer is introduced to effectively handle mixed information.

[0090] Image Tips and event prompts Source feature image features and event characteristics Perform dot multiplication, and the learnable parameter source embedding is independent of the input, that is, the source embedding and , generating refined source features and , and finally pass through the fusion layer to obtain the fusion feature The weights of different modalities are dynamically adjusted using prompts to achieve feature selection, thereby improving the network's adaptability to complex scenes such as stroboscopic and extreme low illumination.

[0091]

[0092]

[0093]

[0094] The fusion layer The refined multi-source features are fused using the addition operation and then passed through a set of convolutional layers. The fusion layer structure is as follows Figure 8 shown.

[0095] The features input to the next module are processed in the following way,

[0096]

[0097]

[0098] in Represents image features and Represents event characteristics, represents the fusion feature, is a learnable parameter initialized to 0.5.

[0099] In order to ensure that the model dynamically retains complementary information while discarding redundant information from multiple source features, the image prompts and event prompts Regularization constraints are imposed to force a balanced information distribution between sources. MIR (Mutual Information Regularization) is defined as follows:

[0100]

[0101] Step S203: Build a cross-scale feature fusion module based on sparse transformer blocks

[0102] Input high-, medium-, and low-resolution image features and event features, and the features of each resolution are processed through a 3x3 convolution layer (Conv3x3) and a sparse transformer block. Features are spliced ​​at different levels and further fused through a 1x1 convolution layer (Conv1x1). Low-resolution features and medium-resolution features are aligned with high-resolution features through upsampling operations. Finally, the fused features pass through another 3x3 convolution layer to generate the final feature representation. This cross-scale processing and fusion process aims to enhance the representation of features by combining information at different scales, and provide cross-scale fused image features and event features for subsequent cross-modal feature fusion. The module structure is as follows: Figure 9 shown.

[0103] The Spatial Window Transformer Block (STB) is a self-attention mechanism that reduces computational complexity and improves processing efficiency by partitioning the input image into multiple local windows and performing self-attention calculations within each window. The Channel-based Transformer Block (CTB) is an important variant of the Transformer in computer vision. In particular, it improves computational efficiency by focusing on the channel dimension while maintaining good modeling of the spatial structure of the image.

[0104] This embodiment designs a sparse block based on STB and CTB, and designs a spatial window-based sparse Transformer block (SparSTB) and a channel window-based sparse Transformer block (SparCTB) based on the sparse block. Figure 10 As shown in Figure 2, the sparse block is used to convert the dimension size of the output of the previous stage into a new token with a new embedding dimension size. The conversion process is to obtain a new embedding representation through a convolutional layer and convert it to a latent token through a linear layer.

[0105] In this embodiment, the convolution kernel and stride size used in the convolution layer are 3 and 1 respectively. Meanwhile, the input features in the linear layer in the third stage are , the output features are 49 to generate a new tag representation. The output obtained from the sparse block (SparseBlock) is ,in is the desired number of new markers, is the number of new embedding dimensions.

[0106] Step S204: Cross-modal fusion decoder based on frequency domain filtering

[0107] This embodiment designs a cross-modal fusion decoder based on frequency domain filtering, event features and image features As input, it is processed by a 1x1 convolution layer and a LeakyReLU activation function, and then enters the low-frequency channel and the high-frequency channel. The low-frequency channel includes a convolution layer, an FFT transform, a Gaussian low-pass filter, and an FFC block for frequency domain filtering, and then passes through an inverse FFT transform and a pixel-by-pixel spatial dynamic filter, and is processed through a frequency adaptive fusion module and a further convolution layer to generate low-frequency features. The high-frequency features are processed through a convolution layer, a Swin-CNN module, and further convolution layers, and fused with the low-frequency features. Finally, the fused features are processed through Swin-CNN and convolution layers to generate a high-resolution, clear, and enhanced image. This multi-stage processing method aims to effectively remove noise, retain important details, and generate clearer and more accurate image representations. The module structure is as follows: Figure 11 shown.

[0108] In low-light conditions, low signal-to-noise ratios (SNRs) can lead to significant noise in images, making it extremely difficult to recover global structural information. Furthermore, events in stroboscopic and extremely low-light scenes tend to generate significant noise. Therefore, a feature fusion enhancement method based on frequency-domain filtering was designed to effectively reduce this high-frequency noise and accurately recover the scene's primary structural information.

[0109] A Gaussian low-pass filter is used to extract low-frequency information. The Gaussian filter is defined as follows:

[0110]

[0111] in, and Represents the spectrum domain Axis and The center point of the axis, represents the standard deviation, 、 are the coordinates in the spectral domain. After low-pass filtering, additional frequency-domain selection is performed using a fast Fourier convolution (FFC) block. Finally, an inverse fast Fourier transform (Inverse-FFT) is applied to generate frequency-filtered features. These frequency-filtered features are combined with the original features via a residual connection. These features, which are emphasized by low-frequency information, often correspond to spatially varying primary structural information. To better enhance the primary structure of the scene, a pixel-by-pixel spatial dynamic filter is applied to the features.

[0112] Because the fusion process prioritizes complementary information between the source images, the residuals of the features of the two source images are first calculated. This residual is then fed into the Swin-CNN module to extract semantic features. Next, a single convolution layer is used to map the multi-channel semantic features into two-channel features. Finally, a SoftMax operation is applied to normalize the features, generating two masks. These masks are then element-wise multiplied with the features of the source images to obtain the fused features.

[0113] After obtaining the low-frequency fusion features and high-frequency fusion features, they will be passed through the convolution layer and Sigmoid to remove irrelevant spatial information respectively. After addition, they will be decoded by Swin-CNN and the convolution layer to obtain a high-resolution and clear enhanced image.

[0114] Step S205: Construct loss function

[0115] The network is supervised by a loss function whose input includes the final network output And the corresponding true value of the main view The loss function designed by the present invention includes two parts: frequency loss and reconstruction losses .

[0116]

[0117] in, ; Frequency loss function , specifically for blurred images, suppressing high-frequency noise, can be expressed as

[0118]

[0119] represents the Fourier transform.

[0120] The reconstruction loss function The reconstruction loss consists of two parts: pixel error and structural consistency constraint. It can be expressed as

[0121]

[0122] use Norm to calculate the enhanced results and the true value Structural Similarity Index To compare the similarity of images.

[0123] Next, a device for clearly enhancing an image under low illumination according to an embodiment of the present invention will be described with reference to the accompanying drawings.

[0124] Figure 12 The figure is a block diagram of a device for fusing images and events to generate a clear enhanced image under low illumination provided by an embodiment of the present invention.

[0125] like Figure 12 As shown, the device 10 for generating a clear enhanced image by fusing images and events under low illumination includes: an acquisition module 301 , an extraction module 302 and a fusion module 303 .

[0126] Among them, the acquisition module 301 is used to acquire image data and event data under low illumination conditions; the extraction module 302 is used to extract multi-resolution image features in the image data and multi-resolution event features in the event data, and for multi-resolution image features and multi-resolution event features, new image features and new event features are selected cross-modally at the same resolution scale; the fusion module 303 is used to perform cross-scale fusion on new image features and new event features of different resolutions to obtain new comprehensive image features and new comprehensive event features, perform cross-modal fusion on the comprehensive image features and comprehensive event features to obtain fused comprehensive features, and generate a clearly enhanced image of the image data based on the fused comprehensive features.

[0127] In an embodiment of the present invention, multi-resolution image features in image data and multi-resolution event features in event data are extracted, and the extraction module 302 is further used to: input the image data and event data into a multi-scale encoder, and the multi-scale encoder outputs multi-resolution image features and multi-resolution event features, wherein the multi-scale encoder includes an image encoder and an event encoder; the image encoder and the event encoder are provided with a processing path at the resolution layer, and the processing path includes a convolution layer and a convolutional neural network Swin-CNN module enhanced by a shifted window; the image encoder downsamples the image data to obtain multi-resolution image features, and the event encoder upsamples the event data to obtain multi-resolution event features.

[0128] In an embodiment of the present invention, for multi-resolution image features and multi-resolution event features, new image features and new event features are selected cross-modally at the same resolution scale, and the extraction module 302 is further used to: input the multi-resolution image features and multi-resolution event features into the cross-modal feature selection module, and the cross-modal feature selection module outputs the new image features and new event features, wherein the cross-modal feature selection module includes a lower projection layer and an upper projection layer, and the upper projection layer maps the features after lower projection by the lower projection layer back to the corresponding modal space.

[0129] In an embodiment of the present invention, the lower projection layer includes a connection layer, a linear layer, a normalization layer, and a fusion layer, wherein the image features and event features are connected using the connection layer; the linear layer and the normalization layer perform feature dimensionality reduction on the connected features to obtain multi-source features, generate image prompts and event prompts based on the multi-source features, and obtain image source embedding and event source embedding; generate image source features based on the image prompts, image source embedding and image features, and generate event source features based on the event prompts, event source embedding and event features; fuse the image source features and event source features to obtain fusion features, generate new image features based on the fusion features, image features and image source embedding, and generate new event features based on the fusion features, event features and event source embedding.

[0130] In an embodiment of the present invention, new image features and new event features of different resolutions are cross-scale fused to obtain new comprehensive image features and new comprehensive event features, respectively. The fusion module 303 is further used to: input the new image features of different resolutions and the new event features of different resolutions into the cross-scale feature fusion module, and the cross-scale feature fusion module outputs the comprehensive image features and the comprehensive event features, wherein the cross-scale feature fusion module includes a convolution layer and a Transformer block; at each resolution scale, local information and global information are extracted through the convolution layer and the Transformer block, features at different resolution scales are aligned through sampling operations, and features at different resolution scales are fused through the convolution layer.

[0131] In an embodiment of the present invention, cross-modal fusion is performed on the comprehensive image features and the comprehensive event features to obtain fused comprehensive features, and a clearly enhanced image of the image data is generated based on the fused comprehensive features. The fusion module 303 is further used to: input the comprehensive image features and the comprehensive event features into a cross-modal fusion decoder, and the cross-modal fusion decoder outputs a clearly enhanced image, wherein the cross-modal fusion decoder includes a convolution layer, an activation function and a Swin-CNN; the convolution layer and the activation function pre-process the comprehensive image features and the comprehensive event features, and the pre-processed features are input into a first processing channel and a second processing channel; the first processing channel obtains enhanced features through FFT transformation, Gaussian low-pass filter, FFC block and inverse FFT transformation, and the second processing channel is processed through Swin-CNN block and convolution layer, and the features processed by the second processing channel are fused with the enhanced features to obtain fused comprehensive features; the fused comprehensive features are processed through Swin-CNN and the convolution layer to obtain a clearly enhanced image of the image data.

[0132] In an embodiment of the present invention, a construction module is also included, wherein the construction module is further used to construct a loss function, and the loss function is used to guide the training of the network, wherein the loss function includes frequency loss, reconstruction loss and total loss, the frequency loss is determined by calculating the norm in the frequency domain, the reconstruction loss is determined by combining the pixel-level error and the structural similarity index, and the frequency loss and reconstruction loss are weighted to determine the total loss.

[0133] It should be noted that the above explanation of the method embodiment for fusing images and events to generate a clear enhanced image under low illumination is also applicable to the device for fusing images and events to generate a clear enhanced image under low illumination in this embodiment, and will not be repeated here.

[0134] According to an embodiment of the present invention, a device for fusing images and events under low illumination to generate a clear enhanced image is proposed. The device obtains image data and event data under low illumination conditions; extracts multi-resolution image features from the image data and multi-resolution event features from the event data; for the multi-resolution image features and multi-resolution event features, cross-modally selects new image features and new event features at the same resolution scale; cross-scale fuses the new image features and new event features of different resolutions to obtain new comprehensive image features and new comprehensive event features, cross-modally fuses the comprehensive image features and comprehensive event features to obtain fused comprehensive features, generates a clearly enhanced image of the image data based on the fused comprehensive features, and utilizes the high-resolution blurred image and low-resolution event fusion to effectively remove noise under stroboscopic and extreme low illumination conditions, thereby generating a high-resolution, clear and enhanced image.

[0135] Figure 13 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device may include:

[0136] Memory 401 , processor 402 , and computer programs stored in the memory 401 and executable on the processor 402 .

[0137] When the processor 402 executes the program, the method for generating a clear enhanced image by fusing images and events under low illumination provided in the above embodiment is implemented.

[0138] Furthermore, the electronic device further includes:

[0139] The communication interface 403 is used for communication between the memory 401 and the processor 402 .

[0140] The memory 401 is used to store computer programs that can be run on the processor 402 .

[0141] The memory 401 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0142] If the memory 401, processor 402, and communication interface 403 are implemented independently, the communication interface 403, memory 401, and processor 402 can be connected to each other via a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 13 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0143] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.

[0144] The processor 402 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0145] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the method for generating a clear enhanced image by fusing images and events under low illumination is implemented as described above.

[0146] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0147] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "N" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0148] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or N executable instructions for implementing a custom logical function or step of a process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0149] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, the steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0150] A person skilled in the art may understand that all or part of the steps carried out in the method for implementing the above-mentioned embodiment may be completed by instructing the relevant hardware through a program, and the above-mentioned program may be stored in a computer-readable storage medium, which, when executed, includes one of the steps of the method embodiment or a combination thereof.

[0151] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for generating a clear enhanced image by fusing images and events under low illumination, characterized in that: The following steps are involved: Acquire image data and event data under low-light conditions; Extracting multi-resolution image features from the image data and multi-resolution event features from the event data, and selecting new image features and new event features across modalities at the same resolution scale for the multi-resolution image features and the multi-resolution event features; New image features and new event features of different resolutions are cross-scale fused to obtain new comprehensive image features and new comprehensive event features, the comprehensive image features and the comprehensive event features are cross-modally fused to obtain fused comprehensive features, and a clearly enhanced image of the image data is generated based on the fused comprehensive features.

2. The method for generating a clear enhanced image by fusing images and events under low illumination according to claim 1, characterized in that: The extracting multi-resolution image features from the image data and multi-resolution event features from the event data includes: Inputting the image data and event data into a multi-scale encoder, the multi-scale encoder outputting multi-resolution image features and multi-resolution event features, wherein the multi-scale encoder includes an image encoder and an event encoder; The image encoder and the event encoder are provided with a processing path at the resolution layer, the processing path including a convolutional layer and a convolutional neural network Swin-CNN module enhanced by a shifted window; The image encoder downsamples the image data to obtain multi-resolution image features, and the event encoder upsamples the event data to obtain multi-resolution event features.

3. The method for generating a clear enhanced image by fusing images and events under low illumination according to claim 1, characterized in that: For the multi-resolution image features and the multi-resolution event features, cross-modal selection of new image features and new event features at the same resolution scale includes: Multi-resolution image features and multi-resolution event features are input into a cross-modal feature selection module, and the cross-modal feature selection module outputs new image features and new event features, wherein the cross-modal feature selection module includes a down-projection layer and an up-projection layer, and the up-projection layer maps the features after down-projection by the down-projection layer back to the corresponding modal space.

4. The method for generating a clear enhanced image by fusing images and events under low illumination according to claim 3, characterized in that: The down-projection layer includes a connection layer, a linear layer, a normalization layer, and a fusion layer, wherein: The image features and the event features are connected using a connection layer; The linear layer and the normalization layer perform feature dimensionality reduction on the connected features to obtain multi-source features, generate image prompts and event prompts according to the multi-source features, and obtain image source embedding and event source embedding; Generate an image source feature based on the image prompt, the image source embedding, and the image feature, and generate an event source feature based on the event prompt, the event source embedding, and the event feature; The image source feature and the event source feature are fused to obtain a fused feature, a new image feature is generated based on the fused feature, the image feature and the image source embedding, and a new event feature is generated based on the fused feature, the event feature and the event source embedding.

5. The method for generating a clear enhanced image by fusing images and events under low illumination according to claim 1, characterized in that: The cross-scale fusion of new image features and new event features of different resolutions to obtain new comprehensive image features and new comprehensive event features includes: Inputting the new image features of different resolutions and the new event features of different resolutions into a cross-scale feature fusion module, the cross-scale feature fusion module outputting comprehensive image features and comprehensive event features, wherein the cross-scale feature fusion module includes a convolutional layer and a Transformer block; At each resolution scale, local information and global information are extracted through the convolutional layer and the Transformer block, features at different resolution scales are aligned through sampling operations, and features at different resolution scales are fused through the convolutional layer.

6. The method for generating a clear enhanced image by fusing images and events under low illumination according to claim 1, characterized in that: The cross-modal fusion of the comprehensive image features and the comprehensive event features to obtain a fused comprehensive feature, and generating a clearly enhanced image of the image data based on the fused comprehensive feature, includes: Inputting the comprehensive image features and the comprehensive event features into a cross-modal fusion decoder, wherein the cross-modal fusion decoder outputs a clear and enhanced image, wherein the cross-modal fusion decoder includes a convolutional layer, an activation function and a Swin-CNN; The convolution layer and the activation function preprocess the comprehensive image features and the comprehensive event features, and the preprocessed features are input into a first processing channel and a second processing channel; The first processing channel obtains enhanced features through FFT transformation, Gaussian low-pass filter, FFC block and inverse FFT transformation, the second processing channel is processed through Swin-CNN block and convolution layer, and the features processed by the second processing channel are fused with the enhanced features to obtain fused comprehensive features; The fused comprehensive features are processed by Swin-CNN and convolutional layers to obtain an image with clear enhancement of the image data.

7. The method for generating a clear enhanced image by fusing images and events under low illumination according to claim 1, characterized in that: Also includes: Construct a loss function and use the loss function to guide the training of the model, wherein the loss function includes frequency loss, reconstruction loss and total loss, the frequency loss is determined by calculating the norm in the frequency domain, the reconstruction loss is determined by combining the pixel-level error and the structural similarity index, and the total loss is determined by weighting the frequency loss and the reconstruction loss.

8. A device for generating a clear enhanced image by fusing images and events under low illumination, characterized in that: include: An acquisition module, used for acquiring image data and event data under low illumination conditions; an extraction module, configured to extract multi-resolution image features from the image data and multi-resolution event features from the event data, and select new image features and new event features across modalities at the same resolution scale for the multi-resolution image features and the multi-resolution event features; A fusion module is used to perform cross-scale fusion on new image features and new event features of different resolutions to obtain new comprehensive image features and new comprehensive event features, perform cross-modal fusion on the comprehensive image features and the comprehensive event features to obtain fused comprehensive features, and generate a clearly enhanced image of the image data based on the fused comprehensive features.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for generating a clear enhanced image by fusing images and events under low illumination as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the method for generating a clear enhanced image by fusing images and events under low illumination as described in any one of claims 1 to 7 is implemented.

Images