Dual-domain interactive guided multi-scale perception night flare removal method

Abstract

This invention provides a dual-domain interactive guided multi-scale perception nighttime flare removal method, comprising: inputting a nighttime flare image into an encoder-decoder end-to-end neural network based on a Transformer architecture, and outputting a flare-removed image after feature extraction and image reconstruction; wherein each Transformer block of the encoder and decoder is composed of a dual-domain interactive guided self-attention module and a multi-scale perception gating module connected in series; the dual-domain interactive guided self-attention module first performs spatial domain self-attention calculation on the input features to obtain spatial domain features, then performs linear projection on the spatial domain features to enhance high-frequency signals and transforms them to the frequency domain, and after frequency domain transformation processing, inversely transforms them back to the spatial domain to output dual-domain fused features; the multi-scale perception gating module uses convolutional kernels with different receptive fields to perform multi-path feature extraction on the dual-domain fused features to obtain multi-scale flare features, and performs adaptive fusion of the multi-scale flare features based on the gating mechanism to output multi-scale perception features.

Description

Technical Field

[0001] This invention belongs to the field of image information processing technology, specifically relating to a dual-domain interactive guided multi-scale perception nighttime flare removal method. Background Technology

[0002] Night flares are artifacts formed in nighttime imaging scenarios after strong light sources are reflected and scattered by the lens's optical system. They severely contaminate image information, leading to loss of image details and decreased contrast. Accurate removal of night flares is a core technological aspect for improving the robustness of nighttime imaging in vision systems such as autonomous driving and intelligent surveillance.

[0003] Currently, technologies related to nighttime flare removal are mainly divided into two categories: physical hardware suppression solutions and algorithmic post-processing solutions.

[0004] Physical hardware suppression solutions achieve flare reduction by optimizing imaging hardware, including lens components such as optical lenses with anti-reflective coatings, aspherical lenses, and multi-layer composite coated lenses, or by blocking scattered light through lens hoods. However, the suppression effect of such solutions is limited by the hardware conditions of the camera, lens, and optical imaging system. They cannot eliminate flare artifacts that have already been imaged, have limited effectiveness in removing flares that have already occurred in complex night scenes, and cannot meet the post-processing requirements of existing vision devices.

[0005] Post-processing algorithms remove flare by optimizing the image after imaging. These methods fall into two categories: traditional computational methods and deep learning methods.

[0006] 1. Traditional computational methods rely on image post-processing techniques to remove flares. For example, Faulkner et al., in their paper "Veiling glare deconvolution of images produced by X-ray image intensifiers," used deconvolution techniques to reduce flare artifacts. However, these methods typically assume that the impact of flares on the image is constant across the entire image, which does not match the uneven distribution and varied shapes of flares in real nighttime scenes, resulting in limited practical application effectiveness. Other methods use image inpainting techniques to reconstruct the damaged flare area based on the shape and location of the flare. However, these methods are only applicable to a limited number of flare types, such as bright spots, and are prone to misidentifying real light sources in the scene as flare artifacts, leading to the loss of effective image information.

[0007] 2. Deep learning-based methods for removing nighttime flares utilize the feature modeling and representation capabilities of neural networks to achieve high-quality flare removal, which is currently the mainstream research direction in this field. Among them, Wu et al. first proposed a semi-synthetic flare dataset and a general flare removal method based on the U-Net architecture in the paper "How to train neural networks for flare removal"; Qiao et al. achieved light source-guided flare removal by synchronously predicting the light source and flare mask through an encoder-decoder architecture in the paper "Light source guided single image flare removal"; Zhou et al. optimized the automatic exposure rules of the ISP synthesis process in the paper "Improving lens flare removal with general-purpose pipeline and multiple light sources recovery", solving the problems of global illumination variation and local saturation in the synthesized image; Zhang et al. extracted global frequency features based on SwinTransformer combined with Fast Fourier Transform in the paper "FFformer: Swin Fourier transformer for nighttime flare removal", achieving good results in the nighttime flare removal task; Song and Bae achieved good results in the paper "Hard-negative sampling with cascaded fine-tuning network to boost flare removal performance in the nighttime". In the article "images", Kotp and Torki improved the performance of flare removal by optimizing the model through cascaded fine-tuning of the network and contrastive learning. In the paper "Flare-free vision: Empowering uformer with depth insights", Kotp and Torki integrated the depth estimation module into the Uformer architecture, realizing plug-and-play flare removal.

[0008] However, in practical applications in real-world nighttime scenes, the aforementioned deep learning solutions still suffer from insurmountable drawbacks: the morphology, size, and distribution range of flares vary greatly under different nighttime scenes, and existing algorithms cannot adequately model and distinguish the feature differences between flare regions and non-flare regions. This easily leads to problems such as incomplete or over-processing of flare regions and misprocessing of non-flare regions, resulting in significant visual differences between the reconstructed image and the real scene. Furthermore, existing solutions lack targeted perception capabilities for flare features of different sizes, requiring complex explicit multi-scale supervision to adapt to multi-morphological flares, resulting in insufficient generalization ability and robustness in complex real-world scenes. How to achieve global-local joint perception of flare damage information in real-world scenes, accurately distinguish between flare regions and non-flare regions, and adaptively adapt to flare features of different sizes to achieve refined flare removal remains an unsolved technical challenge in this field. Summary of the Invention

[0009] To address the shortcomings and deficiencies of existing technologies, this invention provides a dual-domain interactive guided multi-scale perception nighttime flare removal method. It aims to solve the problems of insufficient perception of feature differences between flare and non-flare areas, easy occurrence of flare residue or over-processing, poor adaptability to flares of different sizes and shapes in real scenes, and insufficient generalization ability in existing nighttime flare removal schemes. This invention constructs an end-to-end encoder-decoder processing network based on the Transformer architecture. Each Transformer block in the encoder and decoder is designed as a core structure with a dual-domain interactive guided self-attention module and a multi-scale perception gating module sequentially connected in series, realizing end-to-end extraction of flare features and image reconstruction. Specifically, the dual-domain interactive guided self-attention module first performs spatial domain self-attention calculation on the input features to complete global feature modeling of the flare, then performs linear projection on the obtained spatial features to enhance high-frequency signals and transforms them to the frequency domain for processing, and finally inversely transforms them back to the spatial domain, realizing collaborative interactive learning between the spatial and frequency domains, completing global-local joint perception of flare damage information, and guiding the network to accurately focus on the flare damage area. Through the multi-scale perception gating module, convolutional kernels with different receptive fields are used to perform multi-path parallel extraction of features, and combined with the gating mechanism, flare features of different scales are adaptively fused, internalizing multi-scale perception capabilities into the Transformer block, adapting to different sizes of flare morphologies in real-world scenarios. This invention employs a joint loss function combining reconstruction loss, perception loss, and frequency domain loss to complete end-to-end supervised training. It achieves refined nighttime flare removal without the need for complex explicit multi-scale supervision. In real-world test sets, it outperforms existing advanced methods in multiple image reconstruction evaluation metrics and can be widely applied to the nighttime visual image preprocessing stage of systems such as autonomous driving and intelligent monitoring.

[0010] The specific technical solution adopted by this invention to solve its technical problem is as follows:

[0011] A dual-domain interactive guided multi-scale sensing nighttime flare removal method includes:

[0012] The nighttime flare image is input into an end-to-end encoder-decoder neural network built on the Transformer architecture. After feature extraction and image reconstruction, the output is an image with flare removal.

[0013] Each Transformer block of the encoder and decoder is composed of a dual-domain interactive guided self-attention module and a multi-scale perception gating module connected in series.

[0014] The dual-domain interactive guided self-attention module first performs spatial domain self-attention calculation on the input features to obtain spatial domain features, then performs linear projection on the spatial domain features to enhance high-frequency signals and transforms them to the frequency domain. After frequency domain transformation processing, it is inversely transformed back to the spatial domain and outputs dual-domain fused features.

[0015] The multi-scale perception gating module uses convolutional kernels with different receptive fields to extract multi-path features from the dual-domain fusion features to obtain multi-scale flare features. Based on the gating mechanism, the multi-scale flare features are adaptively fused to output multi-scale perception features.

[0016] Furthermore, the feature processing of the Transformer block follows these rules: first, the input features are processed by layer normalization and then input into the dual-domain interactive guided self-attention module. The module output is then residually connected with the original input features to obtain the dual-domain fused features. Next, the dual-domain fused features are processed by layer normalization and then input into the multi-scale perception gating module. The module output is then residually connected with the dual-domain fused features to obtain the final output features of the Transformer block.

[0017] And it satisfies the following formula:

[0018]

[0019]

[0020] in, The input features are the Transformer block's features, LN is the layer normalization operation, and DGSA is the processing operation of the dual-domain interactive guided self-attention module. For dual-domain fusion features, MAGB is the processing operation of the multi-scale perception gating module. This is the final output feature of the Transformer block.

[0021] Furthermore, the processing flow of the dual-domain interactive guided self-attention module includes: firstly, performing channel context encoding on the input features to generate the query vector, key vector, and value vector required for self-attention calculation; and then performing spatial domain self-attention calculation based on the vectors. After performing linear projection on the spatial features to enhance high-frequency signals, the features are decomposed into real and imaginary parts in the frequency domain by two-dimensional fast Fourier transform. The real and imaginary parts are then concatenated along the channel dimension and linearly modulated. Finally, the features are converted back to the spatial domain by two-dimensional inverse fast Fourier transform, and the dual-domain fused features are output.

[0022] Furthermore, the processing flow of the multi-scale perception gating module includes: first, performing channel expansion on the dual-domain fusion features, then splitting them into multiple groups of sub-features with equal number of channels along the channel dimension, assigning independent convolutional paths to each group of sub-features, and using depth-separable convolution with different kernel sizes for each path to extract multi-scale flare features corresponding to different receptive fields; cross-fusing multiple groups of multi-scale flare features to generate gating features and content features, performing activation on the gating features and fusing them element by element with the content features, and outputting multi-scale perception features after channel compression.

[0023] Furthermore, the cross-fusion and adaptive fusion of the multi-scale flare features satisfy the following formula:

[0024]

[0025]

[0026]

[0027] in, As a gating feature, As a content feature, , , Multi-scale flare features output from different convolutional pathways. , For the model's learnable parameters, For convolution operations, For multi-scale perception features, This is a channel compression operation, where G is the activation function. This is an element-wise multiplication operation.

[0028] Furthermore, the encoder is equipped with a multi-level feature extraction module, in which the feature size and number of channels are adjusted sequentially by downsampling at each level, and a corresponding number of Transformer blocks are stacked at each level; the decoder is equipped with a multi-level feature reconstruction module corresponding to the feature scale of the encoder, in which the feature size is restored sequentially by upsampling at each level, and a corresponding number of Transformer blocks are stacked at each level, and the upsampled features are channel-wise concatenated and fused with the output features of the encoder at the corresponding scale level.

[0029] Furthermore, after inputting the nighttime flare image into the neural network, the initial convolutional feature extraction layer first completes the mapping from the image to the feature space, and then the initial feature map is input into the encoder; after the decoder outputs the reconstructed feature map, the feature refinement module with stacked Transformer blocks first completes the detail optimization, and then the image after flare removal is reconstructed.

[0030] Furthermore, the training process of the neural network employs a joint loss function combining reconstruction loss, perceptual loss, and frequency domain loss for end-to-end supervision. The reconstruction loss is calculated based on the pixel-level difference between the output image and the real flare-free image, the perceptual loss is calculated based on the difference in deep semantic features of the image extracted by the pre-trained convolutional neural network, and the frequency domain loss is calculated based on the difference in the amplitude of the frequency domain spectrum after the image undergoes Fourier transform.

[0031] Furthermore, the method is applied to the nighttime image preprocessing stage of autonomous driving systems or video surveillance systems.

[0032] And, a dual-domain interactive guided multi-scale perception nighttime flare removal system, including an image input module, an encoder-decoder end-to-end neural network module based on the Transformer architecture, and an image output module;

[0033] In the encoder and decoder of the neural network module, each Transformer block is composed of a dual-domain interactive guided self-attention module and a multi-scale perception gating module connected in series.

[0034] The dual-domain interactive guided self-attention module is configured to first perform spatial domain self-attention calculation on the input features to obtain spatial domain features, then perform linear projection on the spatial domain features to enhance high-frequency signals and transform them to the frequency domain, and after frequency domain transformation processing, inversely transform them back to the spatial domain to output dual-domain fused features.

[0035] The multi-scale perception gating module is configured to use convolutional kernels with different receptive fields to extract multi-path features from the dual-domain fusion features to obtain multi-scale flare features, and to perform adaptive fusion of the multi-scale flare features based on the gating mechanism to output multi-scale perception features.

[0036] Compared to existing technologies, this invention and its preferred solution achieve collaborative interactive feature learning in the spatial and frequency domains through a self-attention design guided by dual-domain interaction. This enables global-local joint perception of flare damage information, effectively improving the network's ability to identify feature differences between flare and non-flare regions. This helps reduce problems such as incomplete flare removal, overprocessing, and misprocessing of non-flare regions, better preserving the original details, textures, and structural information of the image while removing flare artifacts. Furthermore, this invention internalizes multi-scale perception capabilities into the core structure of the Transformer block. Through parallel feature extraction with multiple receptive fields and adaptive gating fusion design, it can adapt to flare features of different sizes and shapes in real-world scenarios, achieving multi-type flare characteristics without relying on complex explicit multi-scale supervision. Robust handling of flares significantly improves the generalization ability and applicability of the solution in complex real-world scenarios. This invention constructs an end-to-end encoder-decoder network based on the Transformer architecture, integrating the core dual-domain interaction and multi-scale perception modules into every Transformer block in the entire encoding and decoding process. This achieves seamless processing of flare feature extraction and image reconstruction throughout the entire process. The network structure is simple and efficient, requiring no additional pre- or post-processing modules, making it easy to implement and deploy in engineering. At the same time, this invention achieves end-to-end supervised training through a multi-dimensional joint loss function, which can constrain the optimization direction of the network from multiple dimensions, further ensuring the overall visual quality and real-world scene reproduction of the image after flare removal. It can be widely adapted to the image preprocessing needs of various night vision systems. Attached Figure Description

[0037] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0038] Figure 1 This is a diagram showing the overall network structure of the dual-domain interactive guided multi-scale sensing nighttime flare removal method according to an embodiment of the present invention.

[0039] Figure 2 This is a flowchart illustrating the overall training process of the nighttime flare removal network according to an embodiment of the present invention.

[0040] Figure 3 The figures show the flare removal results of the various methods of the present invention on the Flare7K real nighttime flare test set; in the figures, (a) is the input nighttime flare image to be processed, (b) is the flare removal result of the Flare7K method, (c) is the flare removal result of the Flare7K++ method, (d) is the flare removal result of the GRL method, (e) is the flare removal result of the InternImage method, (f) is the flare removal result of the Kotp et al. method, (g) is the flare removal result of the method of the present invention, and (h) is a real reference image without flares. Detailed Implementation

[0041] To make the features and advantages of the present invention more apparent and understandable, specific embodiments are described below in detail:

[0042] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0043] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0044] Given the unique characteristics of flare damage, the ability to jointly perceive global and local data in images is crucial for flare removal tasks. To address the issue that nighttime flare removal algorithms cannot adequately handle the differences between flare and non-flare regions, this invention proposes a dual-domain interactive guided multi-scale perception method for nighttime flare removal. First, the network introduces interactive processing in the spatial and frequency domains to more effectively model flare damage information from a global to local perspective, guiding the model to focus on processing flare regions. Furthermore, a multi-scale mechanism is introduced into the network to enable it to learn to perceive flare loss information at different scales, achieving more refined modeling capabilities.

[0045] The overall network provided in this invention is designed based on the Transformer encoder-decoder architecture. Specifically, a dual-domain interactive guided attention mechanism is proposed, enabling the network to learn interactively in the spatial and frequency domains. This allows for more effective perception of global-local flare damage information, guiding the model to focus on processing flare regions and minimizing features affecting non-flare regions. To further perceive the impact of flare damage information at different scales, compared to traditional multi-scale explicit supervision methods, a multi-scale mechanism is internalized into the feedforward neural network of the Transformer block. This allows the network to incorporate multi-scale perception capabilities while learning to model flare damage information. Based on this network architecture design, the network achieves high-quality flare removal by jointly perceiving multi-scale global-local flare loss information in real-world scenarios, further extending the flare removal task to adapt to real-world scenarios.

[0046] The core mechanisms upon which the solution of this invention relies include:

[0047] (1) Dual-domain interaction guides self-attention

[0048] Because the standard Transformer globally models all tokens when calculating self-attention, this often introduces noisy interactions between irrelevant features, which is not ideal for nighttime flare removal tasks. To overcome these limitations, this invention leverages the sparsity of neural networks, designing it as an efficient feature extraction unit. Through its spatial domain, it can focus on the most relevant non-local information at each scale, thereby achieving more accurate feature representation. In each Transformer block, given the data from the first... Input characteristics of each module Its encoding process can be described as follows:

[0049]

[0050]

[0051] in, and represents the output of the Dual-Domain Interactive Guided Self-Attention (DGSA) module and the Multi-Scale Perception Gated Module (MAGB) module, respectively, with LN representing layer normalization. Specifically, the Dual-Domain Interactive Guided Self-Attention module first uses a... Convolution followed by a Depthwise separable convolutions are used to encode channel context information. Given a query vector Q, a key vector K, and a value vector V, the attention calculation process is as follows:

[0052]

[0053] in, Let represent the dimension of each attention head, and k represent the number of attention heads. After obtaining the global spatial features in the spatial domain, they are transformed into local features in the frequency domain. To utilize feature differences in the frequency domain and bridge the information gap, this invention focuses on spatial domain features. Apply the Fast Fourier Transform (FFT). For simplicity, consider the single-channel case, i.e. . The two-dimensional fast Fourier transform is defined as follows:

[0054]

[0055] Where u and v represent frequency coordinates in the transform space, and IFFT is represented as... In its implementation, features are analyzed before applying the FFT. First, a linear layer is used for projection to enhance high-frequency signals, thus acting as a high-pass filter. As a further preferred option, this linear projection is implemented through 1×1 convolution, also acting as a high-pass filter. This enhances the high-frequency detail signals corresponding to the flare region, strengthens the network's ability to perceive the feature differences between flare and non-flare regions, and solves the problems of flare residue and overprocessing that are common in existing technologies.

[0056] Subsequently, the features are decomposed into real and imaginary parts using FFT, denoted as (R, I). Mathematically, this can be represented as:

[0057]

[0058] Generate a complex spectrum containing key information. Next, concatenate these real and imaginary parts along the channel dimension to obtain... :

[0059]

[0060] Where [·] indicates splicing along the channel dimension, The first layer represents the features obtained after frequency domain interaction and fusion. The second linear layer further modulates and refines the frequency components, while the inverse FFT transforms them back into the spatial domain for subsequent image processing stages. Through interactive learning in the spatial and frequency domains, the network can comprehensively perceive the distribution range of flare damage information in the image from a global-local perspective, thereby guiding the network to focus on processing the flare region.

[0061] (2) Multi-scale sensing gating module

[0062] Following spatial-frequency domain interactive learning, this invention considers further enhancing the network's ability to perceive flare damage information at different scales from a multi-scale perspective. The multi-scale concept is implicitly internalized into the Transformer's feedforward neural network, enabling the network to incorporate multi-scale perception capabilities while learning to model flare damage information. Features are obtained after processing through a dual-domain interactive guided attention mechanism. After that, first through The convolution is performed with channel expansion, and then the channel number is divided into three equal parts. Then, a multi-scale fusion gating mechanism is adopted, and the specific calculation process is as follows:

[0063]

[0064]

[0065]

[0066]

[0067] in, This indicates a channel splitting operation. After splitting the feature into three equal channels, depthwise separable convolutions of 3×3, 5×5, and 7×7 are used to extract features. This can cover flare morphologies of different sizes in real-world scenes, achieving comprehensive perception of large, medium, and small flares without complex explicit multi-scale supervision. This improves the network's generalization ability in complex real-world scenes. G( ) represents the GELU activation function. , , They represent Depth-separable convolution processing. And... and The learnable parameters in the model, as a further preferred implementation, refer to these learnable parameters. and The weight kernels are 1×1 convolutions, and are optimized end-to-end throughout the network training process. During training, the weights are iteratively updated using the backpropagation algorithm based on the joint loss function composed of reconstruction loss, perceptual loss, and frequency domain loss. and The weight values ​​enable the network to adaptively learn the fusion weights of flare features at different scales, enhancing its adaptability to flares of varying sizes. (Symbol) Hadamard product represents element-wise multiplication, symbol This represents a 1×1 convolution operation used for linear modulation of multi-scale features. Through this design, the multi-scale perception gating module can perceive the range of flare loss at different sizes, avoiding the neglect of global effective information due to focusing only on local details, thereby enabling the network to achieve robust flare removal capabilities in real-world scenarios.

[0068] The aforementioned dual-domain interactive guided self-attention module and multi-scale gated perception module are the core components of the Transformer block. The two are sequentially connected within the Transformer block to perform feature processing: first, the dual-domain interactive guided self-attention module completes the global-local joint perception of the input features, and the output dual-domain fused features are directly input into the multi-scale gated perception module to complete the multi-scale perception and adaptive fusion of flare features of different sizes, and finally output the processing result of the Transformer block.

[0069] (3) Loss function

[0070] To enhance network performance, the framework combines reconstruction loss, perceptual loss, and frequency domain loss during training to construct a loss function. This is applied to the final recovered flare-removed image. , will be compared with real images A comparison is then made. The network uses L1 loss as the reconstruction loss, with the specific formula as follows:

[0071]

[0072] For perceptual loss, the training process uses a pre-trained VGG19 network, from... and ReLU

[0073] The extracted features of the predicted image are then compared with those of the real image. The loss is expressed as:

[0074]

[0075] For frequency domain loss, the flare-removed image and the ground truth image are first transformed to the frequency domain using Fast Fourier Transform, and then the loss is calculated. The specific formula is as follows:

[0076]

[0077] Therefore, the final loss function can be expressed as:

[0078]

[0079] During the training phase, loss weights were adjusted based on experience gained from multiple experiments. , , The values ​​were set to 0.5, 0.5, and 0.01 respectively, which effectively removed flares and generated images with good visual effects.

[0080] The following will, with reference to the accompanying drawings, clearly and completely describe the entire process of network construction, feature processing, training, and inference of the present invention through specific embodiments, wherein... Figure 1 Corresponding to the overall network architecture and core module structure of this embodiment, Figure 2 The specific implementation steps for the network training process in this embodiment are as follows:

[0081] S1: First, a dual-domain interactive guided multi-scale perception nighttime flare removal network is constructed. This network is an end-to-end encoder-decoder architecture based on Transformer. Each layer of the encoder and decoder stacks Transformer blocks as core feature processing units. All Transformer blocks integrate the dual-domain interactive guided self-attention (DGSA) and multi-scale perception gated module (MAGB) proposed in this invention. The network as a whole includes an initial feature extraction layer, an encoding stage, a decoding stage, and a feature refinement module, realizing multi-scale perception removal of nighttime flare damage information under spatial domain and frequency domain guidance.

[0082] S1.1: See Figure 1 Using a 7×7 convolutional layer as the initial feature extraction layer, the input 3-channel flare damage image is processed. Mapping to a 24-dimensional feature space yields the initial feature representation of the network, providing a foundation for feature processing in the encoding stage;

[0083] S1.2: Network encoding stage. The encoding stage sets up four layers of feature extraction modules. Each layer of encoder stacks a corresponding number of Transformer blocks. Within each block, DGSA and MAGB are used to complete the core processing of flare features. Each layer sequentially adjusts the feature size and number of channels through the downsampling module.

[0084] S1.2.1: The first layer encoder stacks two Transformer blocks with 24 feature channels and uses one attention head for feature modeling to complete the first round of dual-domain interaction and multi-scale perception of flare damage features;

[0085] S1.2.2: After processing by the first layer encoder, the feature map space size is reduced to 1 / 2 of the original size and the number of channels is increased to 48 through the downsampling module. It is then input into the second layer encoder, which stacks 4 Transformer blocks, has a feature dimension of 48, and has 2 attention heads.

[0086] S1.2.3: After processing by the second layer encoder, the feature map space size is reduced to 1 / 2 of the original size and the number of channels is increased to 96 through the downsampling module. The feature map is then input into the third layer encoder, which stacks 4 Transformer blocks and uses a 4-head self-attention mechanism to enhance feature modeling.

[0087] S1.2.4: After processing by the third encoder, the feature map space size is reduced to half of the original size and the number of channels is increased to 192 through the third downsampling. It is then input into the fourth latent feature layer, which stacks 6 Transformer blocks and uses an 8-head self-attention mechanism to realize the dual-domain interaction and multi-scale perception of global flare features.

[0088] S1.3: The network decoding stage is built. The decoding stage follows the flowchart logic of progressive upsampling. While restoring the image resolution, the decoder at each level also stacks a corresponding number of Transformer blocks. DGSA and MAGB processing are also performed within the blocks. After upsampling, the features of the corresponding level in the encoding stage are concatenated and fused together.

[0089] S1.3.1: First, the features of the fourth latent layer are restored to the scale of the third layer through the upsampling module. They are then channel-wise concatenated and fused with the output features of the third layer in the encoding stage. After the number of channels is compressed to 96 by 1×1 convolution, they are input into the third layer of the decoder. This layer stacks 4 Transformer blocks to complete feature reconstruction.

[0090] S1.3.2: After processing by the third layer of the decoder, the features are restored to the scale of the second layer through upsampling. After being concatenated with the output features of the second layer of the encoder, they are compressed to 48 channels by 1×1 convolution and input into the second layer of the decoder. This layer stacks 4 Transformer blocks for feature reconstruction.

[0091] S1.3.3: After the second layer of the decoder processes the features, it restores the features to the resolution of the first layer through upsampling. After concatenating the features with the output features of the first layer of the encoder, it is input into the first layer of the decoder. This layer stacks two Transformer blocks to complete further feature fusion.

[0092] S1.4: After the backbone decoding stage, a feature refinement module consisting of four Transformer blocks is built. The output features of the first layer of the decoder are input into this module. Within the block, DGSA and MAGB are still used to optimize feature details, completing the refinement of the detailed structure of the restored image, which is used for the subsequent output of the final flare-removed image. Make preparations;

[0093] S2: See also Figure 1 For features in all Transformer blocks of the encoder and decoder, dual-domain interactive guided self-attention (DGSA) and multi-scale perceptual gating (MAGB) processing are performed sequentially. DGSA is the core attention processing unit, which realizes interactive feature learning between the spatial domain and the frequency domain, guiding the network to accurately focus on the flare-damaged area.

[0094] S2.1: Given a Transformer block, from the first... Input characteristics of each module First, perform layer normalization (LN) on the input, then input it into the DGSA module proposed in this invention for attention feature extraction. The processing result is then concatenated with the original input feature residual to obtain... The process satisfies formula (1):

[0095] S2.2: Within the DGSA module, first perform a 1×1 convolution followed by a 3×3 depthwise separable convolution. The channel context information is encoded to obtain the query vector Q, the key vector K, and the value vector V;

[0096] S2.3: Perform spatial self-attention calculation on Q, K, and V to obtain the global spatial features F in the spatial domain. This process satisfies formula (3).

[0097] S2.4: The spatial feature F is projected through a linear layer to enhance the high-frequency signal to achieve a high-pass filtering effect. Then, it is transformed to the frequency domain through a two-dimensional fast Fourier transform (FFT) to complete the frequency domain feature learning of dual-domain attention. This process satisfies formulas (4) and (5).

[0098] S2.5: Decompose the frequency domain features into real part R and imaginary part I, concatenate R and I along the channel dimension, and then modulate and refine the concatenated frequency components through a linear layer.

[0099] S2.6: Perform an inverse fast Fourier transform (IFFT) on the modulated frequency domain features to convert them back to the spatial domain and obtain the frequency domain interactive fusion features F^. This process satisfies formula (6) and completes the entire processing flow of DGSA attention.

[0100] S2.7: First, perform layer normalization (LN) on the feature F^ output by DGSA, then input it into the MAGB module. The processing result is the same as... The residual connection yields the final output features of the Transformer block. The process satisfies formula (2);

[0101] S2.8: Within the MAGB module, via Convolution expands the channel of feature F^, and then divides it into three equal channels X1, X2, X3 through channel splitting operation. This process satisfies formula (7).

[0102] S2.9: respectively adopt Depthwise separable convolutions are used to extract multi-scale features from X1, X2, and X3, resulting in processed features. The process satisfies formula (8);

[0103] S2.10: Fuse the extracted features at different scales to obtain gated features. and content features The process satisfies formula (9).

[0104] S2.11: Gated features using the GELU activation function Activate, then combine with content features The MAGB output features are obtained by multiplying each element and then compressing the channels by 1×1 convolution. This process satisfies formula (10) and completes all feature processing for a single Transformer block.

[0105] In summary, the core of this invention lies in integrating the proposed Dual-Domain Interactive Guided Self-Attention (DGSA) into all Transformer blocks of the encoder and decoder. This allows each round of feature processing in the encoding and decoding stages to achieve dual-domain attention learning in both the spatial and frequency domains. Combined with the Multi-Scale Perception Gating (MAGB) module, it enables global-local multi-scale perception of flare damage information. This allows the network to effectively preserve the detailed structure of the image while removing nighttime flare artifacts, providing robust flare removal capabilities in real-world scenarios and generating images with superior visual quality.

[0106] For experimental setup and training procedures, please refer to [link / reference]. Figure 2 The network training in this embodiment follows an end-to-end training process, with the following specific settings: the training process uses the Adam optimizer, the batch size is set to 2, and the input flare damage images are uniformly cropped to... The model was trained for a total of 600K iterations, with an initial learning rate set to [value missing]. After 200K iterations, the learning rate is halved, and for the remaining 400K iterations, the learning rate is kept constant after halving. The loss is reconstructed during the training phase. Perceived loss With frequency domain loss To construct the total loss function, we first combine the flare-damaged image... The image is fed into the network during training to obtain a flare-removed image. Compare it with real flare-free images By comparison, the results were calculated using formulas (11), (12), and (13) respectively. Loss. Then, the three losses are integrated using formula (14), where the loss weights ω1, ω2, and ω3 are set to 0.5, 0.5, and 0.01 respectively, to obtain the total loss L. Finally, backpropagation, gradient update, and network parameter update are performed on the network based on the total loss to complete one optimization iteration of the network training process. After 600K iterations, the network's ability to remove flares in real scenes is optimized. The network testing phase of this method is simpler than the training phase, without the need to perform loss calculation and network parameter update operations: the flare-damaged image to be processed is directly... The input is fed into the trained network, and through the sequential processing of the network's encoder, decoder, and feature refinement modules, a high-quality flare removal image is directly reconstructed. This demonstrates the practical application of nighttime flare removal. During the network testing phase, no loss calculation or parameter updates are required; nighttime flare images of any resolution can be directly input, and the network will directly output the flare-removed image after end-to-end processing.

[0107] Based on the above, this embodiment further provides the following experiments for comparative verification:

[0108] As shown in Table 1, the proposed method is compared with other state-of-the-art methods on the Flare7K real-world test set. Experimental results show that the proposed method achieves optimal performance across all evaluation metrics. Compared to existing state-of-the-art methods, this method achieves superior results in both the PSNR metric (reflecting image reconstruction quality) and the SSIM metric (measuring image structure preservation ability), indicating that it effectively removes flare regions while preserving image detail and texture information. Furthermore, this method also achieves the best performance on the G-PSNR and S-PSNR metrics proposed by Flare7K++ for flare region evaluation, further validating its advantages in flare region recovery.

[0109] Table 1. Quantitative results of each method on the Flare7K real-world nighttime flare test set.

[0110]

[0111] To more intuitively demonstrate the performance of different methods, this embodiment also provides a qualitative analysis of the results of each method on the Flare7K++ real-world test set, such as... Figure 3 As shown, this experiment uses a real-world nighttime flare test set from the Flare7K public dataset for validation. PSNR (Peak Signal-to-Noise Ratio) is used to measure image reconstruction quality; SSIM (Structural Similarity) is used to measure image structure preservation; LPIPS (Learned Perceptual Image Patch Similarity) is used to measure image perceptual quality; G-PSNR and S-PSNR are specific reconstruction quality evaluation metrics for flare regions. All metrics are calculated using methods commonly used in this field. Experimental results show that when facing complex flares in real-world scenes, the proposed method can effectively remove flare regions and significantly alleviate the shortcomings of existing methods in flare processing and poor detail and texture restoration. Figure 3 As shown in the third row, in scenarios where global flare damage is severe, existing methods often fail to adequately process the flare region, resulting in significant flare residue. However, the method presented in this paper can achieve precise focusing processing of the flare region through an interactive guidance strategy in the spatial and frequency domains, combined with a multi-scale sensing mechanism that coordinates global and local processes, thereby obtaining higher quality flare removal results.

[0112] As shown above, this invention constructs an end-to-end encoder-decoder network architecture based on Transformer. This network integrates Dual-Domain Interactive Guided Self-Attention (DGSA) and Multi-Scale Perception Gated (MAGB) modules as core units into all Transformer blocks, achieving efficient removal of nighttime flares while accurately preserving the original image structure and details. The core Transformer block of this network consists of DGSA and MAGB connected in series. DGSA guides the network to accurately focus on flare-damaged areas through dual-domain interactive learning of spatial domain global self-attention computation and frequency domain local feature modeling. MAGB, on the other hand, relies on multi-scale depthwise separable convolution to extract flare features from different receptive fields and adaptively fuses multi-scale features using a gating mechanism to achieve comprehensive perception of flare damage information of different sizes. Experimental results demonstrate that the method of this invention exhibits significant superiority in nighttime flare removal tasks. Compared with existing advanced methods, it can effectively remove flare loss information from real-world scenes, and the generated restored images have superior visual effects, providing an efficient technical approach and implementation solution for nighttime image flare removal tasks.

[0113] Based on the same inventive concept, this invention also provides a computer device, comprising: one or more processors, and a memory for storing one or more computer programs; the programs include program instructions, and the processor executes the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, used to implement one or more instructions, specifically for loading and executing one or more instructions stored in a computer storage medium to implement the above-described method.

[0114] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0115] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0116] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

[0117] This invention is not limited to the preferred embodiment described above. Anyone inspired by this invention can derive other forms of dual-domain interactive guided multi-scale sensing nighttime flare removal methods. All equivalent variations and modifications made within the scope of the claims of this invention should be included within the scope of this invention.

Claims

1. A dual-domain interactive guided multi-scale sensing nighttime flare removal method, characterized in that, include: The nighttime flare image is input into an end-to-end encoder-decoder neural network built on the Transformer architecture. After feature extraction and image reconstruction, the output is an image with flare removal. Each Transformer block of the encoder and decoder is composed of a dual-domain interactive guided self-attention module and a multi-scale perception gating module connected in series. The dual-domain interactive guided self-attention module first performs spatial domain self-attention calculation on the input features to obtain spatial domain features, then performs linear projection on the spatial domain features to enhance high-frequency signals and transforms them to the frequency domain. After frequency domain transformation processing, it is inversely transformed back to the spatial domain and outputs dual-domain fused features. The multi-scale perception gating module uses convolutional kernels with different receptive fields to extract multi-path features from the dual-domain fusion features to obtain multi-scale flare features. Based on the gating mechanism, the multi-scale flare features are adaptively fused to output multi-scale perception features.

2. The dual-domain interactive guided multi-scale sensing nighttime flare removal method according to claim 1, characterized in that: The feature processing of the Transformer block follows the following rules: first, perform layer normalization on the input features, then input them into the dual-domain interactive self-attention module, and finally perform residual connection between the module output and the original input features to obtain dual-domain fused features. After performing layer normalization on the dual-domain fusion features, the input is fed into the multi-scale perception gating module. The module output is then residually connected with the dual-domain fusion features to obtain the final output features of the Transformer block. And it satisfies the following formula: wherein, is the input feature of the Transformer block, LN is the layer normalization operation, DGSA is the processing operation of the dual-domain interaction guided self-attention module, is the dual-domain fusion feature, MAGB is the processing operation of the multi-scale perception gating module, is the final output feature of the Transformer block.

3. The dual-domain interactive guided multi-scale perception nightglow removal method of claim 1, wherein: The processing flow of the dual-domain interactive guided self-attention module includes: first, performing channel context encoding on the input features to generate the query vector, key vector, and value vector required for self-attention calculation; then, performing spatial domain self-attention calculation based on the vectors; and finally, performing linear projection on the spatial features to enhance high-frequency signals, decomposing them into real and imaginary parts in the frequency domain using a two-dimensional fast Fourier transform. After concatenating the real and imaginary parts along the channel dimension and performing linear modulation, the features are converted back to the spatial domain using a two-dimensional fast Fourier inverse transform, and the dual-domain fused features are output.

4. The dual-domain interactive guided multi-scale perception nightglow removal method of claim 1, wherein: The processing flow of the multi-scale perception gating module includes: first, performing channel expansion on the dual-domain fusion features, then splitting them into multiple groups of sub-features with equal number of channels along the channel dimension, assigning independent convolutional paths to each group of sub-features, and using depth-separable convolution with different kernel sizes for each path to extract multi-scale flare features corresponding to different receptive fields; cross-fusing multiple groups of multi-scale flare features to generate gating features and content features, performing activation on the gating features and fusing them element by element with the content features, and outputting multi-scale perception features after channel compression.

5. The dual-domain interactive guided multi-scale sensing nighttime flare removal method according to claim 4, characterized in that: The cross-fusion and adaptive fusion of the multi-scale flare features satisfy the following formula: wherein, is a gating feature, is a content feature, , , is a multi-scale flare feature output by different convolution pathways, , is a model learnable parameter, is a convolution operation, is a multi-scale perception feature, is a channel compression operation, G is an activation function, is an element-wise multiplication operation.

6. The dual-domain interactive guided multi-scale perception nightglow removal method of claim 1, wherein: The encoder is equipped with a multi-level feature extraction module. Each level adjusts the feature size and number of channels by downsampling. Each level stacks a corresponding number of Transformer blocks. The decoder is equipped with a multi-level feature reconstruction module corresponding to the feature scale of the encoder. Each level restores the feature size by upsampling. Each level stacks a corresponding number of Transformer blocks. The upsampled features are combined with the output features of the encoder at the corresponding scale level through channel splicing and fusion.

7. The dual-domain interactive guided multi-scale sensing nighttime flare removal method according to claim 1, characterized in that: After the nighttime flare image is input into the neural network, the initial convolutional feature extraction layer first completes the mapping from the image to the feature space, and then the initial feature map is input into the encoder. After the decoder outputs the reconstructed feature map, it first performs detail optimization through a feature refinement module with stacked Transformer blocks, and then reconstructs the image after flare removal.

8. The dual-domain interactive guided multi-scale sensing nighttime flare removal method according to claim 1, characterized in that: The training process of the neural network employs a joint loss function combining reconstruction loss, perceptual loss, and frequency domain loss for end-to-end supervision. The reconstruction loss is calculated based on the pixel-level difference between the output image and the real flare-free image, the perceptual loss is calculated based on the difference in deep semantic features of the image extracted by the pre-trained convolutional neural network, and the frequency domain loss is calculated based on the difference in the amplitude of the frequency domain spectrum after the image undergoes Fourier transform.

9. The dual-domain interactive guided multi-scale sensing nighttime flare removal method according to claim 1, characterized in that: The method is applied to the nighttime image preprocessing stage of autonomous driving systems or video surveillance systems.

10. A dual-domain interactive guided multi-scale sensing nighttime flare removal system, characterized in that, It includes an image input module, an end-to-end encoder-decoder neural network module built on the Transformer architecture, and an image output module; In the encoder and decoder of the neural network module, each Transformer block is composed of a dual-domain interactive guided self-attention module and a multi-scale perception gating module connected in series. The dual-domain interactive guided self-attention module is configured to first perform spatial domain self-attention calculation on the input features to obtain spatial domain features, then perform linear projection on the spatial domain features to enhance high-frequency signals and transform them to the frequency domain, and after frequency domain transformation processing, inversely transform them back to the spatial domain to output dual-domain fused features. The multi-scale perception gating module is configured to use convolutional kernels with different receptive fields to extract multi-path features from the dual-domain fusion features to obtain multi-scale flare features, and to perform adaptive fusion of the multi-scale flare features based on the gating mechanism to output multi-scale perception features.

Images