Image halo removing method based on mixed frequency domain decomposition and improved Transformer

Abstract

The invention relates to the technical field of image processing, in particular to an image halo removing method based on mixed frequency domain decomposition and improved Transformer, the method comprises the steps of constructing and training an image halo removing network model based on mixed frequency domain decomposition and improved Transformer, and the construction of the network model comprises the following steps: converting an input RGB image into an HSI color space; separating a hue component, a saturation component and a brightness component; decomposing the brightness component into high and low frequency components including local details and global illumination through a mixed frequency domain decomposition module; halo suppression is carried out on the low-frequency component by using an improved Transform network in which a mixed window attention mechanism is introduced; performing detail enhancement on the high-frequency component by adopting a residual network; and finally, fusing the processed components through a multi-scale fusion module of a dynamic weight interpolation strategy to reconstruct a high-quality halo-free RGB image. The method aims at efficiently removing the highlight halo in the complex low-light scene, and meanwhile, the detail integrity and the visual authenticity of the image are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to an image halo removal method based on hybrid frequency domain decomposition and improved Transformer, which is especially suitable for efficient removal of strong light halos and image quality enhancement in low-light imaging scenarios. Background Technology

[0002] In low-light imaging, when a point light source or a strong light source is present, light undergoes multiple reflections and scatterings within the lens, easily forming halos, including scattered halos and reflected halos. These optical artifacts significantly degrade image quality, leading to decreased contrast, loss of detail, and color distortion.

[0003] Existing halo removal methods often struggle to maintain local texture and edge integrity while suppressing global halos, and their generalization ability is limited under complex halo morphologies. While the traditional Transformer architecture performs well in image processing tasks, its computational complexity increases quadratically with image size, posing a significant computational burden when processing high-resolution images. Standard self-attention mechanisms lack targeted optimization for local image structures, limiting their effectiveness in tasks like halo removal that require simultaneous consideration of global illumination distribution and local detail preservation.

[0004] Therefore, how to suppress global halos while maintaining the integrity of local textures and edges and reducing computational complexity remains a challenge. Summary of the Invention

[0005] In view of this, the purpose of this invention is to overcome the shortcomings of the prior art and provide an image halo removal method based on hybrid frequency domain decomposition and improved Transformer. By designing an improved Transformer architecture with higher computational efficiency and combining it with a hybrid frequency domain decomposition strategy, efficient halo removal and detail preservation can be achieved.

[0006] To achieve the above objectives, the technical solution created by this invention is implemented as follows:

[0007] An image halo removal method based on hybrid frequency domain decomposition and improved Transformer includes constructing and training an image halo removal network model based on hybrid frequency domain decomposition and improved Transformer, wherein the construction of the network model includes the following steps:

[0008] S11: Convert the input RGB image to the HSI color space to obtain the hue component H, saturation component S, and brightness component I;

[0009] S12: The brightness component I is decomposed into a low-frequency component containing global illumination information and a high-frequency component containing local detail information using a hybrid frequency domain decomposition module;

[0010] S13: Input the low-frequency component into the improved Transformer network for halo suppression to obtain the halo-free low-frequency component;

[0011] S14: The high-frequency components are processed using a residual network (ResNet) to obtain enhanced high-frequency components;

[0012] S15: The de-halo low-frequency component, the enhanced high-frequency component, the hue component H, and the saturation component S are input to the multi-scale fusion module and fused using a dynamic weight interpolation strategy to reconstruct a high-quality, halo-free RGB output image.

[0013] Furthermore, the improved Transformer network adopts a hierarchical encoder-decoder architecture, containing multiple improved Transformer blocks, each of which includes:

[0014] The window partitioning module divides the input feature map into multiple non-overlapping windows to balance computational efficiency and receptive field size.

[0015] The multi-scale window attention module computes self-attention separately within different windows, capturing multi-scale feature representations through parallel attention paths;

[0016] The Local Enhancement Feedforward Network introduces depthwise separable convolution into the standard feedforward network, and enhances the ability to extract local features through 3×3 convolution kernels, making up for the shortcomings of pure self-attention in processing local details.

[0017] Furthermore, the calculation process of the hybrid window attention module specifically includes:

[0018] Input feature map It is also divided into windows of 8×8 size;

[0019] For each window partition, the features within the window are reshaped into a sequence, and the query matrix Q, key matrix K, and value matrix V are calculated respectively.

[0020] Calculate self-attention within each window: Where d is the projection dimension and B is the relative position code;

[0021] Adaptively weighted fuse the attention outputs of the two window sizes: Q = XW Q K = XW K V = XW V ,in These are learnable fusion parameters;

[0022] The fused features are mapped to the original dimensions through a linear projection layer.

[0023] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0024] (1) By designing an improved Transformer architecture and adopting a hybrid window attention mechanism, the computational complexity is reduced from O(HW) of standard self-attention while maintaining powerful global modeling capabilities. 2 Reduced to O(HW(M) 2 +N 2 )), where M and N are the window sizes, which significantly improves computational efficiency.

[0025] (2) By using a multi-scale window partitioning mechanism, the problem of limited attention receptive field of traditional windows is effectively solved, and better global illumination consistency is maintained in the halo removal task.

[0026] (3) The hybrid frequency domain decomposition strategy makes full use of the global frequency domain analysis capability of Fourier transform and the local detail preservation advantage of wavelet transform, providing the most suitable processing path for image components of different properties. Attached Figure Description

[0027] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0028] Figure 1 This is a schematic diagram illustrating the process of constructing an image halo removal network model based on hybrid frequency domain decomposition and improved Transformer, as described in an embodiment of the present invention.

[0029] Figure 2 This is a schematic diagram of the image halo removal network model based on hybrid frequency domain decomposition and improved Transformer as described in an embodiment of the present invention.

[0030] Figure 3 This is a schematic diagram illustrating the process of training an image halo removal network model based on hybrid frequency domain decomposition and improved Transformer, as described in an embodiment of the present invention.

[0031] Figure 4 This is an example image of a nighttime image with a halo as input to the image halo removal network model based on hybrid frequency domain decomposition and improved Transformer described in the embodiments of the present invention.

[0032] Figure 5This is a high-quality halo-free image processed by the image halo removal method based on hybrid frequency domain decomposition and improved Transformer as described in the embodiments of this invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and do not constitute a limitation thereof. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of the invention. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, some operations related to the invention are not shown or described in the specification. This is to avoid obscuring the core parts of the invention with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0034] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined to form various implementations. Furthermore, the order of the steps or actions in the method description can be changed or adjusted in a manner readily apparent to those skilled in the art. Therefore, the various orders in the specification and drawings are merely for the clear description of a particular embodiment and do not imply a mandatory order, unless otherwise stated that a particular order must be followed.

[0035] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0036] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0037] The invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0038] Example 1

[0039] like Figure 1 As shown, this invention provides an image halo removal method based on hybrid frequency domain decomposition and improved Transformer, including constructing and training an image halo removal network model based on hybrid frequency domain decomposition and improved Transformer. The construction of the image halo removal network model based on hybrid frequency domain decomposition and improved Transformer includes the following steps:

[0040] S11: Convert the input RGB image to the HSI color space and separate the hue, saturation and luminance components;

[0041] In this embodiment, the input low-light image with a halo will be used. Converting from the RGB color space to the HSI color space yields three independent components: H, S, and I. H ∈ [0°, 360°] represents the hue component, S ∈ [0, 1] represents the saturation component, and I ∈ [0, 1] represents the lightness component. The color space conversion calculation formula is as follows:

[0042]

[0043] S12: The luminance component is decomposed into high and low frequency components containing local details and global illumination through the hybrid frequency domain decomposition module;

[0044] The input is the luminance component I, denoted as I(x,y). First, a two-dimensional discrete Fourier transform is performed on the input luminance component I to obtain a complex matrix in the frequency domain. Set Gaussian low-pass mask The control cutoff frequency σ = 0.1·min(H,W) is used to extract the low-frequency components. High-frequency components are obtained through frequency domain residual operations.

[0045]

[0046] For low-frequency components and high frequency components Perform inverse discrete Fourier transforms on each image to obtain the low-frequency image I in the spatial domain. L and high-frequency image I high The calculation formula is:

[0047]

[0048] For the output high-frequency image I high A first-order Symlet wavelet transform is performed using the Symlet8 wavelet basis, followed by fine decomposition to obtain four subbands. The calculation formula is as follows:

[0049]

[0050] in, This is a low-frequency approximate sub-band. These are horizontal, vertical, and diagonal high-frequency sub-bands, respectively.

[0051] S13. Use an improved Transformer network with a hybrid window attention mechanism to suppress halo effects on low-frequency components;

[0052] In practice, the input features are divided into two types of windows: 8×8 and 16×16. Self-attention is computed in parallel within the two types of windows and weighted fusion is performed. Window self-attention promotes the flow of global information. 3×3 depthwise separable convolution is introduced into the standard feedforward network.

[0053] S14. Use a residual network to enhance the details of high-frequency components;

[0054] Using a residual network to enhance high-frequency components results in smoother image edges, richer details, and improved overall image quality. In practice, steps S14 and S13 can be performed simultaneously.

[0055] S15. A multi-scale fusion module using a dynamic weight interpolation strategy fuses and reconstructs a high-quality, halo-free RGB image by fusing the processed components.

[0056] Example 2

[0057] Based on the above embodiment 1, step S13 of the image halo removal method based on hybrid frequency domain decomposition and improved Transformer provided in this embodiment specifically includes:

[0058] A 4-stage encoder-decoder architecture is adopted. Each encoder stage contains two sparse window attention layers with a window size of 8×8 and four attention heads. This is followed by convolutional downsampling with a stride of 2, reducing the output feature resolution of the l-th stage to [value missing]. Double the number of channels;

[0059] The low-frequency component of the output in step S12 First, the number of channels is expanded to C=64 using convolutional layers to obtain the initial feature map. feature map Divided into Each window contains 8×8×C features and is non-overlapping.

[0060] Features of the k-th window The query, key, and value are generated through linear transformation, and the calculation formula is as follows:

[0061] Q (k) =X (k) W Q ,K (k) =X (k) W K V (k) =X (k) W V

[0062] in, The learnable parameter matrix is ​​d = C / 4 = 16, which is the projection dimension.

[0063] Attention weights are calculated by scaling the dot product, using the following formula:

[0064]

[0065] Output features This represents the result of reorganizing features within the window;

[0066] By employing a shifted window strategy, adjacent windows share some features, enhancing global dependencies. The calculation formula is as follows:

[0067] X shift =Roll(X,(Δx,Δy))

[0068] Where Δx = Δy = 4 pixels;

[0069] Each level of the decoder is upsampled through transposed convolution and skipped with the corresponding level features of the encoder; at the same time, a channel attention mechanism is introduced into the decoder to adaptively enhance the feature response of the detailed region.

[0070] The feature fusion calculation formula is as follows:

[0071]

[0072] For feature maps The weights are calculated using Global Average Pooling (GAP) and the fully connected layer. The calculation formula is as follows:

[0073] α c =σ(W2·δ(W1·GAP(X)))

[0074] Where σ is the Sigmoid function and δ is the ReLU activation. For fully connected layer parameters; set the channel weights α c Multiplying the original features channel by channel enhances the details of key channels. The calculation formula is as follows:

[0075] X enhanced =X⊙α c

[0076] Low-frequency component I L After processing in step S13, the output is a low-frequency image with halo suppression.

[0077] Example 3

[0078] Based on the above embodiment 1, step S14 of the image halo removal method based on hybrid frequency domain decomposition and improved Transformer provided in this embodiment specifically includes:

[0079] The three high-frequency detail subbands obtained by wavelet decomposition in step S12 A lightweight high-frequency enhancement module consisting of three residual blocks is input to enhance details and suppress any residual local halo noise, due to the low-frequency subband. The energy is extremely low and the information content is minimal, therefore it can be ignored. The calculation formula is as follows:

[0080]

[0081] An inverse wavelet transform is performed on the enhanced high-frequency detail subbands to reconstruct a clean high-frequency component image I. high The calculation formula is:

[0082]

[0083] Example 4

[0084] Based on the above embodiment 1, step S15 of the image halo removal method based on hybrid frequency domain decomposition and improved Transformer provided in this embodiment specifically includes:

[0085] The output I of step S13 low A three-level discrete wavelet transform (DWT) is performed, decomposing the data into a low-frequency baseband L0 and three high-frequency subbands {H1, H2, H3}, each corresponding to detail information at different scales:

[0086]

[0087] Among them, DWT LL and DWT HH These represent the low-pass and high-pass filtering operations of wavelet transform, respectively, where k is the decomposition level;

[0088] The high-frequency component I after output enhancement in step S14 high 'With a step size of 2 k-1 The stepwise downsampling is aligned with the low-frequency decomposition scale to generate the corresponding level of detail subbands. The calculation formula is as follows:

[0089]

[0090] The fusion weights for each scale are calculated using a dynamic weight interpolation mechanism. The calculation formula is as follows:

[0091]

[0092] in, This indicates channel concatenation, where σ is the Sigmoid function and Conv is a 3×3 convolutional layer.

[0093] The repaired high-frequency subband H' k with I high The weighted fusion calculation formula is as follows:

[0094]

[0095] Where ⊙ represents pixel-by-pixel multiplication;

[0096] The image is reconstructed step by step using inverse wavelet transform, and the final output is an image I without halo brightness components. fused The calculation formula is:

[0097] I fused =IDWT(L0,H'1,H'2,H'3)

[0098] Finally, the HSI image is reconstructed and inversely converted to an RGB image to output a high-quality, halo-free image. output The calculation formula is:

[0099]

[0100] I output =Combine(R) out G out B out )

[0101] Example 5

[0102] like Figure 2As shown, the image halo removal network model based on hybrid frequency domain decomposition and improved Transformer provided by the present invention includes a color space conversion module 51, a hybrid frequency domain image decomposition module 52, an improved Transformer halo suppression module 53, a high-frequency image enhancement module 54, and a multi-scale fusion module 55. Specifically, the color space conversion module 51 implements step S11 in any of the above embodiments; the hybrid frequency domain image decomposition module 52 implements step S12 in any of the above embodiments; the improved Transformer halo suppression module 53 implements step S13 in any of the above embodiments; the high-frequency image enhancement module 54 implements step S14 in any of the above embodiments; and the multi-scale fusion module 55 implements step S15 in any of the above embodiments; further details are omitted here.

[0103] Example 6

[0104] Based on any of the above embodiments, the image halo removal method based on hybrid frequency domain decomposition and improved Transformer provided in this embodiment, such as Figure 3 As shown, the training of the image halo removal network model based on hybrid frequency domain decomposition and improved Transformer includes the following steps:

[0105] S21. Prepare a halo image dataset and divide it into training set, validation set and test set according to the proportions;

[0106] The halo image dataset includes halo images, light source images, and background images, wherein the background images are night scene images taken by a camera. Preferably, the halo images and corresponding light source images used in the halo image dataset are randomly selected from camera-captured images and software-synthesized images with a 50% probability each.

[0107] S22. Train the image halo removal network model based on hybrid frequency domain decomposition and improved Transformer using halo image data in the training set;

[0108] Nighttime images with halos were generated from the halo images, light source images, and background images in the training set using image editing software.

[0109] S23. Optimize the parameters of the image halo removal network model based on hybrid frequency domain decomposition and improved Transformer using halo image data from the validation set;

[0110] S24. Input the halo image data from the test set into the optimized image halo removal network model based on hybrid frequency domain decomposition and improved Transformer.

[0111] The halo image data in the test set is used as data sample images and input into the pre-trained network model. The model parameters are fine-tuned to achieve the best halo removal effect. The model parameters are then saved.

[0112] Example 7

[0113] Inputting the image halo removal network model based on hybrid frequency domain decomposition and improved Transformer constructed and trained in this invention into the following: Figure 4 The nighttime image shown has a halo effect; the output image is as follows. Figure 5 As shown. It can be observed that... Figure 4 The nighttime image with a halo shown exhibits numerous scattered and reflected halos around the light source. Furthermore, artifacts caused by light passing through the lens are present in the center of the haloed nighttime image. In contrast, the image processed by the method described in this invention... Figure 5 This method not only effectively removes scattered halo, reflected halo, and unnecessary artifacts while preserving the original lighting and light source, but also significantly enhances the detail information in the image and improves the overall contrast.

[0114] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for removing image halos based on hybrid frequency domain decomposition and improved Transformer, characterized in that, Construct and train an image halo removal network model based on hybrid frequency domain decomposition and improved Transformer, wherein the construction of the image halo removal network model based on hybrid frequency domain decomposition and improved Transformer includes the following steps: S11: Convert the input RGB image to the HSI color space to obtain the hue component, saturation component, and brightness component; S12: The brightness component is decomposed into a low-frequency component containing global illumination information and a high-frequency component containing local detail information using a hybrid frequency domain decomposition module; S13: The low-frequency component is input into the improved Transformer network for halo suppression to obtain the halo-free low-frequency component. The improved Transformer network adopts a hybrid window attention mechanism to reduce computational complexity while maintaining global modeling capability. S14: Use a residual network to perform detail enhancement processing on the high-frequency components to obtain the enhanced high-frequency components; S15: The low-frequency component with de-halo, the enhanced high-frequency component, the hue component, and the saturation component are input to the multi-scale fusion module and fused using a dynamic weight interpolation strategy to reconstruct a high-quality, halo-free RGB output image.

2. The method according to claim 1, characterized in that, The improved Transformer network adopts a hierarchical encoder-decoder architecture, containing multiple improved Transformer blocks, each of which includes: The window partitioning module is used to dynamically divide the input feature map into multiple non-overlapping windows; The multi-scale window attention module calculates self-attention separately within different windows to capture multi-scale features; Locally enhanced feedforward networks introduce depthwise separable convolutions into standard feedforward networks to enhance local feature extraction capabilities.

3. The method according to claim 1, characterized in that, The hybrid frequency domain decomposition module employs a hybrid decomposition strategy combining Fourier transform and wavelet transform, including: First, the brightness component is decomposed into a global low-frequency illumination component and a global high-frequency residual component using Fourier transform; The global high-frequency residual components are then subjected to wavelet transform to decompose them into high-frequency detail subbands in multiple directions.

4. The method according to claim 3, characterized in that, In the Fourier transform decomposition process, a Gaussian low-pass filter is used to extract low-frequency components, and its cutoff frequency parameter is set to σ = 0.1·min(H,W), where H and W are the height and width of the image, respectively.

5. The method according to claim 1, characterized in that, The multi-scale fusion module uses a three-level discrete wavelet transform to decompose the processed low-frequency components into multi-scale sub-bands, and aligns the enhanced high-frequency components to the corresponding scales through downsampling; it calculates the fusion weights of each scale through a dynamic weight interpolation mechanism, and reconstructs the image step by step using inverse wavelet transform.

Images