A method for removing night image halo

Abstract

The present application provides a kind of night image halo removal method, including constructing prior knowledge, according to prior knowledge, construct initialization network and depth unfolding network, using initialization network to extract variable initial value set from night halo pollution image;Night halo pollution image and variable initial value set are input into depth unfolding network and are operated multiple iterations, and the final halo removal image is extracted from night halo pollution image;Wherein, depth unfolding network includes multiple proximal networks, and each proximal network participates in an iteration operation.The present application makes full use of prior knowledge, uses the depth unfolding network with multiple proximal networks to iterate mapping diagram, halo-free feature map, constraint variable and halo removal image, so as to extract the final halo removal image, to better preserve image texture and restore image details, under the premise of not increasing application cost, the halo and stripe artifact in image are preferably removed.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for removing halos from nighttime images. Background Technology

[0002] When photographing at night, strong light scattering or reflection within the lens can produce light spots and streaks, reducing image contrast and detail sharpness, and impacting visual effects and the performance of image processing algorithms. For example, a stereo camera in a nighttime driving scenario might mistake a halo for an obstacle. In a drone tracking an aerial target scenario, it might track the wrong target.

[0003] The first type of approach to reduce halos is to optimize hardware design, such as using specialized lenses or applying anti-reflective coatings to the lenses. However, these hardware improvements cannot completely eliminate halos, increase application costs, and cause unpredictable flare when the lens is contaminated. The second type of approach is to design image processing algorithms to remove halos from images. Recently, learning-based methods have achieved some success in image restoration. Examples include selecting several image restoration models as baseline models; introducing Fast Fourier Transform based on the Swing Transformer to extract global frequency features; and relying on cascaded neural networks combined with fine-tuning strategies, while simultaneously constructing triples and contrastive learning to optimize the model. However, existing methods do not fully utilize prior information when processing images, resulting in incomplete removal of flare and fringe artifacts, affecting visual quality.

[0004] Therefore, there is a need for a method that can make full use of prior information to better preserve image texture and restore image details, and remove light spots and stripe artifacts in images without increasing application costs. Summary of the Invention

[0005] To overcome the problems existing in related technologies, the purpose of this invention is to provide a method for removing halos from nighttime images. This method can make full use of prior information to better preserve image texture and restore image details, and remove light spots and stripe artifacts in images without increasing application costs.

[0006] A method for removing halos from nighttime images, comprising:

[0007] Construct prior knowledge, which includes pixel-level spatial consistency terms, feature-level detail enhancement terms, and feature-level noise suppression terms;

[0008] An initialization network and a deep unfolding network are constructed based on prior knowledge. The initialization network is used to extract a set of initial variable values ​​from the nighttime halo pollution image. The set of initial variable values ​​includes an initial halo mask image, an initial halo removal image, an initial mapping map, an initial halo-free feature map, and initial constraint variables.

[0009] The nighttime halo pollution image and the initial set of variable values ​​are input into a depth unrolling network for multiple iterative operations to extract the final halo-removed image from the nighttime halo pollution image; wherein, the depth unrolling network includes multiple near-end networks, and each near-end network participates in one iterative operation.

[0010] In a preferred embodiment of the present invention, the construction of prior knowledge includes:

[0011] Construct a halo layer decomposition model based on the following formula:

[0012] ;

[0013] Where I is the nighttime halo pollution image, B is the halo removal image, and F is the halo mask image;

[0014] Construct the first optimization function corresponding to the pixel-level spatial consistency term:

[0015] ;

[0016] Where f represents the Fibonacci norm, E1 is the Fibonacci norm term, which is used to ensure the fidelity of image data.

[0017] In a preferred embodiment of the present invention, the construction of prior knowledge includes:

[0018] Feature maps are extracted using a convolutional dictionary based on the following formula:

[0019] ;

[0020] ;

[0021] ;

[0022] Where ZI is the feature map of the nighttime halo pollution image, ZB is the feature map without halo, ZF is the feature map of the halo mask; D is the convolution dictionary. It is a convolution operator;

[0023] Construct a second optimization function that includes the aforementioned feature-level detail enhancement term:

[0024] ;

[0025] Where M is a mapping graph, and * represents element-wise multiplication. The first penalty coefficient, This is the second penalty coefficient. The first regularization coefficient is... E1 is the first prior term, and E2 is the second optimization function.

[0026] In a preferred embodiment of the present invention, the construction of prior knowledge includes:

[0027] Construct the feature-level noise prior terms according to the following formula:

[0028] ;

[0029] in, This is the second regularization coefficient. The norm of the element-wise multiplication of the halo-free feature map and the halo mask feature map;

[0030] Construct the third optimization function for the feature-level noise suppression term:

[0031] ;

[0032] in, E3 is the norm of the convolution of the convolution dictionary and the halo mask image, and the element-wise product of the halo-free feature map, and is the third optimization function.

[0033] In a preferred embodiment of the present invention, before extracting the initial set of variable values ​​from the nighttime halo pollution image using the initialization network, the method further includes:

[0034] Calculate the Charbonnier consistency loss between the halo-removed image and the ground truth to obtain the first loss function;

[0035] Calculate the Charbonnier consistency loss between the halo mask image and the training set to obtain the second loss function;

[0036] The first loss function and the second loss function are added together to obtain the third loss function;

[0037] The fourth loss function is obtained by calculating the Vgg consistency loss between the halo-removed image and the ground truth.

[0038] Calculate the Vgg consistency loss between the halo mask image and the training set to obtain the fifth loss function;

[0039] Adding the fourth and fifth loss functions together yields the sixth loss function;

[0040] The third loss function and the sixth loss function are weighted and summed to obtain the final loss function;

[0041] The final loss function is used to train both the initialization network and the deep unfolded network end-to-end.

[0042] In a preferred embodiment of the present invention, the step of inputting the nighttime halo pollution image and the set of initial variable values ​​into a depthwise unfolded network for multiple iterative calculations includes:

[0043] During the t-th iteration, the gradient of the t-th mapping is calculated. The t-th mapping and its gradient are then input into the t-th proximal network of the deep unrolled network to calculate the (t+1)-th mapping.

[0044] Convolve the convolution dictionary with the halo mask image to obtain the t-th halo mask feature; calculate the gradient of the t-th halo-free feature map; input the t-th halo-free feature map, the gradient of the t-th halo-free feature map, and the t-th halo mask feature into the soft threshold operator to obtain the (t+1)-th halo-free feature map.

[0045] Calculate the gradient of the t-th constraint variable, input the t-th constraint variable and its gradient into the t-th proximal network of the deep unfolded network, and calculate the (t+1)-th constraint variable.

[0046] Calculate the gradient of the t-th halo-removed image, and calculate the (t+1)-th halo-removed image based on the gradient of the t-th halo image and the t-th halo-removed image.

[0047] In a preferred embodiment of the present invention, the step of inputting the gradient of the t-th mapping map and the t-th mapping map into the t-th proximal network of the depthwise unfolded network to calculate the (t+1)-th mapping map includes:

[0048] Calculate the (t+1)th mapping graph using the following formula:

[0049] ;

[0050] Among them, M t+1 For the (t+1)th mapping, M t For the t-th mapping, Let be the gradient of the t-th mapping. Update parameters for the mapping graph, * indicates element-wise multiplication, prox t1 The t1th near-end network is the mapping graph of the deep unfolded network.

[0051] In a preferred embodiment of the present invention, the step of inputting the t-th halo-free feature map, the gradient of the t-th halo-free feature map, and the t-th halo mask feature into a soft threshold operator to obtain the (t+1)-th halo-free feature map includes:

[0052] Calculate the (t+1)th halo-free feature map using the following formula:

[0053] ;

[0054] Among them, ZB t+1 Let t+1 be the halo-free feature map, soft be the soft thresholding operator, and ZB be the value of the halo-free feature map. t For the t-th feature map without halo, Update parameters for the halo-free feature map. Let be the gradient of the t-th halo-free feature map. This is the second regularization coefficient. The first penalty coefficient, D is the second penalty coefficient. t Let F be the dictionary for the t-th convolution, and F be the halo mask image. is the convolution operator, and * represents element-wise multiplication.

[0055] In a preferred embodiment of the present invention, calculating the (t+1)th constraint variable includes:

[0056] Calculate the (t+1)th constraint variable using the following formula:

[0057] ;

[0058] Where, N t+1 For the (t+1)th constraint variable, prox t2 For the t2th proximal network of the constraint variables of the deep unfolded network, N t Let t be the t-th constraint variable. For updating parameters of constraint variables, Let t be the gradient of the t-th constraint variable, and * denote element-wise multiplication.

[0059] In a preferred embodiment of the present invention, the calculation of the (t+1)th halo removal image includes:

[0060] The halo removal image of the (t+1)th time is calculated using the following formula:

[0061] ;

[0062] Among them, B t+1 For the (t+1)th halo removal image, B t Remove the halo image for the t-th halo. Update image parameters to remove halo effect. The gradient of the image is removed for the t-th halo, and * represents element-wise multiplication.

[0063] The beneficial effects of this invention are as follows:

[0064] The nighttime image halo removal method provided by this invention includes constructing prior knowledge, which comprises a pixel-level spatial consistency term, a feature-level detail enhancement term, and a feature-level noise suppression term. The pixel-level spatial consistency term, based on the principle of image additive geometry, decomposes the nighttime halo-polluted image into a halo-removed image and a halo-mask image, thereby simplifying computation. The pixel-level spatial consistency term uses a Fibonacci norm term for constraint, ensuring data fidelity. The feature-level detail enhancement term uses a convolutional dictionary to extract richer features from the nighttime halo-polluted image, the halo-removed image, and the halo-mask image, enhancing the learning capabilities of the initialization network and the deep unfolding network. The feature-level detail enhancement term uses the Fibonacci norm to constrain the relationship between the halo-removed image, the halo-free feature map, and the convolutional dictionary D, thereby adaptively suppressing stripe artifacts in the nighttime halo-polluted image and enhancing details in the halo region. The feature-level noise suppression term uses a 1-norm to constrain the halo-free feature map and the halo-mask feature map, thereby suppressing the halo noise content of the halo-free feature map by increasing the halo-mask feature map. The nighttime halo-polluted image and prior knowledge are input into an initialization network to extract a set of initial variable values. This set includes an initial halo mask image, an initial halo-removed image, an initial mapping map, an initial halo-free feature map, and initial constraint variables. The nighttime halo-polluted image and the initial variable value set are then input into a deep unfolding network for multiple iterative operations to extract the final halo-removed image. Each iteration of the deep unfolding network employs a proximal network to learn implicit priors, combining the advantages of CNNs in local feature modeling with the advantages of Transformers in global information modeling. This invention, by fully utilizing prior knowledge, uses a deep unfolding network with multiple proximal networks to iterate over the mapping map, halo-free feature map, constraint variables, and halo-removed image to extract the final halo-removed image. This better preserves image texture and restores image details, effectively removing light spots and stripe artifacts from the image without increasing application costs. Attached Figure Description

[0065] Figure 1 This is a flowchart of the nighttime image halo removal method of the present invention;

[0066] Figure 2 This is a flowchart of the present invention, which inputs a nighttime halo pollution image and a set of initial variable values ​​into a depth unrolling network for multiple iterative calculations;

[0067] Figure 3 This is a schematic diagram of the network architecture of the UBFormer model of the present invention;

[0068] Figure 4 This is a schematic diagram of the network architecture of LWTB in this invention. Detailed Implementation

[0069] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0070] Example 1

[0071] like Figure 1 As shown, this embodiment provides a method for removing halos from nighttime images, including:

[0072] S1: Construct prior knowledge, which includes pixel-level spatial consistency terms, feature-level detail enhancement terms, and feature-level noise suppression terms.

[0073] S2: Construct an initialization network and a deep unfolding network based on prior knowledge, and use the initialization network to extract a set of initial variable values ​​from the nighttime halo pollution image; wherein, the set of initial variable values ​​includes an initial halo mask image, an initial halo removal image, an initial mapping map, an initial halo-free feature map, and initial constraint variables.

[0074] S3: Input the nighttime halo pollution image and the initial set of variable values ​​into the depth unrolling network for multiple iterative operations to extract the final halo removal image from the nighttime halo pollution image; wherein, the depth unrolling network includes multiple proximal networks, and each proximal network participates in one iterative operation.

[0075] The deep unfolded network of this invention comprises multiple modules, each with a proximal network, and each module performs one iterative operation. Both the initialization network and the proximal network employ the UBFormer model. The UBFormer model utilizes multi-scale features to learn the implicit priors of the image. The UBFormer model includes a residual module and a large-window Transformer module. Nighttime halo pollution images refer to images with halo and stripe artifacts in nighttime scenes. Removing halo and stripe artifacts from nighttime halo pollution images can improve image resolution.

[0076] The construction of prior knowledge includes:

[0077] Construct a halo layer decomposition model based on the following formula:

[0078] (1)

[0079] Where I is the nighttime halo pollution image, B is the halo removal image, and F is the halo mask image.

[0080] The pixel-level spatial consistency term employs a halo layer decomposition model. Based on the principle of image additive properties, this model decomposes a nighttime halo-polluted image into a halo-removed image and a halo mask image. In other words, the nighttime halo-polluted image is composed of the halo-removed image and an additive halo mask image, where the halo mask image is a noise image superimposed on the halo-removed image. Given a known nighttime halo-polluted image I and a halo mask image F estimated via a network, the MAP estimation based on Bayes' theorem can transform the halo removal problem into an energy function minimization problem. The first optimization function corresponding to the pixel-level spatial consistency term is constructed using the following formula:

[0081] (2)

[0082] Where f represents the Fibonacci norm, E1 is the first optimization function, and the Fibonacci norm is used to ensure the fidelity of the image data.

[0083] The construction of prior knowledge includes:

[0084] Feature maps are extracted using a convolutional dictionary based on the following formula:

[0085] (3)

[0086] (4)

[0087] (5)

[0088] Where ZI is the feature map of the nighttime halo pollution image, ZB is the feature map without halo, ZF is the feature map of the halo mask; D is the convolution dictionary. It is a convolution operator;

[0089] Formulas (3) to (5) can be rewritten as:

[0090] (6)

[0091] (7)

[0092] (8)

[0093] Where D is the convolution dictionary, It is a convolution operator, where c represents the image channels, n represents the feature channels, and total represents the total number of feature channels. When c=1, it represents the red channel (R channel) of the image; when c=2, it represents the green channel (G channel) of the image; and when c=3, it represents the blue channel (B channel) of the image.

[0094] Combining pixel-level spatial consistency terms and feature-level detail enhancement terms, while introducing a first penalty coefficient. Second penalty coefficient Construct a second optimization function that includes the feature-level detail enhancement term:

[0095] (9)

[0096] Where M is a mapping graph, and * represents element-wise multiplication. The first penalty coefficient, This is the second penalty coefficient. The first regularization coefficient is . E1 is the first prior term, and E2 is the second optimization function.

[0097] A convolutional dictionary is used to ensure richer feature maps ZI, ZB, and ZF are extracted from I, B, and F, thereby enhancing the model's learning ability. Furthermore, by extracting images into multi-channel feature maps, the convolutional dictionary reduces information loss during each iteration update.

[0098] The difference between the convolutions of ZB and D and B is used to constrain the relationship between the feature ZB and the image B to be restored. This maps the feature ZB, which has learned implicit priors, back to the halo-removed image B, thereby enhancing the detail information of the halo-removed image B. The mapping map M is used to balance the removal of stripe artifacts and the enhancement of detail features. When stripe artifacts are detected, M is larger, resulting in a smaller value for ZB, thus suppressing stripe artifacts. When halo regions are detected, M is smaller, resulting in a larger value for ZB, thus enhancing the details of the halo region.

[0099] By constraining the relationship between the Frobenius norm-constrained mapping map M and the convolution dictionary D, the halo image I and the halo-free feature ZB, ZB can learn this prior constraint, thus adaptively suppressing stripe artifacts and enhancing details in the halo region. Similarly, by constraining the relationship between the halo removal image B, the halo-free feature ZB, and the convolution dictionary D, B can learn this prior constraint from ZB, thus adaptively suppressing stripe artifacts and enhancing details in the halo region.

[0100] The construction of prior knowledge includes:

[0101] Construct the feature-level noise prior terms according to the following formula:

[0102] (10)

[0103] in, This is the second regularization coefficient. The norm of the element-wise multiplication of the halo-free feature map and the halo mask feature map;

[0104] Construct the third optimization function for the feature-level noise suppression term:

[0105] (11)

[0106] in, E3 is the norm of the convolution of the convolution dictionary and the halo mask image, and the element-wise product of the halo-free feature map, and is the third optimization function.

[0107] By using a feature-level noise suppression term, the halo noise content in halo-free feature maps can be suppressed even when the halo mask feature map value is large. A constraint variable N and a third penalty coefficient are introduced. This makes N=ZB, transforming the optimization problem into:

[0108] (12)

[0109] Where I represents the nighttime halo pollution image, B represents the halo removal image, F represents the halo mask image, and D represents the convolution dictionary. Here, ZB represents the convolution operator, M represents the halo-free feature map, and * represents element-wise multiplication. The first penalty coefficient, This is the second penalty coefficient. This is the third penalty coefficient. The first regularization coefficient is . This is the second regularization coefficient. This is the third regularization coefficient. As the first prior, The first prior term is represented by the proximal operator of the mapping graph M, which is estimated by the corresponding proximal network. The proximal operator of the mapping graph M is used to represent the first prior term, and the proximal gradient algorithm is used to solve for the first prior term. The second prior term is represented by the proximal operator of the constraint variable N, which is estimated by the corresponding proximal network. The proximal operator of the constraint variable N is used to represent the second prior term, and the proximal gradient algorithm is used to solve for the second prior term.

[0110] Constructing an initialization network and a deep unrolling network based on prior knowledge involves inputting prior knowledge into these networks. Different modules of the initialization network are interconnected during the extraction of initial variable values ​​from nighttime halo pollution images. During the iterative computation of prior knowledge by the deep unrolling network, different modules within the deep unrolling network are iteratively linked together. This process creates a data processing pathway between the initialization and deep unrolling networks, guiding them to learn prior knowledge. The initialization and deep unrolling networks constructed based on this prior knowledge can effectively remove halo mask images from nighttime halo pollution images, i.e., remove halos and stripe artifacts, resulting in a final halo-removed image.

[0111] The nighttime image halo removal method provided in this embodiment includes constructing prior knowledge, which includes pixel-level spatial consistency terms, feature-level detail enhancement terms, and feature-level noise suppression terms. The pixel-level spatial consistency term, based on the principle of image additive modeling, decomposes the nighttime halo-polluted image into a halo-removed image and a halo-mask image, thereby simplifying computation. The pixel-level spatial consistency term uses a Fibonacci norm term for constraint, ensuring data fidelity. The feature-level detail enhancement term uses a convolutional dictionary to extract richer features from the nighttime halo-polluted image, the halo-removed image, and the halo-mask image, enhancing the learning capabilities of the initialization network and the deep unfolding network. The feature-level detail enhancement term uses the Fibonacci norm to constrain the relationship between the halo-removed image, the halo-free feature map, and the convolutional dictionary D, thereby adaptively suppressing stripe artifacts in the nighttime halo-polluted image and enhancing details in the halo region. The feature-level noise suppression term uses a 1-norm to constrain the halo-free feature map and the halo-mask feature map, thereby suppressing the halo noise content in the halo-free feature map by increasing the halo-mask feature map. The nighttime halo-polluted image and prior knowledge are input into an initialization network to extract a set of initial variable values. This set includes an initial halo mask image, an initial halo-removed image, an initial mapping map, an initial halo-free feature map, and initial constraint variables. The nighttime halo-polluted image and the initial variable value set are then input into a deep unfolding network for multiple iterative operations to extract the final halo-removed image. Each iteration of the deep unfolding network employs a proximal network to learn implicit priors, combining the advantages of CNNs in local feature modeling with the advantages of Transformers in global information modeling. This invention, by fully utilizing prior knowledge, uses a deep unfolding network with multiple proximal networks to iterate over the mapping map, halo-free feature map, constraint variables, and halo-removed image to extract the final halo-removed image. This better preserves image texture and restores image details, effectively removing light spots and stripe artifacts from the image without increasing application costs.

[0112] Example 2

[0113] like Figure 1 As shown, this embodiment provides a method for removing halos from nighttime images. This embodiment describes the differences between it and Embodiment 1. The method includes:

[0114] S1: Construct prior knowledge, which includes pixel-level spatial consistency terms, feature-level detail enhancement terms, and feature-level noise suppression terms.

[0115] S2: Construct an initialization network and a deep unfolding network based on prior knowledge, and use the initialization network to extract a set of initial variable values ​​from the nighttime halo pollution image; wherein, the set of initial variable values ​​includes an initial halo mask image, an initial halo removal image, an initial mapping map, an initial halo-free feature map, and initial constraint variables.

[0116] S3: Input the nighttime halo pollution image and the initial set of variable values ​​into the depth unrolling network for multiple iterative operations to extract the final halo removal image from the nighttime halo pollution image; wherein, the depth unrolling network includes multiple proximal networks, and each proximal network participates in one iterative operation.

[0117] like Figure 2 As shown, the step of inputting the nighttime halo pollution image and the initial set of variable values ​​into a deep unrolled network for multiple iterative calculations includes:

[0118] S31: During the t-th iteration, calculate the gradient of the t-th mapping, input the t-th mapping and its gradient into the t-th proximal network of the deep unrolled network, and calculate the (t+1)-th mapping.

[0119] S32: Convolve the convolution dictionary with the halo mask image to obtain the t-th halo mask feature; calculate the gradient of the t-th halo-free feature map, input the t-th halo-free feature map, the gradient of the t-th halo-free feature map and the t-th halo mask feature into the soft threshold operator to obtain the (t+1)-th halo-free feature map.

[0120] S33: Calculate the gradient of the t-th constraint variable, input the t-th constraint variable and its gradient into the t-th proximal network of the deep unfolded network, and calculate the (t+1)-th constraint variable.

[0121] S34: Calculate the gradient of the t-th halo-removed image, and calculate the (t+1)-th halo-removed image based on the gradient of the t-th halo image and the t-th halo-removed image.

[0122] The mapping map, halo-free feature map, constraint variables, and halo-removed image are iteratively updated using formulas (13)-(16):

[0123] ;

[0124] Among them, M t+1 For the (t+1)th mapping, M t For the t-th mapping, Let be the gradient of the t-th mapping. Update parameters for the mapping graph, * indicates element-wise multiplication, prox t1 Let t1 be the t1th near-end network of the mapping graph of the deep unfolded network, where t1 = t. ZB t+1 Let t+1 be the halo-free feature map, soft be the soft thresholding operator, and ZB be the value of the halo-free feature map. t For the t-th feature map without halo, Update parameters for the halo-free feature map. Let be the gradient of the t-th halo-free feature map. This is the second regularization coefficient. The first penalty coefficient, D is the second penalty coefficient. t Let F be the dictionary for the t-th convolution, and F be the halo mask image. is the convolution operator, and * represents element-wise multiplication. N t+1 For the (t+1)th constraint variable, prox t2 For the t2th proximal network of the constraint variables of the deep unfolded network, t2=t, N t Let t be the t-th constraint variable. For updating parameters of constraint variables, Let be the gradient of the t-th constraint variable, and * denote element-wise multiplication. B t+1 For the (t+1)th halo removal image, B t Remove the halo image for the t-th halo. Update image parameters to remove halo effect. The gradient of the image is removed for the t-th halo, and * represents element-wise multiplication.

[0125] t1 and t2 are numerically equal to t. During the t-th iteration of the deep unfolded network on the prior knowledge, there are two proximal networks, one of which is used to compute M. t+1 Another proximal network is used to compute N. t+1 For example, during the fifth iteration of the deep unfolded network on prior knowledge, the fifth proximal network of the deep unfolded network's mapping graph is used to compute M. 6 The fifth proximal network of the constraint variables of the deep unfolded network is used to calculate N. 6At this point, t1 and t2 are both equal to 5. t1 represents the 5th proximal network in the mapping graph of the deep unfolded network, and t2 represents the 5th proximal network in the constraint variables of the deep unfolded network.

[0126] The calculation formula for the soft operator is as follows:

[0127] (17);

[0128] Where soft(input,r) is the soft operator, sign is the sign function, input is the independent variable of the soft operator, max is the maximum value function, and r is the adjustment parameter of the soft operator. In this embodiment, r=1 is taken as an example. When input>1, the soft operator grows linearly with a slope of 1; when -1≤input≤1, soft(input)=0; when input<-1, the soft operator grows linearly with a slope of 1.

[0129] Calculate using formula (18)-formula (21) , , and :

[0130] ;

[0131] in, , where I is the third penalty coefficient, B is the image with halo pollution at night, and F is the image with halo removal.

[0132] In formulas (13)-(16) and (18)-(21), t represents the number of iterations, 0 ≤ t ≤ num, num+1 represents the total number of iterations, and num ≥ 2. When t = 0, M 0 For the initial mapping graph, ZB 0 For the initial halo-free feature map, N 0 B is the initial constraint variable. 0 The initial halo-removed image is used. Substituting the initial mapping map, initial halo-free feature map, initial constraint variables, initial halo-removed image, and initial convolution dictionary into formulas (18)-(21), the following calculations are performed: , , and Then , , and Substituting into formula (13) - formula (16), calculate , , and Repeat the above process until num+1 iterations are completed to obtain the final halo-removed image. The mapping map, constraint variables, and halo-removed image are all optimized using the proximal gradient descent algorithm, while the halo-free feature map is optimized using both the proximal gradient descent algorithm and the iterative shrinkage threshold algorithm.

[0133] The UBformer model is a 4-layer U-shaped network that introduces multi-scale features to learn implicit prior information. The UBformer model includes a downsampling module, an upsampling module, a residual module, and a large window Transformer module (LWTB).

[0134] like Figure 3 As shown, the overall framework of the UBFormer model is based on an encoder-decoder structure design, given an input feature map. Where H represents the height of the feature map, W represents the width of the feature map, and C represents the number of feature channels. In the backbone network, this paper uses 1×1 convolution to extract features, in order to reduce the computational cost when the number of feature channels C is large.

[0135] For the downsampling and upsampling modules, this invention employs pixel recombination and pixel shuffling operations. For the residual module, this invention replaces the 3×3 convolution module of U-Net with 3×3 depthwise separable convolution DWConv and 1×1 convolution Conv to reduce computation and parameter count, and replaces the ReLU activation function with the LeakyReLU activation function to preserve information corresponding to negative values.

[0136] At the smallest feature layer, this invention designs LWTB, which uses a Transformer model to improve the network's learning ability and a larger window to increase the model's global receptive field. Due to the high parallelism of GPUs, this invention finds that a larger window does not introduce excessive latency and computational cost compared to a smaller window. To balance speed and accuracy, this embodiment sets the window size to 32×32. Although the window Transformer uses multiple heads to increase feature diversity, the features are still relatively similar. Therefore, this invention adds a spatially differentiated feature to each feature in the group after the self-attention mechanism to increase the spatial diversity between features and introduces a gating mechanism to enhance the network's non-linear expressive power.

[0137] At the minimum feature scale layer, this invention employs LWTB. Figure 4 This is a schematic diagram of the LWTB network architecture. The window attention used by LWTB is different from that of Uformer. This invention does not use relative position encoding because relative position encoding will consume more video memory resources in the case of a larger window and has limited performance gain.

[0138] Based on the window-based Transformer mechanism, SABlock's invention designs a DNet (Differentiated Network) to address the low-rank problem of the self-attention mechanism, and designs a Conv module to enhance the model's nonlinear expressive power.

[0139] The initial feature map X0 is transformed into the first feature map X1 through window transformation. The first feature map X1 is processed by SABlock to obtain the second feature map X2. The second feature map X2 is processed by the FFN module to obtain the third feature map X3. The second and third feature maps are calculated using the following formula:

[0140] ;(twenty two)

[0141] ;(twenty three)

[0142] In the self-attention block, this invention first encodes the query matrix Q, key matrix K, value matrix V, and attention score matrix A using a Conv module. The encoding networks for these four feature matrices do not share weights. The Conv module consists of 3×3 depthwise convolutions, a ReLU activation function, and a 1×1 convolution to encode spatial and channel information. Since the self-attention mechanism uses multiple feature-sharing heads, this invention introduces a DNet to enhance the diversity between features after self-attention and improve the model's expressive power. The DNet consists of 3×3 depthwise convolutions, a ReLU activation function, and another 3×3 depthwise convolution to increase the diversity between features while reducing parameter redundancy and computational complexity, all within a lightweight framework.

[0143] The fourth feature map is calculated using the following formula:

[0144] ;(twenty four)

[0145] Where X4 is the fourth feature map, softmax is the normalization exponential function, Q is the query matrix, and K... T Let d be the transpose of the key matrix. k V is the dimension of the key matrix, V is the value matrix, and A is the attention score matrix.

[0146] This embodiment utilizes the near-end network UBFormer to learn implicit priors. UBFormer includes a residual module and a large-window Transformer module. The residual module uses depthwise separable convolutions and pointwise convolutions to efficiently capture information from a large receptive field and prevents gradient vanishing through skip connections. The large-window Transformer module includes a self-attention block and a feedforward neural network. Since the self-attention mechanism uses multiple feature-sharing heads, this invention introduces a DNet to enhance the diversity between features after self-attention, thereby improving the model's expressive power.

[0147] Example 3

[0148] This embodiment provides a method for removing halos from nighttime images. This embodiment describes the differences from Embodiment 1, and the method includes:

[0149] S1: Construct prior knowledge, which includes pixel-level spatial consistency terms, feature-level detail enhancement terms, and feature-level noise suppression terms.

[0150] S2: Construct an initialization network and a deep unfolding network based on prior knowledge, and use the initialization network to extract a set of initial variable values ​​from the nighttime halo pollution image; wherein, the set of initial variable values ​​includes an initial halo mask image, an initial halo removal image, an initial mapping map, an initial halo-free feature map, and initial constraint variables.

[0151] S3: Input the nighttime halo pollution image and the initial set of variable values ​​into the depth unrolling network for multiple iterative operations to extract the final halo removal image from the nighttime halo pollution image; wherein, the depth unrolling network includes multiple proximal networks, and each proximal network participates in one iterative operation.

[0152] Before extracting the initial set of variable values ​​from the nighttime halo pollution image using the initialization network, the method further includes:

[0153] S21': Calculate the Charbonnier consistency loss between the halo-removed image and the ground truth to obtain the first loss function.

[0154] S22': Calculate the Charbonnier consistency loss between the halo mask image and the training set to obtain the second loss function.

[0155] S23': Add the first loss function and the second loss function to obtain the third loss function.

[0156] S24': Calculate the Vgg consistency loss between the halo-removed image and the ground truth to obtain the fourth loss function.

[0157] S25': Calculate the Vgg consistency loss between the halo mask image and the training set to obtain the fifth loss function.

[0158] S26': Add the fourth loss function and the fifth loss function to obtain the sixth loss function.

[0159] S27': The third loss function and the sixth loss function are weighted and summed to obtain the final loss function.

[0160] S28': The final loss function is used to train the initialization network and the deep unfolded network end-to-end simultaneously.

[0161] Calculate the first loss function using the following formula:

[0162] (25)

[0163] in, Let B be the first loss function, B be the halo-removed image, and gt be the true value. The square of the L2 norm of the difference between the image and the true value is used to remove the halo. Adjust the parameters for the loss function.

[0164] The second loss function is calculated using the following formula:

[0165] (26)

[0166] in, Let F be the second loss function, F be the halo mask image, and flare be the training set. Let L be the square of the L2 norm of the difference between the halo mask pattern and the training set.

[0167] The third loss function is calculated using the following formula:

[0168] (27)

[0169] in, For the third loss function, For the first loss function, Let be the second loss function, B be the halo-removed image, gt be the ground truth, F be the halo mask image, and flare be the training set.

[0170] The fourth loss function is calculated using the following formula:

[0171] (28)

[0172] in, The fourth loss function is defined by VggNet(B), which is the output of the Vgg network when the input is B, and VggNet(gt), which is the output of the Vgg network when the input is gt. Let L be the norm of the difference between the output of the Vgg network when the input is B and the output of the Vgg network when the input is gt.

[0173] The fifth loss function is calculated using the following formula:

[0174] (29)

[0175] in, Here, VggNet(F) is the fifth loss function, VggNet(F) is the output of the Vgg network when the input is F, and VggNet(flare) is the output of the Vgg network when the input is flare. Let L be the norm of the difference between the output of the Vgg network with input F and the output of the Vgg network with input flare.

[0176] The sixth loss function is calculated using the following formula:

[0177] (30)

[0178] in, The sixth loss function, This is the fourth loss function. This is the fifth loss function.

[0179] Calculate the final loss function using the following formula:

[0180] (31)

[0181] Among them, l final The final loss function is defined by w1, which is the first adjustment weight, and w2 is the second adjustment weight. In this embodiment, both w1 and w2 are set to 1.

[0182] The final loss function is used to train both the initialization network and the deep unfolding network end-to-end to optimize their network parameters. The nighttime halo removal model in this embodiment is trained on the Flare7K++ dataset, which includes the synthetic dataset Flare7K and the real dataset Flare-R. Flare7K provides 5000 synthetic halo images, while Flare-R has 962 real halo images. This embodiment utilizes the Flare7K++ synthesis pipeline to generate halo images and corresponding halo-removed images. The background images are 23949 images randomly selected from the 24K Flickr dataset, while the halo images and light sources are randomly selected from Flare7K and Flare-R, respectively, each accounting for 50%. The enhanced halo images in the Flare7K++ dataset are obtained by performing gamma correction and image enhancement on the synthetic halo images from Flare7K and the real halo images from Flare-R. Image enhancement includes rotation, translation, cropping, scaling, Gaussian blur, and random flipping. For each background image, the RGB values ​​are multiplied by a random factor and Gaussian noise is added. The result is normalized to [0,1] to serve as the ground truth for the corresponding halo-free image. In this embodiment, the light source is subtracted from the enhanced halo image to generate a halo-removed image. The enhanced halo image is then combined with the background image to generate a nighttime halo-polluted image, which is used as input data to initialize the network. During testing, this embodiment uses the Flare7K test set, which includes 100 pairs of real images, one set of which are nighttime halo-polluted images, and the other set of real images are halo-removed images.

[0183] This embodiment employs an end-to-end training approach, simultaneously training both the initialization network and the deep unfolded network. The Adam optimizer is used for multiple iterative training iterations, with the learning rate constantly set to 2e. -4 This embodiment compares the halo mask image learned by the initialization network with the halo mask image in the Flare7K++ synthesis process, and trains the initialization network using the second and fifth loss functions to adjust its parameters. The initial halo mask image extracted by the initialization network is used as the prior image for the deep unfolding network. The deep unfolding network iterates multiple times over the halo-removed image, mapping map, constraint variables, and halo-free feature map, training it using the first and fourth loss functions to adjust its parameters. This embodiment uses evaluation metrics such as PSNR, SSIM, and LPIPS to assess the image quality of the final halo-removed image, and determines whether to continue or stop training the initialization network and the deep unfolding network based on the evaluation results and the total number of iterations (num).

[0184] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0185] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for removing halos from nighttime images, characterized in that, include: Construct prior knowledge, which includes pixel-level spatial consistency terms, feature-level detail enhancement terms, and feature-level noise suppression terms; An initialization network and a deep unfolding network are constructed based on prior knowledge. The initialization network is used to extract a set of initial variable values ​​from the nighttime halo pollution image. The set of initial variable values ​​includes an initial halo mask image, an initial halo removal image, an initial mapping map, an initial halo-free feature map, and initial constraint variables. The nighttime halo pollution image and the initial set of variable values ​​are input into a depth unrolling network for multiple iterative operations to extract the final halo-removed image from the nighttime halo pollution image; wherein, the depth unrolling network includes multiple proximal networks, and each proximal network participates in one iterative operation; The construction of prior knowledge includes: Construct a halo layer decomposition model based on the following formula: ； Where I is the nighttime halo pollution image, B is the halo removal image, and F is the halo mask image; Construct the first optimization function corresponding to the pixel-level spatial consistency term: ； Where f represents the Fibonacci norm, E1 is the Fibonacci norm term, which is used to ensure the fidelity of image data; E1 is the first optimization function. The construction of prior knowledge includes: Feature maps are extracted using a convolutional dictionary based on the following formula: ； ； ； Where ZI is the feature map of the nighttime halo pollution image, ZB is the feature map without halo, ZF is the feature map of the halo mask; D is the convolution dictionary. It is a convolution operator; Construct a second optimization function that includes the aforementioned feature-level detail enhancement term: ； Where M is a mapping graph, and * represents element-wise multiplication. The first penalty coefficient, This is the second penalty coefficient. The first regularization coefficient is . E2 is the first prior term, and E2 is the second optimization function. The construction of prior knowledge includes: Construct the feature-level noise prior terms according to the following formula: ； in, This is the second regularization coefficient. The norm of the element-wise multiplication of the halo-free feature map and the halo mask feature map; Construct the third optimization function for the feature-level noise suppression term: ； in, E3 is the norm of the convolution of the convolution dictionary and the halo mask image, and the element-wise product of the halo-free feature map, and is the third optimization function.

2. The method for removing halos from nighttime images according to claim 1, characterized in that, Before extracting the initial set of variable values ​​from the nighttime halo pollution image using the initialization network, the method further includes: Calculate the Charbonnier consistency loss between the halo-removed image and the ground truth to obtain the first loss function; Calculate the Charbonnier consistency loss between the halo mask image and the training set to obtain the second loss function; The first loss function and the second loss function are added together to obtain the third loss function; The fourth loss function is obtained by calculating the Vgg consistency loss between the halo-removed image and the ground truth. Calculate the Vgg consistency loss between the halo mask image and the training set to obtain the fifth loss function; Adding the fourth and fifth loss functions together yields the sixth loss function; The third loss function and the sixth loss function are weighted and summed to obtain the final loss function; The final loss function is used to train both the initialization network and the deep unfolded network end-to-end.

3. The method for removing halos from nighttime images according to claim 1, characterized in that, The step of inputting the nighttime halo pollution image and the initial set of variable values ​​into a deep unrolled network for multiple iterative calculations includes: During the t-th iteration, the gradient of the t-th mapping is calculated. The t-th mapping and its gradient are then input into the t-th proximal network of the deep unrolled network to calculate the (t+1)-th mapping. Convolve the convolution dictionary with the halo mask image to obtain the t-th halo mask feature; calculate the gradient of the t-th halo-free feature map; input the t-th halo-free feature map, the gradient of the t-th halo-free feature map, and the t-th halo mask feature into the soft threshold operator to obtain the (t+1)-th halo-free feature map. Calculate the gradient of the t-th constraint variable, input the t-th constraint variable and its gradient into the t-th proximal network of the deep unfolded network, and calculate the (t+1)-th constraint variable. Calculate the gradient of the t-th halo-removed image, and calculate the (t+1)-th halo-removed image based on the gradient of the t-th halo image and the t-th halo-removed image.

4. The method for removing halos from nighttime images according to claim 3, characterized in that, The step of inputting the t-th mapping map and its gradient into the t-th proximal network of the depthwise unfolded network to calculate the (t+1)-th mapping map includes: Calculate the (t+1)th mapping graph using the following formula: ； Among them, M t+1 For the (t+1)th mapping, M t For the t-th mapping, Let be the gradient of the t-th mapping. Update parameters for the mapping graph. For element-wise multiplication, prox t1 The t1th near-end network is the mapping graph of the deep unfolded network.

5. The method for removing halos from nighttime images according to claim 3, characterized in that, The step of inputting the t-th halo-free feature map, the gradient of the t-th halo-free feature map, and the t-th halo mask feature into the soft thresholding operator to obtain the (t+1)-th halo-free feature map includes: Calculate the (t+1)th halo-free feature map using the following formula: ； Among them, ZB t+1 Let t+1 be the halo-free feature map, soft be the soft thresholding operator, and ZB be the value of the halo-free feature map. t For the t-th feature map without halo, Update parameters for the halo-free feature map. Let be the gradient of the t-th halo-free feature map. This is the second regularization coefficient. The first penalty coefficient, D is the second penalty coefficient. t For the t-th convolutional dictionary, For halo mask images, is the convolution operator, and * represents element-wise multiplication.

6. The method for removing halos from nighttime images according to claim 3, characterized in that, The calculation of the (t+1)th constraint variable includes: Calculate the (t+1)th constraint variable using the following formula: ； Where, N t+1 For the (t+1)th constraint variable, prox t2 For the t2th proximal network of the constraint variables of the deep unfolded network, N t Let t be the t-th constraint variable. For updating parameters of constraint variables, Let t be the gradient of the t-th constraint variable, and * denote element-wise multiplication.

7. The method for removing halos from nighttime images according to claim 3, characterized in that, The calculation of the (t+1)th halo-removed image includes: The halo removal image of the (t+1)th time is calculated using the following formula: ； Among them, B t+1 For the (t+1)th halo removal image, B t Remove the halo image for the t-th halo. Update image parameters to remove halo effect. The gradient of the image is removed for the t-th halo, and * represents element-wise multiplication.

Images