Image flare removal method based on flare mask guidance and multi-receptive field perception

Abstract

The present application proposes an image flare removal method based on flare mask guidance and multi-receptive field perception, comprising the following steps: step S1, collecting flare images and performing data preprocessing; step S2, designing a global feature extraction module to extract global features of the flare images; step S3, designing a multi-receptive field dynamic fusion module, which uses an inflation convolution module to extract flare image features of different receptive fields, and uses a dynamic weight fusion module to fuse flare image features of multiple receptive fields; step S4, designing a flare removal network; step S5, using a flare mask to design a loss function for guiding the optimization of network parameters designed in step S4; step S6, training the flare removal network to obtain an image flare removal model; step S7, inputting a to-be-tested flare image into the image flare removal model to predict and generate a flare-removed image; the present application can effectively remove flares on the flare image.

Description

Technical Field

[0001] This invention relates to the fields of image processing and computer vision technology, and in particular to an image flare removal method based on flare mask guidance and multi-receptive field perception. Background Technology

[0002] Lens flares are a common optical phenomenon. When strong light is scattered or reflected within a lens system, it creates unique radial bright areas and spots in the captured image; these are called flares. Flare phenomena can lead to low contrast and suppressed details around the light source, reducing image visual quality and the performance of visual algorithms. For example, in aerial object tracking, bright spots introduced by lens flares can easily mislead algorithms into tracking the flare instead of the flying object. Furthermore, in mobile platforms such as surveillance lenses, smartphone cameras, drones, and autonomous driving cameras, daily wear and tear, fingerprints, and dust act as gratings, exacerbating lens flares and severely impacting the performance of these platforms. Therefore, removing flares from images is of significant research importance for improving the applicability and stability of outdoor computer vision systems. Additionally, flare patterns depend on the optical properties of the lens, the lens protective glass, the infrared cutoff glass, the position and intensity of the light source, manufacturing defects, scratches, and dust accumulated during daily use. Because flares have diverse causes and manifest in various forms—typical flares include halos, streaks, bright lines, saturation points, and reflections—this diversity makes flare removal a very challenging problem.

[0003] Currently, image flare removal solutions are broadly categorized into three main types: hardware solutions, software solutions, and data-driven solutions. Most hardware solutions focus on improving the camera's optical system to eliminate flares, such as optimized lens barrel design, lens covers, or reflective coatings. A widely used technique is to apply anti-reflective coatings to lens elements to reduce internal reflections by disrupting interference. However, in this case, adding anti-reflective coatings to all optical surfaces is expensive, and the thickness of such coatings can only be optimized for specific wavelengths and incident angles. Furthermore, they can only reduce flares during capture but cannot process existing images with flares, and these hardware solutions struggle to eliminate entire flares. Many software-based solutions are essentially two-stage approaches: first, identifying flares, and then restoring the scene in the flare area. For example, binarizing the image using a series of thresholds, calculating the contour features of the binarized image, obtaining a series of potential flare candidate regions, and reconstructing these candidate regions. However, software-based methods are only applicable to a limited number of flare types, easily treating local bright areas as flares, and struggling to distinguish between different types of flares. Data-driven solutions utilize deep learning to train neural networks with large amounts of data, enabling the networks to learn various flare features from this data and thus perform flare removal more effectively. Summary of the Invention

[0004] This invention proposes an image flare removal method based on flare mask guidance and multi-receptive field perception, which can effectively remove flares from flare images.

[0005] The present invention adopts the following technical solution.

[0006] The image flare removal method based on flare mask guidance and multi-receptive field perception includes the following steps: Step S1, collect flare images and perform data preprocessing, the preprocessing methods include data pairing, random data cropping, and data augmentation to obtain a training dataset;

[0007] Step S2: Design a global feature extraction module, using channel attention and spatial attention to extract global features of flare images;

[0008] Step S3: Design a multi-receptive-field dynamic fusion module. This module uses a dilated convolution module to extract flare image features from different receptive fields and a dynamic weight fusion module to fuse flare image features from multiple receptive fields.

[0009] Step S4: Design a flare removal network, which consists of an initial feature extraction module, a U-shaped encoder-decoder module, and an image restoration module;

[0010] Step S5: Use flare masking to design a loss function to guide the optimization of network parameters designed in step S4;

[0011] Step S6: Use the training dataset obtained in step S1 to train the flare removal network in step S4, and obtain a trained image flare removal model based on flare mask guidance and multi-receptive field perception.

[0012] Step S7: Input the image of the flare to be tested into the trained image flare removal model based on flare mask guidance and multi-receptive field perception to predict and generate a flare-removed image.

[0013] The specific implementation steps of step S1 are as follows:

[0014] Step S11: Generate paired flare images, clean images with light sources, and flare masks based on the flare images, light source images, and background images provided in the dataset.

[0015] Step S12: Randomly crop each flare image of size h×w×3 into an image of size p×p×3, and apply the same random cropping method to the corresponding clean image with light source and flare mask, where h and w are the height and width of the flare image, and p is the height and width of the cropped image.

[0016] Step S13: Randomly use one of the following 8 augmentation methods to augment the paired images to be trained: keep the original image, flip vertically, rotate 90 degrees, rotate 90 degrees and then flip vertically, rotate 180 degrees, rotate 180 degrees and then flip vertically, rotate 270 degrees, rotate 270 degrees and then flip vertically.

[0017] The specific implementation steps of step S2 are as follows:

[0018] Step S21: Design the channel attention module and input image features. The channel attention weights Att are obtained by applying GELU activation, adaptive average pooling (AdaptiveAvgPool), 1×1 convolution, GELU activation, 1×1 convolution, and Sigmoid activation. c , With Att c Multiplication yields channel attention-weighted features. The specific formula is expressed as follows:

[0019] Where GELU(*) represents the GELU activation function, Sigmoid(*) represents the Sigmoid activation function, AdaptiveAvgPool(*) represents adaptive average pooling, Conv1×1(*) represents a convolutional layer with a kernel size of 1×1, and Attc represents the channel attention weights.

[0020] Step S22: Design a spatial attention module and input image features. The spatial attention weights Att are obtained by performing 3×3 convolution, 1×1 convolution, GELU activation function, 1×1 convolution, and Sigmoid activation function. s , With Att s Multiplication yields spatial attention-weighted features. The specific formula is expressed as follows:

[0021]

[0022]

[0023] Where GELU(*) represents the GELU activation function, Sigmoid(*) represents the Sigmoid activation function, and Conv 1×1(*) indicates a convolutional layer with a kernel size of 1×1. 3×3 (*) indicates a convolutional layer with a kernel size of 3×3. Att s Indicates spatial attention weights;

[0024] Step S23: Design a global feature extraction module and input image features. After passing through the GroupNorm layer, the channel attention module in step S21, and the spatial attention module in step S22, and... Adding together yields the features After passing through a GroupNorm layer, a 3×3 convolution, a 1×1 convolution, and the GELU activation function, and... Adding them together yields the output features. The specific formula is expressed as follows:

[0025]

[0026]

[0027] Where GELU(*) represents the GELU activation function, Conv 1×1 (*) indicates a convolutional layer with a kernel size of 1×1. 3×3 (*) indicates a convolutional layer with a kernel size of 3×3, GroupNorm(*) indicates a group normalization layer, CAM(*) indicates the channel attention module in step S21, and SAM(*) indicates the spatial attention module in step S22.

[0028] The specific implementation steps of step S3 are as follows:

[0029] Step S31: Design a dilated convolution module and input image features. The output features F are obtained by dilated convolutions with dilation rates of 2, 4, and 8, respectively, followed by the GELU activation function. dial1 F dial2 F dial3 (F dial1 F dial2 , The specific formula is as follows:

[0030]

[0031]

[0032]

[0033] Where GELU(*) represents the GELU activation function, Dilated r＝2 (*) indicates a dilated convolution with a dilation rate of 2. r＝4 (*) indicates a dilated convolution with a dilation rate of 4. r＝8 (*) indicates a dilated convolution with a dilation rate of 8;

[0034] Step S32: Design a dynamic weight fusion module, with the input being the output features F of the three dilated convolution modules in step S31. dial1 F dial2 F dial3 These three features are processed separately using AdaptiveAvgPool and then concatenated along the channel dimension using the cat concatenation function to obtain the final feature. Will The feature F is obtained by performing 1×1 convolution, LELU activation function, 1×1 convolution, GELU activation function, 1×1 convolution and Sigmoid activation function. conv , F conv The channel dimension is segmented using a chunking function to generate three dynamic feature weights w1, w2, and w3(w1, w2, ..., w3). The output features F of the three dilated convolutional modules are... dial1 F dial2 F dial3 The output feature F is obtained by multiplying each feature by the weights of the three dynamic features w1, w2, and w3, and then summing the results. fusion , The specific formula is expressed as follows:

[0035]

[0036]

[0037] w1, w2, w3 = chunk(F conv )

[0038] F fusion =F dial1 ×w1+F dial2 ×w2+F dial3 ×w3

[0039] Where GELU(*) represents the GELU activation function, Sigmoid(*) represents the Sigmoid activation function, AdaptiveAvgPool(*) represents adaptive average pooling, cat(*) represents the concatenation function, chunk(*) represents the splitting function, and Conv...1×1 (*) indicates a convolutional layer with a kernel size of 1×1, where w1, w2, and w3 are three dynamic feature weights;

[0040] Step S33: Design a multi-receptive-field dynamic fusion module, inputting image features. The dilated convolution module in step S31 produces three output features F. dial1 F dial2 F dial3 These three features are processed by the dynamic weight fusion module in step S32 to obtain feature F. fusion , Will With F fusion Adding them together yields the output features.

[0041] The specific implementation steps of step S4 are as follows:

[0042] Step S41: Design the initial feature extraction module, with the flare image I as input. flars , I flars The initial feature F is obtained after 3×3 convolution, GELU activation function, and 3×3 convolution. initial , The specific formula is expressed as follows:

[0043] F initial =Conv 3×3 (GELU(Conv 3×3 (I flare )))

[0044] Where GELU(*) represents the GELU activation function, Conv 3×3 (*) indicates a convolutional layer with a kernel size of 3×3;

[0045] Step S42: Design the encoder module in the U-shaped encoder decoder module, with the initial feature F from step S41 as the input. initial F initial The encoded feature E1 is obtained through the global feature extraction module in step S23 and the multi-receptive field dynamic fusion module in step S33. E1 is processed through downsampling, global feature extraction, and multi-receptive-field dynamic fusion to obtain encoded feature E2. E2 is processed through downsampling, global feature extraction, and multi-receptive-field dynamic fusion to obtain encoded feature E3. E3 is processed through downsampling, global feature extraction, and multi-receptive-field dynamic fusion to obtain encoded feature E4. The specific formula is expressed as follows:

[0046] E1=MFDFM(GFEM(F initial ))

[0047] E2=MFDFM(GFEM(DownSample(E1)))

[0048] E3=MFDFM(GFEM(DownSample(E2)))

[0049] E4=MFDFM(GFEM(DownSample(E3)))

[0050] Wherein, GFEM(*) is the global feature extraction module in step S23, MFDFM(*) is the multi-receptive field dynamic fusion module in step S33, and DownSample(*) is the downsampling operation;

[0051] Step S43: Design the decoder module in the U-shaped encoder decoder module. The input is the encoded feature E4 from step S42. E4 is upsampled and then added to the encoded feature E3. The decoded feature D3 is then obtained through the global feature extraction module in step S23 and the multi-receptive field dynamic fusion module in step S33. D3 is upsampled and then added to the encoded feature E2. The decoded feature D2 is then obtained through the global feature extraction module and the multi-receptive-field dynamic fusion module. D2 is upsampled and then added to the encoded feature E1. The decoded feature D1 is then obtained through the global feature extraction module and the multi-receptive-field dynamic fusion module. The specific formula is expressed as follows:

[0052] D3=MFDFM(GFEM(UpSample(E4)+E3))

[0053] D2=MFDFM(GFEM(UpSample(D3)+E2))

[0054] D1=MFDFM(GFEM(UpSample(D2)+E1))

[0055] Wherein, GFEM(*) is the global feature extraction module in step S23, MFDFM(*) is the multi-receptive field dynamic fusion module in step S33, and UpSample(*) is the upsampling operation;

[0056] Step S44: Design the image restoration module, with the input being the decoded feature D1 from step S43. D1 is processed by 3×3 convolution, GELU activation function, and 3×3 convolution to obtain the restored feature F.restore , Restoration feature F restore and flare image I flare The summation yields the final flare-free image I. deflare , The specific formula is expressed as follows:

[0057] F restore =Conv 3×3 (GELU(Conv 3×3 (D1)))

[0058] I deflare =F restore +I flare

[0059] Where GELU(*) represents the GELU activation function, Conv 3×3 (*) indicates a convolutional layer with a kernel size of 3×3;

[0060] Step S45: Design a flare removal network. The input to the network is the flare image I. flare I flare F is obtained after the initial feature extraction module in step S41. initial F initial The U-shaped encoder and decoder, after steps S42 and S43, obtain the decoding features D1, D1, and I. flare The image restoration module in step S44 obtains the deflare image I. deflare The specific formula is as follows:

[0061] F initial =IFEM(I flare )

[0062] D1 = Decoder(Encoder(F) initial ))

[0063] I deflare =IRM(D1, I flare )

[0064] Wherein, IFEM(*) is the initial feature extraction module in step S41, Encoder(*) is the encoder module in step S42, Decoder(*) is the decoder module in step S43, and IRM(*) is the image restoration module in step S44.

[0065] The specific implementation method of step S5 is as follows: Design the loss function, which is composed of the basic loss L base and mask loss L mask The overall objective loss function L of the network is as follows:

[0066] L = L base +L mask

[0067] L base =L char_base +L ssim_base +λL vgg_base

[0068] L mask =L char_mask +L ssim_mask +λL vgg_mask

[0069] L char_bass =Charloss(I deflare I clear )

[0070] L ssim_bass =1-SSIM(I deflare I clear )

[0071] L vgg_bass =Vggloss(I deflare I clear )

[0072] L char_mask =Charloss(I deflare ×I mask I clear ×I mask )

[0073] L ssim_mask =1-SSIM(I deflare ×I mask I clear ×I mask )

[0074] L vgg_mask =Vggloss(I deflare ×I mask I clear ×I mask )

[0075] Among them, I mask For flare mask images, I deflare To predict flare-free images, I clear For the labeled image, Charloss(*) is the L1 Charbonnier loss function, vggloss(*) is the VGG perceptual loss function, SSIM(*) is the structural similarity measure, λ represents the weight of the loss, and × represents element-wise multiplication.

[0076] The specific implementation method of step S6 is as follows: Divide the flare images into batches N, and divide the training dataset in step S1 into K batches; divide the flare images I flare Input the flare removal network in step S4 to obtain the flare-removed image I. deflare Using the loss function designed in step S5, calculate the flare-reduced image I. deflare The network parameters are updated using Adam optimization until convergence, resulting in an image flare removal model based on flare mask guidance and multi-receptive field perception.

[0077] This invention proposes an image flare removal method based on flare mask guidance and multi-receptive field perception, using a deep learning approach. It constructs a flare removal network using a global feature extraction module, a multi-receptive field dynamic fusion module, and a U-shaped encoder-decoder structure to extract flare features from the image. The training of the flare removal network is supervised by flare mask loss, thus achieving better flare removal results. Attached Figure Description

[0078] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0079] Appendix Figure 1 This is a schematic diagram illustrating the implementation process of the method in an embodiment of the present invention;

[0080] Appendix Figure 2 This is a schematic diagram of the paired flare pattern, light source pattern, and flare mask in an embodiment of the present invention;

[0081] Appendix Figure 3 This is a schematic diagram of the global feature extraction module in an embodiment of the present invention;

[0082] Appendix Figure 4 This is a schematic diagram of the structure of the multi-receptive field dynamic fusion module in an embodiment of the present invention;

[0083] Appendix Figure 5 This is a schematic diagram of the structure of the flare removal network in an embodiment of the present invention. Detailed Implementation

[0084] like Figure 1 As shown, the image flare removal method based on flare mask guidance and multi-receptive field perception includes the following steps:

[0085] Step S1: Collect flare images and perform data preprocessing. Preprocessing methods include data pairing, random data cropping, and data augmentation to obtain a training dataset.

[0086] Step S2: Design a global feature extraction module, using channel attention and spatial attention to extract global features of flare images;

[0087] Step S3: Design a multi-receptive-field dynamic fusion module. This module uses a dilated convolution module to extract flare image features from different receptive fields and a dynamic weight fusion module to fuse flare image features from multiple receptive fields.

[0088] Step S4: Design a flare removal network, which consists of an initial feature extraction module, a U-shaped encoder-decoder module, and an image restoration module;

[0089] Step S5: Use flare masking to design a loss function to guide the optimization of network parameters designed in step S4;

[0090] Step S6: Use the training dataset obtained in step S1 to train the flare removal network in step S4, and obtain a trained image flare removal model based on flare mask guidance and multi-receptive field perception.

[0091] Step S7: Input the image of the flare to be tested into the trained image flare removal model based on flare mask guidance and multi-receptive field perception to predict and generate a flare-removed image.

[0092] The specific implementation steps of step S1 are as follows:

[0093] Step S11: Generate paired flare images, clean images with light sources, and flare masks based on the flare images, light source images, and background images provided in the dataset; such as Figure 2 As shown,

[0094] Step S12: Randomly crop each flare image of size h×w×3 into an image of size p×p×3, and apply the same random cropping method to the corresponding clean image with light source and flare mask, where h and w are the height and width of the flare image, and p is the height and width of the cropped image.

[0095] Step S13: Randomly use one of the following 8 augmentation methods to augment the paired images to be trained: keep the original image, flip vertically, rotate 90 degrees, rotate 90 degrees and then flip vertically, rotate 180 degrees, rotate 180 degrees and then flip vertically, rotate 270 degrees, rotate 270 degrees and then flip vertically.

[0096] like Figure 3 As shown, the specific implementation steps of step S2 are as follows:

[0097] Step S21: Design the channel attention module and input image features. The channel attention weights Att are obtained by applying GELU activation, adaptive average pooling (AdaptiveAvgPool), 1×1 convolution, GELU activation, 1×1 convolution, and Sigmoid activation. c , With Att c Multiplication yields channel attention-weighted features. The specific formula is expressed as follows:

[0098]

[0099]

[0100] Where GELU(*) represents the GELU activation function, Sigmoid(*) represents the Sigmoid activation function, AdaptiveAvgPool(*) represents adaptive average pooling, and Conv... 1×1 (*) indicates a convolutional layer with a kernel size of 1×1, Att c Indicates channel attention weights;

[0101] Step S22: Design a spatial attention module and input image features. The spatial attention weights Att are obtained by performing 3×3 convolution, 1×1 convolution, GELU activation function, 1×1 convolution, and Sigmoid activation function. s , With Att s Multiplication yields spatial attention-weighted features. The specific formula is expressed as follows:

[0102]

[0103]

[0104] Where GELU(*) represents the GELU activation function, Sigmoid(*) represents the Sigmoid activation function, and Conv 1×1 (*) indicates a convolutional layer with a kernel size of 1×1. 3×3 (*) indicates a convolutional layer with a kernel size of 3×3. Att s Indicates spatial attention weights;

[0105] Step S23: Design a global feature extraction module and input image features. After passing through the GroupNorm layer, the channel attention module in step S21, and the spatial attention module in step S22, and... Adding together yields the features After passing through a GroupNorm layer, a 3×3 convolution, a 1×1 convolution, and the GELU activation function, and... Adding them together yields the output features. The specific formula is expressed as follows:

[0106]

[0107]

[0108] Where GELU(*) represents the GELU activation function, Conv 1×1 (*) indicates a convolutional layer with a kernel size of 1×1. 3×3 (*) indicates a convolutional layer with a kernel size of 3×3, GroupNorm(*) indicates a group normalization layer, CAM(*) indicates the channel attention module in step S21, and SAM(*) indicates the spatial attention module in step S22.

[0109] like Figure 4 As shown, the specific implementation steps of step S3 are as follows:

[0110] Step S31: Design a dilated convolution module and input image features. The output features F are obtained by dilated convolutions with dilation rates of 2, 4, and 8, respectively, followed by the GELU activation function. dial1 F dial2 F dial3 (F dial1 F dial2 , The specific formula is as follows:

[0111]

[0112]

[0113]

[0114] Where GELU(*) represents the GELU activation function, Dilated r＝2 (*) indicates a dilated convolution with a dilation rate of 2. r＝4 (*) indicates a dilated convolution with a dilation rate of 4. r＝8(*) indicates a dilated convolution with a dilation rate of 8;

[0115] Step S32: Design a dynamic weight fusion module, with the input being the output features F of the three dilated convolution modules in step S31. dial1 F dial2 F dial3 These three features are each processed by AdaptiveAvgPool and then concatenated along the channel dimension using the cat concatenation function to obtain the final feature. Will Feature F is obtained by performing 1×1 convolution, GELU activation function, 1×1 convolution, GELU activation function, 1×1 convolution and Sigmoid activation function. conv , F conv The channel dimension is segmented using a chunking function to generate three dynamic feature weights w1, w2, and w3(w1, w2, ..., w3). The output features F of the three dilated convolutional modules are... dial1 F dial2 F dial3 The output feature F is obtained by multiplying each feature by the weights of the three dynamic features w1, w2, and w3, and then summing the results. fusion , The specific formula is expressed as follows:

[0116]

[0117]

[0118] w1, w2, w3 = chunk(F conv )

[0119] F fusion =F dial1 ×w1+F dial2 ×w2+F dial3 ×w3

[0120] Where GELI(*) represents the GELU activation function, Sigmoid(*) represents the Sigmoid activation function, AdaptiveAvgPool(*) represents adaptive average pooling, cat(*) represents the concatenation function, chunk(*) represents the splitting function, and Conv... 1×1 (*) indicates a convolutional layer with a kernel size of 1×1, where w1, w2, and w3 are three dynamic feature weights;

[0121] Step S33: Design a multi-receptive-field dynamic fusion module, inputting image features. The dilated convolution module in step S31 produces three output features F. dial1 F dial2 F dial3 These three features are processed by the dynamic weight fusion module in step S32 to obtain feature F. fusion , Will With F fusion Adding them together yields the output features.

[0122] like Figure 5 As shown, the specific implementation steps of step S4 are as follows:

[0123] Step S41: Design the initial feature extraction module, with the flare image I as input. flare , I flare The initial feature F is obtained after 3×3 convolution, GELU activation function, and 3×3 convolution. initial , The specific formula is expressed as follows:

[0124] F initial =Conv 3×3 (GELU(Conv 3×3 (I flare )))

[0125] Where GELU(*) represents the GELU activation function, Conv 3×3 (*) indicates a convolutional layer with a kernel size of 3×3;

[0126] Step S42: Design the encoder module in the U-shaped encoder decoder module, with the initial feature F from step S41 as the input. initial F initial The encoded feature E1 is obtained through the global feature extraction module in step S23 and the multi-receptive field dynamic fusion module in step S33. E1 is processed through downsampling, global feature extraction, and multi-receptive-field dynamic fusion to obtain encoded feature E2. E2 is processed through downsampling, global feature extraction, and multi-receptive-field dynamic fusion to obtain encoded feature E3. E3 is processed through downsampling, global feature extraction, and multi-receptive-field dynamic fusion to obtain encoded feature E4. The specific formula is expressed as follows:

[0127] E1=MFDFM(GFEM(F initial ))

[0128] E2=MFDFM(GFEM(DownSample(E1)))

[0129] E3=MFDFM(GFEM(DownSample(E2)))

[0130] E4=MFDFM(GFEM(DownSample(E3)))

[0131] Wherein, GFEM(*) is the global feature extraction module in step S23, MFDFM(*) is the multi-receptive field dynamic fusion module in step S33, and DownSample(*) is the downsampling operation;

[0132] Step S43: Design the decoder module in the U-shaped encoder decoder module. The input is the encoded feature E4 from step S42. E4 is upsampled and then added to the encoded feature E3. The decoded feature D3 is then obtained through the global feature extraction module in step S23 and the multi-receptive field dynamic fusion module in step S33. D3 is upsampled and then added to the encoded feature E2. The decoded feature D2 is then obtained through the global feature extraction module and the multi-receptive-field dynamic fusion module. D2 is upsampled and then added to the encoded feature E1. The decoded feature D1 is then obtained through the global feature extraction module and the multi-receptive-field dynamic fusion module. The specific formula is expressed as follows:

[0133] D3=MFDFM(GFEM(UpSample(E4)+E3))

[0134] D2=MFDFM(GFEM(UpSample(D3)+E2))

[0135] D1=MFDFM(GFEM(UpSample(D2)+E1))

[0136] Wherein, GFEM(*) is the global feature extraction module in step S23, MFDFM(*) is the multi-receptive field dynamic fusion module in step S33, and UpSample(*) is the upsampling operation;

[0137] Step S44: Design the image restoration module, with the input being the decoded feature D1 from step S43. D1 is processed by 3×3 convolution, GELU activation function, and 3×3 convolution to obtain the restored feature F. restore , Restoration feature F restore and flare image I flare The summation yields the final flare-free image I. deflare , The specific formula is expressed as follows:

[0138] F restore =Conv 3×3 (GELU(Conv 3×3 (D1)))

[0139] I deflare =F restore +I flare

[0140] Where CELU(*) represents the GELU activation function, Conv 3×3 (*) indicates a convolutional layer with a kernel size of 3×3;

[0141] Step S45: Design a flare removal network. The input to the network is the flare image I. flare I flare F is obtained after the initial feature extraction module in step S41. initial F initial The U-shaped encoder and decoder, after steps S42 and S43, obtain the decoding features D1, D1, and I. flare The image restoration module in step S44 obtains the deflare image I. deflare The specific formula is as follows:

[0142] F initial =IFEM(I flare )

[0143] D1 = Decoder(Encoder(F) initial ))

[0144] I deflare =IRM(D1, I flare )

[0145] Wherein, IFEM(*) is the initial feature extraction module in step S41, Encoder(*) is the encoder module in step S42, Decoder(*) is the decoder module in step S43, and IRM(*) is the image restoration module in step S44.

[0146] The specific implementation method of step S5 is as follows: Design the loss function, which is composed of the basic loss L bass and mask

[0147] Loss L mask The overall objective loss function l of the network is as follows:

[0148] L = L base +L mask

[0149] L base=L char_base +L ssim_bass +λL vgg_base

[0150] l mask =L char_mask +L ssim_mask +λL vgg_mask

[0151] L char_base =Charloss(I deflare I clear )

[0152] L ssim_bass =1-SSIM(I deflare I clear )

[0153] L vgg_bass =Vhhloss(I deflare I clear )

[0154] L char_mask =Charloss(I deflare ×I mask I clear ×I mask )

[0155] L ssim_mask =1-SSIM(I deflare ×I mask I clear ×I mask )

[0156] L vgg_mask =Vggloss(I deflare ×I mask I clear ×I mask )

[0157] Among them, I mask For flare mask images, I deflare To predict flare-free images, I clear For the labeled image, Charloss(*) is the L1 Charbonnier loss function, Vggloss(*) is the VGG perceptual loss function, SSIM(*) is the structural similarity measure, λ represents the weight of the loss, and × represents element-wise multiplication.

[0158] The specific implementation method of step S6 is as follows: Divide the flare images into batches N, and divide the training dataset in step S1 into K batches; divide the flare images I flare Input the flare removal network in step S4 to obtain the flare-removed image I.deflare Using the loss function designed in step S5, calculate the flare-reduced image I. deflare The network parameters are updated using Adam optimization until convergence, resulting in an image flare removal model based on flare mask guidance and multi-receptive field perception.

Claims

1. An image flare removal method based on flare mask guidance and multi-receptive field perception, characterized in that: Includes the following steps: Step S1: Collect flare images and perform data preprocessing. Preprocessing methods include data pairing, random data cropping, and data augmentation to obtain a training dataset. Step S2: Design a global feature extraction module, and use channel attention and spatial attention to extract global features of flare images; Step S3: Design a multi-receptive-field dynamic fusion module. This module uses a dilated convolution module to extract flare image features from different receptive fields and a dynamic weight fusion module to fuse flare image features from multiple receptive fields. Step S4: Design a flare removal network, which consists of an initial feature extraction module, a U-shaped encoder-decoder module, and an image restoration module; Step S5: Design a loss function using flare masking to guide the optimization of network parameters designed in step S4; Step S6: Train the flare removal network in step S4 using the training dataset obtained in step S1 to obtain a trained image flare removal model based on flare masking and multi-receptive field perception. Step S7: Input the image of the flare to be tested into the trained image flare removal model based on flare mask guidance and multi-receptive field perception, and predict and generate a flare-removed image; The specific implementation steps of step S3 are as follows: Step S31, design the dilated convolution module, input the image features respectively by the dilated convolution with expansion rate of 2, 4, 8 and The activation function obtains the output features , , , The specific formula is as follows: in, This represents the GELU activation function. This represents a dilated convolution with a dilation rate of 2. This represents a dilated convolution with a dilation rate of 4. This represents a dilated convolution with a dilation rate of 8; Step S32: Design a dynamic weight fusion module, with the input being the output features of the three dilated convolution modules in step S31. These three features are then subjected to adaptive average pooling. Then through the channel dimension Features are obtained by concatenating functions. , ,Will go through convolution, Activation function convolution, Activation function Convolution and Activation function obtains features , , Passing through the channel dimension The segmentation function generates three dynamic feature weights. , The output features of the three dilated convolutional modules are processed. and 3 dynamic feature weights , , The output features are obtained by multiplying and then adding them together. The specific formula is as follows: in, This represents the GELU activation function. This represents the Sigmoid activation function. Indicates adaptive average pooling. This represents the concatenation function. Represents the partitioning function. This indicates a convolutional layer with a kernel size of 1×1. The weights are 3 dynamic features; Step S33: Design a multi-receptive-field dynamic fusion module, inputting image features. , , The dilated convolution module outputs three features obtained after step S31. These three features are processed by the dynamic weight fusion module in step S32 to obtain the features. ,Will Adding them together yields the output features. , ; The specific implementation steps of step S4 are as follows: Step S41: Design the initial feature extraction module, with the flare image as input. go through convolution, Activation function and Convolution yields initial features The specific formula is as follows: in, This represents the GELU activation function. This indicates a convolutional layer with a kernel size of 3×3; Step S42: Design the encoder module in the U-shaped encoder decoder module, with the initial features from step S41 as the input. , The encoded features are obtained through the global feature extraction module in step S23 and the multi-receptive field dynamic fusion module in step S33. , , The encoded features are obtained through downsampling, global feature extraction, and multi-receptive field dynamic fusion modules. , , The encoded features are obtained through downsampling, global feature extraction, and multi-receptive field dynamic fusion modules. , , The encoded features are obtained through downsampling, global feature extraction, and multi-receptive field dynamic fusion modules. , The specific formula is as follows: in, It is the global feature extraction module in step S23. It is the multi-receptor field dynamic fusion module in step S33. It is a downsampling operation; Step S43: Design the decoder module in the U-shaped encoder decoder module, with the input being the encoded features from step S42. , After upsampling and then combining with encoded features The features are then added together, and then decoded using the global feature extraction module in step S23 and the multi-receptive field dynamic fusion module in step S33. , , After upsampling and then combining with encoded features The features are then added together and then processed by the global feature extraction module and the multi-receptive field dynamic fusion module to obtain the decoded features. , , After upsampling and then combining with encoded features The features are then added together and then processed by the global feature extraction module and the multi-receptive field dynamic fusion module to obtain the decoded features. , The specific formula is as follows: in, It is the global feature extraction module in step S23. It is the multi-receptor field dynamic fusion module in step S33. It is an upsampling operation; Step S44: Design the image restoration module, with the input being the decoded features from step S43. , , go through convolution, Activation function and Convolution yields restored features , Restoration features and flare images The summation yields the final flare-free image. , The specific formula is as follows: in, This represents the GELU activation function. This indicates a convolutional layer with a kernel size of 3×3; Step S45: Design a flare removal network. The input to the network is a flare image. , After the initial feature extraction module in step S41, the following is obtained: , The U-shaped encoder and decoder obtain the decoding features after steps S42 and S43. , and The image restoration module in step S44 obtains the flare-reduced image. The specific formula is as follows: in, It is the initial feature extraction module in step S41. It is the encoder module in step S42. It is the decoder module in step S43. This is the image restoration module in step S44.

2. The image flare removal method based on flare mask guidance and multi-receptive field perception according to claim 1, characterized in that: The specific implementation steps of step S1 are as follows: Step S11: Generate paired flare images, clean images with light sources, and flare masks based on the flare images, light source images, and background images provided in the dataset. Step S12: Divide each sheet into sheets of size [missing information]. The flare image was randomly cropped into The image of the size is processed, and its corresponding clean image with the light source and the flare mask are subjected to the same random cropping method, wherein, , It is the height and width of the flare image. It involves cropping the height and width of the image; Step S13: Randomly use one of the following 8 augmentation methods to augment the paired images to be trained: keep the original image, flip vertically, rotate 90 degrees, rotate 90 degrees and then flip vertically, rotate 180 degrees, rotate 180 degrees and then flip vertically, rotate 270 degrees, rotate 270 degrees and then flip vertically.

3. The image flare removal method based on flare mask guidance and multi-receptive field perception according to claim 1, characterized in that: The specific implementation steps of step S2 are as follows: Step S21: Design the channel attention module and input image features. , , go through Activation function, adaptive average pooling , convolution, Activation function Convolution and Activation function yields channel attention weights , , and Multiplication yields channel attention-weighted features. , The specific formula is as follows: in, This represents the GELU activation function. This represents the Sigmoid activation function. Indicates adaptive average pooling. This indicates a convolutional layer with a kernel size of 1×1. Indicates channel attention weights; Step S22: Design a spatial attention module and input image features. , , go through convolution, convolution, Activation function Convolution and Activation function yields spatial attention weights , ; and Multiplication yields spatial attention-weighted features. , The specific formula is as follows: in, This represents the GELU activation function. This represents the Sigmoid activation function. This indicates a convolutional layer with a kernel size of 1×1. This indicates a convolutional layer with a kernel size of 3×3. Indicates spatial attention weights; Step S23: Design a global feature extraction module and input image features. , , go through After the layer, the channel attention module in step S21, and the spatial attention module in step S22, and... Adding together yields the features , , go through layer, convolution, Convolution and After activation function and Adding them together yields the output features. , The specific formula is as follows: in, This represents the GELU activation function. This indicates a convolutional layer with a kernel size of 1×1. This indicates a convolutional layer with a kernel size of 3×3. Indicates the group normalization layer, This refers to the channel attention module in step S21. This represents the spatial attention module in step S22.

4. The image flare removal method based on flare mask guidance and multi-receptive field perception according to claim 1, characterized in that: The specific implementation method of step S5 is as follows: Design the loss function, which is composed of the basic loss. and mask Composition, the network's overall objective loss function as follows: in, For flare mask images, To predict flare-free images, For the label image, The L1 Charbonnier loss function is used. The loss function for VGG is... For structural similarity measurement, Represented as the weight of the loss, This indicates element-wise multiplication.

5. The image flare removal method based on flare mask guidance and multi-receptive field perception according to claim 1, characterized in that: The specific implementation method of step S6 is as follows: according to each batch The flare images are divided into segments, and the training dataset from step S1 is divided into segments. One batch; flare images Input the flare removal network in step S4 to obtain the flare-removed image. Using the loss function designed in step S5, calculate the flare-free image. The network parameters are updated using Adam optimization until convergence, resulting in an image flare removal model based on flare mask guidance and multi-receptive field perception.

Images