Image restoration system and method based on flash cue and neural network

Abstract

The application discloses an image restoration system and method based on a flashlight clue and a neural network, and the image restoration system comprises a reflection training atlas, a reflection-removing convolutional neural network and an optimization training module; the reflection-removing convolutional neural network comprises a reflection image estimation subnetwork and a transmission image estimation subnetwork; the reflection training atlas is processed to generate an initial image I by an ambient light image containing reflection, a flashlight image, a transmission image and a reflection image fo ; the reflection image estimation subnetwork generates a reflection image R by taking the ambient light image I a and the initial image I fo as input; the transmission image estimation subnetwork generates a transmission image T by taking the reflection image R and the ambient light image I a as input; and the optimization training module trains the reflection-removing neural network by a linear combination of an L1 norm loss function and a perception loss function to obtain an optimized reflection-removing neural network; and the application can quickly and accurately restore a reflection-free image by using the assistance of the flashlight image.

Description

Technical Field

[0001] This invention relates to the fields of image processing technology and deep learning technology, and in particular to an image restoration system and method based on flash cues and neural networks. Background Technology

[0002] Reflection is a common optical phenomenon. When taking a photograph through glass or other transparent media, the resulting image will blend two different scenes: transmission and reflection, located on either side of the medium. Unlike other image degradation problems, the distortion introduced by reflection is non-uniform. Some areas of strong reflection make it difficult for the human eye to judge information from the original scene, and reflected images can also interfere with the understanding of semantic information. Therefore, reflection not only affects the visual quality of an image but also has a significant impact on downstream visual tasks such as recognition. With the widespread use of smartphones and the extensive application of visual information in intelligent surveillance, the requirements for image quality are increasing, making the restoration of reflected images highly valuable.

[0003] In image dereflection tasks, environmental images containing reflections are included. It can be represented as a transmission image. and reflection diagram linear combination Image dereflection is the process of refining a composite image. Extracting the transmission image from the middle .

[0004] Image dereflection methods can be divided into traditional image processing methods and deep learning-based methods. Traditional image processing methods utilize additional reference information to achieve image dereflection. For example, Agrawl et al. used flash images and gradient projection to remove artifacts in images; Wan et al. used depth maps to identify reflective edges in images and eliminated reflections in images under the guidance of edge information.

[0005] Current deep learning-based dereflection methods target single images, requiring only a single image with reflection as input to train a convolutional neural network to estimate the transmission scene from the input image. For example, Fan et al. pioneered a reflection removal algorithm based on edge maps and deep neural networks. This algorithm uses a two-stage framework to first predict edge information from the non-reflective image, and then uses the obtained edge information to reconstruct the transmission image. Yang et al. proposed a cascaded neural network structure that alternately estimates transmission and reflection images. This network first estimates a coarse transmission map from the input image. Then, the generated coarse transmission map was used. To estimate the reflection map Finally, input image and estimated reflection map As input, the final transmission map is generated. Similarly, Li et al. implemented a cascaded refining network using LSTM, iteratively refining the generated transmission map each time. and reflection diagram This data is continuously updated as input for the next iteration, resulting in a final and improved transmission map. and reflection diagram However, current single-image reflection removal methods are not robust enough and struggle to handle reflections in complex environments or with strong reflections.

[0006] Flash-cue-based dereflection methods use an additional image taken under flash conditions as input. Capturing the same scene under both natural and flash lighting conditions only affects the imaging illumination of the transmissive scene; the reflective scene, located behind the camera, is unaffected by the flash. Therefore, flash imaging can provide more cues for dereflection, helping to overcome the limitations of traditional single-image dereflection methods. Lei et al. proposed a novel flash-cue-based image dereflection method. First, they subtract two images of the same scene taken under ambient and flash lighting conditions respectively to eliminate the influence of reflection, obtaining a flash-only image (equivalent to an image without natural light, containing only the flash image). This flash-only image is then used as a guide for a convolutional neural network to recover a higher-quality transmissive image.

[0007] Since flash units are widely integrated into smartphones, the difference in illumination produced when the flash is on or off can provide helpful cues for eliminating reflections, which helps to achieve better restoration results than single-image de-reflection. Summary of the Invention

[0008] To address the problems existing in the prior art, this invention provides an image restoration system and method based on flare cues and neural networks. This invention utilizes a deep neural network for image dereflection, and the dereflection neural network contains two identical subnetworks to generate reflection maps respectively. and transmission diagram The method employs a linear combination of L1 norm loss function and perceptual loss function to train the dereflection neural network; by utilizing the aid of flash image, this method can quickly and accurately recover non-reflection images.

[0009] To address the problems existing in the prior art, the present invention adopts the following technical solution:

[0010] An image restoration system and method based on flare cues and neural networks, wherein the image restoration system includes a reflection training dataset, a de-reflection convolutional neural network, and an optimization training module; the de-reflection convolutional neural network includes a reflection map estimation subnetwork and a transmission map estimation subnetwork; wherein:

[0011] The reflection training map set generates an initial image by processing the environment map, flash map, transmission map, and reflection map containing reflections using the following formula. :

[0012]

[0013] Among them: ambient lighting image and flash images ;

[0014] The reflectance estimation subnetwork uses ambient light images. and initial image Generate a reflection image as input. ;

[0015] The transmission map estimation subnetwork uses the reflection image and ambient lighting images Transmission image is generated as input. ;

[0016] The optimization training module trains the dereflective neural network through a linear combination of the L1 norm loss function and the perceptual loss function to obtain an optimized dereflective convolutional neural network.

[0017] Furthermore, both the reflectance estimation subnetwork and the transmission estimation subnetwork consist of an encoder and a decoder; the encoder includes convolutional layers, downsampling layers, and multi-branch residual layers; the decoder includes 8 convolutional layers and 4 upsampling layers.

[0018] Furthermore, the multi-branch residual layer also includes shallow residual units; the steps of the multi-branch residual layer in processing the input image features are as follows:

[0019] 201. The multi-branch residual layer will incorporate the first-level features. The following formula is used to segment the channel dimension and generate three independent image features;

[0020]

[0021] 202. The shallow residual units are respectively applied according to the following formulas. X 1[2] and X 1[3] Perform convolution and residual connection processing:

[0022]

[0023]

[0024] in: and These represent convolutional layers with kernels of 3×3 and 5×5, respectively.

[0025] 203. The multi-branch residual layer is processed according to the following formula. X 1[2] and X 1[3] The results after shallow residual unit processing are spliced ​​in the channel dimension to generate intermediate features.

[0026]

[0027] in: This indicates a splicing operation on the channel;

[0028] 204. The multi-branch residual layer applies the following formula to the intermediate features. Perform channel mixing operations and The convolutional layer features are fused to generate the first fused feature. F ,

[0029]

[0030] in: Indicates channel mixing operation; The convolutional layer is denoted as ;

[0031] 205. The multi-branch residual layer fuses features using the following formula. F Compared with first-level features Adding them together yields the second-level features. ,

[0032]

[0033] 206. Repeat steps 201-205 above for the second-level features. Three independent features are generated by segmentation along the channel dimension. , X 2[2] and X 2[3] Generating third-level features through shallow residual units ;

[0034] 207. Repeat steps 201-205 above for the third-level features. Three independent features are generated by segmentation along the channel dimension. , X 3[2] and X 3[3] Generate fourth-level features through shallow residual units ;

[0035] 208. The multi-branch residual layer is processed according to the following formula. The features are concatenated along the channel dimension and then fused again through a 1×1 convolutional layer to generate a second fused feature. ,

[0036]

[0037] 209. The multi-branch residual layer divides the first-level features according to the following formula. Features of the second fusion The final features are obtained by concatenating the channels again. O ,

[0038] .

[0039] The image restoration system and method based on flashlight cues and neural networks are characterized in that: the cost function obtained by the optimization training module for optimizing the de-reflection convolutional neural network is:

[0040] The L1 norm loss function is shown in the following equation:

[0041]

[0042] In the formula, This is a true reflection or transmission image. The reflection or transmission map estimated by the aforementioned neural network;

[0043] The perceptual loss function is shown in equation (8):

[0044]

[0045] In the formula, This refers to the output of the input feature map after passing through the ReLU1_2 and ReLU2_2 layers of the VGG19 network. , and These are the indexes of the width, height, and number of channels of the feature map, respectively. , and These represent the width, height, and number of channels of the feature map, respectively.

[0046] The overall loss function is a linear combination of the above two terms, as shown in the following equation:

[0047]

[0048] In the formula, and They are respectively and The weight.

[0049] This invention can also be implemented using the following technical solutions:

[0050] A system-based image restoration method based on flashlight cues and neural networks includes the following steps:

[0051] The reflection training map set generates an initial image by processing the environment map, flash map, transmission map, and reflection map containing reflections using the following formula. :

[0052]

[0053] Among them: ambient lighting image and flash images ;

[0054] The reflectance estimation subnetwork uses ambient light images. and initial image Generate a reflection image as input. ;

[0055] The transmission map estimation subnetwork uses the reflection image and ambient lighting images Transmission image is generated as input. ;

[0056] The optimization training module trains the dereflection neural network using a linear combination of the L1 norm loss function and the perceptual loss function to obtain an optimized dereflection convolutional neural network; wherein:

[0057] The steps of the multi-branch residual layer in processing the input image features are as follows:

[0058] 201. The multi-branch residual layer will incorporate the first-level features. The following formula is used to segment the channel dimension and generate three independent image features;

[0059]

[0060] 202. The shallow residual units are respectively applied according to the following formulas. X 1[2] and X 1[3] Perform convolution and residual connection processing:

[0061]

[0062]

[0063] in: and These represent convolutional layers with kernels of 3×3 and 5×5, respectively.

[0064] 203. The multi-branch residual layer is processed according to the following formula. X 1[2] and X 1[3] The results after shallow residual unit processing are spliced ​​in the channel dimension to generate intermediate features.

[0065]

[0066] in: This indicates a splicing operation on the channel;

[0067] 204. The multi-branch residual layer applies the following formula to the intermediate features. Perform channel mixing operations and The convolutional layer features are fused to generate the first fused feature. F ,

[0068]

[0069] in: Indicates channel mixing operation; The convolutional layer is denoted as ;

[0070] 205. The multi-branch residual layer fuses features using the following formula. F Compared with first-level features Adding them together yields the second-level features. ,

[0071]

[0072] 206. Repeat steps 201-205 above for the second-level features. Three independent features are generated by segmentation along the channel dimension. , X 2[2] and X 2[3] Generating third-level features through shallow residual units ;

[0073] 207. Repeat steps 201-205 above for the third-level features. Three independent features are generated by segmentation along the channel dimension. , X 3[2] and X 3[3] Generate fourth-level features through shallow residual units ;

[0074] 208. The multi-branch residual layer is processed according to the following formula. The features are concatenated along the channel dimension and then fused again through a 1×1 convolutional layer to generate a second fused feature. ,

[0075]

[0076] 209. The multi-branch residual layer divides the first-level features according to the following formula. Features of the second fusion The final features are obtained by concatenating the channels again. O ,

[0077] .

[0078] Beneficial effects:

[0079] 1. This invention can directly learn the transformation relationship between ambient lighting images, flash lighting images, and transmission images through neural networks, thereby achieving high-quality image dereflection;

[0080] 2. Once the neural network is trained, only two images of a scene taken under ambient lighting and flash lighting conditions are needed to remove reflections;

[0081] 3. The present invention has a good de-reflection effect, and has good effect on high-resolution images and complex scenes containing reflections. Attached Figure Description

[0082] Figure 1 A flowchart of an image restoration method based on flashlight cues and neural networks;

[0083] Figure 2 This is a schematic diagram of the overall structure of the dereflection neural network;

[0084] Figure 3 A schematic diagram of the subnetwork structure for estimating reflection and transmission maps;

[0085] Figure 4-6 The image shows the effect of dereflection on an image taken in a real-world scene. The left image is an ambient light image with reflections, and the right image is a transmission image estimated by a neural network (i.e., the dereflection result). Detailed Implementation

[0086] To achieve realistic image dereflection, this invention proposes an image restoration method based on flash cues and neural networks. (See [link]). Figure 1 See the description below for details:

[0087] 101: Based on optical principles, environmental images including reflections, flash images, transmission images, and reflection images are acquired. Flash-only images are obtained through basic image processing, and a flash-based de-reflection training image set is constructed. In this invention example, the dataset constructed by Lei et al. is used; wherein,

[0088] The specific steps for building the training set in step 101 are as follows:

[0089] 1) Construct an image containing ambient lighting. and flash images The training image set constituted, according to and Generate flash-only initial image :

[0090]

[0091] 2) Scale all images in the training set to a multiple of 32, and resize the environmental images in the training set. and initial image It is input into the de-reflection neural network.

[0092] 102: Construct a de-reflection convolutional neural network. The network structure contains two identical sub-networks: a reflection map estimation sub-network and a transmission map estimation sub-network; wherein, the reflection map estimation sub-network and the transmission map estimation sub-network are:

[0093] The dereflection network processes the ambient lighting images and initial images in the training image set using the following steps:

[0094] The reflectance estimation subnetwork uses ambient light images. Generate a reflection image using the initial image as input. ;

[0095] The transmission map estimation subnetwork uses the reflection image Generate a transmission image using the ambient lighting image as input. ;

[0096] 1) The dereflection neural network structure includes a reflection map estimation subnetwork and a transmission map estimation subnetwork, with the structure as follows: Figure 2 As shown, the two have the same structure and are connected in a cascaded manner, representing the original ambient lighting image. and initial image A reflection image is generated using a reflection map estimation subnetwork. The transmission image estimation subnetwork utilizes the reflection image and ambient lighting images Generate the final transmission map ;

[0097] 2) Both the reflection map estimation subnetwork and the transmission map estimation subnetwork consist of an encoder and a decoder;

[0098] 3) The encoder includes convolutional layers, downsampling layers, and four multi-branch residual layers; for example... Figure 3 As shown in the figure. Each multi-branch residual unit can be viewed as three repeated cascaded structures, as shown in the specific structure below. Figure 3 As shown, to avoid loss of generality, the input image size is set to... The feature map size output by the feature extraction module is... .

[0099] 4) The input of the multi-branch residual layer is denoted as ,Will The channel dimension will be divided into three parts at a ratio of 0.25, 0.375, and 0.375: The latter two parts X 1[2] and X 1[3] is fed into two shallow residual units with kernels of 3×3 and 5×5 respectively, and the two shallow residual layers are denoted as and The outputs of the two shallow residual layers are concatenated along the channel dimension to obtain intermediate features. ,Right now:

[0100]

[0101] Then, intermediate features Perform channel mixing operations and The convolutional layers are used for feature fusion, The convolutional layer is denoted as The output of the convolutional layer is the first fused feature. F ,Right now:

[0102]

[0103] This example uses the channel mixing method proposed in the literature.

[0104] Fusion features F Input features Adding them together gives the characteristics of the next level structure. ,Right now:

[0105] (4)

[0106] The new features obtained Further processing, including channel segmentation, shallow residual module, residual connection, and channel blending, yields the desired result. And so on, for Repeat the above process to generate features Next, The features are concatenated along the channel dimension and then fused again through a 1×1 convolutional layer to generate new features. ,Right now:

[0107] (5)

[0108] Finally, the original input features are... and Then, the channel dimensions are concatenated again to obtain the final output features of the multi-branch residual layer, namely:

[0109]

[0110] 5) Each shallow residual layer in 4) above consists of a convolutional layer with a specified kernel size and a LeakyRELU activation function, forming a residual structure to process the input features. ,Right now:

[0111]

[0112] 6) The decoder is used to generate the reflection map. and transmission diagram It consists of 8 convolutional layers and 4 upsampling layers. These 8 convolutional layers and 4 upsampling layers can be divided into 4 cascaded networks, each consisting of two convolutional layers and one upsampling layer. The decoder is based on the encoder output size. The feature maps are used as input, and directly fed into the next four cascaded networks. The final output is of size [size missing]. The reflection or transmission image.

[0113] 7) The activation function used throughout the network is the LeakyReLU function, and the Leaky value is 0.2.

[0114] 103: Training a reflexive neural network using a linear combination of the L1 norm loss function and the perceptual loss function;

[0115] The specific steps in step 103, which involves training the image restoration neural network using a linear combination of the L1 norm loss function and the perceptual loss function, are as follows:

[0116] 1) The L1 norm loss function is shown in the following equation:

[0117]

[0118] In the formula, This is a true reflection or transmission image. The reflection or transmission map estimated by the aforementioned neural network.

[0119] 2) The perceptual loss function is shown in equation (8):

[0120]

[0121] In the formula, This refers to the output of the input feature map after passing through the ReLU1_2 and ReLU2_2 layers of the VGG19 network. , and These are the indexes of the width, height, and number of channels of the feature map, respectively. , and These represent the width, height, and number of channels of the feature map, respectively.

[0122] 4) The overall loss function is a linear combination of the above two terms, as shown in the following equation:

[0123]

[0124] In the formula, and They are respectively and The weights. In this embodiment, we take... , .

[0125] 104: Input the ambient lighting image and the flash lighting image into the trained de-reflection neural network to obtain the transmission image. .

[0126] Specifically, step 104 involves using the trained model parameters, taking the ambient lighting image and the flash lighting image as input to the dereflection neural network, to obtain the transmission map. T .

[0127] Figures 4-6 The dereflection removal results of this algorithm in a real-world scene are presented. As can be seen from the figure, this algorithm can effectively improve image quality, remove almost all reflections in the image from the transmission map, and the dereflection removal process does not introduce significant distortion.

[0128] This invention is not limited to the embodiments described above. The above description of specific embodiments is intended to illustrate and explain the technical solutions of this invention. The specific embodiments described above are merely illustrative and not restrictive. Without departing from the spirit and scope of the claims, those skilled in the art can make many specific modifications based on the teachings of this invention, and these modifications all fall within the scope of protection of this invention.

Claims

1. An image restoration system based on flashlight cues and neural networks, characterized in that: The image restoration system includes a reflection training dataset, a dereflection convolutional neural network, and an optimization training module. The dereflection convolutional neural network includes a reflection map estimation subnetwork and a transmission map estimation subnetwork. Both the reflection map estimation subnetwork and the transmission map estimation subnetwork consist of an encoder and a decoder. The encoder includes convolutional layers, downsampling layers, and multi-branch residual layers. The multi-branch residual layers also include shallow residual units. The decoder includes 8 convolutional layers and 4 upsampling layers. The reflection training map set generates an initial image by processing the environment map, flash map, transmission map, and reflection map containing reflections using the following formula. : Among them: ambient lighting image and flash images ; The reflectance estimation subnetwork uses ambient light images. and initial image Generate a reflection image as input. ; The transmission map estimation subnetwork uses the reflection image and ambient lighting images Transmission image is generated as input. ; The optimization training module trains the dereflection neural network through a linear combination of the L1 norm loss function and the perceptual loss function to obtain an optimized dereflection convolutional neural network.

2. The image restoration system based on flashlight cues and neural networks according to claim 1, characterized in that: The steps of the multi-branch residual layer in processing the input image features are as follows: S201, the multi-branch residual layer will use the first-level feature The following formula is used to segment the channel dimension and generate three independent image features; S202, the shallow residual units are respectively processed according to the following formulas. X 1[2] and X 1[3] Perform convolution and residual connection processing: in: and These represent convolutional layers with kernels of 3×3 and 5×5, respectively. S203, the multi-branch residual layer is processed according to the following formula. X 1[2] and X 1[3] The results after shallow residual unit processing are spliced ​​in the channel dimension to generate intermediate features. in This indicates a splicing operation on the channel; S204, the multi-branch residual layer processes the intermediate features according to the following formula. Perform channel mixing operations and The convolutional layer features are fused to generate the first fused feature. F , in: Indicates channel mixing operation; The convolutional layer is denoted as ; S205, The multi-branch residual layer fuses features using the following formula. F Compared with first-level features Adding them together yields the second-level features. , S206, Repeat steps 201-205 for the second-level features. Three independent features are generated by segmentation along the channel dimension. , X 2[2] and X 2[3] Generating third-level features through shallow residual units ; S207, Repeat steps 201-205 for the third-level features. Three independent features are generated by segmentation along the channel dimension. , X 3[2] and X 4[3] Generate fourth-level features through shallow residual units ; S208, the multi-branch residual layer is processed according to the following formula: The features are concatenated along the channel dimension and then fused again through a 1×1 convolutional layer to generate a second fused feature. , S209, The multi-branch residual layer divides the first-level features according to the following formula. Features of the second fusion The final features are obtained by concatenating the channels again. O , 。 3. The image restoration system based on flashlight cues and neural networks according to claim 2, characterized in that: The cost function used by the optimized training module to optimize the de-reflection convolutional neural network is: The L1 norm loss function is shown in the following equation: In the formula, Y This is a true reflection or transmission image. The reflection or transmission map estimated by the aforementioned neural network; The perceptual loss function is shown in the following formula: In the formula, This refers to the output of the input feature map after passing through the ReLU1_2 and ReLU2_2 layers of the VGG19 network. , and These are the indexes of the width, height, and number of channels of the feature map, respectively. , and These represent the width, height, and number of channels of the feature map, respectively. The overall loss function is a linear combination of two loss functions, as shown in the following equation: In the formula, and They are respectively and The weight.

4. A method for image restoration based on flashlight cues and neural networks, using the system described in any one of claims 1-3, characterized in that: Includes the following steps: The reflection training map set generates an initial image by processing the environment map, flash map, transmission map, and reflection map containing reflections using the following formula. : Among them: ambient lighting image and flash images ; The reflectance estimation subnetwork uses ambient light images. and initial image Generate a reflection image as input. ; The transmission map estimation subnetwork uses the reflection image and ambient lighting images Transmission image is generated as input. ; The optimization training module trains the dereflection neural network using a linear combination of the L1 norm loss function and the perceptual loss function to obtain an optimized dereflection convolutional neural network; wherein: The steps of the multi-branch residual layer in processing the input image features are as follows: S101, the multi-branch residual layer will use the first-level feature The following formula is used to segment the channel dimension and generate three independent image features; S102, the shallow residual units are respectively processed according to the following formulas. X 1[2] and X 1[3] Perform convolution and residual connection processing: in and These represent convolutional layers with kernels of 3×3 and 5×5, respectively. S103, the multi-branch residual layer is processed according to the following formula. X 1[2] and X 1[3] The results after shallow residual unit processing are spliced ​​in the channel dimension to generate intermediate features. , in This indicates a splicing operation on the channel; S104, the multi-branch residual layer processes the intermediate features according to the following formula. Perform channel mixing operations and The convolutional layer features are fused to generate the first fused feature. F , in: Indicates channel mixing operation; The convolutional layer is denoted as ; S105, The multi-branch residual layer fuses features using the following formula. F Compared with first-level features Adding them together yields the second-level features. , S106. Repeat steps 101-105 for the second-level features. Three independent features are generated by segmentation along the channel dimension. , X 2[2] and X 2[3] Generating third-level features through shallow residual units ; S107. Repeat steps 101-105 for the third-level features. Three independent features are generated by segmentation along the channel dimension. , X 3[2] and X 3[3] Generate fourth-level features through shallow residual units ; S108, the multi-branch residual layer is processed according to the following formula. The features are concatenated along the channel dimension and then fused again through a 1×1 convolutional layer to generate a second fused feature. , S109, The multi-branch residual layer divides the first-level features according to the following formula. Features of the second fusion The final features are obtained by concatenating the channels again. O , 。

Images