A method, system, electronic device, and storage medium for de-reflection under low-light conditions

Abstract

This invention discloses a low-light conditional dereflection method, system, electronic device, and storage medium. The method includes the following steps: First, a low-light conditional reflection image dataset is synthesized using the characteristics of reflective images under low-light conditions, and a real-world low-light reflection image dataset is obtained. Then, the synthesized low-light reflection image is used as a training image I, and input into a low-light conditional dereflection model along with the corresponding illumination attention map. After processing by an illumination enhancement module and a dereflection module, an estimated image R' of the normal light reflection layer and an estimated image T' of the normal light background layer are obtained. The network is trained using a backpropagation algorithm to obtain a trained low-light conditional dereflection model. Finally, a real-world low-light reflection image is input into the trained low-light conditional dereflection model to obtain the corresponding estimated image R' of the normal light reflection layer and the estimated image T' of the normal light background layer.

Description

Technical Field

[0001] This invention relates to the field of digital image restoration technology, and in particular to a method, system, electronic device, and storage medium for dereflection under low-light conditions. Background Technology

[0002] Computer vision technology has developed rapidly in recent years, and is widely used in fields such as autonomous driving and drones, making outstanding contributions to technological innovation and improving human life. Through photos taken by cameras, computers can "see" the world and learn from the information in the photos through extensive training, thereby making judgments and outputs, mimicking human visual perception of the outside world. However, this simulation of visual perception places high demands on the quality of the input image data. If there are reflections, raindrops, or other obstructions in the image, or if the image was captured in insufficient light or was poorly exposed, the image quality will degrade, posing a significant challenge to computer computation and training. More seriously, using these low-quality images as input in real-world applications may lead to safety risks and cause great harm to users. For example, in nighttime autonomous driving, if the road condition image input to the computer is low-light and reflective, it will affect the performance of the relevant object detection model, thus affecting the system's judgment of road conditions. Therefore, effectively handling images affected by low light and reflection is an important issue for promoting the development and progress of vision applications.

[0003] Research in image reflection removal has developed rapidly in recent years, with many deep learning-based image reflection removal methods proposed. Fan et al. (Q. Fan, J. Yang, G. Hua, B. Chen, and DPWipf, “A generic deep architecture for single image reflection removal and image smoothing.” in IEEE International Conference on Computer Vision, 2017.) first designed a reflection removal model using deep neural networks. It first regresses and generates the edge map of the background layer, and then reconstructs the background layer. Wei et al. (K. Wei, J. Yang, Y. Fu, “Single image reflection removal exploiting misaligned training data and network enhancements.” in IEEE Conference on Computer Vision and Pattern Recognition, 2019) improved the network structure based on the research of Fan et al., added contextual information to the network, and the model can be trained on both aligned and unaligned data. Yang et al. (J. Yang, D. Gong, L. Liu, Q. Shi, “Seeing deeply and bidirectionally: A deep learning approach for single image reflection removal.” in European Conference on Computer Vision, 2018.) proposed a multi-level network with two sequential estimation layers, where the reflection layer predicted in the previous stage serves as auxiliary information to guide the reconstruction of the transport layer in the next stage. Li et al. (C. Li, Y. Yang, K. He, “Single image reflection removal through cascaded refinement.” in IEEE Conference on Computer Vision and Pattern Recognition, 2020.) proposed a hierarchical optimization network structure, a recurrent network based on long short-term memory, which enables cross-level transfer of image information.Lei et al. (C. Lei, Q. Chen, “Robust Reflection Removal with Reflection-free Flash-only Cues.” in IEEE Conference on Computer Vision and Pattern Recognition, 2021.) proposed using flash images to perform dereflection removal. Through the principles of subtraction and physical optics, they obtained images containing only flash after removing ambient light, providing a new approach to image dereflection removal. Dong et al. (Z. Dong, K. Xu, Y. Yang, “Location-aware Single Image Reflection Removal.” in IEEE International Conference on Computer Vision, 2021.) recently proposed a location-aware image dereflection model. This model uses Laplacian kernel parameters to emphasize reflection boundaries and employs a reflection detection module to regress and predict the probability reflection confidence map, improving the quality of the dereflection results. These methods focus more on eliminating strong reflections under normal lighting conditions, while research on weak reflections under low-light conditions is relatively limited.

[0004] For low-light image enhancement, existing deep learning methods mainly rely on paired training, where most low-light images are synthesized from normal-light images. Lore et al. (K. G. Lore, A. Akintayo, S. Sarkar, “Llnet: A deep autoencoder approach to natural low-light image enhancement.” In Pattern Recognition, 2017.) proposed a stacked autoencoder to learn patch-level joint denoising and low-light enhancement, which was the first network to truly apply deep learning to image enhancement. Wei et al. (C. Wei, W. Wang, W. Yang, “Deep retinex decomposition for low-light enhancement.” arXiv preprint arXiv:1808.04560, 2018.) provided an end-to-end framework for combining Retinex theory and deep networks, including networks for decomposition and enhancement networks for illumination enhancement. Gharbi et al. (M. Gharbi, J. Chen, J.T. Barron, “Deep bilateral learning for real-time image enhancement.” in ACM Transactions on Graphics (TOG), 2017.) combined deep networks with bilateral mesh processing and local affine color transformation with pairwise supervision to train convolutional neural networks to predict the coefficients of local affine models in bilateral space. Chen et al. (C. Chen, Q. Chen, J. Xu, “Learning to see in the dark.” arXiv preprint arXiv:1805.01934, 2018.) proposed a model that directly uses raw sensor data and replaces a large number of traditional image processing procedures. It focuses more on avoiding artifact amplification during low-light enhancement by learning pipelines for color transformation, demosaicing, and denoising. The visual effects obtained by training this model are impressive. Summary of the Invention

[0005] To address the gaps and deficiencies in existing research on dereflection under low-light conditions, this invention provides a method, system, electronic device, and storage medium for dereflection under low-light conditions. By utilizing information from the reflected image and the estimated image of the reflective layer, the invention achieves the goal of dereflection under low-light conditions.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a method for dereflection under low-light conditions, comprising the following steps:

[0008] Generate a first reflection image dataset, the first reflection image dataset containing a first reflection image, the first reflection image being generated from a reflection layer image and a background layer image under a first illumination condition, and / or, a reflection layer image and a background layer image under a second illumination condition, the first illumination condition being weaker than the second illumination condition; generate a corresponding first illumination attention map using the first reflection image dataset;

[0009] A low-light conditional dereflection model is generated by inputting the first reflection image dataset and the first illumination attention map into the low-light conditional dereflection model. The low-light conditional dereflection model includes an illumination enhancement module and a dereflection module. After processing by the illumination enhancement module and the dereflection module, the reflection layer image estimate and the normal light background layer image are obtained. The low-light conditional dereflection model is trained by the backpropagation algorithm to obtain the trained low-light conditional dereflection model.

[0010] A second reflection image dataset is obtained, and a corresponding second illumination attention map is generated using the second reflection image dataset. The second reflection image dataset and the second illumination attention map are input into the low-light condition de-reflection model to obtain a normal light reflection layer image and a normal light background layer image.

[0011] In a second aspect, the present invention provides a low-light condition de-reflection system, comprising:

[0012] A first processing unit is configured to generate a first reflection image dataset, the first reflection image dataset containing a first reflection image, the first reflection image being generated from a reflection layer image and a background layer image under a first illumination condition, and / or, a reflection layer image and a background layer image under a second illumination condition, wherein the first illumination condition is weaker than the second illumination condition; and to generate a corresponding first illumination attention map using the first reflection image dataset.

[0013] The second processing unit is used to generate a low-light conditional dereflection model. The first reflection image dataset and the first illumination attention map are input into the low-light conditional dereflection model. The low-light conditional dereflection model includes an illumination enhancement module and a dereflection module. After processing by the illumination enhancement module and the dereflection module, the reflection layer image estimate and the normal light background layer image are obtained. The low-light conditional dereflection model is trained by the backpropagation algorithm to obtain the trained low-light conditional dereflection model.

[0014] The processing unit is used to acquire a second reflection image dataset, generate a corresponding second illumination attention map using the second reflection image dataset, and input the second reflection image dataset and the second illumination attention map into the low-light condition de-reflection model to obtain a normal light reflection layer image and a normal light background layer image.

[0015] Thirdly, the present invention provides an electronic device comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the low-light condition de-reflection method as described above.

[0016] Fourthly, the present invention provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the low-light condition de-reflection method as described above.

[0017] Compared with existing technologies, the advantages of this invention are as follows: This invention fills the gap and deficiency in research on low-light dereflection, explores methods for synthesizing low-light reflective images, and provides new ideas and methods for this field; this invention fully utilizes the characteristics of low-light reflective images, enhances image brightness, distinguishes between the background layer and the reflective layer, and improves image quality and clarity, which is beneficial for subsequent advanced visual tasks. This invention uses deep learning methods, combining the U-net model and a dereflection module with a reflection-aware guided multi-scale encoding and decoding structure, effectively utilizing information from the reflective image and the estimated image of the reflective layer to achieve the purpose of low-light dereflection. This invention can produce good restoration results for real low-light reflective images. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the process of synthesizing low-reflectance images in an embodiment of the present invention;

[0020] Figure 2 This is a flowchart of the low-light condition de-reflection method in an embodiment of the present invention;

[0021] Figure 3This is an example image of a real-world low-light reflection image and its corresponding background layer image in an embodiment of the present invention;

[0022] Figure 4 This is a rendering of a normal light reflection image obtained after processing by the light enhancement module according to an embodiment of the present invention;

[0023] Figure 5 This is a rendering of the normal light background layer image obtained after processing by the dereflection module in an embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0025] Example:

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, in the embodiments of this invention are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] This invention primarily studies a low-light dereflection technique based on deep learning methods. Targeting reflective images captured in low-light environments, it fully utilizes the characteristics of these images to enhance image brightness and separate the background and reflective layers. The purpose of this invention is to fill the gaps and deficiencies in existing research on low-light dereflection, explore methods for synthesizing low-light reflective images, and propose a low-light dereflection method. This method uses a U-net-based illumination enhancement module and a multi-scale encoding / decoding-based dereflection module to obtain estimated background and reflective layer images under normal lighting conditions, thus achieving dereflection reduction in low-light conditions.

[0028] See Figure 1 and Figure 2 , Figure 1 A flowchart illustrating the process of synthesizing low-reflectance images in an embodiment of the present invention is shown; Figure 2A flowchart illustrating the low-light de-reflection method in an embodiment of the present invention is shown.

[0029] A method for dereflection under low-light conditions may specifically include the following steps:

[0030] Step 1: Generate a first reflection image dataset, which contains a first reflection image. The first reflection image is generated from a reflection layer image and a background layer image under a first illumination condition, and / or a reflection layer image and a background layer image under a second illumination condition, wherein the first illumination condition is weaker than the second illumination condition; generate a corresponding first illumination attention map using the first reflection image dataset.

[0031] In this step, the first lighting condition is also called the low-light condition, and the second lighting condition is also called the normal light condition. Specifically, this step requires synthesizing a composite low-light reflectance image dataset with a reflectance layer image and a background layer image under low-light conditions, as well as corresponding reflectance layer images and background layer images under normal light conditions. This embodiment collects the publicly available VOC2012 dataset and the ExDark dataset for dataset synthesis. The synthesis process is as follows: Figure 1 As shown. Since the VOC2012 dataset is under normal lighting conditions, it requires reflection synthesis and brightness reduction processing; while the ExDark dataset is under low lighting conditions, it only requires reflection synthesis processing.

[0032] In some implementations, synthesizing a dataset of reflection images with out-of-focus reflection characteristics using a reflection image synthesis method may specifically include the following steps:

[0033] Step 101: Obtain the background layer image T and the reflection layer image R;

[0034] Step 102: Normalize the color channel values ​​of the reflection layer image R and the background layer image T to the range of 0 to 1 respectively;

[0035] Step 103: Blur the reflection layer image R using a Gaussian filter to obtain a new reflection layer image.

[0036] Step 104: Combine the background layer image T and the new reflection layer image Perform matrix superposition Let m be the average value of the values ​​outside the RGB color range in the superimposed image;

[0037] Step 105: According to the formula Obtain the image of the new reflection layer Value, where γ is the adjustment coefficient;

[0038] Step 106: Crop the new reflection layer image The values ​​of each color channel are limited to the range of 0 to 1;

[0039] Step 107: Combine the background layer image T and the new reflection layer Perform matrix superposition The first reflection image is obtained by cropping the image to limit the values ​​of each channel to the range of 0 to 1. The first reflection image dataset consists of multiple first reflection images.

[0040] In some implementations, part of the synthesized reflective layer image is still under normal light, and the brightness of this part of the reflective layer image needs to be adjusted. An illuminance map is generated from the coordinates of the bright areas in the synthesized first reflective image in the above steps. The brightness of the reflective image is reduced according to the low-light image synthesis method, which may specifically include the following steps:

[0041] The obtained first reflection image is blurred using a Gaussian filter, and a thresholding method is applied to filter out bright areas;

[0042] Let the center coordinates of the bright area be the center of the circle, and randomly select the radius and light intensity to generate a smoothly transitioned illuminance map;

[0043] Based on the illuminance map, the illuminance in the first reflectance image is adjusted using a random illumination method, as follows:

[0044] I = I normal L

[0045] Where I normal The image refers to the reflection image under normal light conditions, where L is the illuminance image and I is the reflection image under low light conditions.

[0046] Step 2: Use the synthesized low-light reflection image from the previous step and the image under normal lighting as training image I, and combine them with the generated lighting attention image. Figure 1 The input is fed into the low-light conditional dereflection model. After processing by the illumination enhancement module based on the U-net structure and the dereflection module based on the multi-scale encoding and decoding structure, the estimated image R′ of the normal light reflection layer and the estimated image T′ of the normal light background layer are obtained. The network is trained by the backpropagation algorithm to obtain the trained low-light conditional dereflection model.

[0047] In this step, a low-light conditional dereflection model is generated. The first reflection image dataset and the illumination attention map are input into the low-light conditional dereflection model. The low-light conditional dereflection model includes an illumination enhancement module and a dereflection module. After processing by the illumination enhancement module and the dereflection module, the reflection layer image estimate and the normal light background layer image are obtained. The low-light conditional dereflection model is trained by the backpropagation algorithm to obtain the trained low-light conditional dereflection model.

[0048] In some embodiments, the illumination enhancement module (based on the U-net structure) includes a convolutional layer (Conv), an average pooling layer (AvgPool), a max pooling layer (MaxPool), a deconvolutional layer (Deconv), a normalization layer (Batch Norm), an activation function (LeakyReLU), and a skip connection (Skip Conm), while also introducing an attention mechanism.

[0049] In low-light reflection images, rather than enhancing the entire region, this invention aims to enhance only the low-light areas while maintaining the original brightness of the bright areas, ensuring that the final output image is neither underexposed nor overexposed. To this end, the low-light dereflection model introduces an attention mechanism. For example, it uses a threshold to filter out the bright and dark areas of the image to form an attention map, and adjusts the size of the attention map using a 2×2 max-pooling layer to adapt to feature maps of different scales, multiplying it with all intermediate feature maps and the output image.

[0050] Specifically, in the illumination enhancement module (based on the U-net structure), the input is a first reflection image of size W×H×3 and the corresponding illumination attention map. Feature extraction is performed using five convolutional blocks consisting of a 3×3 kernel convolutional layer, a LeakyReLU activation function, and a normalization layer. Each convolutional block is connected by a downsampling layer, which is composed of an average pooling layer with a 2×2 pooling kernel. The first reflection image output from the downsampling layer is encoded and input into five deconvolutional layers with a 3×3 kernel, a LeakyReLU activation function, and a normalization layer. Feature recovery is performed in deconvolutional blocks composed of the liveness function LeakyReLU and normalization layers. Upsampling between each deconvolutional block is performed using bilinear interpolation with a factor of 2. A staggered skip connection is set between the convolutional and deconvolutional blocks. The features of all upper convolutional blocks are downsampled and concatenated with the features of the current layer. After being multiplied by the illumination attention map, the concatenation is added to the corresponding deconvolutional block. Finally, the feature map obtained is multiplied by the illumination attention map and added to the input to output a normal light reflection image I′ with an illumination enhancement of size W×H×3.

[0051] In some embodiments, the dereflection module (based on a multi-scale encoding and decoding structure) includes multiple encoders and decoders. Each encoder and decoder can be divided into a convolutional layer (Conv), an average pooling layer (AvgPool), a max pooling layer (MaxPool), a transposed convolutional layer (ConvTranspose), a normalization layer (Batch Norm), an activation function (ReLU), and an activation function (Sigmoid), and uses feature-generated masks to guide the restoration of the reflection image.

[0052] For ease of description, let the feature of the intermediate image I′ of the normal light reflection layer be denoted as F. I′ The features of the reflection layer image are denoted as F. R′(Also known as the features of the decoder), the features of the background layer image are denoted as F. T′ .

[0053] Specifically, five encoders are used to extract multi-scale features of the normal light reflection image I′ obtained from the illumination enhancement module. The encoder performs a 3×3 kernel convolution operation on the image and uses the ReLU activation function. At the same time, it extracts the features of the selected layer of the VGG19 model and normalizes them. The downsampling operation between each encoder layer consists of a max pooling layer with a 2×2 pooling kernel.

[0054] Features F of the bottom reflection layer image R′ The feature F of the normal light reflection image of the corresponding layer I′ The features F of the remaining three reflective layer images are stitched together. R′ Then, simultaneously with the characteristic F of a normal light reflection image I′ Features F of the background layer image T′ The image is stitched together, and a mask is calculated using convolutional layers, the Sigmoid activation function, the ReLU activation function, and average pooling with a 3×3 pooling kernel to guide the restoration of reflective areas.

[0055] Calculate the feature F of a normal light reflection image I′ Features F of the reflection layer image R′ The characteristic differences between F I′ -F R′ The feature differences are multiplied by the mask and then upsampled to obtain the feature F of the next background layer image. T′ The upsampling layer consists of one or three sets of convolutional blocks, each of which includes a 3×3 kernel convolutional layer, a normalization layer, and a ReLU activation function.

[0056] The features F of the last reflection layer image are processed by a convolution operation with a kernel size of 1×1. R′ The image is restored to 3 channels, and the estimated image of the normal light reflection layer, R′, is output. The final background layer feature is F. T′ The normal light background layer estimated image T′ is also output through a 1×1 convolution operation.

[0057] In some embodiments, the low-light conditional dereflection model is trained using a backpropagation algorithm as a deep learning method. Deep learning methods employ several loss functions, which help the model to perform better training.

[0058] The illumination enhancement module based on the U-net architecture utilizes a global-local discriminator to guide the network in improving global illumination and adaptively enhancing local regions, without requiring paired low-light-normal-light image pairs. The discriminator's convolutional blocks consist of 4×4 kernel convolutional layers and the LeakyReLU activation function. In both the global and local discriminators, PatchGAN is used for real / false image detection, and the overall discrimination is derived by combining the results from multiple parts. To enable adaptive enhancement of local low-light regions, n local image patches are randomly cropped from the output image and the real normal-light image as input to the local discriminator.

[0059] For the loss function, the model uses least-squares GAN loss. The loss functions for the global discriminator D and generator G in the final model are as follows:

[0060]

[0061]

[0062] x r and x f The distribution of the enhanced images, D, represents the distribution of the images after illumination enhancement, specifically the distribution of the images under normal lighting conditions and the distribution of the images under low light reflection conditions. Ra It is the standard function of the relativistic discriminant. Represents the mathematical expectation. This represents a probability distribution.

[0063] Similarly, for the local discriminator, an adversarial loss is also set, and the cropped local image patches are respectively called fake-patches and real-patches:

[0064]

[0065]

[0066] Let the aforementioned adversarial loss be denoted as Represented as:

[0067]

[0068] This method also calculates the VGG feature distance between the input image and the corresponding output image, called the perceptual loss, which is expressed as: The definition is as follows:

[0069]

[0070] Where φ represents the VGG-19 model, and l represents the index of the selected VGG-19 feature layers conv1_2, conv2_2, conv3_2, conv4_2, and conv5_2. Weights {kl This is used to balance different layers. The self-feature preservation loss described above is used for regularization in both the global and local discriminators, expressed as follows: and In the dereflection module, a perceptual loss is also applied to the output background image, denoted as . Therefore, the total perceptual loss of the model is expressed as:

[0071]

[0072] Feature differences in the dereflection module based on a multi-scale encoding / decoding structure can better suppress reflections. A threshold is used to limit the mask image generated by these feature differences, guiding the network to generate a more accurate mask image. The mask loss is set here and defined as follows:

[0073]

[0074]

[0075]

[0076] Where i represents the i-th level, ||·||1 represents the l1 norm, and M[condition] represents the part of M that satisfies the condition in square brackets. ξ is the threshold for defining the heavily reflective region, while ξ represents the threshold for the region with very little reflection.

[0077] The reconstruction loss is the pixel difference between the output background layer T′ and reflection layer R′ and the corresponding ground truth values, expressed as:

[0078]

[0079] Since the edges of the actual reflection layer and the background layer typically do not overlap, an exclusion loss is introduced to reduce the correlation between them, denoted as:

[0080]

[0081] in λ T and λ R Represents the normalization factor. Represents the gradient. ||·|| F It is the F-norm. T′ ↓n and R′ ↓n Represents T′ and R′ obtained by downsampling by n times, where T′ ↓0 and R′ ↓0 This is the raw input.

[0082] Combining the above loss functions, the total loss of the low-light condition dereflection method in this invention is expressed as follows:

[0083]

[0084] The network, including a lighting enhancement module based on the U-net structure and a de-reflection module based on a multi-scale encoding and decoding structure, is trained using the backpropagation algorithm to obtain a well-trained low-light condition de-reflection model.

[0085] Step 3: Obtain the second reflection image dataset, generate the corresponding second illumination attention map using the second reflection image dataset, and input the second reflection image dataset and the second illumination attention map into the low-light condition de-reflection model to obtain the normal light reflection layer image and the normal light background layer image.

[0086] In this step, the second reflection image dataset refers to a real low-light reflection image dataset composed of reflection layer images and background layer images under low-light conditions in the real world. To obtain this dataset, the phone is fixed with a tripod, and the camera's focus and exposure are locked. A timed burst of shots is used to obtain 10 images. Before shooting, the reflector is placed. The burst shooting process takes approximately one second. To obtain image pairs with and without reflection in a single burst, the reflector needs to be quickly removed before the burst ends. Finally, images with complete and clear reflection and images without reflection are selected from the 10 images as one image pair. This example obtained a total of 52 pairs of real images. Figure 3 Example images of a pair of real-world low-light reflectance images and their corresponding background layers are provided. The real low-light reflectance images are input into the low-light conditional dereflection model trained in step 2 to obtain the estimated image R′ of the normal light reflectance layer and the estimated image T′ of the normal light background layer.

[0087] Figure 4 yes Figure 3 The image shows the result of processing the low-light reflection image in step 2 using the illumination enhancement module to obtain the normal light reflection image. Figure 5 yes Figure 4 The image shows the result of the normal light background layer image after processing by the dereflection module in step 2. It is evident that the method provided by this invention can effectively accomplish the task of dereflection under low-light conditions.

[0088] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0089] The above embodiments are merely illustrative of the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made based on the essence of the content of the present invention should be covered within the scope of protection of the present invention.

[0090] Based on the same inventive concept, embodiments of the present invention also provide a low-light condition dereflection system, which includes:

[0091] A first processing unit is configured to generate a first reflection image dataset, the first reflection image dataset containing a first reflection image, the first reflection image being generated from a reflection layer image and a background layer image under a first illumination condition, and / or, a reflection layer image and a background layer image under a second illumination condition, wherein the first illumination condition is weaker than the second illumination condition; and to generate a corresponding first illumination attention map using the first reflection image dataset.

[0092] The second processing unit is used to generate a low-light conditional dereflection model. The first reflection image dataset and the first illumination attention map are input into the low-light conditional dereflection model. The low-light conditional dereflection model includes an illumination enhancement module and a dereflection module. After processing by the illumination enhancement module and the dereflection module, the reflection layer image estimate and the normal light background layer image are obtained. The low-light conditional dereflection model is trained by the backpropagation algorithm to obtain the trained low-light conditional dereflection model.

[0093] The processing unit is used to acquire a second reflection image dataset, generate a corresponding second illumination attention map using the second reflection image dataset, and input the second reflection image dataset and the second illumination attention map into the low-light condition de-reflection model to obtain a normal light reflection layer image and a normal light background layer image.

[0094] Since this system corresponds to the low-light condition de-reflection method in the embodiments of the present invention, and the principle of solving the problem in this system is similar to that of the method, the implementation of this system can refer to the implementation process of the above method embodiments, and the repeated parts will not be described again.

[0095] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, the electronic device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to realize the low-light condition de-reflection method as described above.

[0096] It is understood that the memory may include random access memory (RAM) or read-only memory. Optionally, the memory may include non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, etc.; the stored data area may store data created according to the use of the server, etc.

[0097] A processor may include one or more processing cores. The processor connects to various parts of the server via various interfaces and lines, executing instructions, programs, code sets, or instruction sets stored in memory, and accessing data stored in memory to perform various server functions and process data. Optionally, the processor may be implemented using at least one of the following hardware forms: Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor may integrate one or more of the following: Central Processing Unit (CPU) and Modem. The CPU primarily handles the operating system and applications; the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor.

[0098] Since this electronic device is the electronic device corresponding to the low-light condition de-reflection method of the present invention, and the principle of solving the problem by this electronic device is similar to that of the method, the implementation of this electronic device can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.

[0099] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the low-light condition de-reflection method as described above.

[0100] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0101] Since this storage medium is the storage medium corresponding to the low-light condition de-reflection method of the present invention, and the principle of the storage medium in solving the problem is similar to that of the method, the implementation of this storage medium can refer to the implementation process of the above method embodiments, and the repeated parts will not be described again.

[0102] In some possible implementations, various aspects of the methods of the embodiments of the present invention can also be implemented as a program product comprising program code that, when run on a computer device, causes the computer device to perform the steps of the low-light condition de-reflection methods according to the various exemplary embodiments of this application described above. The executable computer program code or "code" for performing the various embodiments can be written in high-level programming languages ​​such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0103] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0104] The above embodiments are merely illustrative of the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made based on the essence of the content of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A low-light condition anti-reflecting method characterized by, The method comprises the following steps: generating a first reflection image dataset containing a first reflection image generated from a reflection layer image and a background layer image under a first lighting condition, or a reflection layer image R and a background layer image T under the first lighting condition and a reflection layer image and a background layer image under a second lighting condition, the first lighting condition being a low-light condition, and the second lighting condition being a normal-light condition; generating a corresponding first lighting attention map using the first reflection image dataset; generating a low-light condition deflection model, the low-light condition deflection model comprising a lighting enhancement module and a deflection module, the lighting enhancement module containing five convolutional blocks each consisting of a convolutional layer with a kernel size of 3x3, an activation function LeakyReLU, and a normalization layer, each convolutional block being connected by a down-sampling layer consisting of an average pooling layer with a 2x2 pooling kernel, and further containing five deconvolutional blocks each consisting of a deconvolutional layer with a kernel size of 3x3, an activation function LeakyReLU, and a normalization layer, each deconvolutional block being up-sampled by a bilinear interpolation with a multiple of 2, and the convolutional blocks and the deconvolutional blocks being arranged in staggered layers with skip connections, the deflection module containing multiple encoders and decoders, each encoder performing a convolution operation with a kernel size of 3x3 and using an activation function ReLU, extracting features of selected layers of a VGG19 model and normalizing them, and each encoder being down-sampled by a maximum pooling layer with a 2x2 pooling kernel; inputting the first reflection image dataset and the first lighting attention map into the low-light condition deflection model to obtain a reflection layer estimation image and a normal-light background layer estimation image after processing by the lighting enhancement module and the deflection module, and training the low-light condition deflection model by a backpropagation algorithm to obtain a trained low-light condition deflection model; obtaining a second reflection image dataset, generating a corresponding second lighting attention map using the second reflection image dataset, and inputting the second reflection image dataset and the second lighting attention map into the low-light condition deflection model to obtain a normal-light reflection layer image and a normal-light background layer image.

2. The low-light condition de-reflecting method of claim 1, wherein, The method for generating a first reflection image dataset specifically comprises the following steps: obtaining a background layer image T and a reflection layer image R; normalizing the values of each color channel of the reflection layer image R and the background layer image T to the interval of 0 to 1, respectively; blurring the reflectivity layer image R with a Gaussian filter to obtain a new reflectivity layer image The background layer image T and the new reflection layer image Matrix superposition is performed Let m be the average of the values of the superimposed image that exceed the RGB color range; According to the formula The numerical value of the new reflection layer image γ is the adjustment coefficient; limiting each color channel value in the new reflection layer image to the interval 0 to 1 by a clipping operation; The background layer image T and the new reflection layer Matrix multiplication is performed and by clipping the image channel values to the 0 to 1 interval, the first reflection image is obtained, the first reflection image dataset consisting of a plurality of the first reflection images.

3. The low-light condition de-reflecting method of claim 2, wherein, The method for adjusting the lighting intensity in the first reflection image dataset specifically comprises the following steps: blurring the obtained first reflection image using a Gaussian filter and applying a thresholding method to screen out bright areas; setting the center coordinates of the bright areas as the center of a circle, randomly selecting a radius length and a lighting intensity, and generating a transition smooth illumination map; adjusting the lighting intensity in the first reflection image according to the illumination map by a random lighting method.

4. The low-light condition de-reflecting method of claim 1, wherein, In the illumination enhancement module, a first reflection image with an input size of WxHx3 and a corresponding illumination attention map are subjected to feature extraction by five convolution blocks composed of a convolution layer with a kernel size of 3x3, an activation function LeakyReLU and a normalization layer, and are connected by a down-sampling layer between each convolution block, and the down-sampling layer is composed of an average pooling layer with a 2x2 pooling kernel; the first reflection image output by the down-sampling layer is encoded and input into five deconvolution blocks composed of a deconvolution layer with a kernel size of 3x3, an activation function LeakyReLU and a normalization layer for feature recovery, and the up-sampling between each deconvolution block is performed by bilinear interpolation with a multiple of 2; staggered connection is arranged between the convolution blocks and the deconvolution blocks, the features of all upper convolution blocks are down-sampled and spliced with the features of the current layer, multiplied by the illumination attention map and added to the corresponding deconvolution block of the current layer; the final feature map is multiplied by the illumination attention map and added to the input to output a normal light reflection estimation image I' after illumination enhancement with a size of WxHx3.

5. The low-light condition de-reflecting method of claim 1, wherein, In the anti-reflection module, the feature of the normal light reflection estimation image I' is denoted as F I' , the feature of the reflection layer image is denoted as F R' , and the feature of the background layer image is denoted as F T' , and the anti-reflection module is provided with a plurality of encoders and decoders, A plurality of encoders are used to extract multi-scale features of the normal light reflection image I' obtained in the illumination enhancement module, wherein the encoder performs convolution operation with a kernel size of 3x3 on the image and uses the activation function ReLU, extracts the features of the selected layers of the VGG19 model and performs normalization, and the down-sampling operation between each layer of the encoder is composed of a maximum pooling layer with a 2x2 pooling kernel; The features F of the image of the bottommost reflective layer R' The features F of the normal light reflection image of the corresponding layer I' Splicing, the features F of the image of the remaining three reflective layers R' The features F of the normal light reflection image of the corresponding layer I' The features F of the image of the background layer T' Splicing, the features F of the image of the remaining three reflective layers calculating the features F of the normal light reflection image I' the feature difference F between the features F of the reflection layer image R' I' R' multiplying the feature difference with the mask to be up-sampled, thereby obtaining the features F of the next background layer image T' The up-sampling layer is composed of 1 group or 3 groups of convolution blocks, and each convolution block includes a convolution layer with a kernel size of 3x3, a normalization layer and an activation function ReLU.​​ The features F of the last layer of the reflectance layer image are outputted by a convolution operation with a kernel size of 1x1 R' The output normal light reflectance layer estimation image R' is outputted by restoring to 3 channels, and the background layer features F T' The output normal light background layer estimation image T' is also outputted by a 1x1 convolution operation.

6. The low-light condition de-reflecting method of claim 1, wherein, The low-light condition de-reflection model is trained by a back propagation algorithm, and the deep learning method is provided with an adversarial loss, a perception loss, a mask loss, a reconstruction loss and an exclusion loss, wherein The loss function of the global discriminator is: The loss function of the generator is: where x r and x f are the distributions of the real normal illumination image and the weak light reflection image after illumination enhancement, respectively, D Ra is the standard function of the relativistic discriminator, E represents the mathematical expectation, denotes the probability distribution; The loss function of the local discriminator is: True block: False block: Adversarial loss for low-light condition de-reflecting model is represented as: The perceptual loss is determined according to the following formula where φ denotes the VGG-19 model, / denotes the index of the selected VGG-19 feature map layers convl_2, conv2_2, conv3_2, conv4_2 and conv5_2, and the weights {κ l} are used to balance the different layers; By analogy, the perceptual loss in the low-light condition de-reflecting model can be obtained The perception loss of the low-light condition de-reflection model is represented as: Mask loss for low-light condition de-reflecting model is represented as: where i denotes the i-th layer, ||•||1 denotes the l1 norm, M[condition] denotes the portion of M that satisfies the condition in the brackets, is a threshold value that demarcates the heavy reflection region, and ξ denotes a threshold value of a region in which reflection is little; Reconstruction loss for low-light condition de-reflecting model is represented as: Exclusion loss for low-light condition de-reflecting model is represented as: wherein λ T and λ R denotes a normalization factor, denotes a gradient; || · || F is the F-norm; T' ↓n and R' ↓n represent n-fold down-sampled versions of T' and R', respectively, where T' ↓0 and R' ↓0 is the original input; The total loss of the low-light condition de-reflection method is represented as, 7. A low-light condition anti-reflective system characterized by, Comprising: A first processing unit for generating a first reflection image data set containing a first reflection image, the first reflection image being generated from a reflection layer image and a background layer image under a first illumination condition, or a reflection layer image R and a background layer image T under a first illumination condition and a reflection layer image and a background layer image under a second illumination condition, the first illumination condition being a low-light condition, and the second illumination condition being a normal light condition; a corresponding first illumination attention map is generated using the first reflection image data set; The second processing unit is configured to generate a low-light condition de-reflection model, the low-light condition de-reflection model comprising an illumination enhancement module and a de-reflection module, the illumination enhancement module comprising five convolution blocks each consisting of a convolution layer with a kernel size of 3x3, an activation function LeakyReLU, and a normalization layer, and connected by a down-sampling layer consisting of an average pooling layer with a 2x2 pooling kernel, and further comprising five de-convolution blocks each consisting of a de-convolution layer with a kernel size of 3x3, an activation function LeakyReLU, and a normalization layer, and connected by an up-sampling layer with a bilinear interpolation factor of 2, and the convolution blocks and the de-convolution blocks being connected by staggered layer skip connections, and the de-reflection module comprising a plurality of encoders and decoders, each encoder performing a convolution operation with a kernel size of 3x3 and using an activation function ReLU, extracting features of selected layers of a VGG19 model and normalizing, and each encoder being connected by a maximum pooling layer with a 2x2 pooling kernel for down-sampling; the first reflection image dataset and the first illumination attention map being input to the low-light condition de-reflection model, and after being processed by the illumination enhancement module and the de-reflection module, a reflection layer estimation image and a normal-light background layer estimation image being obtained, and the low-light condition de-reflection model being trained by a back propagation algorithm to obtain a trained low-light condition de-reflection model. The processing unit is configured to obtain a second reflection image dataset, generate a corresponding second illumination attention map using the second reflection image dataset, input the second reflection image dataset and the second illumination attention map to the low-light condition de-reflection model, and obtain a normal-light reflection layer image and a normal-light background layer image.

8. An electronic device, comprising: The electronic device comprises a processor and a memory, and the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the low-light condition de-reflection method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the low-light condition de-reflection method according to any one of claims 1 to 6.

Images