Removing reflections from images

Abstract

The invention relates to a method for removing reflections from an image. The method detects one or more reflection regions in the image, wherein each reflection region comprises a reflection. Furthermore, the method extracts the one or more reflection regions from the image and removes the reflection from each extracted reflection region.

Description

Technical Field

[0001] The present invention relates to a method and apparatus for removing unwanted reflections from an image. Background Technology

[0002] In some cases, photographs are taken through surfaces such as glass. In these situations, the visual quality of the resulting image can be significantly degraded due to unwanted reflections. In fact, there are many instances where taking a sharp image without any reflections is challenging. For example, photographs taken on airplanes or trains are often marred by unwanted reflections. Another common example involves photographs of people wearing glasses, where, in selfies or similar situations, lights or phone screens cause reflections in the glasses within the resulting image.

[0003] Several traditional methods have been proposed for removing reflections from images. For example, one approach is to decompose the image into two layers. However, this is a highly ill-posed problem because the number of unknown parameters is twice the number of given values. Therefore, without additional assumptions, there are almost an infinite number of variations for extracting the background and reflection layers. Consequently, in earlier writings (e.g., see "Single Image Reflection Suppression" by N. Armanitopoulos et al., July 2017, IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1752–1760), the task was considered an optimization problem with constraints derived from different image priors, suggesting a single solution. Besides producing poor results and only working in a limited number of cases, most optimization techniques are too slow for use in real-time systems, especially smartphones.

[0004] Due to the success of deep learning methods in many computer vision problems, this approach has also been used for reflection removal. For example, in their 2017 paper, "Ageneric deep architecture for single image reflection removal and imagesmoothing," Fan et al. proposed training an end-to-end convolutional neural network (CNN) to estimate the background scene using a two-stage pipeline. First, an edge map is predicted, resulting in a blended image. Then, a background layer is generated, providing the edges and the input image. Due to the lack of real-world paired data, synthetic data was used to train the CNN. In later works, modern deep learning techniques are used to improve visual quality: perceptual loss and adversarial loss.

[0005] Existing methods (e.g., see the aforementioned work by Fan et al. or Wei et al.'s 2019 paper, "Single image reflection removal exploiting misaligned training data and network enhancements," published in the Proceedings of the IEEE International Conference on Computer Vision and Patterns, pp. 8178-8187) apply a CNN to the entire image, expecting the CNN to detect reflective regions and remove them simultaneously. Figure 1 As shown. This has several drawbacks. In fact, since the CNN is applied to the entire image, the processing time is enormous even if the actual image region that includes reflections is small. Furthermore, because the network is required to detect and remove reflections, a large portion of the CNN's capacity is spent on the detection task, resulting in poor reflection removal quality. In addition, these methods cannot ideally detect reflections; for example, they produce unwanted changes in areas where there are no reflections.

[0006] Therefore, there is a need for an improved method and apparatus for removing reflections from images. Summary of the Invention

[0007] In view of the above-mentioned problems and disadvantages, the object of the present invention is to improve the removal of unwanted reflections from images. Therefore, the object of the present invention is to provide an improved apparatus and method for removing unwanted reflections.

[0008] The object of the invention is achieved by means of the embodiments provided in the appended independent claims. Advantageous implementations of the embodiments are further defined in the dependent claims.

[0009] According to a first aspect, the present invention relates to a method for removing reflections from an image, the method comprising: detecting one or more reflective regions in the image, wherein each reflective region includes a reflection; extracting the one or more reflective regions from the image; and removing the reflection from each extracted reflective region.

[0010] The first aspect of the method has the advantage of efficiently removing unwanted reflections from images (e.g., photographs). The first aspect of the method works under many different reflection conditions. Furthermore, the method is fast enough to be used in real-time systems, particularly in smartphones. The first aspect of the method also avoids generating unwanted changes in areas without reflection. These advantages are specifically achieved through a two-stage approach: detecting reflections in the first stage and removing reflections from the reflective regions only in the second stage.

[0011] In an implementation of the first aspect, after removing the reflection from each extracted reflection region, the extracted reflection regions without the reflection are reinserted into the image to replace the reflection regions with the reflection, respectively.

[0012] This provides the advantage of obtaining a processed image without reflections.

[0013] In an implementation of the first aspect, a first trained model is used to detect the one or more reflective regions; and / or a second trained model is used to remove the reflection from each extracted reflective region.

[0014] This offers the advantage of efficiently and accurately performing the detection and removal of reflections in images using trained models. Specifically, each trained model can be specifically trained. That is, a first trained model can be specifically trained to detect reflections in an image, and a second trained model can be specifically trained to remove reflections from reflective regions (rather than the entire image). Therefore, both the training and inference phases can be performed faster, and the quality of the results is improved.

[0015] In an implementation of the first aspect, the first trained model includes a first CNN.

[0016] This offers the advantage of being able to use known, trained models. Furthermore, it offers the advantage of enabling improvements in the execution time of reflection removal (e.g., compared to...). Figure 1 Compared to traditional methods, this approach applies the CNN only to the region containing the reflection, rather than to the entire image. Furthermore, by using only the CNN to process the cropping to remove unwanted artifacts, alteration of non-reflective regions is prevented. Additionally, by targeting the CNN to specific regions without requiring it to detect the reflective regions themselves, this method improves the quality of the resulting dereflection.

[0017] In an implementation of the first aspect, the first CNN includes a semantic segmentation CNN for performing semantic segmentation of the image.

[0018] In the implementation of the first aspect, semantic masks are used to detect the one or more reflection regions.

[0019] The above implementation provides a simple yet efficient method for detecting reflective regions in an image.

[0020] In an implementation of the first aspect, the second trained model includes a generative adversarial network (GAN).

[0021] This offers the advantage of being able to remove reflections from images using known, trained models. The advantages of GANs can be leveraged.

[0022] In an implementation of the first aspect, the GAN includes a conditional GAN.

[0023] This offers the advantage that known, trained models can be used to remove reflections from an image.

[0024] In the implementation of the first aspect, the second trained model includes a second CNN.

[0025] In an implementation of the first aspect, the image is a photograph of a person wearing glasses, wherein the step of detecting the one or more reflective regions includes: detecting the person's face in the image, detecting the glasses in the image based on the detected face, and detecting the one or more reflective regions located within the glasses detected in the image.

[0026] In an implementation of the first aspect, the segmentation CNN is used to detect the glasses in the image.

[0027] In an implementation of the first aspect, the glasses detected in the image are segmented by the segmentation CNN: the obtained glasses segments are extracted from the image, the one or more reflective regions located within the glasses are detected by removing glasses segments without reflection from the extracted glasses segments, and the reflection is removed from each extracted glasses segment with reflection.

[0028] The above implementation provides a particularly efficient method for detecting and removing reflections in images that include people wearing glasses.

[0029] According to a second aspect, the present invention relates to an apparatus for removing reflections from an image, the apparatus being configured to: detect one or more reflective regions in the image, wherein each reflective region includes a reflection; extract the one or more reflective regions from the image; and remove the reflection from each extracted reflective region.

[0030] Typically, the device of the second aspect is therefore used to execute the method of the first aspect. The device of the second aspect can also have an implementation according to the implementation of the first aspect. That is, the device of the second aspect can be used to execute the method according to any implementation of the first aspect. Therefore, the device of the second aspect achieves all the advantages and effects of the method of the first aspect.

[0031] According to a third aspect, the present invention relates to a computer program comprising program code for performing a method according to the first aspect or any implementation thereof.

[0032] It should be noted that all devices, elements, units, and apparatuses described in this application can be implemented in software or hardware elements or any combination thereof. All steps performed by the various entities described in this application, and the functions to be performed by the various entities described, are intended to indicate that the respective entities are suitable for or used to perform the respective steps and functions. Even in the following description of specific embodiments, if a specific function or step to be performed by an external entity is not reflected in the detailed description of the specific element of the entity performing that specific step or function, it should be understood by those skilled in the art that these methods and functions can be implemented in the corresponding software or hardware elements or any combination thereof. Attached Figure Description

[0033] The above aspects and implementations of the invention will be set forth in the following description of specific embodiments in conjunction with the accompanying drawings, in which:

[0034] Figure 1 A schematic diagram of a conventional pipeline for removing reflections from an image is shown;

[0035] Figure 2 A schematic diagram of a method for removing reflections from an image, provided in an embodiment, is shown.

[0036] Figure 3 A schematic diagram of a pipeline for removing reflections from an image, provided in an embodiment, is shown.

[0037] Figure 4 A schematic diagram of a pipeline for removing reflections from an image, provided in an embodiment, is shown.

[0038] Figure 5 A schematic diagram of a pipeline for removing reflections from an image, provided in an embodiment, is shown.

[0039] Figure 6 An embodiment of the apparatus for removing reflections from an image is shown. Detailed Implementation

[0040] Figure 2 A schematic diagram of a method 200 for removing reflections from an image, provided by an embodiment, is shown. Method 200 can be provided by device 600 (e.g., see [link to documentation]). Figure 6 The method can be executed, for example, by a processor in a consumer electronic device such as a smartphone, tablet, or digital camera. Method 200 can also be executed by a device 600 capable of capturing images 500.

[0041] Method 200 includes the step 201 of detecting one or more reflective regions in image 500. Each reflective region includes at least one reflection. That is, method 200 can specifically detect the location of these reflections in image 500 and can accordingly determine the image region that includes or consists of these reflections.

[0042] Furthermore, method 200 includes a step 202 of extracting the one or more reflective regions from image 500. For example, the one or more detected reflective regions can be cropped from image 500 or copied from image 500. Further processing then need not be applied to the entire image 500, but can be applied only to the extracted reflective regions. Therefore, method 200 (especially reflection removal) can work faster.

[0043] Method 200 also includes a step 203 of removing reflections from each extracted reflection region. For this purpose, conventional reflection removal algorithms can be used. Alternatively, a trained model trained for this purpose can be used. This trained model can be specifically trained to operate on reflection regions (i.e., specific image segments) rather than on the entire image.

[0044] Furthermore, method 200 may also include a re-insertion step. Specifically, after removing reflection 202 from each extracted reflection region, method 200 may include the step of re-inserting the extracted reflection regions without reflection into image 500 to replace the reflection regions with reflection, respectively. Thus, a processed image without reflection, or at least a processed image with a significantly reduced amount of reflection, can be obtained.

[0045] Figure 3 A schematic diagram of a pipeline for removing reflections from an image 500, provided by an embodiment, is shown. This pipeline can be used to perform method 200. The pipeline can be part of a device 600 performing method 200. The pipeline may include a reflection detection module 302 and a reflection removal module 304.

[0046] exist Figure 2 In the exemplary embodiment shown, reflections are removed from the input image 500 (see also...) Figure 5The image 500 can be composed of two modules, 302 and 304. In the first reflection detection module 302, reflective regions (i.e., reflective areas) of the input image 500 are detected. That is, the reflection detection module 302 can implement step 201 of method 200. Then, the detected reflective regions are extracted or cropped from the image 500 (i.e., method step 202 is executed, for example, also by the reflection detection module 302), and the extracted reflective image segmentation or cropping 303 is provided to the reflection removal module 304. That is, method step 203 can be executed by the reflection removal module 304. The second reflection removal module 304 may include a CNN, which can be applied to the image segmentation or cropping 303 to remove reflections therein. The result (i.e., the reflective regions with reflections removed) can then be pasted back into the original image 500 to obtain a processed image 500' with reflections removed.

[0047] In one embodiment, a first trained model can be used to detect one or more reflective regions 201, i.e., it can be used in the reflectance detection module 302. Furthermore, a second trained model can be used to remove reflections 202 from each extracted reflective region, i.e., it can be used in the reflectance removal module 304. The first trained model may include a first CNN. As described above, the second trained model may also include a (second) CNN. Furthermore, the first CNN may include a semantic segmentation CNN for performing semantic segmentation of image 500.

[0048] Specifically, detection step 201 can be performed by a semantic segmentation CNN that detects reflection regions. For example, a semantic mask can be used to find connected components (identified as reflections). Their minimum bounding rectangles can then be determined to define the reflection regions. Multiple such rectangles can define multiple reflection regions. As mentioned above, reflection regions can also be used to crop (as an example) image 500. Then, as described above, reflection removal step 202 can be applied to each crop (i.e., each reflection region), and the result (i.e., the reflection regions with reflections removed) can be pasted back into image 500.

[0049] Embodiments of the present invention offer several advantages: First, improvements are achieved in the execution time of removing reflections from image 500 because reflection removal is applied only to the reflective regions. Second, in the reflection removal step (e.g., implemented by a CNN), unwanted artifacts are removed by processing only the cropped or extracted (reflective regions), preventing alteration of non-reflective areas. Third, the dereflection quality is improved by performing reflection removal only in the reflective regions, without requiring reflection removal step 202 to detect reflections throughout image 500.

[0050] Furthermore, to train the first and / or second trainable (or trained) model (e.g., CNN or GAN), method 200 can also obtain initial images without reflections from a pool of images without reflections, and can use a reflection generator to generate synthetic reflections on the initial images to obtain training images. Then, method 200 can process the training images using the generator of the first and / or second trainable model to obtain synthetic images. Finally, method 200 can compute the loss function of the first and / or second trainable model based on the initial images and the synthetic images.

[0051] Figure 4 A schematic diagram of a pipeline for removing reflections from image 500, provided by an embodiment, is shown. This pipeline can be used to perform method 200. The pipeline is established on... Figure 3 The assembly line shown. Figure 4 The pipeline can also be part of the device 600 that performs method 200. Figure 4 The pipeline can also include a reflection detection module 302 and a reflection removal module 304.

[0052] exist Figure 4 In the embodiments, similar to Figure 3 In one embodiment, a reflected input image 500 is provided as input to module 302. Module 302 is used to detect 201 regions including reflections. In this embodiment, reflection detection 201 is performed using a CNN. Furthermore, the output of module 302 includes cropped portions (extracted reflection regions) with reflections, which are provided as input to module 304. Module 304 is used to perform 202 removal of reflections from the extracted reflection regions. This can be accomplished using a CNN or a GAN. In one embodiment, the GAN can be a conditional GAN.

[0053] Figure 5 A schematic diagram of another pipeline for removing reflections from image 500, provided by an embodiment, is shown, which can be used to perform method 200. The pipeline may be part of a device for performing method 200.

[0054] for Figure 5 The pipeline shown, as an example, uses image 500 as a photograph of a person wearing glasses. The pipeline includes a first stage 501 for detecting the face of the person in image 500. Furthermore, the pipeline may include a second stage 502 for detecting glasses in image 500 based on the detected face. For example, a segmentation CNN can be used to detect glasses in image 500. Then, a third stage 503 of the pipeline can detect one or more reflective regions located within the glasses detected in image 500.

[0055] also, Figure 5The pipeline can perform the following steps. Second stage 502 can, for example, further segment the glasses detected in image 500 using a segmentation CNN. Third stage 503 can further extract or crop the glasses segments obtained from image 500, and can detect the one or more reflective regions 201 located within the glasses by removing glasses segments without reflection from the extracted glasses segments. Furthermore, fourth stage 504 can remove reflections from each extracted glasses segment with reflections, and can paste the regions with removed reflections back 504 to obtain the processed image 500'.

[0056] Therefore, in this embodiment, reflections are removed only from the glasses area, not from any other part of image 500. That is, the first stage 501 can, for example, use face detection and face parsing tools to detect the glasses. Then, a CNN can be applied to the glasses cropping in the fourth stage 504, and the result can be pasted back.

[0057] Figure 6 An embodiment of the device 600 is illustrated. The device 600 can be used to perform method 200, including steps 201, 202, and 203, to remove reflections from an image 500. The output can be a processed image 500' with less or no reflection. The device 600 may include a camera to acquire the image 500.

[0058] Furthermore, device 600 may include a processor or processing circuitry (not shown) for performing, conducting, or initiating various steps of the method 200 described herein. The processing circuitry may include hardware and / or may be software-controlled. The hardware may include analog or digital circuitry, or both. The digital circuitry may include components such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), or multi-purpose processors.

[0059] The device 600 may also include memory circuitry that stores one or more instructions that can be executed by a processor or processing circuitry (specifically, under software control). For example, the memory circuitry may include a non-transitory storage medium storing executable software code that, when executed by a processor or processing circuitry, causes method 200 to be executed.

[0060] In one embodiment, the processing circuitry includes one or more processors and non-volatile memory connected to the one or more processors. The non-volatile memory may carry executable program code that, when executed by the one or more processors, causes the device 600 to perform, conduct, or initiate the method 200 described herein.

[0061] Specifically, the processor or processing circuitry of device 600 is used to detect 201 one or more reflective regions in image 500, wherein each reflective region includes a reflection. Furthermore, the device is used to extract 202 the one or more reflective regions from image 500 and remove 203 the reflection from each extracted reflective region.

[0062] In summary, compared with traditional first-order methods (e.g., see...), Figure 1 In contrast, embodiments of the present invention perform a two-stage reflection removal process (see...). Figures 3 to 6 First, detect 201 regions with reflections; second, apply only reflection removal 202 to the found reflection regions that have already been extracted 203 from image 500.

[0063] This invention has been described in conjunction with various embodiments as examples and implementations. However, based on a study of the drawings, the invention, and the independent claims, those skilled in the art will be able to understand and implement other variations when practicing the claimed subject matter. In the claims and the description, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. A single element or other unit may fulfill the function of several entities or items recited in the claims. The mere fact that certain measures are cited in dissimilar dependent claims does not indicate that a combination of these measures cannot be used advantageously.

Claims

1. A method (200) for removing reflections from an image (500), characterized in that, The method (200) includes: Detect (201) one or more reflective regions in the image (500), wherein each reflective region includes a reflection; In the reflection detection module (302), a first trained model is used to extract (202) the one or more reflection regions from the image (500); the image (500) serves as the input to the reflection detection module (302), and the output of the reflection detection module (302) contains only image segments or cropping of the one or more reflection regions; The reflection is removed (203) from each extracted reflection region using a second trained model in the reflection removal module (304); the extracted reflection region is used as the input of the reflection removal module (304), and the output of the reflection removal module (304) is the reflection region from which the reflection is removed. The reflective areas that have had their reflections removed are reinserted into the image (500) to replace the reflective areas that have reflections.

2. The method (200) according to claim 1, characterized in that, The first trained model and / or the second trained model obtains a training image by acquiring an initial image without reflection from a pool of images without reflection and generating a synthetic reflection on the initial image using a reflection generator; The training images are processed using the generator of the first trained model and / or the second trained model to obtain synthetic images; The loss function of the first trained model and / or the second trained model is calculated based on the initial image and the synthesized image.

3. The method (200) according to claim 2, characterized in that, The first trained model includes a first convolutional neural network (CNN).

4. The method (200) according to claim 3, characterized in that, The first CNN includes a semantic segmentation CNN for performing semantic segmentation of the image, wherein the first CNN is used to use a semantic mask to find the connected components identified as reflections and to determine the minimum bounding rectangle of the reflections to determine the reflection region.

5. The method (200) according to any one of claims 2 to 4, characterized in that, The second trained model includes a Generative Adversarial Network (GAN).

6. The method (200) according to claim 5, characterized in that, The GAN includes conditional GAN.

7. The method (200) according to any one of claims 2 to 4, characterized in that, The second trained model includes a second CNN.

8. The method (200) according to claim 4, characterized in that, The image (500) is a photograph of a person wearing glasses, and the detection (201) of one or more reflective regions in the image (500) includes: detecting the face of the person in the image (500), using a segmentation CNN in the image (500) to detect glasses based on the detected face, and detecting (201) the one or more reflective regions located within the glasses detected in the image (500).

9. The method (200) according to claim 8, characterized in that, Also includes: The glasses detected in the image are segmented using the segmentation CNN (502); The eyeglass segments obtained from the image (500) are extracted (503); Detection (504) is performed on one or more reflective areas located within the glasses. Remove the reflection (504) from each extracted eyeglass segment with reflection.

10. An apparatus (600) for removing reflections from an image (500), characterized in that, The device (600) is used for: Detect (201) one or more reflective regions in the image (500), wherein each reflective region includes a reflection; In the reflection detection module (302), a first trained model is used to extract (202) the one or more reflection regions from the image (500); the image (500) serves as the input to the reflection detection module (302), and the output of the reflection detection module (302) contains only image segments or cropping of the one or more reflection regions; The reflection is removed (203) from each extracted reflection region using a second trained model in the reflection removal module (304); the extracted reflection region is used as the input of the reflection removal module (304), and the output of the reflection removal module (304) is the reflection region from which the reflection is removed. The reflective areas that have had their reflections removed are reinserted into the image (500) to replace the reflective areas that have reflections.

11. A computer program product containing instructions, characterized in that, When the instructions are executed on a computer, the computer causes the computer to perform the method (200) as described in any one of claims 1 to 9.

Images