REFLECTION REMOVED FROM AN IMAGE

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image processing in particular to a method and device for removing undesired reflections from an image.BACKGROUND

[0002] In some situations, photographs are taken through a glass surface or the like. In such situations, the visual quality of the obtained images can decrease dramatically due to the appearance of undesired reflections. There are, in fact, many situations, in which taking a clear image without any reflections is challenging. For example, photographs from an airplane or a train are often corrupted by undesired reflections. Another common example concerns photographs taken of people wearing eyeglasses, wherein reflections in the eyeglasses in the obtained image are caused by lamps or phone screens when taking, for example, a selfie photo.

[0003] Several conventional methods are proposed to remove reflections from an image. For example, by decomposing the image into two layers. However, this is a highly ill-posed problem, since the number of unknown parameters is twice as many as giving values. Without additional assumptions there is thus an almost infinite number of variants to extract the background and reflection layers. Therefore, in early works (see, e.g., "Single image reflection suppression'', IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1752-1760, July 2017, by N. Arvanitopoulos et al.), this task was considered as an optimization problem with constraints arising from the different image priors proposed to take a single solution. Besides the fact that these methods produce poor results and can work only on a limited amount of cases, most of the optimization techniques are too slow for using them in real-time systems, in particular, in smartphones.

[0004] Due to the success of deep learning methods for many computer vision problems, such methods are also used for reflection removal. For example, Fan et al. in "A generic deep architecture for single image reflection removal and image smoothing", 2017, propose to train an end-to-end convolutional neural network (CNN) to estimate a background scene using a two-staged pipeline. Firstly, an edge map is predicted giving a mixture image. Afterwards, a background layer is produced giving edges and the input picture. Because of the lack of real paired data, synthetic data is used for training the CNN. At later works, modern deep learning techniques were used to improve the visual quality: perceptual loss, and adversarial loss.

[0005] Current methods (see, e.g., the above mentioned work of Fan et al. or "Single image reflection removal exploiting misaligned training data and network enhancements." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8178-8187, 2019, by Wei et al.) apply the CNN to the full image, and expect the CNN to detect areas with reflections and remove them simultaneously, as is illustrated in Fig. 1. This has several disadvantages. In fact, since the CNN is applied to the whole image, the processing time is huge, even if the image area that actually includes reflections is small. Further, since the network is required to be able to detect and remove reflections, a large amount of the CNN capacity is spent on the detection task, which leads to a worse reflection removal quality. Moreover, these methods cannot detect reflections ideally, for instance, they produce undesirable changes in areas without reflections.

[0006] Thus, there is a need for an improved method and device for removing reflections from images.

[0007] Document CN 111 325 698 A provides an image processing method comprising the steps of carrying out region segmentation of a target image, and obtaining a target region in the target image; and inputting the target image and the target area of the target image into an image processing model, so that the image processing model performs image processing on the target area of the target image to obtain a processed image corresponding to the target image.SUMMARY

[0008] In view of the above-mentioned problems and disadvantages, the present disclosure aims to improve the removal of undesired reflections from an image. The present disclosure has thereby the object to provide for an improved device and method for removing the undesired reflections.

[0009] The object of the present invention is achieved by the embodiments provided in the enclosed independent claims. Advantageous implementations of the embodiments are further defined in the dependent claims.

[0010] According to a first aspect, the disclosure relates to a method for removing reflections from an image, the method comprising: detecting one or more reflection areas in the image, wherein each reflection area includes a reflection, extracting the one or more reflection areas from the image, and removing the reflection from each of the extracted reflection areas.

[0011] The method of the first aspect has the advantage that undesirable reflections can efficiently be removed from the image, e.g., a photograph. The method of the first aspect works in many different cases of reflections. Further, the method is fast enough for being used in a real-time system, in particular, in a smartphone. The method of the first aspect also avoids undesirable changes in areas without reflections. These advantages are particularly achieved because of the two stage approach, i.e., the detection of reflections in the first stage, and the removal of reflections from the reflection areas only in the second stage.

[0012] In an implementation form of the first aspect, after removing the reflection from each of the extracted reflection areas, reinserting the extracted reflection areas without the reflection into the image to replace, respectively, the reflection areas with the reflection.

[0013] This provides the advantage that a processed image without reflections can be obtained.

[0014] In an implementation form of the first aspect, a first trained model is used to detect the one or more reflection areas; and a second trained model is used to remove the reflection from each of the extracted reflection areas.

[0015] This provides the advantage that the detection and the removal of the reflections in the image can be efficiently and accurately performed by means of trained models. In particular, each trained model can be specifically trained. That is, the first trained model can be specifically trained to detect reflections in an image, and the second trained model can be specifically trained to remove reflections from reflection areas (not the entire image). Thus, both the training phase and the inference phase can be performed faster, and the quality of the results is improved. In an implementation form of the first aspect, the first trained model comprises a first CNN.

[0016] This provides the advantage that well-known trained models may be used. Moreover, this provides the advantage that an improvement of the execution time for the reflection removal may achieved (compared to the conventional method shown in Fig. 1, for example), due to the fact that the CNN is applied only to the areas comprising reflections and not to the entire image. Moreover, the removal of undesirable artefacts by processing only crops with the CNN may prevent areas without reflections from being changed. Further, it improves the resulting de-reflection quality by targeting the CNN on specific areas, and not requiring it to detect reflection areas as such.

[0017] In an implementation form of the first aspect, the first CNN comprises a semantic segmentation CNN configured to perform a semantic segmentation of the image.

[0018] In an implementation form of the first aspect, the one or more reflection areas are detected using a semantic mask.

[0019] The above implementation forms provide a simple but efficient way to detect the reflection areas in the image.

[0020] In an implementation form of the first aspect, the second trained model comprises a generative adversarial network (GAN).

[0021] This provides the advantage that a well-known trained model may be used to remove the reflections on the image. The advantages of the GAN may be employed.

[0022] In an implementation form of the first aspect, the GAN comprises a conditional GAN.

[0023] This provides the advantage that a well-known trained model may be used to remove the reflections on the image.

[0024] In an implementation form of the first aspect, the second trained model comprises a second CNN.

[0025] In an implementation form of the first aspect, the image is a photograph of a person wearing eyeglasses, and wherein the step of detecting the one or more reflection areas comprises: detecting a face of the person in the image, detecting the eyeglasses in the image based on the detected face, and detecting the one or more reflection areas located within the eyeglasses detected in the image.

[0026] In an implementation form of the first aspect, the eyeglasses are detected in the image using the segmentation CNN.

[0027] In an implementation form of the first aspect, segmenting the eyeglasses detected in the image by the segmentation CNN: extracting the obtained eyeglass segments from the image, detecting the one or more reflection areas located within the eyeglasses by removing eyeglasses segments without reflection from the extracted eyeglasses segments, and removing the reflection from each of the extracted eyeglasses segments with reflection.

[0028] The above implementation forms provide a particularly efficient way to detect and remove reflections in the case of images including persons wearing eyeglasses.

[0029] According to a second aspect, the disclosure relates to a device for removing reflections from an image, the device being configured to: detect one or more reflection areas in the image, wherein each reflection area includes a reflection, extract the one or more reflection areas from the image, and remove the reflection from each of the extracted reflection areas.

[0030] Generally, the device of the second aspect is thus configured to perform the method of the first aspect. The device of the second aspect may further have implementation forms according to the implementation forms of the first aspect. That is, the device of the second aspect may be configured to perform the method according to any implementation form of the first aspect. Accordingly, the device of the second aspect achieves all advantages and effects of the method of the first aspect.

[0031] According to a third aspect, the disclosure relates to a computer program comprising a program code for performing the method according to the first aspect or any one of the implementation forms thereof.

[0032] It has to be noted that all devices, elements, units and means described in the present application could be implemented in the software or hardware elements or any kind of combination thereof. All steps which are performed by the various entities described in the present application as well as the functionalities described to be performed by the various entities are intended to mean that the respective entity is adapted to or configured to perform the respective steps and functionalities. Even if, in the following description of specific embodiments, a specific functionality or step to be performed by external entities is not reflected in the description of a specific detailed element of that entity which performs that specific step or functionality, it should be clear for a skilled person that these methods and functionalities can be implemented in respective software or hardware elements, or any kind of combination thereof.BRIEF DESCRIPTION OF DRAWINGS

[0033] The above described aspects and implementation forms of this disclosure will be explained in the following description of specific embodiments in relation to the enclosed drawings, in which: FIG. 1shows a schematic representation of a conventional pipeline for reflection removal from an image; FIG. 2shows a schematic representation of a method for reflection removal from an image according to an embodiment; FIG. 3shows a schematic representation of a pipeline for reflection removal from an image according to an embodiment; FIG. 4shows a schematic representation of a pipeline for reflection removal from an image according to an embodiment; FIG. 5shows a schematic representation of a pipeline for reflection removal from an image according to an embodiment; and FIG. 6shows a device for reflection removal from an image according to an embodiment. DETAILED DESCRIPTION OF EMBODIMENTS

[0034] FIG. 2 shows a schematic representation of a method 200 for reflection removal from an image, according to an embodiment. The method 200 may be performed by a device 600 (see e.g. FIG. 6), for instance by a processor of an electronic consumer device like a smartphone, tablet or digital camera. The method 200 may be performed by a device 600 that is also capable of taking the image 500.

[0035] The method 200 comprises a step 201 of detecting one or more reflection areas in the image 500. Each reflection area includes at least one reflection. That is, the method 200 detects specifically where the reflections are located in the image 500, and determines accordingly image areas comprising or consisting of these reflections.

[0036] Further, the method 200 comprises a step 202 of extracting the one or more reflection areas from the image 500. The one or more detected reflection areas are cropped from the image 500, or may be copied from the image 500. Further processing does then not have to be applied to the whole image 500, but is applied only to the extracted reflection areas. Thus, the method 200, particularly the reflection removal, may work faster.

[0037] The method 200 further comprises a step 203 of removing the reflection from each of the extracted reflection areas. To this end, a conventional reflection removal algorithm may be used. Also, a trained model, trained for this purpose, may be used. This trained model may be specifically trained to operate on reflection areas, i.e., specific image segments, and not on entire images.

[0038] Moreover, the method 200 also comprises a reinsertion step. In particular, after the removal 202 of the reflections from each of the extracted reflection areas, the method 200 comprises a step of reinserting the extracted reflection areas without the reflection into the image 500, in order to replace, respectively, the reflection areas with the reflection. Thus, a processed image without reflections, or at least with a significantly reduced amount of reflections, may be obtained.

[0039] FIG. 3 shows a schematic representation of a pipeline for reflection removal from an image 500, which may be used for performing the method 200 according to an embodiment. The pipeline may be part of a device 600 that performs the method 200. The pipeline may comprise a reflection detection module 302 and a reflection removal module 304.

[0040] In the exemplary embodiment shown in Fig. 2, the reflection removal from an input image 500 (see also Fig. 5) may consist of the two modules 302 and 304. In the first reflection detection module 302, the areas of the input image 500 with reflections (i.e., the reflection areas) are detected. That is, the reflection detection module 302 can implement the step 201 of the method 200. Afterwards, the detected areas with reflection are extracted or cropped from the image 500 (i.e. the method step 202 is performed, e.g., also by the reflection detection module 302), and the extracted image segments or crops 303 with reflections are provided to the reflection removal module 304. That is, the method step 203 can be performed by the reflection removal module 304. The second reflection removal module 304 may comprise a CNN, which can be applied on the image segments or crops 303, in order to remove the reflections therein. The results, i.e. the reflection areas with reflections removed, are then pasted back into the original image 500, in order to obtain a processed image 500' with removed reflections.

[0041] In an embodiment, a first trained model is used to detect 201 the one or more reflection areas, i.e., it is used in the reflection detection module 302. Further, a second trained model is used to remove 202 the reflections from each of the extracted reflection areas, i.e., it is used in the reflection removal module 304. The first trained model can comprise a first CNN. As described above, also the second trained module may comprise a (second) CNN. Moreover, the first CNN can comprise a semantic segmentation CNN, which is configured to perform a semantic segmentation of the image 500.

[0042] In particular, the detection step 201 can be performed by a semantic segmentation CNN, which detects the reflection areas. For example, a semantic mask may be used to find connected components (identified as reflections). Then, their minimum circumscribing rectangle can be determined to determine the reflection areas. Multiple such rectangles may define multiple reflection areas. The reflection areas may further be used to crop (as example) the image 500 as described above. Afterwards, the reflection removal step 202 is applied to each crop (i.e., to each reflection area), and the result (i.e., the reflection area with reflections removed) is pasted back to the image 500 as described above.

[0043] The embodiments of this disclosure provide several advantages: Firstly, an improvement of the execution time for the reflection removal from the image 500 is achieved, due to the fact that the reflection removal is applied only to reflection areas. Secondly, the removal of undesirable artefacts by processing only crops or extractions (reflection areas) in the reflection removal step (e.g., implemented by a CNN), prevents areas without reflections from being changed. Thirdly, a de-reflection quality may be improved by performing the reflection removal only in the reflection areas, and not requiring the reflection removal step 202 to detect reflections in the entire image 500.

[0044] Moreover, in order to train the first and / or second trainable (or trained) model (e.g., CNN or GAN), the method 200 can further take an initial image without a reflection from a pool of images without reflections, and can generate a synthetic reflection on the initial image with a reflection generator to obtain a training image. Then, the method 200 may process the training image with a generator of the first and / or second trainable model to obtain a synthetic image. Finally, the method 200 may calculate a loss function of the first and / or second trainable model on the basis of the initial image and the synthetic image.

[0045] Fig. 4 shows schematic representation of a pipeline for reflection removal from an image 500 that may be used for performing the method 200 according to an embodiment. The pipeline builds on the pipeline shown in Fig. 3. The pipeline of Fig. 4 may again be part of a device 600 that performs the method 200. The pipeline of Fig. 4 may again comprise the reflection detection module 302 and the reflection removal module 304.

[0046] In the embodiment of Fig. 4, analogously to the embodiment of Fig. 3, an input image 500 comprising reflections is given as input to module 302. The module 302 is configured to detect 201 the areas comprising reflection. In this embodiment, the reflection detection 201 is performed by means of a CNN. Moreover, the output of the module 302 comprises crops with reflections (extracted reflection areas), which are given as input to the module 304. The module 304 is configured to perform a reflection removal 202 from the extracted reflection areas. This can be done either by means of a CNN or by means of a GAN. In an embodiment, the GAN can be a conditional GAN.

[0047] FIG. 5 shows a schematic representation of another pipeline for removing reflections from an image 500 that may be used to perform the method 200 according to an embodiment. The pipeline may be part of a device that performs the method 200.

[0048] For the pipeline shown in Fig. 5, the image 500 is exemplarily a photograph of a person wearing eyeglasses. The pipeline includes a first stage 501 for detecting a face of the person in the image 500. Further, the pipeline may include a second stage 502 for detecting the eyeglasses in the image 500 based on the detected face. For instance, the eyeglasses can be detected in the image 500 using a segmentation CNN. A third stage 503 of the pipeline may then detect the one or more reflection areas located within the eyeglasses detected in the image 500.

[0049] Furthermore, the pipeline of Fig. 5 can perform the following steps. The second stage 502 may further segment the eyeglasses detected in the image 500, for instance, by using the segmentation CNN. The third stage 503 may further extract or crop the obtained eyeglass segments from the image 500, and may detect 201 the one or more reflection areas located within the eyeglasses by removing eyeglasses segments without reflection from the extracted eyeglasses segments. Further, a fourth stage 504 may remove the reflection from each of the extracted eyeglasses segments with reflection and may paste back 504 the areas with removed reflections to obtain the processed image 500'.

[0050] Accordingly, in this embodiment, the reflections are removed only from eye-glasses area, not from any other part of image 500. That is, the first stage 501 may detect the eyeglasses, for example, using face detection and face parsing tool. Then, a CNN can be applied to the eyeglasses crops in the fourth stage 504 and the result can be pasted back.

[0051] Fig. 6 shows a device 600 according to an embodiment. The device 600 may be configured to perform the method 200, including the steps 201, 202 and 203, in order to remove reflections from an image 500. The output may be the processed image 500' with less reflections or no reflections. The device 600 may comprise a camera to obtain the image 500.

[0052] Further, the device 600 may comprise a processor or processing circuitry (not shown) configured to perform, conduct or initiate the various steps of the method 200 described herein. The processing circuitry may comprise hardware and / or the processing circuitry may be controlled by software. The hardware may comprise analog circuitry or digital circuitry, or both analog and digital circuitry. The digital circuitry may comprise components such as application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), digital signal processors (DSPs), or multi-purpose processors.

[0053] The device 600 may further comprise memory circuitry, which stores one or more instruction(s) that can be executed by the processor or by the processing circuitry, in particular under control of the software. For instance, the memory circuitry may comprise a non-transitory storage medium storing executable software code which, when executed by the processor or the processing circuitry, causes the method 200 to be performed.

[0054] In one embodiment, the processing circuitry comprises one or more processors and a non-transitory memory connected to the one or more processors. The non-transitory memory may carry executable program code which, when executed by the one or more processors, causes the device 600 to perform, conduct or initiate the method 200 described herein.

[0055] In particular, the processor or processing circuitry of the device 600 is configured to detect 201 one or more reflection areas in the image 500, wherein each reflection area includes a reflection. Further, it is configured to extract 202 the one or more reflection areas from the image 500, and to remove 203 the reflection from each of the extracted reflection areas.

[0056] In summary, contrary to a conventional one-stage approach (see, e.g., Fig. 1), embodiments of the present disclosure perform a two-stage reflection removal procedure (see Figs. 3-6). First, the areas with the reflections are detected 201, and, second, the reflection removal 202 is applied only to found reflection areas that have been extracted 203 from the image 500.

[0057] The present disclosure has been described in conjunction with various embodiments as examples as well as implementations. However, other variations can be understood and effected by those persons skilled in the art and practicing the claimed matter, from the studies of the drawings, this disclosure and the independent claims. In the claims as well as in the description the word "comprising" does not exclude other elements or steps and the indefinite article "a" or "an" does not exclude a plurality. A single element or other unit may fulfill the functions of several entities or items recited in the claims. The mere fact that certain measures are recited in the mutual different dependent claims does not indicate that a combination of these measures cannot be used in an advantageous implementation.

Claims

1. A method (200) for removing reflections from an image (500), the method (200) comprising: - detecting (201) one or more reflection areas in the image (500) and extracting (202) the one or more reflection areas from the image (500) by using a first trained model in a reflection detection module (302), wherein the image (500) is given as input to the reflection detection module (302), and the output of the reflection detection module (302) extracting the one or more reflection areas from the image (500) comprises crops from the image (500) of only the one or more reflection areas, wherein each reflection area includes a reflection, - removing (203) the reflection from each of the extracted reflection areas by using a second trained model in a reflection removal module (304), - wherein the method is characterized by further comprising: only the extracted reflection areas are given as input to the reflection removal module (304), and the output of the reflection removal module (304) is the extracted reflection areas without the reflection, and - reinserting the extracted reflection areas without the reflection into the image (500) to replace, respectively, the detected reflection areas with the reflection.

2. The method (200) of claim 1, wherein: the first and / or second trained model are / is trained by taking an initial image without a reflection from a pool of images without reflections, and generating a synthetic reflection on the initial image with a reflection generator to obtain a training image, and processing the training image with a generator of the first and / or second trained model on the basis of the initial image and the synthetic image, and finally calculating a loss function of the first and / or second trained model on the basis of the initial image and the synthetic image.

3. The method (200) of claim 2, wherein the first trained model comprises a first convolutional neural network, CNN.

4. The method (200) of claim 3, wherein the first CNN comprises a semantic segmentation CNN configured to perform a semantic segmentation of the image to detect the one or more reflection areas using a semantic mask, wherein the semantic mask is used to find connected components identified as reflections, and to further find a minimum circumscribing rectangle of the reflections to determine the reflection areas.

5. The method (200) of any one of the claims 2 to 4, wherein the second trained model comprises a generative adversarial network, GAN.

6. The method (200) of claim 5, wherein the GAN comprises a conditional GAN.

7. The method (200) of any one of the claims 2 to 4, wherein the second trained model comprises a second CNN.

8. The method (200) of claim 4, wherein the image (500) is a photograph of a person wearing eyeglasses, and wherein the step of detecting (201) the one or more reflection areas comprises: - detecting (501) a face of the person in the image (500), - detecting the eyeglasses in the image (500) by using the segmentation CNN to segment the eyeglasses based on the detected face and face parsing tool, and - detecting (201) the one or more reflection areas located within the eyeglasses detected in the image (500).

9. The method (200) of claim 8, further comprising: - segmenting (502) the eyeglasses detected in the image by the segmentation CNN, - extracting (503) the obtained eyeglass segments from the image (500), - detecting (201, 504) the one or more reflection areas located within the eyeglasses detected in the image (500), and - removing (203, 504) the reflection from each of the extracted eyeglasses segments with reflection.

10. A device (600) for removing reflections from an image (500), the device (600) comprising means being configured to perform the method (200) according to any one of the claims 1 to 9.

11. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method (200) according to any one of the claims 1 to 9.