Method for identifiying reflections in image data, method for correcting image data containing a reflection, and method for detecting target objects

Abstract

A method for identifying reflections in image data includes detecting presence of a reflective object in the image data 304, constructing a radiance field of an environment around the reflective obje

Description

TECHNICAL FIELD

[0001] Various embodiments relate to methods for identifying reflections in image data, methods for correcting image data containing a reflection, and methods for detecting target objects in an image. BACKGROUND

[0002] Object detection is an important function in advanced or autonomous driving assistance systems. The object detection function may prevent road accidents by detecting pedestrians, obstacles, or road signs. The function may also include categorizing the detected objects into various classifications. For example, vehicles may be categorized into categories such as 'regular,' 'emergency,' and 'school bus.' If a vehicle is classified as an emergency vehicle, the system may monitor for the presence of sirens and flashing lights and make decisions accordingly. However, illumination conditions heavily influence the visual information available to camera sensors. In particular, shiny surfaces on the road that result in reflections may substantially decrease the accuracy of object detection. Existing driving assistance systems typically mitigate this challenge through sensor data fusion by combining camera images with sensor data from other sensors such as radar and LiDAR. However, such solutions may be costly and complex. SUMMARY

[0003] According to various embodiments, there is provided a computer-implemented method for identifying reflections in image data. The method includes detecting presence of a reflective object in the image data, constructing a radiance field of an environment around the reflective object based on visual information on the reflective object, and generating a recovered image based on the radiance field. The recovered image represents an undistorted view of the visual information. The method further includes constructing a 3D representation of the environment based on a multi-view image dataset, and generating a plurality of rendered images based on the 3D representation. Each rendered image of the plurality of rendered images corresponds to a respective camera viewpoint. The method further includes determining whether the visual information on the reflective object comprises a reflection, based on comparing the recovered image to each rendered image.

[0004] According to various embodiments, there is provided a computer-implemented method for correcting image data containing a reflection. The method includes identifying the reflection in the image data according to the abovementioned method for identifying reflections in image data. The method further includes detecting a reflection region in the image data where the reflection lies, masking the reflection region, and inpainting the reflection region based on contextual information around the reflection region.

[0005] According to various embodiments, there is provided a computer-implemented method for detecting target objects in an image. The method includes correcting the image according to the abovementioned method for correcting image data containing a reflection, to result in a corrected image. The method further includes detecting target objects in the corrected image.

[0006] According to various embodiments, there is provided a use of the method for detecting target objects in an image, for an advanced driving assistance function.

[0007] According to various embodiments, there is provided a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out any one of the abovementioned methods.

[0008] According to various embodiments, there is provided a data processing apparatus comprising means for carrying out any one of the abovementioned methods.

[0009] According to various embodiments, there is provided a computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out any one of the abovementioned methods.

[0010] Additional features for advantageous embodiments are provided in the dependent claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In the drawings, like reference characters generally refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various embodiments are described with reference to the following drawings, in which:

[0012] FIG. 1 represents a camera image that captured a reflection.

[0013] FIG. 2 shows a flow chart of a computer-implemented method for detecting and correcting reflections in camera images, according to various embodiments.

[0014] FIG. 3 shows a flowchart of the reflection identification process according to various embodiments.

[0015] FIG. 4 shows a flowchart of the corrective action process according to various embodiments.

[0016] FIG. 5 shows a flow diagram of a computer-implemented method for identifying reflections in image data, according to various embodiments.

[0017] FIG. 6 shows a flow diagram of a computer-implemented method for correcting image data containing a reflection, according to various embodiments.

[0018] FIG. 7 shows a flow diagram of a computer-implemented method for detecting target objects in an image according to various embodiments.

[0019] FIG. 8 shows a block diagram of a data processing apparatus according to various embodiments. DESCRIPTION

[0020] Embodiments described below in context of the devices are analogously valid for the respective methods, and vice versa. Furthermore, it will be understood that the embodiments described below may be combined, for example, a part of one embodiment may be combined with a part of another embodiment.

[0021] It will be understood that any property described herein for a specific device may also hold for any device described herein. It will be understood that any property described herein for a specific method may also hold for any method described herein. Furthermore, it will be understood that for any device or method described herein, not necessarily all the components or steps described must be enclosed in the device or method, but only some (but not all) components or steps may be enclosed.

[0022] The term “coupled” (or “connected”) herein may be understood as electrically coupled or as mechanically coupled, for example attached or fixed, or just in contact without any fixation, and it will be understood that both direct coupling or indirect coupling (in other words: coupling without direct contact) may be provided.

[0023] In this context, the device as described in this description may include a memory which is for example used in the processing carried out in the device. A memory used in the embodiments may be a volatile memory, for example a DRAM (Dynamic Random Access Memory) or a non-volatile memory, for example a PROM (Programmable Read Only Memory), an EPROM (Erasable PROM), EEPROM (Electrically Erasable PROM), or a flash memory, e.g., a floating gate memory, a charge trapping memory, an MRAM (Magnetoresistive Random Access Memory) or a PCRAM (Phase Change Random Access Memory).

[0024] In order that the invention may be readily understood and put into practical effect, various embodiments will now be described by way of examples and not limitations, and with reference to the figures.

[0025] FIG. 1 represents a camera image 100 that captured a reflection 102. The camera image 100 captures the visual image of a bus 102. The bus 102 have glass panels 104 that reflect off light from their surfaces. A car travelling on a different lane from the bus 102 is visible on the glass panels 104. In other words, the camera image 100 includes a reflection 102 that shows the car. While the human eye may be able to identify the visual image of the car on the bus 102 as a reflection 102, the camera image 100 may confuse an object detection software in a driver assistance system. The object detection software may erroneously classify the reflection 102 as another car, instead of disregarding the reflection 102.

[0026] FIG. 2 shows a flow chart of a computer-implemented method 200 for detecting and correcting reflections in camera images, according to various embodiments. The method 200 may address the problem of reflections in camera images affecting the accuracy of object detection algorithms, such as the example described with respect to FIG. 1, without the need for other types of sensors. Reflections depend jointly on object geometry, material properties, the three-dimensional (3D) environment, and the observer viewing direction. The method 200 may include an image data collection process 202, a reflection identification process 210 and a corrective action process 220.

[0027] The image data collection process 202 may include obtaining image data captured by a plurality of cameras. The plurality of cameras may be set up such that they capture a scene from various viewpoints. The set up of the plurality of cameras may be referred herein as a multi-view image capture set up. The image data collected may referred herein as a multi-view image dataset. The camera images may be processed in real-time using a perception stack at each frame. In some embodiments, the multi-view image capture set up may be configured to capture an at least substantially 360 degrees scene around an object. In an automotive application, the multi-view image capture set up may include surround view cameras installed on a vehicle. The surround view cameras may be configured to capture an at least substantially 360 degrees scene around the vehicle.

[0028] The reflection identification process 210 may include identifying reflective materials and their specific regions in image data obtained from the image data collection process 202, using a neural network. The image data may include, or may be a subset of the multi-view image dataset. The reflection identification process 210 may further include reconstructing an image shown on the reflective materials, to determine whether the reflective materials reflect another object.

[0029] The corrective action process 220 may identify and locate areas in the image data that are affected by the reflections, and then generate image(s) devoid of the unwanted reflections. The corrective action process 220 may detect the area of reflected images on reflective materials or surfaces using an object detection algorithm in conjunction with the Intersection over Union (loU) metric to accurately identify reflections on reflective materials. The corrective action process 220 may further mask the detected areas and paint them with appropriate pixels based on the surrounding context, to generate an image without any reflections.

[0030] FIG. 3 shows a flowchart of the reflection identification process 210 according to various embodiments. The reflection identification process 210 determines whether visual information, i.e. image, present on a reflective object shows another object. The reflection identification process 210 may include inputting a multi-view image dataset 302 to a reflective object identification network 304. The reflective object identification network 304 may be a neural network that is trained to identify and locate reflective objects in images. If the reflective object identification network 304 ascertains the presence of reflective objects in the multi-view image dataset 302, the output of the reflective object identification network 304 may trigger activation of a reflection recovery process 330 and activation of a viewpoint rendering process 340. Reflective objects are objects that have reflective surfaces that sends off most of the light that lands on it. For example, glossy and / or smooth materials such as glass, metal, stone etc. may be reflective.

[0031] The reflection recovery process 330 may include reconstructing the image observed on the surface of the reflective object. To this end, the reflection recovery process 330 transforms the reflective object with an unfamiliar geometry and texture into a plurality of radiance-field cameras that capture the radiance field of the environment surrounding the reflective object. Referring back to the example of FIG. 1, the bus 102 may be considered to be a reflective object while the image observed on the reflective object is that of a car. The reflection recovery process 330 may further include reconstructing the reflective object’s geometry and diffuse radiance, along with capturing the depth and radiance information from the object’s perspective in its surroundings.

[0032] The reflection recovery process 330 includes a radiance field transformation process 306. In the radiance field transformation process 306, based on the multi-view image dataset 302, reflections on the object’s surface are decomposed to construct a radiance field, capturing the 3D world. The constructed radiance field may be used to synthesize beyond field-of-view novel viewpoints in a recovery process 308. The output of the recovery process 308 may be recovered images. The recovered images present undistorted views of the reflections on the reflective object.

[0033] An example of the implementation of the reflection recovery process 330 is disclosed in “ORCa: Glossy Objects as Radiance-Field Cameras” by Tiwary et. al., accessible at https: / / export.arxiv.org / abs / 2212.04531, which is incorporated herein by reference.

[0034] The viewpoint rendering process 340 may include constructing a 3D NeRF scene using the multi-view image dataset 302, then rendering two-dimensional (2D) images from the NeRF scene based on predefined camera viewpoint configurations. The viewpoint rendering process 340 may include a novel view synthesis process 312 and a rendering process 314. The novel view synthesis process 312 may include converting the multi-view image dataset 302 into a 3D continuous function, for example, using Neural Radiance Fields (NeRF). The novel view synthesis process 312 may involve interpreting reflections to reconstruct a full scene. By representing the surrounding environment as a 3D continuous function, comprehensive light information, including light information of areas not directly visible from the camera may be captured, effectively 'seeing' the environment from the reflective object's perspective.

[0035] The rendering process 314 includes generating a rendered image based on a predefined camera viewpoint. In other words, the rendered image is rendered from a perspective as defined by the camera viewpoint. The rendered image may be an input to a similarity score determination process 310. Further, the recovered image generated from the recovery process 308 may also be presented as a 3D continuous function through the novel view synthesis process 312, to thereby enable the rendering of the recovered image from various perspectives through the rendering process 314. By rendering these images from different perspectives, the rendering process 314 may create a complete picture of the reflected scene, facilitating the computation of image similarity scores.

[0036] The similarity score determination process 310 may compare the rendered image generated by the rendering process 314, to the recovered image generated by the recovery process 308. The similarity score determination process 310 may determine an image similarity score based on the comparison. Next, the image similarity score may be compared against a similarity threshold in a check process 318. If the image similarity score of the rendered image exceeds the similarity threshold, the reflection identification process 210 may output a confirmation that the reflection of another object is present on the reflective object, in 320. If the image similarity score of a rendered image does not exceed the similarity threshold, the reflection identification process 210 returns to the rendering process 314 to generate another rendered image at another predefined camera viewpoint. The image similarity determination process 310 and the check process 318 are repeated against based on comparing this other rendered image to the recovered image. The rendering process 314, the image similarity determination process 310 and the check process 318 may be repeated until the image similarity score of one rendered 2D image exceeds the similarity threshold, or until the rendering process 314 has generated images for all of the predefined camera viewpoints. If none of the rendered 2D image has an image similarity score that exceeds the similarity threshold, the reflection identification process 210 may output a result that there is no reflection of another object observed on the reflective object, in 322.

[0037] According to various embodiments, the similarity score determination process 310 may include employing a content-based metric to evaluate the similarity or dissimilarity of images by leveraging deep feature representations of the images. Examples of suitable content-based metrics may include Learned Perceptual Image Patch Similarity (LPIPS), Structured Similarity Index Metric (SSIM, MSSIM) and Peak Signal-to-Noise Ratio (PSNR).

[0038] An implementation of the LPIPS is disclosed in "The unreasonable effectiveness of deep features as a perceptual metric" by Zhang, Richard, et al., published in the Proceedings of the IEEE conference on computer vision and pattern recognition. 2018, which is incorporated herein by reference. An implementation of the SSIM, MSSIM is disclosed in "Image quality assessment: from error visibility to structural similarity" by Wang, Zhou, et al, published in the IEEE transactions on image processing 13.4 (2004): 600-612, which is incorporated herein by reference. Details of the PSNR is disclosed in “Peak Signal-to-Noise Ratio” accessible at https: / / en.wikipedia.org / wiki / Peak signal-to-noise ratio. which is incorporate herein by reference.

[0039] An example of the implementation of the novel view synthesis process 312 using NeRF is disclosed in “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis” by Mildenhall et. al., accessible at https: / / arxiv.org / pdf / 2003.08934, which is incorporated herein by reference.

[0040] FIG. 4 shows a flowchart of the corrective action process 220 according to various embodiments. The corrective action process 220 may include an object detection process 402, an Intersection over Union (loU) calculation process 404, an extraction process 406, a masking process 408, and an inpainting process 410. The object detection process 402 may detect and locate objects within an image, using an object detection algorithm. Examples of the objects detected may include vehicles, traffic lights, road signs, and road mirrors, among others. The object detection process 402 may include generating bounding boxes for the detected objects. For reflections, the reflected object will be fully contained within the reflective object. In other words, the bounding box for the reflected object will be fully contained within the bounding box for the reflective object. The loU calculation process 404 may calculate an loU score between the smaller bounding boxes and the larger bounding boxes. The loU score indicates an extent of overlap between two bounding boxes. The greater the extent of overlap, the greater the loU. If the loU score of any pair of bounding boxes is close to 1, for example, more than 0.9, one of the bounding boxes may be at least substantially within the other bounding box. The loU score may be given by the formula: Area of intersection of two boxes loU =---------------------------------- Area of union of two boxes

[0041] If the loU score is determined to be close to 1, the extraction process 406 may be carried out. The extraction process 406 may include removing the region bounded by the smaller bounding box, also referred herein as the reflective region or the masked region, from the image. The masking process 408 may be carried out to fill the reflective region with noise to eliminate reflection traces. Next, the inpainting process 410 may be carried out to inpaint the masked region based on contextual information, i.e. information from the area around the masked region. The inpainting process 410 may be carried out, for example, using a diffusionbased method.

[0042] An example of the implementation of the object detection process 402 is disclosed in “Progressive Mirror Detection” by Lin et. al., accessible at https: / / www.cs.cityu.edu.hk / ~rynson / papers / cvpr20c.pdf, which is incorporated herein by reference.

[0043] An example of the implementation of the inpainting process 410 is disclosed in “Palette: Image-to-Image Diffusion Models” by Saharia et. al., accessible at https: / / arxiv.org / pdf / 2111.05826, which is incorporated herein by reference.

[0044] FIG. 5 shows a flow diagram of a computer-implemented method 500 for identifying reflections in image data, according to various embodiments. The method 500 includes processes 502, 504, 506, 508, 510 and 512. The process 502 may include detecting presence of a reflective object in the image data. The process 502 may, for example, include the reflective object identification process 304. The process 504 may include constructing a radiance field of an environment around the reflective object based on visual information on the reflective object. The process 504 may include, for example, the radiance field transformation process 306. The process 506 may include generating a recovered image based on the radiance field. The recovered image may represent an undistorted view of the visual information. The process 506 may include, for example, the recovery process 308. The process 508 may include constructing a three-dimensional representation of the environment based on a multi-view image dataset. The process 508 may include, for example, the novel view synthesis process 312. The process 510 may include generating at least one rendered image based on the 3D representation. Each rendered image of the at least one rendered image may correspond to a respective predefined camera viewpoint. The process 510 may include, for example, the rendering process 314. The process 512 may include determining whether the visual information on the reflective object includes a reflection of another object, based on comparing the recovered image to the at least one rendered image. The process 512 may include, for example, the similarity score determination process 310 and the check process 318. By synthesizing the image of the reflection using two different methods and then comparing the respective outputs, the method 500 may accurately detect reflected objects solely using camera sensors.

[0045] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, the multi-view image dataset may include a plurality of images capturing a scene from a corresponding plurality of viewpoints, wherein the reflective object is present in the scene. The multi-view image dataset may provide visual information of the surroundings of an object, such as a vehicle or a robot, such that the object may be provided with information about its environment.

[0046] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, the multi-view image dataset may include the image data.

[0047] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, the multi-view image dataset may be captured by a set of surround view cameras mounted on a vehicle.

[0048] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, the set of surround view cameras may capture a 360-degree scene around the vehicle.

[0049] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, determining whether the visual information on the reflective object includes a reflection may include for each rendered image, determining a similarity score based on comparing the recovered image to the rendered image, and determining whether the similarity score exceeds a similarity threshold. The process of determining the similarity score may include the similarity score determination process 310. Determining whether the similarity score exceeds a similarity threshold may include the check process 318.

[0050] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, determining whether the visual information on the reflective object includes a reflection may further include determining that the visual information on the reflective object includes a reflection based on determining that the similarity score exceeds the similarity threshold for at least one rendered image.

[0051] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, determining whether the visual information on the reflective object includes a reflection further includes determining that the visual information on the reflective object does not include a reflection based on determining that the similarity score does not exceed the similarity threshold for all rendered images of the plurality of rendered images.

[0052] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, detecting presence of a reflective object in the image data may include inputting the image data into a neural network, wherein the neural network comprise at least one of Convolutional Neural Network (CNN) and Transformer network. The neural network may include at least 20000 or 300000 nodes.

[0053] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, constructing the radiance field of the environment around the reflective object may include modelling a surface of the reflective object as at least one virtual sensor that captures a 2D projection of a 5D environment radiance field surrounding the reflective object. Recovering a 5D radiance field may prove beneficial in scenarios involving nearby objects, where typical occlusions inadequately modeled by 2D environment maps may occur. The estimation of the radiance field enables imaging beyond occlusions through the sampling of novel viewpoints, exemplified by the translated viewpoints. Additionally, this approach facilitates the representation of depth to surroundings from these virtual viewpoints.

[0054] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, modelling the surface of the reflective object as at least one virtual sensor may include: for each local point of a plurality of points on the surface, estimating position of the local point using neural rendering, and converting the local point into a virtual pixel that captures radiance from the environment.

[0055] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, constructing the 3D representation of the environment based on surround view images may include converting the surround view images into a 3D continuous function using NeRF. Using NeRF allows the construction of the 3D representation by accumulating the radiance along rays passing through the scene.

[0056] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, the method 500 may further include converting the recovered image into a further 3D continuous function, and combining the further 3D continuous function with the 3D continuous function for construction the 3D representation of the environment.

[0057] FIG. 6 shows a flow diagram of a computer-implemented method 600 for correcting image data containing a reflection, according to various embodiments. The method 600 includes processes 602, 604, 606, 608, 610 and 612. The method 600 may remove a reflection from an image, so that the corrected image may be used by perception systems without false errors resulting from identifying the reflections as objects. The process 602 may include identifying the reflection in the image data according to the method 500. The process 604 may include detecting a reflection region in the image data where the reflection lies. The process 604 may include, for example, the object detection process 402, the loU calculation process 404 and the extraction process 406. The process 606 may include masking the reflection region. The process 606 may include, for example, the masking process 408. The process 608 may include inpainting the reflection region based on contextual information around the reflection region. The process 608 may include, for example, the inpainting process 410.

[0058] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, detecting the reflection region in the image where the object reflection lies may include using an object detection algorithm in conjunction with the loU metric. The object detection algorithm may perform the object detection process 402.

[0059] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, detecting the region in the image where the object reflection lies may further include generating bounding boxes on the image using the object detection algorithm. The bounding boxes may include a large bounding box representing the reflective object and further include a small bounding box representing a reflection of another object.

[0060] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, detecting the region in the image where the object reflection lies may include determining the loU score between the small bounding box and the large bounding box, and determining that the small bounding box includes a reflection on a surface of the reflective object based on determining that the loU score is at least substantially close to 1.

[0061] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, masking the reflective region may include fdling the reflective region with noise to eliminate reflection traces.

[0062] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, inpainting the masked region based on contextual information around the masked region may include using a generative network, preferably a diffusion network. The generative network may include at least 20000 or 300000 nodes.

[0063] FIG. 7 shows a flow diagram of a computer-implemented method 700 for detecting target objects in an image according to various embodiments. The method 700 includes processes 702 and 704. The process 702 may include correcting the image according to the method 600, to result in a corrected image. The process 704 may include detecting target objects in the corrected image. Minimizing or removing reflections from images may improve the accuracy of object detection and recognition, contributing to improved overall perception capabilities.

[0064] According to various embodiments, a computer program may include instructions which, when the program is executed by a computer, cause the computer to carry out at least one of the methods 500, 600 or 700.

[0065] According to various embodiments, a computer-readable medium may include instructions which, when executed by a computer, cause the computer to carry out at least one of the methods 500, 600 or 700.

[0066] FIG. 8 shows a block diagram of a data processing apparatus 800 according to various embodiments. The data processing apparatus 800 may include means for carrying out at least one of the methods 500, 600 or 700. The data processing apparatus 800 may include at least one processor 802. The data processing apparatus 800 may further include a plurality of sensors 804 and at least one memory 806. The plurality of sensors 804 may include camera sensors configured to capture the multi-view image dataset. The memory 806 may store therein the instructions for the processor 802 to perform at least one of the methods 500, 600 or 700. The at least one processor 802, the plurality of sensors 804 and the at least one memory 806 may be coupled to one another, for example, mechanically or electrically, via the coupling line 810. Various aspects described with respect to the data processing apparatus 800 may be applicable to the computer-readable medium.

[0067] According to various embodiments, the method 700 for detecting target objects in an image may be used for an advanced driving assistance function. As the method 700 only requires camera images, the overall cost of the perception system used with the advanced driving assistance function may be relatively low, as compared to perception systems that use a combination of sensor types. Also, having only one sensor type streamlines the hardware configuration and reduces the time required for pre-processing signals from all sensors.

[0068] According to various embodiments, the data processing apparatus 800 may be part of an advanced driving assistance system (ADAS). By removing reflections from the camera images, the data processing apparatus 800 may reduce unnecessary alerts or interventions, thereby enhancing the efficiency of the ADAS.

[0069] The method 700 for detecting target objects in an image may be implemented in other applications. For example, in surveillance systems, augmented or virtual reality applications, robotics, medical imaging or in consumer electronics.

[0070] Reflections in surveillance camera footage may impede the accurate detection of objects or individuals. Overcoming reflection challenges enhances the reliability of surveillance systems in areas such as security and public safety.

[0071] In AR and VR applications, reflections on surfaces may disrupt the immersive experience. Removing the reflections may contribute to creating more realistic and visually appealing virtual environments.

[0072] Robots equipped with sensors may encounter reflection-related challenges, affecting their ability to navigate and interact with the environment. Addressing reflections may improve the overall performance of robots in various applications, including industrial automation and exploration.

[0073] Reflections may occur in medical imaging, affecting the quality and interpretation of diagnostic images. Removing reflections in medical images may be aid in achieving accurate medical diagnoses.

[0074] Consumer electronics such as smartphones and smart cameras may benefit from solutions that remove reflections from images, for example, to generate better photographs and videos.

[0075] Examples of suitable training datasets include datasets of natural illumination maps the BRDF database is the KITTI object detection 3D dataset available at httDs: / / www.cvlibs.net / datasets / kitti / eval obiect.Dhp?obi benchmark=3d.

[0076] Neural networks, such as the reflective object identification network 304, neural networks used in the reflection recovery process 330, the viewpoint rendering process 340, the object detection process 402, the inpainting process 410 and other neural networks used in the above-described methods 500, 600 and 700 may be trained according to the following steps.

[0077] As an example, a training dataset may be provided to the neural network and the following standard steps for training a general neural network may be followed.

[0078] Before training the neural network, the weights of the neural network may be randomly initialized to numbers between 0.01 and 0.1, while the biases are randomly initialized to numbers between 0.1 and 0.9.

[0079] Subsequently, the first observations of the dataset are loaded into the input layer of the neural network and the output value(s) is generated by forward-propagation of the input values of the input layers. Afterwards the following loss function is used to calculate loss with the output value(s):

[0080] Subsequently, the first observations of the dataset may be loaded into the input layer of the neural network and the output value(s) is generated by forward-propagation of the input values of the input layers. Afterwards the following loss function may be used to calculate loss with the output value(s): 1 2

[0081] Mean Square Error (MSE): MSE = ~S(y — y) , where n represents the number of neurons in the output layer and y represents the real output value and y represents the predicted output. In other words, y — y represents the difference between actual and predicted output.

[0082] The weights and biases may subsequently be updated by an AdamOptimizer with a learning rate of 0.001. Other parameters of the AdamOptimizer may be set to default values. For example: beta_l = 0.9 beta_2 = 0.999 eps = le-08 weight_decay = 0

[0083] The steps described above may be repeated with the next set of observations until all the observations are used for training. This represents the first training epoch. This may be repeated until 10 epochs are completed.

[0084] While embodiments of the invention have been particularly shown and described with reference to specific embodiments, it should be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. The scope of the invention is thus indicated by the appended claims and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced. It will be appreciated that common numerals, used in the relevant drawings, refer to components that serve a similar or the same purpose.

[0085] It will be appreciated to a person skilled in the art that the terminology used herein is for the purpose of describing various embodiments only and is not intended to be limiting of the present invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0086] It is understood that the specific order or hierarchy of blocks in the processes / flowcharts disclosed is an illustration of exemplary approaches. Based upon design preferences, it is understood that the specific order or hierarchy of blocks in the processes / flowcharts may be rearranged. Further, some blocks may be combined or omitted. The accompanying method claims present elements of the various blocks in a sample order, and are not meant to be limited to the specific order or hierarchy presented.

[0087] The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects. Thus, the claims are not intended to be limited to the aspects shown herein, but is to be accorded the full scope consistent with the language claims, wherein reference to an element in the singular is not intended to mean “one and only one” unless specifically so stated, but rather “one or more.” The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects. Unless specifically stated otherwise, the term “some” refers to one or more. Combinations such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, C, or any combination thereof’ include any combination of A, B, and / or C, and may include multiples of A, multiples of B, or multiples of C. Specifically, combinations such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, C, or any combination thereof’ may be A only, B only, C only, A and B, A and C, B and C, or A and B and C, where any such combinations may contain one or more member or members of A, B, or C. All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims.

Claims

1. A computer-implemented method (500) for identifying reflections in image data, the method (500) comprising:detecting presence of a reflective object in the image data;constructing a radiance field of an environment around the reflective object, based on visual information on the reflective object;generating a recovered image based on the radiance field, wherein the recovered image represents an undistorted view of the visual information;constructing a three-dimensional representation of the environment based on a multiview image dataset;generating at least one rendered image based on the three-dimensional representation, wherein each rendered image of the at least one rendered image corresponds to a respective predefined camera viewpoint; anddetermining whether the visual information on the reflective object comprises a reflection of another object, based on comparing the recovered image to the at least one rendered image.

2. The method (500) of claim 1, wherein the multi-view image dataset (302) comprises a plurality of images capturing a scene from a corresponding plurality of viewpoints, wherein the reflective object is present in the scene.

3. The method (500) of any preceding claim, wherein the multi-view image dataset (302) comprises the image data.

4. The method (500) of any preceding claim, wherein the multi-view image dataset (302) is captured by a set of surround view cameras mounted on a vehicle.

5. The method (500) of claim 4, wherein the set of surround view cameras capture a 360-degree scene around the vehicle.

6. The method (500) of any preceding claim, wherein generating the at least one rendered image based on the 3D representation comprisesgenerating a rendered image based on a respective predefined camera viewpoint of a plurality of predefined camera viewpoints until a rendered image of the at least one rendered image is determined to be similar to the recovered image, or until the rendered images are generated for all predefined camera viewpoints of the plurality of predefined camera viewpoints.

7. The method (500) of claim 6, wherein determining whether the visual information on the reflective object comprises a reflection comprisescomparing the at least one rendered image to the recovered image, to thereby determine a similarity score for each rendered image of the at least one rendered image, wherein the similarity score indicates a level of similarity between the rendered image and the recovered image; anddetermining that the visual information on the reflective object comprises a reflection of another object based on determining that the similarity score of one or more rendered image exceeds a similarity threshold.

8. The method (500) of claim 7, wherein determining whether the visual information on the reflective object comprises a reflection further comprisesdetermining that the visual information on the reflective object does not comprise a reflection based on determining that the similarity score does not exceed the similarity threshold for all rendered images of the at least one rendered image, wherein the at least one rendered image comprises rendered images generated for all predefined camera viewpoints of the plurality of predefined camera viewpoints.

9. The method (500) of any preceding claim, wherein detecting presence of a reflective object in the image data comprises inputting the image data into a neural network, wherein the neural network comprise at least one of Convolutional Neural Network and Transformer network.

10. The method (500) of any preceding claim, wherein constructing the radiance field of the environment around the reflective object comprisesmodelling a surface of the reflective object as at least one virtual sensor that captures a 2D projection of a 5D environment radiance field surrounding the reflective object.

11. The method (500) of any preceding claim, wherein modelling the surface of the reflective object as at least one virtual sensor comprisesfor each local point of a plurality of points on the surface, estimating position of the local point using neural rendering, and converting the local point into a virtual pixel that captures radiance from the environment.

12. The method (500) of any preceding claim, wherein constructing the 3D representation of the environment based on surround view images comprises converting the surround view images into a 3D continuous function using NeRF.

13. A computer-implemented (600) method for correcting image data containing a reflection, the method (600) comprising:identifying the reflection in the image data according to the method (500) of any preceding claim;detecting a reflection region in the image data where the reflection lies;masking the reflection region; andinpainting the reflection region based on contextual information around the reflection region.

14. The method (600) of claim 13, wherein detecting the reflection region in the image where the object reflection lies comprises using an object detection algorithm in conjunction with Intersection over Union metric.

15. The method (600) of claim 14, wherein detecting the region in the image where the object reflection lies further comprisesgenerating bounding boxes on the image using the object detection algorithm, wherein the bounding boxes comprise a large bounding box representing the reflective object and further comprises a small bounding box representing a reflection of another object.

16. The method (600) of claim 15, wherein detecting the region in the image where the object reflection lies comprisesdetermining the Intersection over Union score between the small bounding box and the large bounding box, anddetermining that the small bounding box comprises a reflection on a surface of the reflective object based on determining that the Intersection over Union score is at least substantially close to 1.

17. The method (600) of any one of claims 13 to 16, wherein masking the reflective region comprises filling the reflective region with noise to eliminate reflection traces.

18. The method (600) of any one of claims 13 to 17, wherein inpainting the masked region based on contextual information around the masked region comprises using a generative network, preferably a diffusion network.

19. A computer-implemented method (700) for detecting target objects in an image, the method (700) comprising:correcting the image according to the method (600) of any one of claims 13 to 18, to result in a corrected image; anddetecting target objects in the corrected image.

20. Use of the method (700) of claiml9 for an advanced driving assistance function.

21. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method (500, 600, 700) of any one of claims 1 to 19.

22. A data processing apparatus comprising means for carrying out the method (500, 600, 700) of any one of claims 1 to 19.

23. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method (500, 600, 700) of any one of claims 1 to 19.