Apparatus and method for image processing

Abstract

An apparatus comprising an image processor device for implementing an image-based computational model as part of an end-to-end processing pipeline. The processing pipeline is for operating by receiving color-specific image data representing a scene; receiving depth data for the scene; processing the color-specific image data using the image-based computational model to form a feature map for the scene; forming an illumination map from the feature map and the depth data, the illumination map representing an illumination estimate for a set of three-dimensional locations in the scene.

Description

Technical Field

[0001] This invention relates to illumination estimation for object rendering in a scene. Background Technology

[0002] Augmented Reality (AR) applications are more popular than ever, with users demanding more realistic augmentations every day. Coordination is the task of matching the appearance of two images by combining a foreground image extracted from one image with the background of another, resulting in a realistic final composite effect. This is a key concept in AR, allowing applications to add elements to a scene without destroying the illusion. A crucial part of this task is retrieving illumination information from the background and seamlessly applying it to the foreground.

[0003] To successfully adjust these appearances, three major challenges are faced: appearance coordination, which involves matching global image features such as brightness and saturation; geometric coordination, which involves ensuring geometric consistency between the fully enclosed foreground and the background; and illumination coordination, which involves restoring illumination information and correctly relighting the foreground.

[0004] Illuminance estimation is an ill-posed problem because it requires consideration of inverted light sources, geometric reflections and occlusion, and the effects of camera properties. Due to the 3D nature of the scene, the task of estimating light at a specific point in space is inherently 3D-varying, meaning that the illuminance changes relative to the 3D position of that point.

[0005] Several existing methods exist for retrieving global illumination or camera-centric directional lighting. However, these methods do not meet the requirements of augmented reality applications. Some recent spatial variation methods can retrieve lighting direction using dense or sparse approaches. However, although lighting is not a 2D task, none of them allow for the free movement of added objects in 3D.

[0006] Recovering illumination in the real world is very expensive, often requiring extensive scene preparation or large equipment. Existing methods all use adversarial examples to bridge the gap between synthetic and real images. Illumination retrieval techniques are typically used to recover illumination and are provided in a representation that can be quickly simulated by existing commercial renderers, including 2D spatial variation estimation.

[0007] An existing method has been proposed that extracts illuminance from a smartphone's front-facing camera for screen brightness purposes. However, the illumination information retrieved from this method is only related to scene brightness and does not measure direction or color. On the other hand, other methods require multiple views as input. Conceptually, these existing methods retrieve a global representation of scene illumination, centered on the camera's location. These methods largely diverge in their use of the CNNs and data for optimizing the light information representation, for example, using environment maps versus light sources. These methods cannot extract illuminance information from other points in space and require moving the sensor in space to make new estimates. Existing methods struggle to extract illumination information from local light or soft surface reflections. Other existing methods retrieve egocentric illumination estimates based on panoramic retrieval of the input image.

[0008] Some existing methods can provide illumination information for 2D spatial variations, requiring the recovery of illumination information on visible geometric surfaces. This task is simplified because color information in this region of an image contains key indicators of light information; for example, bright areas likely indicate strong light focusing. However, these methods are not only limited to 2.5D ray estimation but also cannot predict light at points in space where there are no nearby surfaces.

[0009] Other existing methods include recovering light source locations. These methods are limited in the number of existing light sources that can be extracted and cannot recover strong reflections and area lighting. Some existing methods require inserting known objects into the scene. The light information at that location can be recovered through the reflections of the known objects. None of the existing works mentioned above address full 3D scene sampling for illumination estimation.

[0010] A method is needed to accurately estimate the 3D lighting conditions within a scene. Summary of the Invention

[0011] According to one aspect, the present invention provides an apparatus including an image processor means for implementing an image-based computational model as part of an end-to-end processing pipeline, the processing pipeline being configured to operate by: receiving color-specific image data representing a scene; receiving depth data of the scene; processing the color-specific image data using the image-based computational model to form a feature map of the scene; and forming an illumination map based on the feature map and the depth data, the illumination map representing illumination estimates for a set of three-dimensional locations in the scene. This arrangement enables the integration of depth information into the illumination estimates without requiring direct measurement across the entire three-dimensional space. In an embodiment, the image-based computational model may be a neural network model.

[0012] In one embodiment, the color-specific image data can be received from the device's camera. The depth data can be received from the device's depth sensor or as an estimate based on the color-specific image data. This arrangement enables the use of three-dimensional illumination maps even if the device lacks depth measurement capabilities.

[0013] In an embodiment, determining the illuminance at a selected location within the scene may include: shifting the reference frame of the illuminance map to center on the selected location, and combining these illumination points based on the spatial distribution of the illumination points in the illuminance map around the selected location at the center of the reference frame. The ability to shift the reference frame of the illuminance map, rather than recalculating for different coordinate sets within the scene, significantly reduces the computational overhead required. This is especially true when multiple selected locations need to be represented simultaneously, as multiple selected locations can also be represented by shifting the reference frame.

[0014] In an embodiment, the illuminance map includes a plurality of illumination points, each illumination point representing one of the following for a corresponding pixel of the color-specific image: (i) illuminance level and (ii) illuminance hue. The illuminance map may also include data representing the depth corresponding to each illumination point. In an embodiment, a feature vector representation of the illuminance at the selected location is extracted from the illuminance map by an extraction neural network model. These features work individually or in combination to provide an efficient and computationally inexpensive system for estimating lighting conditions in real or virtual scenes.

[0015] According to a second aspect, the present invention provides a computer-implemented method for processing images via an image processor device, the image processor device being configured to implement an image-based computational model as part of an end-to-end processing pipeline, the method comprising: receiving color-specific image data representing a scene; receiving depth data of the scene; processing the color-specific image data using the image-based computational model to form a feature map of the scene; and forming an illuminance map based on the feature map and the depth data, the illuminance map representing illuminance estimates at a set of three-dimensional locations in the scene. By combining the depth and illuminance data, illuminance at points between three-dimensional scene surfaces can be estimated.

[0016] In an embodiment, the method may further include determining the illuminance at a selected location within the scene by: shifting the reference frame of the illuminance map to center on the selected location; and combining the illumination points based on the spatial distribution of the illumination points in the illuminance map around the selected location at the center of the reference frame. The ability to shift the reference frame of the illuminance map, rather than recalculating for a different coordinate set within the scene, significantly reduces the computational overhead required for processing.

[0017] In an embodiment, the method may further include extracting a feature vector representation of the illuminance at the selected location from the illuminance map by extracting a neural network model. Furthermore, the method may include processing the feature vector representation to generate a color-specific spherical harmonic representation, a depth spherical harmonic representation, and a geometric distance estimate indication of the illuminance at the selected location. The geometric distance estimate indication may include one or more spherical harmonic coefficients, and each spherical harmonic representation includes 36 coefficients representing a corresponding approximation, each coefficient multiplied by three color channels.

[0018] In an embodiment, the method may include implementing a discriminator neural network to verify the output of the processing pipeline by: distinguishing the feature vector corresponding to a synthetic image from the feature vector corresponding to a real image to generate a gradient; processing the gradient through a gradient inversion layer; and optimizing the image-based computational model and the extraction neural network model using the processed gradient. The image-based computational model may be a neural network. Therefore, the method can be optimized with minimal actual data recording while still generating an illuminance map that accurately estimates illuminance in three-dimensional space. Attached Figure Description

[0019] The invention will now be described by way of example with reference to the accompanying drawings. In the drawings:

[0020] Figure 1 An overview diagram of the front half of the proposed pipeline is shown;

[0021] Figure 2 An overview diagram of the latter half of the proposed pipeline is shown;

[0022] Figure 3 This demonstrates how to collect lighting information by rendering a cube map around a selected location;

[0023] Figure 4 An example scene is shown, in which an artificial object is located at a selected position within the scene;

[0024] Figure 5 A series of man-made objects located at selected locations within the scene are shown;

[0025] Figure 6 An example device for implementing the proposed image processing pipeline is shown. Detailed Implementation

[0026] The proposed method is an end-to-end 3D spatially variable lighting estimation pipeline that retrieves illuminance information at any 3D location within the scene. The pipeline uses color images and depth information received directly from sensors or estimated from images as input. The output is provided directly as a standard representation, natively supported by off-the-shelf renderers. The proposed method supports real-world AR applications through adversarial learning optimization, thus eliminating the need to collect expensive real-world illuminance data.

[0027] Specifically, the proposed method utilizes images and depth measurements to construct a 3D feature structure describing spatial variations in 3D lighting. This 3D feature structure (also known as an illuminance map) can be sampled at any location.

[0028] The proposed method is an end-to-end deep learning pipeline that includes discriminative projection operations that allow the use of color-specific input images (e.g., RGB, CMYK, etc.) and pixel-by-pixel depth measurements. The proposed method can sample lighting information at any 3D location in the scene, thereby performing complete 3D sampling of all visible points in the scene, not just the visible surfaces.

[0029] The sampling method for extracting information from the 3D scene relies on locating a feature structure (also known as a point cloud or illumination map) relative to the target point or selected location to be sampled. This requires no additional learnable parameters or model overhead, reducing memory requirements for memory-constrained handheld devices. By utilizing the shifting of the point cloud reference frame, multiple 3D locations can be sampled simultaneously. That is, multiple selected locations can be represented simultaneously by implementing the shifting of the reference frame. This can also improve the potential throughput of the application.

[0030] The proposed method can be optimized using color-specific image data and depth maps for adversarial examples to support practical applications. Results show that the proposed method can recover the egocentric orientation of the illumination source.

[0031] The core of the proposed method is that it allows for illumination estimation via spherical harmonic coefficient prediction and 3D sampling control. This method does not require multi-view or panoramic images. Furthermore, the method does not require the presence of known objects whose reflections are used for illumination retrieval.

[0032] Figure 1An overview of the first half 100 of the proposed pipeline is shown. This part of the pipeline includes the generation of the illuminance map 110. A color-specific image 102 provides raw illuminance data to an image-based computational model 104. The image-based computational model 104 acquires the color-specific image data 102 and generates a feature map 106 of the scene in the image. The feature map 106 includes the extraction of illuminance information contained in the color-specific image data 102. The image-based computational model 104 may be a neural network model. Depth data 108 is received by the pipeline and can then be combined with the feature map 106 to form the illuminance map 110. The illuminance map 110 represents an illuminance estimate for a set of three-dimensional locations in the scene. That is, the illuminance map includes information that enables the estimation of illuminance quality at various locations in three-dimensional space within the scene.

[0033] The color-specific image data can be received from the camera of the device performing the scene rendering or a camera connected to it. The depth data can be received from the device's depth sensor or received as an estimate based on the color-specific image data.

[0034] The proposed 3D variable illumination estimation method first receives a 2D color-specific image (e.g., 640x480 in size) as input.

[0035] The proposed method first feeds the color-specific image into a CNN to provide a feature map. In an exemplary implementation, the first four blocks of an existing neural network, DenseNet, are used for this purpose. The image data is then encoded in this step to provide a 20x15x256 feature map. The specific size of the feature map can be determined through ablation studies. The feature map 106 is optimized to describe scene lighting information.

[0036] Approximately midway through the pipeline, the proposed method uses depth measurement 108 to improve illumination retrieval and allow for target sampling with 3D spatial variations. The first two dimensions of the feature map (representing the spatial structure of the input image) are then projected into 3D space to generate a point cloud or illumination map. In the exemplary implementation described above, this would produce an illumination map of size 300 x (256 + 3), where 300 is the number of projected spatial locations and 256 is their corresponding features plus three egocentric spatial dimensions.

[0037] The generated illuminance map may include multiple illumination points, each representing one of the following for a corresponding pixel of the color-specific image: (i) illuminance level and (ii) illuminance hue. In three-dimensional space, the illuminance map may also include data representing the depth corresponding to each illumination point.

[0038] Figure 2 An overview of the latter half 200 of the proposed pipeline is shown. This portion of the pipeline determines the illuminance at a selected location 202 within the scene. To achieve this, the created illuminance map can be re-centered at any target location by setting the origin of the illuminance map 110 to the selected location 202 and creating a shifted illuminance map. The illuminance can then be extracted from the intermediate feature vector representation of the illuminance map 110 using an extraction neural network 204 (e.g., a PointNet network). That is, determining the illuminance at the selected location within the scene involves: shifting the reference frame of the illuminance map to center it at the selected location; and combining these illumination points based on the spatial distribution of the illumination points in the illuminance map around the selected location at the center of the reference frame. The feature vector representation of the illuminance at the selected location can then be extracted from the illuminance map using an extraction neural network model.

[0039] The generated feature vectors can then be processed. These feature vector representations are processed to generate a color-specific spherical harmonic representation 206, a depth spherical harmonic representation 208, and a geometric distance estimate of the illuminance at the selected location. Therefore, as a result of this processing, the proposed pipeline outputs a spherical harmonic (SH) representation of the scene's illuminance at the selected location, along with additional outputs related to the geometric distance estimate, also in the form of one or more SH coefficients. The additional outputs related to the geometric distance estimate can contribute to better occlusion estimation. Both SH representations 206 and 208 can use 36x3 coefficients, where 3 refers to the number of color channels in the color-specific image data, and 36 refers to the approximation of the color-specific image data.

[0040] Furthermore, the proposed method utilizes adversarial learning techniques to bridge the gap between synthetic and real images. To this end, a discriminator neural network is employed, trained to distinguish intermediate feature vectors corresponding to synthetic or real images. The gradients generated by this network are then processed through a gradient reversal layer (GRL) 210 before being used in the image-based computational model (when a neural network is included) and the extraction neural network. Specifically, implementing the discriminator neural network to verify the output of the processing pipeline includes: distinguishing the feature vectors corresponding to synthetic images from those corresponding to real images to generate gradients; processing the gradients through the gradient reversal layer; and optimizing the image-based computational model and the extraction neural network model using the processed gradients.

[0041] The proposed method generates synthetic data by combining a GPU-based ray tracing engine with realistic interior scenes. The interior 2D rendering is generated from a scene with randomized parameters. These randomized parameters can include various layouts, such as bedrooms, living rooms, bathrooms, and kitchens; walls and floors with randomized textures and material properties; random placement of objects (e.g., taken from SceneNet) with appropriate randomized textures and material properties; and different lighting arrangements, colors, and intensities.

[0042] For each of these renders, the light information can be sampled at four locations in the view space. The proposed method for generating the 2D dataset samples a point by passing light through each of the four quadrants of the render. While existing methods can then select locations closer to the surface, the 3D dataset proposed in this paper can be randomly sampled at any distance from the camera within the scene.

[0043] Figure 3 This demonstrates how to collect the lighting information by rendering a cube map around a selected location. The cube map can then be used to generate the target SH coefficients. Specifically, Figure 3 Three distinct selected locations within the image data 302a-c are shown. Each selected location 302a-c then has its own cubemap 304a-c, which are rendered centered on the corresponding selected location 302a-c. Figure 3 The resulting SH representations 306a-c for each of these cubic diagrams 304a-c are shown.

[0044] During training, rendering can be enhanced by performing different color corrections, adding Gaussian and salt-and-pepper noise, and horizontal flipping. The cube diagram is then modified accordingly.

[0045] The created training dataset can include 16,000 uncorrected images. Four probes are collected for each image, providing a total of 64,000 probes. These probes are collected from both above and below the surface. This provides overall 3D control. The training data includes real room layouts with random surface textures (e.g., 10 per room) and random light sources (e.g., 10 per room). Additionally, there is randomly placed furniture.

[0046] The loss function used to optimize the proposed pipeline is a combination of multiple losses, each of which will be explained in detail below.

[0047] First, the proposed method is optimized to estimate illumination information in SH format. To achieve this, the L2 distance between the predicted and ground truth 36x3 coefficients representing colored light "SH" is minimized to the greatest extent possible, according to the following equation:

[0048]

[0049] Where SH' and SH are the predicted and ground truth depth SH coefficients, respectively, and i refers to the SH order.

[0050] Furthermore, the distance between the depth SH coefficients "DSH" is minimized to the greatest extent possible according to the following equation:

[0051]

[0052] Where DSH' and DSH are the predicted and ground truth depth SH coefficients, respectively, and i refers to the SH order.

[0053] An adversarial task optimized using the GRL is employed. For example, the loss function used could be a simple binary cross-entropy loss:

[0054] L Adverserial = -log(c′)*c

[0055] Where c' and c are the binarization markers for prediction and ground truth domain classification.

[0056] These losses can be combined with equal weights.

[0057] Figure 4 An example scene 400 is shown, in which an artificial object 402 is located at a selected position within the scene. The object 402 is a sphere, rendered and illuminated according to its position within the scene. It can be seen that the object 402 is not visibly close to any particular surface, and its illumination is that of an object located in free space between objects within the scene (rather than in the location of another existing object). The left side of the sphere has a brighter area to match the brighter light in the left foreground of the scene, while the right and top surfaces of the sphere have more shadows to match the low ceiling and darker far corners of the scene.

[0058] Figure 5A series of man-made objects 502a-c are shown located at selected locations within scene 500. These objects 502a-c are spheres, with the smaller spheres farther from the viewer than the larger ones. The spheres have been rendered with varying levels of illumination based on their positions within the scene. It can be seen that objects 502a-c are not visibly close to any specific surface, and their illuminance is the same as that of objects in free space between objects within the scene (rather than in the location of another existing object). Similarly, the closer sphere 502a is brighter, representing the illumination of the front half of the room. The furthest sphere 502c is illuminated more to match the almost unlit rear of the room. It can be seen that sphere 502c is illuminated not only based on the background directly behind it, but also because the sphere has a different depth than the ground behind it; therefore, sphere 502c has been rendered with surface illuminance to maintain its 3D position within the scene.

[0059] Figure 6 An example camera is shown, which is used to implement the image processor to process images captured by the image sensor 1102 in camera 1101. Such a camera 1101 includes some onboard processing capabilities. This may be provided by the processor 1104. The processor 1104 may also be used for the basic functions of the device. The camera typically also includes a memory 1103.

[0060] Transceiver 1105 is capable of communicating with other entities 1110 and 1111 via a network. These entities may be physically located away from camera 1101. The network may be a publicly accessible network, such as the Internet. Entities 1110 and 1111 may be cloud-based. In one example, entity 1110 is a computing entity, and entity 1111 is a command and control entity. These entities are logical entities. In practice, each of them may be provided by one or more physical devices (e.g., servers and data storage areas), and the functionality of two or more of the entities may be provided by a single physical device. Each physical device used to implement an entity includes a processor and memory. The device also includes a transceiver for sending and receiving data to and from transceiver 1105 of device 1101. The memory stores code in a non-transitory manner, which can be executed by the processor to implement the corresponding entity in the manner described herein.

[0061] The command and control entity 1111 can train the neural network model used in the proposed method. This is typically a computationally intensive task, even though the obtained model can be efficiently described, thus enabling efficient development of the algorithm in the cloud, where abundant energy and computing resources are readily available. This is predictably more efficient than forming such a model on a typical camera. However, the above description, within the context of the currently proposed method, outlines the conditions for implementation on devices with limited memory resources.

[0062] In one implementation, after the algorithm is developed in the cloud, the command and control entity can automatically generate a corresponding model and transmit it to the relevant camera device. In this example, a set of pre-trained neural network models are implemented at the camera 1101 by the processor 1104.

[0063] In another possible implementation, the image can be captured by the camera sensor 1102, and the image data can be sent to the cloud by the transceiver 1105 for processing in the system. The generated target image can then be sent back to the camera 1101.

[0064] In another possible implementation, the image can be captured by the camera sensor 1102, and the image data and the depth data can be processed directly by the image processor of the device without the assistance of an external system.

[0065] Therefore, the method can be deployed in various ways, such as in the cloud, on the device, or in dedicated hardware. As mentioned above, cloud facilities can perform training to develop new algorithms or improve existing ones. Depending on the computing power near the data corpus, the training can be performed near the source data or in the cloud, for example, using an inference engine. The system can also be implemented at the camera, on dedicated hardware, or in the cloud.

[0066] The applicant hereby discloses individually each individual feature described herein, as well as any combination of two or more such features. With ordinary knowledge of those skilled in the art, such features or combinations can be implemented as a whole based on this specification, regardless of whether such features or combinations of features solve any of the problems disclosed herein, and without limitation to the scope of the claims. This application demonstrates that aspects of the invention can be constituted by any such individual features or combinations of features. Various modifications that can be made within the scope of the invention, given the foregoing description, will be apparent to those skilled in the art.

Claims

1. An apparatus (1101) comprising an image processor device, characterized in that, The image processor apparatus is configured to implement an image-based computational model (104) as part of an end-to-end processing pipeline for operating by: receiving color-specific image data (102) representing a scene; receiving depth data (108) for the scene; processing the color-specific image data using the image-based computational model to form a feature map (106) for the scene; forming an illumination map (110) from the feature map and the depth data, the illumination map representing an estimate of illumination at a set of three-dimensional locations in the scene; determining illumination at a selected location (202) within the scene comprises shifting a frame of reference of the illumination map to center it on the selected location, and combining light points of the illumination map based on their spatial distribution around the selected location at the center of the frame of reference.

2. The apparatus of claim 1, wherein, The image-based computational model (104) is a neural network model.

3. The apparatus of claims 1-2, wherein, The color-specific image data is received from a camera (1102) of the device.

4. The apparatus of any one of claims 1 to 3, wherein, The depth data is received from a depth sensor of the device.

5. The apparatus of any one of claims 1 to 3, wherein, The depth data is received as an estimate based on the color-specific image data.

6. The apparatus of claim 1, wherein, Multiple selected locations can be represented simultaneously by implementing the shifting of the frame of reference.

7. The apparatus of any one of claims 1 to 6, wherein, The illumination map (110) includes a plurality of light points, each representing one of (i) an illumination level and (ii) an illumination hue for a respective pixel of the color-specific image.

8. The apparatus of claim 7, wherein, The illumination map further includes data representing a depth corresponding to each light point.

9. The apparatus of any one of claims 1 to 8, wherein, A feature vector representation of the illumination at the selected location is extracted from the illumination map by a feature extraction neural network model (204).

10. A computer-implemented method for processing an image by an image processor device (1101), characterized in that, The image processor apparatus is configured to implement an image-based computational model (104) as part of an end-to-end processing pipeline (100, 200), the method comprising: receiving color-specific image data (102) representing a scene; receiving depth data (108) for the scene; processing the color-specific image data using the image-based computational model to form a feature map (104) for the scene; forming an illumination map (110) from the feature map and the depth data, the illumination map representing an estimate of illumination at a set of three-dimensional locations in the scene; determining illumination at a selected location within the scene by: shifting a frame of reference of the illumination map to center it on the selected location (202); combining light points of the illumination map based on their spatial distribution around the selected location at the center of the frame of reference.

11. The method of claim 10, wherein, includes extracting a feature vector representation of the illumination at the selected location from the illumination map by a feature extraction neural network model (204).

12. The method of claim 11, wherein, includes processing the feature vector representation to generate a color-specific spherical harmonic representation (206), a depth spherical harmonic representation (208), and a geometric distance estimate indication of illumination at the selected location.

13. The method of claim 12, wherein, The geometric distance estimate indication comprises one or more spherical harmonic coefficients.

14. The method according to claim 12 or 13, characterized in that, The spherical harmonic representations each comprise 36 coefficients representing respective degrees of approximation, each multiplied by 3 color channels.

15. The method according to any one of claims 11 to 14, characterized in that, comprising implementing a discriminator neural network (210) to validate an output of the processing pipeline by: differentiating the feature vector corresponding to a synthetic image from the feature vector corresponding to a real image to generate a gradient; processing the gradient through a gradient reversal layer; optimizing the image-based computational model and the extraction neural network model using the processed gradient.

16. The method according to any one of claims 10 to 15, characterized in that, the image-based computational model (104) is a neural network.

Images