Method for processing a 3D scene, corresponding device, system and computer program

Abstract

A method for processing a 3D scene, a corresponding device, system and computer program are disclosed. In an exemplary embodiment, the disclosed method comprises: obtaining an image captured by at least one camera and including at least a lowest point view of the 3D scene; detecting at least one shadow projected in the image by at least one object acting as a support for the at least one camera in the 3D scene; and determining a direction of at least one real light source based on the at least one detected shadow and at least information representing the object.

Description

[0001] Cross reference section

[0002] This application claims the benefit of European Patent Application No. 19305435.0, filed on April 2, 2019, entitled “A Method for Processing a 3D Scene, and Corresponding Device, System, and Computer Program”, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to the field of image processing.

[0004] The present invention provides a technique for processing a 3D scene, and in particular a technique for estimating at least one light source in a 3D scene. Such an estimation can be used to enhance the rendering of the 3D scene, for example when a virtual object is inserted into the 3D scene. Background Art

[0005] Mixed or augmented reality generally refers to adding computer-generated objects to a view of the real world. In order for the synthetic view to be realistic, the computer-generated objects must usually be associated with virtual shadows that are consistent with real shadows. This requires, for example, estimating the real light sources in the scene to reproduce them in the virtual world.

[0006] However, the real lighting of a scene can be very complex. Several solutions have been proposed to estimate the light source through image analysis.

[0007] For example, document EP3352138 discloses a solution according to which real shadows are detected in a standard image and then the light source is estimated, assuming that the camera pose of the image and the 3D geometric model of the scene are known.

[0008] However, this solution requires a pre-processing step to model the scene and estimate the image camera pose. Therefore, this solution is complex, resource-consuming and time-consuming. Summary of the invention

[0009] There is a need for a new technique for processing 3D scenes, and in particular for estimating at least one light source in a 3D scene, which technique overcomes at least one disadvantage of known techniques.

[0010] According to one aspect of the present disclosure, a method for processing a 3D scene is disclosed. The method comprises:

[0011] - Obtain an image of at least a nadir view of the 3D scene captured by at least one camera,

[0012] - Detect at least one shadow projected in the image by at least one object that serves as a support for one of the cameras in the 3D scene, and

[0013] - Determine the direction of at least one real light source based on the detected shadow(s) and at least information representing the object.

[0014] According to the present disclosure, it is thus possible to capture at least one view of the shadow generated by an object (e.g., a tripod) supporting a camera. Since some information representing the object is known (e.g., its height or the 3D model of the object), the direction of the light source(s) in the real 3D scene can be determined based on the shadow of the object (or based on the shadow of a set including the camera and the object serving as the support for the camera) and based on the information representing the object (e.g., the height of the object, or the height of the set including the camera and the object).

[0015] In particular, according to at least one embodiment of the present disclosure, such a method does not require any prior knowledge of the 3D scene.

[0016] According to another embodiment of the present invention, a device for processing a 3D scene is disclosed. Such a device includes at least one processor configured to execute:

[0017] - Obtain an image of at least a nadir view of the 3D scene captured by at least one camera,

[0018] - Detect at least one shadow projected in the image by at least one object that serves as a support for one of the cameras in the 3D scene, and

[0019] - Determine the direction of at least one real light source based on the detected shadow(s) and information representing the object.

[0020] Such a device may be particularly adapted to implement the above method for processing a 3D scene. For example, such a device is a mobile device, such as a smart phone, a tablet, smart glasses, a head-mounted display, etc. In particular, the camera that captures the 3D scene in the nadir view may be part of the device. It may be, for example, a simple camera, a wide-field-of-view camera, or a 360-degree camera.

[0021] Another aspect of the present disclosure relates to at least one computer program product downloadable from a communication network and / or recordable on a computer-readable medium and / or executable by a processor, comprising software code adapted to execute a method for processing a 3D scene, wherein the software code is adapted to execute at least one of the steps of the above-described method.

[0022] Furthermore, another aspect of the present disclosure relates to a non-transitory computer-readable medium comprising a computer program product recorded thereon and capable of being run by a processor, the program product comprising program code instructions for implementing a method for processing the previously described 3D scene.

[0023] At least one embodiment of the present invention also relates to a system comprising at least one camera and a device for processing a 3D scene. The camera comprises at least one processor configured to execute:

[0024] - capturing an image of at least a lowest point view of the 3D scene,

[0025] - sending the image to the device,

[0026] and the device comprises at least one processor configured to execute:

[0027] - receiving the image of the 3D scene,

[0028] - detecting at least one shadow projected by at least one object acting as a support for the camera in the 3D scene in the image, and

[0029] - determining the direction of at least one real light source based on the detected shadow(s) and information representing the object. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The present disclosure will be better understood and illustrated by the following embodiments and examples, with reference to the accompanying drawings, which in no way limit the present disclosure, wherein:

[0031] Figure 1 An example of a camera according to some embodiments is shown, the camera being located on an object acting as a support for the camera;

[0032] Figure 2 is a flowchart showing example steps of a method for processing a 3D scene according to some embodiments;

[0033] Figure 3 Another example of a camera located on an object acting as a support for the camera according to some embodiments is shown.

[0034] Figure 4Shows an example system for processing 3D scenes according to some embodiments;

[0035] Figure 5 is a block diagram of an example device for implementing a method for processing 3D scenes according to some embodiments.

[0036] In the drawings, the represented blocks are purely functional entities and do not necessarily correspond to physically separate entities. That is, they can be developed in the form of software, hardware, or implemented in one or more integrated circuits including one or more processors. Detailed implementation

[0037] It should be understood that the drawings and descriptions of the present disclosure have been simplified to show elements relevant to a clear understanding of the present disclosure, while many other elements found in typical cameras and mobile devices have been eliminated for clarity.

[0038] The present disclosure relates to the field of image processing. More specifically, the present invention proposes a technique for processing 3D scenes and specifically for estimating at least one light source in a 3D scene. For example, when a virtual object is inserted into a 3D scene, such an estimation can be used to enhance the rendering of the 3D scene.

[0039] The present disclosure is particularly applicable to any application in which a 3D scene must be rendered, such as mixed or augmented reality applications.

[0040] Main principle

[0041] Figure 1 Shows an example of a camera positioned on an object serving as a support for a camera according to some embodiments. As Figure 1 schematically shown, according to this example embodiment, at least one camera 11 is located on top of the object 12 such that the camera is not directly on the reference surface 13 on which the object 12 is placed (e.g., on the ground), but is elevated relative to the reference surface 13. The distance between the camera 11 and the reference surface 13 should be sufficient to enable the detection of the shadow(s) cast by the object 12 on the reference surface 13. This distance can vary according to the position of the light source in the 3D scene. As an example, when the light source is close to the horizon, a short distance between the camera and the reference surface is usually sufficient (e.g., a few centimeters). As another example, when the light source is close to the zenith, a longer distance between the camera and the reference surface should generally be used.

[0042] Furthermore, according to an embodiment of the present disclosure, the camera 11 is part of a device such as a smartphone. The smartphone can be positioned vertically, i.e., for example, in a position where its bottom side is resting on the reference surface 13 and its main height is perpendicular to the reference surface 13. In this case, the camera 11 is located at a certain distance from the reference surface 13, e.g., close to the main height of the smartphone. Such a camera can be, for example, a 360-degree camera. In this particular embodiment, the object 12 on which the camera rests can be the body of the smartphone.

[0043] Figure 2 An example method for processing a 3D scene according to some embodiments is shown. Such a 3D scene is a real scene illuminated by a set of lights, and the set of lights will be estimated, for example, in order to solve augmented / mixed reality applications. By way of example, reference Figure 1 is made to Figure 2 illustrate the

[0044] In step 21, an image of at least the lowest point view of the 3D scene is obtained. For example, the image is at least captured by the camera 11. In some embodiments, such an image at least represents the lowest point view of the 3D scene, i.e., for example, a view from the camera 11 in a direction pointing below the camera 11 (or towards the center of the earth). In this way, the shadow(s) cast by the object 12 or by the set including the camera 11 and the object 12 are visible in the image. The camera support actually typically produces one or several longitudinal shadows on the reference surface below the camera. The image can be, for example, a standard image, a large field of view image, a 360° image obtained from one or more images captured by one or more cameras.

[0045] According to an embodiment of the present invention, an image of at least the lowest point view of the 3D scene can be obtained by combining a standard image representing the lowest point view of the camera 11 with a large field of view image representing the environment captured by the camera or another camera.

[0046] In step 22, at least one shadow 14 is detected in the image, which is cast by the object 12 or by the set including the camera 11 and the object 12. Different techniques can be used to detect or extract shadows in the image, such as those disclosed by the following: I. Sato et al. in "Illumination from Shadows" (PAMI’02), A. Panagopoulos et al. in "Robust Shadow and Illumination Estimation Using a Mixture Model" (CVPR’09), or the techniques disclosed by the following: A. Panagopoulos et al. in "Illumination Estimation and Cast Shadow Detection through a Higher-order Graphical Model" (CVPR’11).

[0047] In step 23, based on the detected shadow(s) 14 and at least the information representing the object 12 (e.g., the height of the object 12 or the height of the set including the camera 11 and the object 12, or the 3D model of the object 12 or the 3D model of the set including the camera 11 and the object 12), the direction D of at least one real light source is determined (see Figure 1 ). The color and / or intensity of the light source(s) can also be estimated.

[0048] For example, as Figure 1 shown, the object 12 supporting the camera 11 is a vertical support with a main vertical axis. It can include three legs resting on the reference surface 13 for stability. The length of the shadow 14 cast by the object 12 (or the set of the camera plus the object) is measured in the image. By knowing the information representing the object 12 (or the set of the camera plus the object), such as its height or 3D model, the direction D of the light source can thus be determined.

[0049] It should be noted that the reference surface may be planar or non-planar. If the reference surface is planar, it may be easier to measure the length of the shadow cast by the object, but if the reference surface is non-planar, such as if the 3D model of the reference surface is known, such a length can also be determined. In the latter case, once the shadow cast by the object acting as the support of the camera is detected, the 3D position of the end of the shadow can be derived from the 3D model of the reference surface. Then, the 3D orientation of the light source can be derived from the 3D positions of both the end of the shadow and the top of the object. The 3D model of the reference surface can be obtained, for example, from a depth sensor associated with the camera.

[0050] However, in some embodiments, it is desirable that the height of the point located at the end 141 of the shadow 14 cast by the object 12 (see Figure 1 ) is the same as the height of the reference surface 13 at the position where the object 12 rests on the reference surface 13. According to this example, the reference surface 13 is mainly horizontal.

[0051] The object 12 supporting the camera 11 may also have a main vertical axis and a planar base, such as a circular planar base, which at least partially rests on the reference surface 13. According to this embodiment, the shadow(s) of the object 12 may be cast on the planar base.

[0052] According to another aspect of the present disclosure, the position of the real light source(s) can be determined from the detected shadow(s). The color and / or intensity of the light source(s) can also be estimated. For example, the position, color, and / or intensity of the real light source(s) can be determined by considering the shape of the object and the detected shadow or potentially the shape of the reference surface. Figure 3 Another example of a camera located on an object acting as a support for the camera according to some embodiments is shown.

[0053] According to this aspect, as Figure 3 illustrated by way of example, the object 32 acting as the support for the camera 31 has a vertical axis but also has at least one branch 321 in another plane (e.g., horizontal).

[0054] In this way, as shown in the figure, the object 32 projects a number of shadows, at least one shadow 34 is projected by the vertical axis of the object 32, and at least one shadow 341 is projected by the at least one branch 321. Using the information representing the object 32 and such shadows, the 3D position 35 of the light source can be determined, for example, at the intersection of the direction of the light source estimated from the shadow 34 projected according to the vertical axis of the object 32 and the direction of the light source estimated from the shadow 341 projected by the at least one branch 321. Different techniques can be used to determine the 3D position of the light source(s), such as those disclosed in the previously mentioned document EP3352138.

[0055] In other words, according to at least one embodiment of the present disclosure, at least one of the direction, color, and intensity of the light source(s) of the real 3D scene can be estimated using at least the information representing the object, in a reference system defined by the position of the object that serves as the support of the camera. If the object that serves as the support of the camera is not just a vertical leg (for example, it also has branches), the 3D position of the light source(s) (such as a street lamp) can be estimated.

[0056] Furthermore, in some embodiments, in order to obtain a more realistic rendering of the 3D scene, the estimation of the light source can be combined with the determination of the environment map.

[0057] In this way, according to another embodiment of the present invention, the image of at least the lowest point view of the 3D scene is a large field of view image, such as a 360° image. For example, such an image can be obtained by combining several images obtained by the same or different cameras. In one example, using one camera, a large view image or a standard image is captured in the lowest point view of the camera (i.e., in the direction pointing below the camera or towards the center of the earth), and a large view image is captured in the highest point view of the camera (i.e., in the opposite direction, pointing above the camera).

[0058] According to this embodiment, the environment map can be determined from the image of the 3D scene (such as a 360° image). Such an environment map is a texture image that can be used, for example, to approximate the appearance of reflective surfaces. In this way, the present invention provides a solution for simultaneously estimating the light source direction (for consistent shadows in the rendering of the 3D scene) and determining the environment map (for consistent reflections in the rendering of the 3D scene).

[0059] Specifically, according to one embodiment, determining the environment map includes: removing the object(s) that serve as the support of the camera(s) and the detected shadow(s) from the image of the 3D scene and processing the occluded regions.

[0060] According to this embodiment, when rendering a 3D scene on, for example, the screen of a device, the object serving as the support of the camera and its shadow can be removed and further replaced to provide a realistic aspect of the 3D scene.

[0061] According to another embodiment, at least one bright area is detected in the environmental map. The direction of at least one light source candidate can be determined from this bright area, and at least one virtual shadow cast by the object illuminated by the light source candidate can be detected in the representation of the 3D scene. Also, based on the virtual shadow(s), the direction of the at least one real light source is determined.

[0062] Such an embodiment aims to increase the robustness of light source direction estimation by combining the use of the environmental map with a shadow-based method.

[0063] For example, at least two cameras can be used: one for capturing the 3D scene at least in a nadir view and one for capturing the environment.

[0064] According to this embodiment, the environmental map can be used to detect candidate light sources corresponding to the bright areas in the environmental map. The observation of the environment can actually be used to identify the light sources and possibly the direction of the light sources (usually area lights). These area lights can be extracted and discretized, and a set of distributed point lights can be selected on each area light. Each point light selected in the environmental map provides a 3D direction.

[0065] The virtual scene corresponding to the 3D scene can be illuminated with such point lights, thereby generating shadows in the virtual scene, especially the shadow(s) cast by the object serving as the support of the camera in the 3D scene. For example, by comparing the virtual shadow map and the real shadow map obtained from the image in the nadir view, the shadow(s) in the virtual scene can be compared with the shadow(s) in the real 3D scene. When the shadow(s) in the virtual scene are similar to the shadow(s) in the real scene, the direction or position of the 3D point light is confirmed.

[0066] According to the first example, it can be assumed that the point lights are at a given distance that may be infinite, and the virtual scene is illuminated using this set of virtual point lights.

[0067] According to a second example, the 3D position of a point light is estimated. Each 3D direction is in turn discretized: this defines candidate 3D positions for the corresponding point lights. Thus, a set of candidate 3D point lights can be selected by discretizing a set of 3D directions. Then, this set of candidate 3D point lights can be used as an input for 3D point light estimation (e.g., as disclosed in document EP3352138).

[0068] According to a third example, the area light is considered to be approximately planar, and all the selected point lights of a given area light may also lie at the same distance from the camera of the capture environment. Thus, the selection of the 3D point lights is constrained, and the resulting set is used to estimate the 3D point lights as described above.

[0069] According to a fourth example, the detected area lights and their 3D directions are used to define a volume containing the 3D point lights, and this volume is directly discretized to define the set of candidate 3D point lights, which is subsequently used as an input for the 3D point light estimation.

[0070] According to a fifth example, a pair of cameras is used to capture the environment, and a third camera is used to capture the 3D scene. In this case, stereo vision can be applied to the pair of cameras in order to estimate the 3D position of the light regions. For example, the corners of the boundaries of the area lights can be used to estimate their 3D positions, and a planar constraint can be added independently for each such estimate of an area light. Then, each area light can be discretized in the environment map (as described above), now resulting in point light sources whose 3D positions are derived from the 3D positions of the area lights. Then, this set of 3D point lights can be the input of a linear system, and their intensities can be estimated.

[0071] Since this allows a significant reduction in the number of 3D point light candidates, each area light can be discretized possibly more densely. Then, in this case, a shadow-based method can be used as in EP3352138, for example, to select the best point lights and estimate their intensities.

[0072] Note that the estimation of the (one or more) light sources can be updated periodically by capturing new images. If the light source is static, the camera and the object serving as the support for the camera can be removed.

[0073] According to some embodiments, a method for processing a 3D scene and in particular the estimation of the (one or more) light sources of a 3D scene according to different aspects described herein can be implemented by a camera including, for example, a camera SDK (Software Development Kit). According to some embodiments, it can be implemented by an external image processing toolbox. In addition, according to some embodiments, it can be implemented by a mobile device to process at least one image captured by at least one camera. In the latter case, the estimation of the (one or more) light sources can be provided to the rendering engine of the mobile device.

[0074] Applied to augmented reality

[0075] The estimation of the (one or more) light sources of a 3D scene and a possible environment map as described in the various embodiments disclosed herein can be used to obtain a realistic rendering of the 3D scene, especially in a mixed or augmented reality application where virtual objects are inserted into the 3D scene.

[0076] Figure 4 An example system for processing a 3D scene according to some embodiments is shown. As Figure 4 shown, the system includes at least one camera 41 located on top of an object, such as Figure 1 or as shown in 3 (but not explicitly shown in Figure 4 ), and a device 42, such as a mobile phone, a tablet, a head-mounted display, etc. This example system can be used, for example, in an augmented reality (AR) application.

[0077] In Figure 4 the embodiment illustrated, the camera 41 and the mobile device 42 are shown as different devices. However, according to another embodiment, the camera 41 can alternatively be the camera of the mobile device 42 itself.

[0078] The camera 41 can be placed in the region of interest. The camera 41 can capture an image of at least the lowest point view of the 3D scene and can send the image to the device 42. To do so, the camera 41 and the device 42 can communicate, for example, using WiFi or Bluetooth technology, either wired or wirelessly. The image capture can be triggered manually, periodically, or using, for example, the device 42. In addition, for example, the camera 41 can be a 360-degree camera, and the image captured by the camera 41 can be a 360° image.

[0079] According to at least one embodiment described herein, the device 42 may receive an image of a 3D scene captured by at least the camera 41, detect at least one shadow projected in the image by at least one object of the 3D scene that serves as a support for the camera 41, and determine the direction of at least one real light source based on the detected shadow(s) and at least information representing the object and known to the device 42. These steps may be implemented in the lighting estimator 421 of the device 42.

[0080] According to at least one embodiment described herein, the device 42 may also determine an environment map from images captured by the camera 41 and / or by another camera. According to at least one embodiment, such an environment map may be used to determine the direction of the light source(s) in the scene.

[0081] According to one aspect of the present disclosure, when the light source(s) of a 3D scene have been estimated, at least one of the following may be stored in the memory of the device 42: (i) a lighting model determined based on at least one of the direction, position, or intensity of the real light source(s), and (ii) the environment map.

[0082] For example, both the lighting model and the environment map may be defined in a reference frame that may be attached to an object that serves as a support for the camera 41, or may be attached to the camera 41.

[0083] The lighting model and / or the environment map locally stored on the device 42 may then be accessed by, for example, a rendering engine 422 installed on the device 42 for augmented reality applications such as inserting an object or an animation at a given location.

[0084] For example, an image captured by the camera 41 and displayed on the screen of the device 42 may be mixed with virtual objects. To insert a virtual object in the 3D scene and obtain a realistic rendering of the resulting 3D scene, the virtual object should ideally be illuminated in a manner consistent with the real lighting. To this end, the lighting model may be registered with the pose of the camera 41.

[0085] Thus, one aspect of the present disclosure includes: (i) positioning the environment map and at least one virtual light source determined from the lighting model in a virtual scene, (ii) inserting a virtual object in the virtual scene, and (iii) rendering the virtual scene.

[0086] Such positioning includes, for example, determining a transfer function from a reference frame attached to the camera 41 or to an object that serves as a support for the camera 41 to a reference frame attached to the device 42.

[0087] According to one embodiment, an object serving as a support for the camera 41 is used to define a reference 3D coordinate system, with reference to which the direction / position of the 3D light source and the orientation of the environmental map are defined.

[0088] For example, a physical image marker can be used to link the light source direction estimated by the camera 41 with the light source direction used by the device 42. This marker located on the reference surface or attached to the object serving as the support can be observed by the devices 41 and 42. Based on the analysis of the image of the device 41, the 3D direction / position of the light source is represented in the reference coordinate system attached to the marker. The camera of the device 41 can be pre-calibrated to obtain its intrinsic parameters. Then, based on the analysis of the image of the device 42, the pose of its camera is estimated in the same reference coordinate system.

[0089] According to a particular embodiment, the object serving as a support for the camera 41 includes an axis, such as a vertical leg, and the positioning includes: aligning the axis of the object with a direction known to the device 42.

[0090] According to a first example, a physical marker image can be aligned with the axis of the object serving as a support for the camera 41 to achieve the 3D coordinate system in which the light source is modeled. The AR application can be configured to detect the marker and align the virtual scene with the marker.

[0091] According to a second example, the axis or branch of the object serving as a support for the camera 41 can be configured to face the north direction indicated by the device 42 (e.g., by its compass). According to this example, the coordinate system in which the light source is modeled can be known in advance.

[0092] According to a third example, the estimation of the (one or more) light sources of the 3D scene can also include a registration step, in which the user aligns the direction of his / her device (such as the device 42) with the direction of the axis or branch of the object serving as a support for the camera 41 (e.g., via the display of the device's camera), and then presses a button to record the direction.

[0093] Once the environmental map and / or the light source have been correctly positioned in the virtual scene, virtual objects can be inserted into the virtual scene and correctly illuminated to provide a realistic aspect of the 3D scene.

[0094] When the transfer function between the reference frame of the device and the reference frame of the object serving as a support for the camera has been determined, the object serving as a support for the camera and the camera can be removed from the real 3D scene.

[0095] In some embodiments, the object serving as the support for camera 41 and camera 41 itself can also be removed from the virtual 3D scene. It should be understood that there are several known techniques for removing camera equipment or camera supports from, for example, a 360° lens.

[0096] As described above, the estimation regarding the light source(s) can be updated periodically, for example, by capturing and sending new images to device 42. If the light source is static, camera 41 and the object serving as the support for the camera can be removed.

[0097] For illustrative purposes, examples of the applications of the various embodiments described herein are now given.

[0098] Suppose a user wants to play an augmented reality (AR) game on his / her mobile phone, where virtual objects are inserted in front of him / her. Before starting the game, the user places his / her 360° camera on a support (such as a tripod) in the area (3D scene) where the user wants to play. The user can use his / her mobile phone to estimate the light source(s) of the 3D scene, which can be done, for example, by using a dedicated application that includes the various methods described herein. Such an application can be configured to connect the phone to the 360° camera, trigger the capture of a 360° image, upload the image and then calculate both the environmental map and the main light source direction. Then, this information can be locally stored on the phone. Additionally, when the user starts the game, the rendering engine (e.g., rendering engine 422) obtains the lighting information from the phone itself to generate realistic shadows and reflections.

[0099] Figure 5 is a block diagram of an example device for implementing a method for processing a 3D scene according to some embodiments.

[0100] As Figure 5 shown, such a device includes, for example, a non - volatile memory 53 (e.g., read - only memory (ROM) (as shown) or a hard disk), a volatile memory 51 (e.g., random access memory (RAM), as shown), and at least one processor 52, such as processor P1. The non - volatile memory 53 is a non - transitory computer - readable carrier medium. The memory 53 can be configured to store executable program code instructions that are executed by the processor 52 to implement the methods described herein according to various embodiments.

[0101] In particular, in some embodiments, the processor 52 is configured to perform: (i) obtain an image of at least a lowest point view of a 3D scene captured by at least one camera, (ii) detect at least one shadow projected in the image by at least one object of the 3D scene that serves as a support for the at least one camera, and (iii) determine the direction of at least one real light source based on the detected shadow(s) and at least information representing the object(s).

[0102] In some embodiments, at initialization, the above program code instructions are transferred from the non-volatile memory 53 to the volatile memory 51 for execution by the processor 52. The volatile memory 51 also includes registers for storing variables and parameters required for this execution.

[0103] As a general overview, various embodiments have been described above.

[0104] According to some embodiments, a method for determining information representing a light source in a 3D scene may include: obtaining an image captured by at least one camera that includes at least a lowest point view of the 3D scene; detecting at least one shadow projected in the image by at least one object of the 3D scene that serves as a support for the at least one camera; and determining the direction of at least one real light source based on at least one detected shadow and at least information representing the object(s).

[0105] In some embodiments, the method may further include: determining an environment map from the image of the 3D scene, the environment map including a texture image for approximating the appearance of a reflective surface. In some embodiments, determining the environment map includes: removing the at least one object and the at least one detected shadow from the image of the 3D scene, and replacing the at least one object and the at least one detected shadow to provide a realistic aspect of the 3D scene.

[0106] Furthermore, in some embodiments, the method may also include: detecting at least one bright region in the environment map; determining the direction of at least one light source candidate from the bright region; and detecting at least one virtual shadow projected by the object in the representation of the 3D scene and illuminated by the at least one light source candidate, wherein the direction of at least one real light source is further determined based on the at least one virtual shadow. In some embodiments, the method may also include: determining at least one of the position and intensity of the at least one real light source based on the at least one detected shadow.

[0107] Additionally, in some embodiments, the method may include storing at least one of the following: (i) an illumination model determined from at least one of the direction, position, and intensity of the at least one real light source, and (ii) the environmental map defined in a reference frame. In this regard, in some embodiments, the method may further include: positioning the environmental map and at least one virtual light source determined from the illumination model in a virtual scene; inserting virtual objects in the virtual scene; rendering the virtual scene.

[0108] In some embodiments, the information representing the object includes the height of the object. Additionally, in some embodiments, the image is a 360° image. Additionally, in some embodiments, the object serving as the support for the camera has an axis, and the positioning in the virtual scene includes: aligning the axis of the object with a direction known to the device associated with the at least one camera.

[0109] According to some embodiments, a non - transitory computer - readable medium may be configured to store program instructions that, when executed by a processor, cause the processor to perform any of the methods disclosed herein.

[0110] According to some embodiments, a device for determining information representing a light source in a 3D scene may include at least one processor configured to perform any of the methods disclosed herein.

[0111] Furthermore, according to some embodiments, a device for rendering a virtual scene may include a processor configured to determine information about a light source using any suitable method disclosed herein, wherein the processor may further be configured to: position the environmental map and at least one virtual light source determined from the illumination model in the virtual scene; insert virtual objects in the virtual scene; render the virtual scene. In such a device, in some embodiments, the object serving as the support for the camera has an axis, and the positioning includes: aligning the axis of the object with a direction known to the device.

[0112] Furthermore, in the devices disclosed above, the camera may be part of any such device.

[0113] Additionally, according to some embodiments, a system may include: at least one camera; and the device disclosed above. In this regard, in some embodiments, the at least one camera includes at least one processor configured to perform: capturing the image including the at least lowest - point view of the 3D scene; and sending the image to the device.

[0114] Note that, in some embodiments, the various steps of the method for processing a 3D scene according to at least one embodiment of the present invention as described above can also be implemented by the following:

[0115] - By executing a set of program code instructions performed by a reprogrammable computer such as a PC-type device, a DSP (Digital Signal Processor), or a microcontroller. The program code instructions can be stored in a removable (e.g., floppy disk, CD-ROM, or DVD-ROM) or non-removable non-transitory computer-readable carrier medium; or

[0116] - By a dedicated machine or component, such as an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or any dedicated hardware component.

[0117] However, note that the present disclosure is not limited to a purely software-based implementation in the form of computer program instructions operating on a processor, but rather it can also be implemented in hardware form or any form combining a hardware part and a software part.

[0118] The flowcharts and / or block diagrams in the accompanying drawings illustrate the possible configurations, operations, and functions of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram can represent a module, segment, or portion of code that includes one or more executable instructions for implementing the specified logical function(s).

[0119] It should also be noted that, in some alternative implementations, the functions recited in the blocks may not occur in the order mentioned in the figures. For example, two consecutive blocks shown may actually be executed substantially simultaneously, or these blocks may sometimes be executed in the reverse order, or these blocks may be executed in an alternative order, depending on the functions involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by a dedicated hardware-based system or a combination of dedicated hardware and computer instructions performing the specified functions or actions. Although not explicitly described, the embodiments of the present invention can be employed in any combination or sub-combination.

Claims

1. A method for determining information representing a light source in a 3D scene, the method comprises: obtaining an image captured by at least one camera and including at least a lowest point view of the 3D scene; detecting at least one shadow projected in the image by at least one object serving as a support for the at least one camera in the 3D scene; and determining a direction of at least one real light source based on the at least one detected shadow and based on information of at least a 3D model including the object representing the object.

2. The method according to claim 1, further comprises: determining, from the image of the 3D scene, an environment map, the environment map including a texture image to approximate an appearance of a reflective surface.

3. The method according to claim 2, wherein determining the environment map comprises: removing the at least one object and the at least one detected shadow from the image of the 3D scene, and replacing the at least one object and the at least one detected shadow to provide a realistic aspect of the 3D scene.

4. The method according to claim 2 or 3, further comprises: detecting at least one bright area in the environment map; determining a direction of at least one light source candidate from the bright area; and detecting at least one virtual shadow projected by the object and illuminated by the at least one light source candidate in a representation of the 3D scene, wherein the direction of the at least one real light source is further determined based on the at least one virtual shadow.

5. The method according to any one of claims 1 to 3, further comprises: determining at least one of a position and an intensity of the at least one real light source based on the at least one detected shadow.

6. The method according to claim 5, further comprises storing an illumination model determined based on the direction of the at least one real light source and at least one of the position and the intensity of the at least one real light source.

7. The method according to any one of claims 1 to 3, wherein the information representing the object includes a height of the object.

8. The method according to any one of claims 1 to 3, wherein the image is a 360° image.

9. The method according to claim 6, further comprises: locating at least one virtual light source determined from the illumination model in a virtual scene; inserting virtual objects in the virtual scene; and rendering the virtual scene.

10. The method according to claim 9, wherein the object serving as a support for the camera has an axis, and wherein the locating comprises aligning the axis of the object with a direction known to a device associated with the at least one camera.

11. A non - transitory computer - readable medium configured to store program instructions, the program instructions when executed by a processor cause the processor to execute the method according to any one of claims 1 - 3.

12. A device for determining information representing a light source in a 3D scene, wherein the device includes at least one processor configured to execute: Obtain an image of the at least lowest point view of the 3D scene captured by at least one camera; Detect at least one shadow projected by at least one object that serves as a support for the at least one camera in the 3D scene in the image; And Determine the direction of at least one real light source based on the at least one detected shadow and based on information representing at least a 3D model of the object that includes the object.

13. The apparatus according to claim 12, Wherein, The at least one processor is further configured to execute, from the image of the 3D scene, to determine an environment map that includes a texture image to approximate the appearance of a reflective surface.

14. The apparatus according to claim 13, wherein determining the environment map Includes: Removing the at least one object and the at least one detected shadow from the image of the 3D scene, and replacing the at least one object and the at least one detected shadow to provide a realistic aspect of the 3D scene.

15. The apparatus according to claim 13 or 14, Wherein, The at least one processor is further configured to execute: Detect at least one bright area in the environment map; Determine the direction of at least one light source candidate from the bright area; and Detect at least one virtual shadow projected by the object and illuminated by the at least one light source candidate in a representation of the 3D scene, wherein the direction of at least one real light source is further determined based on the at least one virtual shadow.

16. The apparatus according to any one of claims 12 to 14, Wherein, The at least one processor is further configured to determine at least one of the position and intensity of the at least one real light source based on the at least one detected shadow.

17. The apparatus according to claim 16, wherein the at least one processor is further configured to execute storing an illumination model determined from the direction of the at least one real light source and at least one of the position and intensity of the at least one real light source.

18. The apparatus according to any one of claims 12 to 14, Wherein, The information representing the object includes the height of the object.

19. The apparatus according to any one of claims 12 to 14, wherein the image is a 360° image.

20. An apparatus for rendering a virtual scene, comprising a processor configured to use the method according to claim 6 to determine information about a light source, wherein the processor is further configured to: Locate at least one virtual light source determined according to the illumination model in the virtual scene; Insert virtual objects in the virtual scene; and Render the virtual scene.

21. The apparatus according to claim 20, Wherein, The object that serves as a support for the camera has an axis, and wherein the locating includes aligning the axis of the object with a direction known to the apparatus.

22. The apparatus according to any one of claims 12 to 14, Wherein, The camera is part of the apparatus.

23. A system, Comprising: At least one camera; And The apparatus according to any one of claims 12 to 14.

24. The system according to claim 23, wherein the at least one camera includes at least one processor configured to perform: capturing the image including the at least lowest point view of the 3D scene, and sending the image to the apparatus.

Images