Method and apparatus for adjusting scene rendering

Abstract

It is disclosed that, according to embodiments, a (e.g., multiple) lighting model can be computed from a scene model by a processing device. The model (e.g., a geometric model of a scene supplemented by a lighting model) can be stored, e.g., on the processing device. The processing device can be coupled with a user interface running on either the processing device or another (e.g., Tenderer) device. According to embodiments, a (e.g., particular) scene, e.g., of a set of possible scenes, can be selected via the user interface for rendering by an AR application on either the processing device and (e.g., different) Tenderer device. According to embodiments, (e.g., particular, virtual) outdoor lighting conditions can be selected via the user interface, and rendering of the selected scene can be adjusted by the AR application according to the selected outdoor lighting conditions.

Description

1. Technical Field

[0001] This disclosure relates to the field of environmental modeling, and in particular to light modeling for various applications such as augmented reality applications. 2. Background Technology

[0002] Interior design applications help users select specific furniture for a room by rendering a mixed scene of the room, where different virtual furniture elements can be virtually integrated into the room. When the room's interior includes openings to the outdoor environment (e.g., windows), the lighting effects on the furniture elements can influence the design choices. Users can select furniture elements for a room by running the interior design application within the room, for example, during cloudy and rainy days. After acquiring the furniture elements, the user might then be disappointed with the lighting effects of the elements, for example, during sunny days. This disclosure was designed with the foregoing in mind. 3. Summary of the Invention

[0003] According to the implementation scheme, a processing device can compute (e.g., multiple) lighting models from a scene model. Models (e.g., geometric models of the interior scene supplemented by the lighting models) can be stored on the processing device, for example. The processing device can be coupled to a user interface running on either the processing device or another (e.g., a renderer) device. According to the implementation scheme, a (e.g., specific) interior scene from a set of possible interior scenes can be selected via the user interface for rendering by an AR application on either the processing device or a (e.g., different) renderer device. According to the implementation scheme, (e.g., specific, virtual) outdoor lighting conditions can be selected via the user interface, and the rendering of the selected interior scene can be adjusted by the AR application based on the selected outdoor lighting conditions. 4. Description of the attached drawings

[0004] - Figure 1 This is an illustration demonstrating an example of augmented reality applications based on multiple simulated scene lighting conditions;

[0005] - Figure 2 This is an illustration of an example of a (e.g., indoor) lighting estimation processing module;

[0006] - Figure 3A This is an illustration of an example of a processing device used to adjust the rendering of an interior scene;

[0007] - Figure 3B This is an illustration of another example of a processing device used to adjust the rendering of an interior scene;

[0008] - Figure 4 It means Figure 3A and Figure 3B An illustration of an exemplary architecture of a processing device in any of the figures.

[0009] - Figure 5 This is an illustration of an example of a method used to adjust scene rendering.

[0010] It should be understood that the accompanying drawings are intended to illustrate the concepts of this disclosure and are not necessarily the only possible configurations for illustrating this disclosure. 5. Detailed Implementation

[0011] It should be understood that the elements illustrated in the figures can be implemented in various forms of hardware, software, or combinations thereof. Preferably, these elements are implemented in a combination of hardware and software on one or more appropriately programmed general-purpose devices, which may include a processor, memory, and input / output interfaces. In this document, the term "interconnect" is defined as a direct connection or an indirect connection via one or more intermediate components. Such intermediate components may include both hardware-based and software-based components. The term "interconnect" is not limited to wired interconnects but also includes wireless interconnects.

[0012] All examples and conditional language described herein are intended for pedagogical purposes to help the reader understand the principles of this disclosure and the concepts contributed by the inventors to the advancement of the art, and should be interpreted as not being limited to such specific examples and conditions.

[0013] Furthermore, those skilled in the art will understand that the block diagrams presented herein represent conceptual diagrams of illustrative circuits embodying the principles of this disclosure. Similarly, it should be understood that any flow charts, flow diagrams, state transition diagrams, pseudocode, etc., represent various processes that can be substantially represented in a computer-readable medium and executed by a computer or processor, whether or not such computer or processor is explicitly depicted.

[0014] The functionality of the various components illustrated in the figure can be provided using dedicated hardware and hardware capable of executing software in association with appropriate software. When provided by a processor, the functionality can be provided by a single dedicated processor, a single shared processor, or multiple separate processors, some of which may be shared. Furthermore, the explicit use of the terms "processor" or "controller" should not be construed as referring to hardware capable of executing software, and may implicitly include, but is not limited to, digital signal processor (DSP) hardware, read-only memory (ROM), random access memory (RAM), and non-volatile storage devices for storing software.

[0015] Other conventional and / or custom hardware may also be included. Similarly, any switches illustrated in the figures are conceptual only. Their functionality can be achieved through the operation of programmable logic, through dedicated logic, through interaction between programmable control and dedicated logic, or even manually, as can be selected by the implementer using specific technologies, as understood more specifically from the context.

[0016] In the claims herein, any element expressed as a means for performing a specified function is intended to cover any manner in which that function is performed, including, for example, a) a combination of circuit elements performing that function, or b) software of any form, thus including firmware, microcode, or the like, combined with appropriate circuitry for performing that software to perform that function. The disclosure as defined in these claims lies in the fact that functions provided by the various described means are combined and brought together in the manner claimed in the claims. Therefore, any means that can provide those functions is considered equivalent to those illustrated herein.

[0017] It should be understood that, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the following “ / ,” “and / or,” and “at least one” is intended to cover selecting only the first listed option (A), or only the second listed option (B), or selecting both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” such phrases are intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or selecting all three options (A, B, and C). As will be apparent to those skilled in the art and related fields, this can be extended to as many items as possible listed.

[0018] The implementation schemes described herein can relate to any of augmented, mixed, and diminished reality applications. For clarity, the implementation schemes described herein are illustrated with examples of augmented reality applications, but they can also be applied to any of mixed and diminished reality applications. Augmented reality (AR) applications enable interactive experiences in real-world environments, where objects residing in the real world can be augmented through computer-generated perceptual information, such as virtual objects inserted into either images or models of the real-world environment.

[0019] According to the implementation plan, a hybrid interior scene can include a modeled real interior scene and additional virtual furniture. For example, realistic rendering of such a hybrid interior scene under various lighting conditions can allow for improved realism and efficiency in interior design applications. For example, realistic lighting triggered by outdoor lighting and constrained by a real interior scene during an experience can allow for improved realism in AR applications. In another example, outdoor lighting can be simulated (e.g., modeled) at any time of day and / or under various weather conditions (sunny, cloudy, etc.), in sunlight, and / or in combination with artificial lighting to obtain a lighting model targeted at a specific interior scene.

[0020] According to the implementation plan, the AR application could be, for example, an interior room design application that allows for the realistic rendering of a hybrid interior scene consisting of a modeled real interior scene and attached virtual furniture on a display device (e.g., a tablet) under various lighting conditions. The AR application can allow users to evaluate possible interior design solutions using various furniture, surface properties, and lighting conditions.

[0021] AR indoor scene lighting and high-quality rendering can be based on baking techniques, where, for example, lighting effects can be baked as textures during the modeling phase (e.g., the texture of an object in the model can be calculated based on the current lighting of the corresponding object in the scene). For example, baking lighting effects in the texture of the modeled scene can prevent AR applications from adjusting lighting effects at runtime without any computationally intensive model updates. For example, baking lighting effects in the texture can prevent AR applications from introducing new virtual (e.g., realistically lit) objects, thus moving existing objects in the modeled scene (to different lighting areas) without any computationally intensive model updates.

[0022] Throughout the implementation described herein, the terms “part of the model,” “model element,” and “basic model” are used interchangeably to describe any fragment or subset of a scene’s model that may include information about a particular part of the scene modeling (e.g., functionality, focus).

[0023] According to the implementation scheme, either direct or ambient lighting can be obtained (e.g., estimated) from an image of the scene based on either, for example, shadows cast by objects and specular effects on surfaces. For example, shadows cast by objects onto adjacent surfaces can be detected. A set of virtual shadows (e.g., via 3D rendering from scene geometry) cast by a set of virtual point lights at different (e.g., predefined) 3D locations in the scene can be obtained and compared with detected shadows. Any point light that can produce virtual shadows similar to realistically detected shadows can be selected to represent scene lighting. For example, either the color and intensity of any point light or ambient lighting can be based on analysis of surfaces including areas with and without cast shadows.

[0024] According to the implementation scheme, scene lighting can be obtained based on the observed environment (e.g., processing images of the environment), for example, using any of the cameras (e.g., fisheye, wide-angle) and observing the light probe (e.g., a metal ball) from a standard camera. For example, specular reflections of light can be detected in images captured from different viewpoints, and the 3D position of point lights in the scene can be obtained from these various observations (e.g., images) based on the geometry of the scene and based on the viewpoint pose.

[0025] According to the implementation plan, a set of outdoor lighting conditions can be represented as a set (e.g., finite) models, which can be obtained (e.g., calculated) based on a combination (e.g., specific) meteorological conditions (e.g., any one of sunny, cloudy, rainy, etc.), a day of the year, and a time of day.

[0026] According to the implementation scheme, the outdoor model may include the direction of the sun, which may depend on the 3D location of the indoor scene (e.g., any of latitude or longitude), a day of the year, and a time of day. The terms "3D location" and "location" are used interchangeably to specify the location of the indoor scene on Earth (e.g., a real location). The location of the indoor scene may be represented, for example, by any of latitude, longitude, a set of GPS coordinates, etc. Optionally, the location of the indoor scene may also include altitude (e.g., relative to sea level). The terms "3D location" and "location" are used interchangeably to specify the location within the indoor scene (e.g., any point, object). For example, the location of a point within the indoor scene may be relative to the center of the indoor scene. For example, the location of the indoor scene (latitude, longitude, any of a set of GPS coordinates) may be the same as the location of the center of the indoor scene. In another example, the location of the center of the indoor scene may be represented as the location relative to the indoor scene.

[0027] According to the implementation, the direction of the sun can be evaluated based on (e.g., knowing) the 3D location of the indoor scene, a day of the year, and the time of day. In a first example, the 3D location of the indoor scene can be configured via a user interface. In another example, the 3D location of the indoor scene can be retrieved from a (e.g., geometric) model of the indoor scene. For example, the outdoor lighting model at the 3D location and time can include any of the sun's direction, color, and intensity, and any of the sky's color and intensity. According to the implementation, an indoor lighting model (e.g., a set of indoor lighting models) can be obtained (e.g., calculated, received) based on a model of the indoor scene (any of geometry, texture, and reflectivity) and (e.g., a set of) lighting conditions. According to the implementation, (e.g., indoor) lighting can be referred to herein as lighting in an indoor scene induced by outdoor lighting. For example, for (e.g., each) indoor lighting model, a set of any point lights and an environment map illuminating the indoor scene, along with their parameter values, can be determined. In another example, (e.g., indoor) lighting models can include ambient light with parameters (e.g., any of color and intensity). The parameters of a point light can be, for example, location (e.g., in the scene), color, and intensity. The parameters of an environment map can be, for example, location (e.g., in the scene) and image.

[0028] According to the implementation scheme, the lighting model can be sent in response to a user request (e.g., receive and send).

[0029] In the first example, which may be referred to herein as “real-time navigation,” the user is physically present in an indoor scene. This allows the user to navigate and simulate various (e.g., different) outdoor lighting conditions. In AR (e.g., immersive) applications, the device can display a view of the indoor scene that the device's camera can (e.g., currently) capture. The camera's pose in the indoor scene can be estimated, for example (e.g., continuously). The captured image can be re-illuminated based on a (e.g., registered) model (e.g., geometry, lighting) of the indoor scene, modified according to parameters (e.g., configurable by the user via a user interface), and the result can be displayed (e.g., rendered) on the device. In this example, the re-illuminated texture can be either the texture of the captured image of a visible real object, or the texture of the geometry attached to a virtual portion of the indoor scene (e.g., any of inserted virtual objects, modified surface textures, etc.).

[0030] In the second example, which may be referred to as "virtual navigation" in this paper, the user can be absent from the scene. This allows the user to virtually navigate within a relit scene rendered from its model (any of its geometry, texture, reflectivity, or lighting).

[0031] In the third example, for example, by turning rain into sunshine (e.g., or vice versa), or by changing the experience from night to morning (e.g., or vice versa), users can be enabled to change the atmosphere of a room (in either real-time or virtual navigation).

[0032] In what may be termed a fourth example of mixed reality telepresence, at least two users can communicate (via audio and video data) despite being in different time zones and conditions. One user (e.g., located in the United Kingdom (UK)) may wish to adjust the rendering of their own indoor scene to the virtual outdoor conditions of the other user (e.g., located in Australia). In another example, any local indoor scene rendering may be adjusted based on virtual outdoor conditions, such as the location where the users wish they would meet (e.g., a shared location) (e.g., a virtual Maui meeting using the current time and weather in Maui).

[0033] According to the implementation scheme, any number of lighting models can be selected (e.g., via a user interface) to evaluate (or simulate) the interior of a room (e.g., design) under various lighting conditions. According to the implementation scheme, the user can be enabled to modify other elements of the real-world scene, such as (e.g., remove, insert) any object, surface texture, etc.

[0034] According to the implementation scheme, users can run an AR application for navigating an indoor scene, rendering the indoor scene on a rendering (e.g., display) device under various (e.g., varying) lighting conditions. The AR application can correspond to a real-world scene (e.g., a room where the user can stand) or any other modeled scene. The model of the indoor scene can be obtained using any modeling techniques known to those skilled in the art, including, for example, any of the following: geometry, texture, and reflectivity information.

[0035] In a first example of a modeling technique, photogrammetry (the science of measuring from photographs) can be used. Photogrammetry infers the geometry of a scene from a set of unordered captured images or videos. The captured images can be viewed as a projection of a 3D scene onto a 2D plane, thus losing depth information. Photogrammetry can also be viewed as the reverse of this process. For example, from a set of (e.g., large) captured (e.g., color) images of a scene, and knowing the intrinsic parameters of the camera that captured the scene, one can obtain (e.g., estimate) a camera pose and depth map associated with (e.g., each) image. Either the geometry or texture of the model can be obtained based on the captured images and the estimated depth map. The accuracy of this method can depend on either the number of captured images or the texturing variations in the captured images of the scene. The modeling technique may include a scaling step to obtain a model that fits the dimensions of the real-world scene.

[0036] In the second example, a scene model can be obtained by capturing images of the scene with additional depth information using, for example, either a color sensor or a depth sensor. Additional hardware, such as an inertial measurement unit (IMU) and either a simultaneous localization and mapping (SLAM) algorithm, can allow retrieving (e.g., each) the pose of the images and obtaining (e.g., estimating) the scene model (e.g., either geometric or texture information). In a variant, the specular reflectivity of the scene's surfaces can also be estimated by processing the captured images of the scene.

[0037] In a real-time navigation example, for instance, a (e.g., 3D textured) model of the scene can be obtained based on a set of data acquired in a room (e.g., before the execution of the AR application). For example, any image in the scene, the associated camera pose, or a reference to a specific object that allows for obtaining the model's center and orientation can be acquired. The model's center and orientation can then allow the modeled scene to be rendered correctly in front of the user device based on its position and orientation within the scene during rendering (e.g., in real or virtual contexts).

[0038] According to the implementation scheme, the model of an indoor scene may include any one of the scene's location (e.g., longitude, latitude), orientation (e.g., relative to north), and height (e.g., relative to sea level). The scene's location, orientation, and height can be obtained, for example, from geographic map information, building plans, and other means. In another example, for each image of the scene, the attitude, height, and orientation of the capturing device can be obtained from the device capturing the scene image. The scene's location, orientation, and height can be obtained (e.g., modeled) based on a set of captured images, wherein at least one captured image can be associated with any one of the location, orientation, and height of the device that has already captured the image. The device's location, orientation, and height can be obtained from various sensors of the device (e.g., IMU, GPS, compass, altimeter, etc.).

[0039] According to the implementation scheme, a model of the indoor scene (e.g., including any of geometry, texture, and reflectivity information) can be obtained by the processing device. For example, the model can be calculated by the processing device based on any of the techniques described previously. In another example, the model can be received by the processing device from another device that has already calculated it. According to the implementation scheme, the model of the scene can be stored on the processing device after being obtained. According to the implementation scheme, the indoor scene can correspond to a set of indoor scenes, wherein the indoor scenes can be identified by an identifier. For example, a request can be made to the processing device (via either a user interface or a network interface) to provide (e.g., a specific) relit indoor scene. The processing device can send a (e.g., indoor) lighting model corresponding to the requested indoor scene and the requested lighting conditions. In another example, a request can be made to the processing device (via either a user interface or a network interface) to render (e.g., a specific) relit indoor scene. The processing device can render the requested indoor scene relit according to the requested lighting conditions.

[0040] According to the implementation scheme, the processing device can process (e.g., indoor) lighting models for a scene illuminated by outdoor lighting. For example, the processing device can obtain (e.g., indoor) lighting models of an indoor scene based on a model of the indoor scene and based on instances of outdoor lighting conditions (e.g., identified by a scene identifier). Any number of (e.g., indoor) lighting models of an indoor scene can be obtained based on any number of outdoor lighting conditions (e.g., indoor lighting models of the indoor scene corresponding to instances of outdoor lighting conditions). For example, multiple instances of outdoor lighting conditions can include any of a day of the year, the time of day, and weather conditions (e.g., sunny, cloudy, rainy, stormy, etc.). According to the implementation scheme, the 3D position of the indoor scene (e.g., configured via a user interface) can provide (e.g., allow determination) the direction of the sun associated with the current (e.g., each) outdoor lighting condition of the indoor scene (e.g., azimuth relative to north and tilt relative to the horizontal plane at the 3D position).

[0041] According to the implementation scheme, the model of an indoor scene can include any of geometric shape, texture, and reflectivity information. In a first example, a lighting model (e.g., indoor, which may be referred to herein as a geometric model) can be obtained based on a scene model that includes only the geometric information of the scene. Geometric information can include any of the shape, size, orientation, position, etc., of objects in the scene. In a second example, a lighting model (e.g., indoor) can be obtained based on a scene model that includes information about geometry and texture. In a third example, a lighting model (e.g., indoor) can be obtained based on a scene model that includes information about the geometry, texture, and reflectivity of the scene surface.

[0042] According to the implementation scheme, upon receiving a request for an indoor scene illuminated under specific outdoor lighting conditions, the processing device can select and send an indoor scene (e.g., an indoor) lighting model corresponding to the requested outdoor lighting conditions. In a variant, the processing device can send the scene model along with the scene's (e.g., indoor) lighting model. In another variant, any number (e.g., all) of the scene's (e.g., indoor) lighting models can be sent by the processing device to the rendering device. In a variant, requests can be received from the processing device's local user interface, and the processing device can render the scene model according to the requested (e.g., indoor) lighting model. The user can navigate within the indoor scene using the processing device and can observe the indoor scene under the requested outdoor lighting conditions. According to the implementation scheme, virtual objects can be inserted into the indoor scene. According to the implementation scheme, virtual objects can include lighting objects (e.g., table lamps) that influence (e.g., supplement) the indoor scene's (e.g., indoor) lighting model.

[0043] Figure 1This is an illustration illustrating an example of an augmented reality application based on multiple simulated scene lighting conditions. According to an embodiment, general lighting models 11, 12, 13, and 14 can be obtained (e.g., parametrically) based on outdoor lighting conditions 100. According to an embodiment, processing module 110 can calculate indoor scene (e.g., indoor) lighting models 111 and 112 based on (e.g., parametrically) lighting models 11, 12, 13, and 14 and on a scene model 10 including any of the scene's geometry, 3D position, and texture information. According to an embodiment, processing module 110 can receive (e.g., a set of) indoor scene models. According to an embodiment, processing module 110 can receive (e.g., a set of parametrically) lighting models 11, 12, 13, and 14 representing different outdoor lighting conditions. According to the implementation scheme, for each indoor scene (e.g., each) model 10, a set of scene-dependent (e.g., indoor) lighting models 111, 112 can be obtained based on (e.g., a set of parameterized) lighting models 11, 12, 13, 14 and scene-dependent (e.g., indoor) lighting models 111, 112 corresponding to (e.g., specific) outdoor lighting condition instances. According to the implementation scheme, scene-dependent (e.g., indoor) lighting models 111, 112 associated with different outdoor lighting conditions can be stored on the processing device running the processing module 110. According to the implementation scheme, an external renderer device 120 can send a request to the processing module 110 for an indoor scene under virtual outdoor lighting conditions. The request can include, for example, information indicating the requested virtual outdoor lighting conditions. If the processing device is configured to process more than one scene, the request can further include, for example, information indicating the scene to be rendered. The request can also include, for example, the 3D location of the scene to be rendered. In another example, the 3D location of the scene can be initially obtained by the processing device (e.g., via user configuration and either during scene modeling). According to the implementation scheme, the processing module 11 The processing module 10 can send a scene model 10 supplemented with (e.g., indoor) lighting models corresponding to the requested virtual outdoor lighting conditions of indoor scenes 111, 112. The external renderer device 120 can render the indoor scene illuminated by the requested virtual outdoor conditions. In a variant, the processing module 110 can request, for example, via a user interface, to render the indoor scene illuminated by the virtual outdoor conditions, and can locally render the indoor scene based on (e.g., indoor) lighting models 111, 112. According to the implementation, this allows a user to navigate within the indoor scene under specific simulated (e.g., virtual) outdoor lighting conditions.

[0044] Figure 2This is an illustration of an example of an indoor lighting estimation processing module. Processing module 220 can be configured to obtain a scene-dependent (e.g., indoor) lighting model 210 from a general outdoor lighting model 22 and a model of the indoor scene 20. In a first example, the model of the indoor scene 20 may include the 3D location 21 of the scene. In a second example, the 3D location 21 of the indoor scene may be configurable (e.g., via a user interface), and the scene-dependent (e.g., indoor) lighting model 210 may be obtained based on the general outdoor lighting model 22, the model of scene 20, and the 3D location 21 of the indoor scene.

[0045] According to the implementation scheme, an outdoor lighting model corresponding to any of the following can be obtained: (e.g., general, scene-independent) weather conditions, time of day, and number of days in a year. For example, at a given time of day (e.g., in a year) and under specific weather conditions, the general outdoor lighting model can include any of the direction, color, and intensity of the sun (e.g., derived from the time of day and the 3D position of the scene). For example, (e.g., at a given time of day and under specific weather conditions), the general outdoor lighting model can include any of the color and intensity of the sky. For example, sky lighting can be represented in the outdoor lighting model by a (e.g., single) color parameter controlled by parameters (e.g., derived from a slider in the user interface), ranging from blue to milky white, gray, and dark gray. In the first example, the (e.g., general) outdoor lighting model can include a set (e.g., sample, discrete) model instances sampled from various possible cases. In the second example, the (e.g., general) outdoor lighting model can be (e.g., single, parametric) models whose instances can be obtained via parameterization (e.g., parametric tuning). According to the implementation scheme, instances of such models can be configured (e.g., selected) by the user via a user interface (e.g., using a slider). According to the implementation scheme, a general outdoor lighting model can be parameterized by scene (e.g., target) location parameters (e.g., latitude, longitude, altitude). For example, either sky or solar lighting can be further adjusted based on the target location.

[0046] The direction of the sun can be defined by a parametric model that takes into account the lighting experience, the 3D location, the day of the year, and the time of day. For example, the time zone can be identified from the 3D location. Then, from the 3D location, time zone, and the day of the year and time of day provided by the user, the direction of the sun can be obtained, for example.

[0047] According to the implementation scheme, a suitable (e.g., scene-specific) (e.g., indoor) lighting model can be obtained (e.g., computed) based on (e.g., a general) outdoor lighting model, the 3D location of the scene, and (e.g., an instance) of the scene's (e.g., geometric) model. For example, the (e.g., indoor) lighting model may include either (e.g., directional light derived from sunlight) or (e.g., a set of) environment maps located within the scene. The environment map may approximate the appearance of reflective surfaces around a (e.g., 3D) location within the scene. The environment map may be represented as a spherical image or a cube map. The image may have a coarse (e.g., limited, reduced) resolution to (e.g., only) display the primary illumination reflections (e.g., without any texture details). Lighting information may, for example, be encoded as spherical harmonic fundamental functions. According to the implementation scheme, an environment map can be obtained (e.g., computed) based on (e.g., a selected) instance of the outdoor lighting model. According to the implementation scheme, in addition to direct lighting, the (e.g., indoor) lighting model may include ambient lighting, for example, instead of the environment map. Unlike other lighting models, ambient lighting may combine the corresponding ambient color with surface reflectivity without considering any surface orientation.

[0048] For example, an interior scene may include openings to the outside. An opening may be either a window or a door. More generally, any external opening that allows (e.g., external lighting conditions) to influence the lighting of the interior scene can be used in the embodiments described herein. According to the embodiments, any of the following can be obtained from a model of the interior scene (e.g., the geometry of the model): the opening (e.g., either a window or a door), its orientation (e.g., relative to either a north or vertical axis), its location (e.g., either a latitude or longitude location), and its geometry (e.g., size, shape). For example, the geometric model of the interior scene may include information about the geometry of the opening. Information about the geometry of the opening (which may be referred to herein as the model of the opening) may include information about the opening's orientation (e.g., relative to either a north or vertical axis), its location (e.g., either a latitude or longitude location), and its geometry (e.g., size, shape).

[0049] For example, a lighting model for an indoor scene (e.g., indoor) may include a portion of a model, which may also be referred to herein as a lighting model element associated with (e.g., corresponding to) an opening. The lighting model element associated with an opening can model the lighting of the indoor scene induced by the opening. The lighting model element associated with an opening can be based on a model of the opening and on virtual outdoor conditions. For example, any of the direction, color, and intensity of the sun in the (e.g., indoor) lighting model can be obtained based on any of the direction, color, and intensity of the sun in an instance of an outdoor lighting model applied at a 3D location in the scene and based on any of the location, orientation, and geometry of the opening in the scene.

[0050] According to the implementation, the lighting model (e.g., indoor) can be obtained (e.g., supplemented) based on the geometry of the scene's environment, such as adjacent buildings that may (e.g., partially) block the sun. Either the directional light and color of the sun in the area (e.g., influenced by the environment) can be derived accordingly. According to the implementation, sunlight can illuminate the indoor scene through openings (either windows or doors). According to the implementation, potential adjacent outdoor buildings can act as (e.g., treated as) additional shading masks applied to the scene openings. For example, the diffuse skylight intensity can be weighted via the percentage of visible sky from the scene openings in the presence of adjacent buildings.

[0051] According to the implementation scheme, the lighting model (e.g., indoor) may include (e.g., additionally) either point light or area light to complement the model by taking into account the diffuse aspects of the current direct lighting (e.g., soft shadows) and the geometry of the openings (e.g., windows). For example, either complementary point light or area light can be obtained by any lighting estimation method, for example, taking into account that the openings may also diffuse skylight inside the indoor scene.

[0052] According to the implementation scheme, ambient lighting can be used in (e.g., indoor) lighting models to illuminate surfaces of models that do not receive direct illumination. For example, either the color or intensity of the ambient lighting can depend on any of the direct illumination, geometry, texture, and reflectivity of the indoor scene. For example, either the color or intensity of the ambient lighting can be set to a constant value throughout the scene.

[0053] Depending on the implementation, an environment map (e.g., a set) can be used for an indoor lighting model to account for indirect lighting (e.g., illuminating surfaces of a model that do not receive direct light). The number of environment maps can be any number, depending on the complexity of the scene (e.g., geometry, texture, reflectivity) and the desired rendering quality. For example, a single environment map can allow for realistic rendering of simple scenes (e.g., an empty single room with a relatively uniform texture on surfaces (e.g., a white surface). Higher-density environment maps can allow for improved realism in rendering more complex scenes.

[0054] According to the implementation scheme, a lighting model of a scene (e.g., a single indoor unit) can be obtained based on images of an indoor scene captured at (e.g., different) locations under (e.g., a single) outdoor conditions. The obtained (e.g., indoor) lighting model can be associated with outdoor conditions (e.g., the actual outdoor conditions corresponding to the images capturing the indoor scene). The (e.g., indoor) lighting model can be estimated from the captured images based on any lighting model estimation technique known to those skilled in the art.

[0055] According to the implementation scheme, a set of (e.g., indoor) lighting model instances of a scene can be obtained based on (e.g., a set) of images of the scene captured in an indoor scene under various (e.g., different) outdoor lighting conditions (e.g., at different times and / or under different weather conditions). Each (e.g., indoor) lighting model instance can be associated with (e.g., different) outdoor lighting conditions (e.g., real outdoor lighting conditions corresponding to a corresponding image of the captured scene). The outdoor lighting conditions for capturing a set of images to estimate (e.g., a specific, predetermined indoor) lighting model instance can be referred to herein as the captured outdoor lighting conditions. According to the implementation scheme, a (e.g., indoor) lighting model of a scene can be obtained based on (e.g., user-selected) virtual outdoor conditions by selecting (e.g., a pre-selected) pre-selected (e.g., indoor) lighting model instances associated with the closest (e.g., most similar) virtual outdoor conditions. In the second example, an indoor lighting model instance can be obtained based on interpolation of at least two predetermined indoor lighting model instances associated with at least two (e.g., most) similar captured outdoor lighting conditions. According to the implementation, the similarity between outdoor lighting conditions can be defined by vectorizing the parameter values ​​of the outdoor lighting conditions and calculating either the Hamming or Euclidean distance between two vectorized outdoor lighting conditions.

[0056] According to the implementation plan, in real-time navigation, users can access a scene through their devices and display images corresponding to the scene under user-specified lighting conditions. For example, the displayed image may correspond to (e.g., the current) device orientation, allowing users to align their devices with specific locations within the scene and see the corresponding virtual scene on a display. In the first example, the displayed image can be generated by rendering based on (e.g., the geometry and texturing model of the scene) and based on (e.g., an indoor) lighting model (e.g., similar to virtual navigation, where the rendering viewpoint can be determined based on the device pose). In the second example, the device can display the currently captured image (e.g., video) after it has been relit from (e.g., an indoor) lighting model (e.g., based on geometry and possible specular reflectivity data). In the second example, the texture information of the scene can be extracted from the current image (e.g., rather than from a model of the scene as in the first example). The rendering process can combine the extracted texture with any light-related components generated by geometry (surface orientation) and (e.g., indoor) lighting modeling (direction, color, intensity, etc.). The second example can be applied if the (e.g., current) lighting of the image (e.g., video) is soft (without any strong shadows or highlights). If the captured image (e.g., video) includes (e.g., strong, significant) light effects (shadows, highlights), the lighting effects can be removed before relighting the image (e.g., video).

[0057] According to the implementation scheme, virtual objects can be introduced into a scene (e.g., by a user). The virtual objects can be illuminated based on a lighting model (e.g., indoor) that reflects the lighting of a real indoor scene. In a first example, the lighting model (e.g., indoor) can be obtained (e.g., estimated) from an image of the indoor scene captured by a device (e.g., a camera). For example, lighting parameter values ​​of the lighting model can be estimated from the captured image and the scene model. The estimated lighting parameter values ​​can be used to relight the inserted virtual objects. In a second example, a set of parameter values ​​(e.g., lighting model instances) from a set of lighting model instances that are closest to a set of estimated parameter values ​​can be selected for relighting the virtual objects.

[0058] According to the implementation scheme, the device can render (e.g., display) an image of a scene, including areas in the scene model that have not yet been modeled. Such areas may, for example, appear in the field of view of a device running an AR application, even though they have not yet been modeled. In a first example, the corresponding areas of the image can be color-corrected according to a (e.g., indoor) lighting model to reduce potential differences between (e.g., unmodeled) areas of the scene and modeled, relit portions. In a second example, an ambient lighting component can be applied to reduce potential differences between (e.g., unmodeled) areas of the scene and modeled, relit portions.

[0059] According to the implementation scheme, artificial lighting (e.g., outdoor streetlights, indoor ceiling lights, table lamps, etc.) can be considered during the rendering process by associating a lighting model with artificial lighting. For example, the (e.g., indoor) lighting model may include (e.g., additional) basic lighting models representing artificial light sources. In a first example, the indoor scene may include artificial light sources that can be detected and modeled as a basic artificial light model in the scene's (e.g., indoor) lighting model. In a second example, the basic artificial light model may be available to the user via a user interface (e.g., for adding to the (e.g., indoor) lighting model). According to the implementation scheme, the basic artificial light model may be, for example, pre-defined and stored in a device running the AR application. According to the implementation scheme, the basic artificial light model may be stored in a database and accessible from the database.

[0060] According to the implementation, the (e.g., indoor) scene can be any of, for example, an apartment, a house, or a building. The (e.g., geometric) model can, for example, define the concepts of indoor and outdoor. In this context, any outward opening can be assimilated into a window. For example, the (e.g., geometric) model can include an explicit designation of an opening (e.g., any of a window, door, etc.). According to the implementation, an opening can have its own explicit model (e.g., any of a window shape and size). According to the implementation, a window can be equipped with a shade or blinds that can be modeled using a variable (e.g., parametric) model (e.g., shading intensity, openness, etc.). According to the implementation, the state of such equipment can be adjusted via a user interface (e.g., virtually) to achieve a wide range of ambient lighting.

[0061] According to the implementation scheme, the AR application can allow simulation of the effect of scene orientation relative to a base point, such as experimenting with the effect of sunlight on (e.g., indoor) lighting. According to the implementation scheme, the scene orientation (e.g., relative to the base point) can be adjustable. According to the implementation scheme, virtual obstacles (e.g., representing adjacent buildings) can be added to the scene model, for example via a user interface, allowing the user to test the effect of the hypothetical obstacle's presence on outdoor lighting adjacent to the scene.

[0062] Example of a user interface for adjusting configurations in scene rendering.

[0063] This document describes an example of a user interface for inputting information used to configure (e.g., parameterize) scene rendering adjustments. For example, the user interface may include a first part that can be dedicated to configuring indoor scenes and a second part that can be dedicated to configuring outdoor lighting models.

[0064] For example, a processing device (e.g., which can run an AR application) can be located. For example, a model of the scene (e.g., in which a processing device can be located) can be available and can be displayed (e.g., rendered) via a first part of the user interface. For example, a list of scenes (e.g., models) can be displayed, along with a selection device that allows the user to select a scene (e.g., model) from other scenes.

[0065] For example, an indoor scene can be located (e.g., belong to) any of an apartment, house, or building. For example, the location can be available. An indoor scene can correspond to, for example, an existing apartment (e.g., a house, building) or an apartment that does not yet exist, but for example, plans and models can be available. For example, the environment of the scene can be predetermined (e.g., pre-configured). The environment (e.g., buildings, trees, etc.) can be adjusted via a user interface (e.g., to anticipate the potential impact of environmental changes).

[0066] For example, a building, including an interior scene, can be moved to another location via a user interface, placing it in a different environment. This allows for the simulation of the impact of the other environment on the lighting model of the interior scene.

[0067] According to the implementation scheme, an indoor scene can be selected via a user interface. For example, either the location and orientation (e.g., horizontal orientation relative to north) of the indoor scene (e.g., default 3D) or the environment of the indoor scene (e.g., default) can be available. For example, either the location, orientation, or environment can be adjusted (e.g., configurable, modifiable) via the user interface. This allows the user to position or orient the scene differently, or, for example, to add and remove any of the nearby obstacles (e.g., buildings or trees). For example, the environment can be displayed as a top-down view of the area and includes a (e.g., 2D) map of the scene's buildings and other nearby buildings. The (e.g., 2D) map can correspond to any of, for example, an existing environment, a planned environment, and a virtual environment (e.g., where the user can add buildings). Buildings can be constructed, for example, of parallelepipeds, which can be any of those placed on the map and overlapping each other, and can be configured (e.g., including dimensions) via the user interface.

[0068] According to the implementation scheme, the model of the indoor scene may at least include information describing the geometry of the scene. For example, the model may further include either texture or reflectivity information (e.g., parameters) of the indoor scene. If the model does not contain any texture or reflectivity information, a default texture (e.g., and reflectivity) may be applied. For example, the user interface may suggest (e.g., display suggested information) a set of possible textures (e.g., any of various intensities and colors) and reflectivities (e.g., specular level) to the user for selection of (e.g., different) surfaces included in (e.g., the geometry) the model.

[0069] According to the implementation scheme, information describing the geometry of the indoor scene may include, for example, a description of an opening (e.g., to the outside) having either (e.g., in world space) the scene's 3D position or orientation. For example, a set of possible occlusion objects may be assigned to (e.g., associated with) the opening via a user interface, such as any of a light-blocking shade, blinds, or shutter. For instance, parameters of these occlusion objects may be adjustable (e.g., tunable) via the user interface, such as the degree to which the blinds (e.g., or the shutter) are closed.

[0070] According to the implementation, a second part of the user interface (e.g., which may be dedicated to outdoor lighting model configuration) may display, for example, a list of weather conditions (e.g., sunny, cloudy, rainy, etc.), from which a specific instance can be selected. For example, either the time of day or any day of the year can be selected (e.g., configurable) via the user interface. According to any implementation described herein, the outdoor lighting model can be derived based on these selections (e.g., configuration).

[0071] According to the implementation scheme, the lighting models of the selected indoor scene and the outdoor lighting model, which are associated with the outdoor environment (e.g., illuminated by outdoor lighting that can be generated by the outdoor environment), may have been pre-calculated and can be used for AR applications. In another example, the lighting model of the selected indoor scene may be unavailable, for example, because the current (e.g., lighting) configuration of any of the scene, its location, its orientation, the outdoor environment, and the outdoor lighting model may not have been pre-calculated. According to the implementation scheme, the lighting model of the indoor scene (e.g., illuminated by outdoor lighting that can be generated by any of the outdoor environment and virtual outdoor conditions (e.g., the outdoor lighting model) can be obtained (e.g., calculated) based on these components (e.g., parameters describing these components).

[0072] According to the implementation scheme, any number of artificial indoor lights (e.g., any of indoor ceiling lights, table lamps, etc.) can be included in the model of the (e.g., indoor) scene via a user interface. For example, the (e.g., indoor) lighting model of the (e.g., indoor) scene can be adjusted (e.g., recalculated, supplemented) when indoor light is introduced into (e.g., or removed from) the scene.

[0073] Example of relighting adjustments in scene rendering

[0074] According to the implementation scheme, instances of a (e.g., selected, computed) lighting model can be applied to an indoor scene, for example, based on a scene model (e.g., any of 3D geometry, texture, and reflectivity). For example, (e.g., each) image of a (e.g., captured) video of the scene can be replaced by a (e.g., computer graphics-based) rendered image of the scene at the (e.g., current) viewpoint, based on the scene model and the (e.g., current indoor) lighting model. In a first example, the indoor scene can be enhanced first, for example, by inserting virtual objects (e.g., including any of geometry, texture, and reflectivity) into the scene model. In a second example, the indoor scene can be modified (e.g., before rendering), for example, by removing objects that can render the visible reconstructed area.

[0075] For example, (e.g., current real-world) lighting in an image of an indoor scene (e.g., which may have already been captured by a camera) can be uniform. In the absence of, for example, strongly lit areas contrasting with nearby shadow areas, the lighting in the image can be described as uniform. When the lighting is uniform in an image of an indoor scene, a new (e.g., virtual) lighting instance can be (e.g., directly) applied to the image of a video displayed by a processing device. For example, an image of an indoor scene (e.g., which may have already been captured by a camera) can be used as a texture for a scene model. For example, virtual objects can be inserted into the scene model, and the resulting blended image can be used as a texture and modulated by a new (e.g., virtual) lighting instance applied to the blended scene. For example, before applying the new (e.g., virtual) lighting instance, an image of the indoor scene (e.g., colors) can be shifted to a reference that may correspond to the reference ambient lighting used for the texture model.

[0076] For example, uniform lighting in a scene (e.g., a real indoor scene) can be detected by comparing an input image of the scene (e.g., captured by a camera) with a version rendered by a scene model (e.g., by a GPU) (from the same viewpoint). The input image of the scene (e.g., captured by a camera) can be referred to herein as I... c And the corresponding image rendered based on the scene model (e.g., texture) can be referred to as I in this paper. t For example, in the input image Ic and texture-based rendering image I t The differences between them can be identified through affine transformations (e.g., I...). t =AI c In the case of modeling with +B, where A and B are affine parameters, for example, within an acceptable error range, it can be determined that a new (e.g., virtual) lighting instance can be applied using the input image, for example, instead of the model texture. For example, on an image (I t -AI c -B) 2 If the error is below (e.g., a threshold) the value, the error can be acceptable.

[0077] For example, before applying a new (e.g., virtual) lighting instance, the input image can be color-shifted (e.g., by transforming AI). c +B). For example, color shifting an input image can include obtaining a color-shifted image of the input image, where the color values ​​of pixels or the color-shifted image can be obtained by (e.g., affine) transformation of the color values ​​of the corresponding pixels in the input image.

[0078] Relighting images of video (e.g., captured by a camera from an indoor scene) by generating texture-based images of lighting based on a lighting model rather than (e.g., from a GPU) can allow for a better sense of realism in scenes displayed on a processing device screen, for example, even when there are (e.g., significant) errors in the scene geometry.

[0079] For example, in cases where lighting is uneven in an image of an indoor scene (e.g., which may have already been captured by a camera), the texture of the scene model can be used instead of the image texture. For example, the scene can be rendered based on images (e.g., computer graphics), which can be generated based on a scene model (e.g., geometric, textured) and on instances of the lighting model.

[0080] Figure 3A This is an illustration of an example of a processing device 3A used for adjusting scene rendering. According to an embodiment, the processing device 3A may include a network interface 30 for connecting to a network. The network interface 30 may be configured to send and receive data packets. According to an embodiment, the network interface 30 may be any of the following:

[0081] - Wireless LAN interface, such as Bluetooth, any form of Wi-Fi, or any kind of wireless interface of the IEEE 802 series network interface;

[0082] -A wired LAN interface, such as Ethernet, IEEE 802.3, or any wired interface of the IEEE 802 series network interface;

[0083] - Wired bus interface, such as USB, FireWire, or any kind of wired bus technology.

[0084] - Broadband cellular wireless network interfaces, such as 2G / 3G / 4G / 5G cellular wireless network interfaces that conform to any version of the 3GPP specification;

[0085] - Wide area network interface, such as xDSL, FFTx or WiMAX interface.

[0086] More generally, any network interface that allows sending and receiving data packets is compatible with the implementation described herein.

[0087] According to an embodiment, network interface 30 may be coupled to processing module 32, which is configured to obtain, for example, information indicating an indoor scene to be rendered under virtual outdoor conditions, received via network interface 30. According to an embodiment, information indicating an indoor scene to be rendered under virtual outdoor conditions may be received via a local user interface (not shown). According to an embodiment, processing module 32 may be further configured to obtain an indoor scene (e.g., indoor) lighting model based on a geometric model of the indoor scene and based on virtual outdoor conditions. According to an embodiment, processing module 32 may be further configured to adjust scene rendering by sending the scene's (e.g., indoor) lighting model to an external renderer device via network interface 30. According to an embodiment, the external renderer device may render the indoor scene (e.g., display an image of the indoor scene) based on a model of the scene supplemented by the received (e.g., indoor) lighting model. For example, both the indoor scene's (e.g., geometric, texturized) model and the scene's (e.g., indoor) lighting model may be sent by processing module 32 to the external renderer device. In another example, only the (e.g., indoor) lighting model may be sent by processing module 32 to the external renderer device. The scene's (e.g., geometry, texture) model can be made available to the external renderer device via any other means (received from another device, pre-configured, etc.). In yet another example, only the geometry model and (e.g., interior) lighting model can be sent to the external renderer device by processing module 32. The textured model of the interior scene can be made available to the external renderer device via any other means (received from another device, pre-configured, etc.).

[0088] According to the implementation scheme, the processing device may be coupled to (e.g., shown) a user interface that runs locally on the processing device 3A (not shown). According to the implementation scheme, the user interface may run on another device that communicates with the processing device 3A via a network interface 30. The user interface may allow the processing device 3A to interact with a user, for example, to receive any of the following: receiving a captured image of the scene, receiving a model of the scene, adjusting some parameters of the model, or receiving a request to render the scene under outdoor conditions.

[0089] Figure 3B This is an illustration of an example of a processing device 3B configured to adjust scene rendering. According to an embodiment, the processing device 3B may include a processing module 32 configured to obtain, for example, information via an optional network interface 30 (e.g., such as...). Figure 3A The processing module 32 receives information indicating an indoor scene to be rendered under virtual outdoor conditions, as described in the embodiment. According to the embodiment, the information indicating an indoor scene to be rendered under virtual outdoor conditions can be received via a local user interface coupled to the display device 34. According to the embodiment, the processing module 32 can be further configured to obtain a scene (e.g., indoor) lighting model based on the scene's geometric model and the virtual outdoor conditions. According to the embodiment, the processing module 32 can be further configured to adjust scene rendering by rendering the scene on the display device 34 using a scene-based (e.g., geometric and texturing) model and a scene-based (e.g., indoor) lighting model.

[0090] According to the implementation scheme, the user interface can run on another device that communicates with the processing device 3B via the network interface 30. The user interface can allow the processing device 3B to interact with the user, for example, to receive any of the following: receive a captured image of the scene, receive a model of the scene, adjust some parameters of the model, or receive a request to render the scene under outdoor conditions.

[0091] According to the embodiment, the display device 34 may be located either inside or outside the processing device 3B. According to the embodiment, the display device 34 may be a screen based on any display technology (e.g., LCD, LED, OLED, etc.).

[0092] Figure 4This indicates an exemplary architecture of either of the processing devices 3A or 3B described herein. Processing devices 3A or 3B may include one or more processors 410 and internal memory 420 (e.g., any of RAM, ROM, or EPROM), the one or more processors being, for example, any of CPU, GPU, or DSP (Digital Signal Processor). Processing devices 3A or 3B may include any number of input / output interfaces 430 (e.g., any of keyboard, mouse, touchpad, webcam, or display), adapted to send output information and / or allow user input of commands and / or data, and / or send / receive data via a network interface; and a power supply 440 that may be located externally to processing devices 3A or 3B.

[0093] According to the implementation scheme, processing devices 3A and 3B may further include a computer program stored in memory 420. The computer program may include instructions that, when executed by processing devices 3A and 3B, particularly by processor 410, cause processing devices 3A and 3B to perform reference... Figure 5 The processing method described. According to a variant, the computer program may be externally stored on a non-transitory digital data carrier, such as an external storage medium, like an SD card, HDD, CD-ROM, DVD, read-only and / or DVD drive, or DVD read / write drive, all of which are known in the art. The processing devices 3A and 3B may include an interface for reading the computer program. Additionally, the processing devices 3A and 3B may access any number of Universal Serial Bus (USB) type storage devices (e.g., "Memory Stick") via a corresponding USB port (not shown).

[0094] According to the implementation plan, processing devices 3A and 3B can be any of the following: server, desktop computer, laptop computer, networked device, television, tablet computer, smartphone, set-top box, internet gateway, and game console.

[0095] Figure 5 This is an illustration of an example method for adjusting scene rendering. According to the implementation scheme, in step S52, information indicating the indoor scene to be rendered under virtual outdoor conditions can be obtained. This information can be obtained from either a local user interface or a network interface.

[0096] According to the implementation scheme, in step S54, an indoor lighting model (e.g., indoor) can be obtained based on the 3D position and orientation of the indoor scene, the geometric model of the indoor scene, and virtual outdoor conditions. In the first example, either the 3D position or orientation of the indoor scene can be configured via a user interface. In the second example, either the 3D position or orientation of the indoor scene can be included in the geometric model of the indoor scene (e.g., acquired during scene modeling).

[0097] For example, a general lighting model can be obtained based on virtual outdoor conditions, and an indoor lighting model (e.g., indoor) can be obtained based on the 3D location and orientation of the scene, based on the general lighting model and the geometric model of the scene.

[0098] According to the implementation scheme, an interior scene may include openings to the outside. For example, openings in the interior scene may be obtained, for instance, from the scene's geometry. The opening may be associated with a model of the opening (e.g., an element) (e.g., representing any of size, shape, location, or orientation). The model of the opening (e.g., an element) may be included in the scene's geometry.

[0099] According to the implementation scheme, a portion of a lighting model (e.g., a lighting model element) corresponding to an opening (e.g., an indoor lighting model) can be obtained based on virtual outdoor conditions and a model (e.g., an element) of the opening. For example, the lighting model element may include information modeling the lighting of an indoor scene that can be induced by outdoor lighting through the opening.

[0100] According to the implementation, the model of the opening (e.g., element) may include the orientation of the opening relative to either north or vertical, and a portion of the lighting model corresponding to the opening (e.g., indoor) may be based on virtual outdoor conditions and on the orientation of the opening.

[0101] According to the implementation scheme, the model of the opening (e.g., element) may include the location of the opening (e.g., any location in latitude and longitude), and the portion of the lighting model corresponding to the opening (e.g., indoor) may be based on virtual outdoor conditions and based on the location of the opening (e.g., a location in either latitude or longitude of the scene).

[0102] According to the implementation plan, virtual outdoor conditions can be obtained from the user interface.

[0103] According to the implementation plan, virtual outdoor conditions can include indications of either a day of the year or a time of day.

[0104] According to the implementation plan, virtual outdoor conditions may include information indicating weather conditions.

[0105] According to the implementation scheme, in step S56, the rendering of the scene can be adjusted by either sending an indoor scene (e.g., indoor) lighting model to an external device for rendering the indoor scene by the external device, or rendering the indoor scene based on an indoor scene model supplemented by the indoor scene (e.g., indoor) lighting model.

[0106] According to the implementation scheme, an image of the indoor scene can be captured (e.g., by a camera), and the indoor scene can be rendered, for example, by rendering an image captured by a camera, and relit based on an indoor lighting model (e.g., indoor).

[0107] According to the implementation scheme, lighting effects can be removed from images (e.g., captured by a camera) before applying an indoor scene (e.g., an indoor) lighting model to an image.

[0108] According to the implementation plan, virtual objects can be inserted and illuminated (e.g., processed) in the rendered image based on the lighting model of the indoor scene (e.g., indoor).

[0109] According to the implementation scheme, at least one region of the captured image may not be modeled in the geometric model of the indoor scene. The region may be color-corrected based on (e.g., indoor) lighting model.

[0110] According to the implementation scheme, the opacity of the opening can be configured via a user interface and thus reflected in (e.g., indoor) lighting models (e.g., corresponding model elements of the opening in the lighting model).

[0111] Although not explicitly described, embodiments of the invention can be employed in any combination or sub-combination. For example, the principles of the invention are not limited to the described variations, and any arrangement of variations and embodiments can be used. Furthermore, the embodiments described herein are not limited to the lighting models (any of direct light, ambient light, point light, or environment map) and parameters (e.g., any of position, orientation, color, and intensity) described herein, and any other type of lighting model and / or parameters may be compatible with the embodiments described herein.

[0112] Furthermore, any features, variations, or embodiments described for the method are compatible with: apparatus for processing elements of the disclosed method, apparatus including a processor configured to process the disclosed method, computer program products including program code instructions, and non-transitory computer-readable storage media storing program instructions.

[0113] Although features and elements have been described above in specific combinations, those skilled in the art will understand that each feature or element may be used alone or in any combination with other features and elements. Furthermore, the methods described herein may be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of non-transitory computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media (such as internal hard disks and removable disks), magneto-optical media, and optical media (such as CD-ROM disks and digital versatile optical discs (DVDs)).

[0114] Furthermore, the above embodiments specify processing platforms, computing systems, controllers, and other devices including processors. These devices may include at least one central processing unit (“CPU”) and memory. According to the practice of those skilled in the art of computer programming, references to symbolic representations of actions and operations or instructions can be executed by various CPUs and memories. Such actions and operations or instructions can be considered as being “executed,” “computer-executed,” or “CPU-executed.”

[0115] Those skilled in the art will recognize that the actions and symbols representing operations or instructions include the CPU's manipulation of electrical signals. The electrical system represents data bits, which can lead to the final transformation or reduction of electrical signals and the retention of data bits at memory locations in the memory system, thereby reconfiguring or otherwise altering the CPU's operation and performing other signal processing. The memory location holding the data bits is a physical location having specific electrical, magnetic, optical, or organic properties corresponding to or representing the data bits. It should be understood that representative embodiments are not limited to the platforms or CPUs described above, and other platforms and CPUs may also support the provided methods.

[0116] Data bits may also be stored on a computer-readable medium, including disks, optical disks, and any other CPU-readable volatile (e.g., random access memory (“RAM”)) or non-volatile (e.g., read-only memory (“ROM”)) mass storage system. The computer-readable medium may include cooperative or interconnected computer-readable media that are uniquely present on the processing system or distributed across multiple interconnected processing systems, which may be local or remote relative to the processing system. It should be understood that representative embodiments are not limited to the memory described above, and other platforms and memories may also support the method described.

[0117] In exemplary embodiments, any of the operations, processes, etc., described herein may be implemented as computer-readable instructions stored on a computer-readable medium. These computer-readable instructions may be executed by a processor of a mobile unit, network element, and / or any other computing device.

[0118] There is little difference between the hardware and software implementations of various aspects of the system. The use of hardware or software typically (but not always, as the choice between hardware and software may become important in certain contexts) represents a design choice that weighs cost against efficiency. Various media (e.g., hardware, software, and / or firmware) may exist to implement the processes and / or systems and / or other technologies described herein, and the preferred media may vary depending on the context of the deployment of the processes and / or systems and / or other technologies. For example, if the implementer determines that speed and accuracy are most important, the implementer may choose a media that is primarily hardware and / or firmware. If flexibility is most important, the implementer may choose a primarily software implementation. Alternatively, the implementer may choose some combination of hardware, software, and / or firmware.

[0119] The above detailed description has illustrated various embodiments of the apparatus and / or process using block diagrams, flowcharts, and / or examples. Where such block diagrams, flowcharts, and / or examples contain one or more functions and / or operations, those skilled in the art will understand that each function and / or operation within such block diagrams, flowcharts, or examples can be implemented individually and / or collectively by a wide range of hardware, software, firmware, or virtually any combination thereof. Suitable processors include (by way of example) general-purpose processors, special-purpose processors, conventional processors, digital signal processors (DSPs), multiple microprocessors, one or more microprocessors associated with a DSP core, controllers, microcontrollers, application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), field-programmable gate array (FPGA) circuits, any other type of integrated circuit (IC) and / or state machine.

[0120] Although features and elements have been provided above in specific combinations, those skilled in the art will understand that each feature or element may be used alone or in any combination with other features and elements. This disclosure is not limited to the specific embodiments described in this patent application, which are intended as examples of various aspects. Many modifications and variations are possible without departing from the spirit and scope of the invention, as will be apparent to those skilled in the art. Unless expressly stated otherwise, no element, action, or description used in this specification should be construed as essential or necessary to the invention. Based on the foregoing description, functionally equivalent methods and apparatus within the scope of this disclosure, other than those listed herein, will be apparent to those skilled in the art. Such modifications and variations are intended to fall within the scope of the appended claims. This disclosure is limited only to the terms of the appended claims and the full scope of equivalents of such claimed claims. It should be understood that this disclosure is not limited to any particular method or system.

[0121] In some representative embodiments, portions of the subject matter described herein may be implemented via application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), and / or other integration formats. However, those skilled in the art will recognize that some aspects of the embodiments disclosed herein are, wholly or partially, equivalently implemented in an integrated circuit as one or more computer programs running on one or more computers (e.g., one or more programs running on one or more computer systems), one or more programs running on one or more processors (e.g., one or more programs running on one or more microprocessors), firmware, or virtually any combination thereof, and that designing circuitry and / or writing software and / or firmware code according to this disclosure will be entirely within the skill of those skilled in the art. Furthermore, those skilled in the art will understand that the mechanisms of the subject matter described herein can be distributed as program products in various forms, and the exemplary embodiments of the subject matter described herein apply regardless of the specific type of signal-bearing medium used to actually implement that distribution. Examples of signal-bearing media include, but are not limited to, the following: recordable media (such as floppy disks, hard disks, CDs, DVDs, digital magnetic tapes, computer memory, etc.); and transmission media (such as digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, etc.)).

[0122] The topics described herein sometimes illustrate different components contained within or connected to different other components. It should be understood that such depicted architectures are merely examples, and many other architectures can in fact achieve the same functionality. Conceptually, any arrangement of components achieving the same function is effectively “associated” to enable the desired functionality. Therefore, any two components combined herein to achieve a particular function can be considered “associated” with each other to enable the desired functionality, regardless of the architecture or intermediate components. Similarly, any two such associated components can also be considered “operably connected” or “operably coupled” to each other to achieve the desired functionality, and any two components that can be suchly associated can also be considered “operably coupled” to each other to achieve the desired functionality. Specific examples of operably coupled components include, but are not limited to, components that can physically cooperate and / or physically interact and / or components that can wirelessly interact and / or logically interact and / or logically interact.

[0123] Regarding virtually any plural and / or singular terms used herein, those skilled in the art can appropriately convert them from plural to singular and / or from singular to plural depending on the context and / or application. For clarity, various singular / plural permutations may be explicitly listed herein.

[0124] Those skilled in the art will understand that, in general, the terminology used herein, particularly in the appended claims (e.g., the body of the appended claims), is typically intended as “open-ended” terms (e.g., the term “comprising” should be interpreted as “including but not limited to,” the term “having” should be interpreted as “having at least,” the term “including” should be interpreted as “including but not limited to,” etc.). Those skilled in the art will also understand that if it is intended to specify a particular number of introduced claim objects, such intention will be explicitly stated in the claims, and if no such claim objects are present, such intention will not exist. For example, the term “single” or similar language may be used where only one item is anticipated. To aid understanding, the appended claims and / or the description herein may contain the use of the introductory phrases “at least one” and “one or more” to introduce claim objects. However, the use of such phrases should not be construed as implying that any particular claim containing such introduced claim objects is limited to an embodiment containing only one such claim object by using the indefinite articles “a” or “an.” This is true even when the same claim includes the introductory phrase "one or more" or "at least one" and indefinite articles such as "a" or "an" (e.g., "a" and / or "an" should be interpreted as meaning "at least one" or "one or more"). The same applies to the use of definite articles used to introduce the subject matter of a claim. Furthermore, even when a specific number of the introduced subject matter of a claim is explicitly stated, those skilled in the art will recognize that such a statement should be interpreted as meaning at least the stated number (e.g., a bare statement of "two subject matters" without other modifiers means at least two subject matters, or two or more subject matters).

[0125] Furthermore, in instances where the convention of "at least one of A, B, and C" is used, generally speaking, such a construction implies that a person skilled in the art will understand the convention (e.g., "a system having at least one of A, B, and C" includes, but is not limited to, systems having A alone, having B alone, having C alone, having both A and B, having both A and C, having both B and C, and / or having both A, B, and C). In instances where the convention of "at least one of A, B, or C" is used, generally speaking, such a construction implies that a person skilled in the art will understand the convention (e.g., "a system having at least one of A, B, or C" includes, but is not limited to, systems having A alone, having B alone, having C alone, having both A and B, having both A and C, having both B and C, and / or having both A, B, and C). A person skilled in the art should also understand that, in fact, any separate words and / or phrases presenting two or more alternative terms, whether in the specification, claims, or drawings, should be understood to contemplate the possibility of including one term, any one of the terms, or both of the terms. For example, the phrase “A or B” will be understood to include the possibility of “A” or “B” or “A and B”. Additionally, as used herein, the term “any one of…” followed by a list of multiple items and / or multiple item categories is intended to include items alone or in combination with other items and / or other item categories, “any one of,” “any combination,” “any multiple,” and / or “any combination of multiples of.” Furthermore, as used herein, the term “group” or “cluster” is intended to include any number of items, including zero. Additionally, as used herein, the term “quantity” is intended to include any quantity, including zero.

[0126] Furthermore, where features or aspects of this disclosure are described in accordance with the Markush Group, those skilled in the art will recognize that this disclosure is also described in accordance with any individual member of the Markush Group or a subgroup of its members.

[0127] Furthermore, unless otherwise stated, the claims should not be construed as being limited to the order or elements provided. Additionally, the use of the term "means for..." in any claim is intended to invoke 35 U.S.SC §112. The claim format is either device plus function, and any claim without the term "device for..." is not intended to be so.

Claims

1. A method for adjusting scene rendering, the method comprising: - Obtain virtual outdoor conditions from the user interface, wherein the virtual outdoor conditions are defined by user-selectable parameters, which include at least one of a day of the year, the time of day, and weather conditions; - Obtain information indicating the indoor scene to be rendered under virtual outdoor conditions; - A general lighting model is obtained based on the virtual outdoor conditions, wherein the general lighting model is independent of the indoor scene; - Based on the geometric model of the indoor scene, the general lighting model, and the lighting model of the indoor scene obtained based on the virtual outdoor conditions; and - Adjust the scene rendering using any of the following: ○ Send the lighting model of the indoor scene to an external device for rendering; and ○ Render the indoor scene based on the lighting model of the indoor scene.

2. The method of claim 1, wherein the indoor scene includes an opening to the outside, and the geometric model of the indoor scene includes a model of the opening.

3. The method of claim 2, wherein the method includes obtaining a lighting model element corresponding to the opening based on the virtual outdoor conditions and based on the model of the opening.

4. The method of claim 3, wherein the model of the opening includes the orientation of the opening relative to either north or vertical, and the lighting model element corresponding to the opening is based on the virtual outdoor conditions and based on the orientation.

5. The method of claim 3, wherein the model of the opening includes the location of the opening, and the lighting model element corresponding to the opening is based on the virtual outdoor conditions and the location.

6. The method of claim 1, wherein the indoor scene is associated with a first geographic location, and wherein the virtual outdoor conditions are associated with a second geographic location different from the first geographic location.

7. The method of claim 1, the method comprising capturing an image of the indoor scene, wherein rendering the indoor scene comprises rendering the image of the indoor scene based on the lighting model of the indoor scene.

8. The method of claim 7, wherein the method includes removing lighting effects from the image before applying the lighting model of the indoor scene.

9. The method of claim 7, wherein the method includes illuminating virtual objects inserted into a rendered image according to the lighting model of the indoor scene.

10. The method of claim 7, wherein at least one region of the image is not modeled in the geometric model, and wherein the at least one region is color-corrected based on the lighting model.

11. The method of claim 2, wherein the opacity of the opening can be configured via a user interface.

12. An apparatus for adjusting scene rendering, the apparatus comprising at least one processor configured to: - Obtain virtual outdoor conditions from the user interface, wherein the virtual outdoor conditions are defined by user-selectable parameters, which include at least one of a day of the year, the time of day, and weather conditions; - Obtain information indicating the indoor scene to be rendered under virtual outdoor conditions; - A general lighting model is obtained based on the virtual outdoor conditions, wherein the general lighting model is independent of the indoor scene; - Based on the geometric model of the indoor scene, the general lighting model, and the lighting model of the indoor scene obtained based on the virtual outdoor conditions; and - Adjust the scene rendering using any of the following: ○ Send the lighting model of the indoor scene to an external device for rendering; and ○ Render the indoor scene based on the lighting model of the indoor scene.

13. The apparatus of claim 12, wherein the interior scene includes an opening to the outside, and the geometry of the interior scene includes a model of the opening.

14. The apparatus of claim 13, wherein a lighting model element corresponding to the opening is obtained based on the virtual outdoor conditions and based on the model of the opening.

15. The apparatus of claim 14, wherein the model of the opening includes an orientation of the opening relative to either north or vertical, and the lighting model element corresponding to the opening is based on the virtual outdoor conditions and the orientation.

16. The apparatus of claim 14, wherein the model of the opening includes the location of the opening, and the lighting model element corresponding to the opening is based on the virtual outdoor conditions and the location.

17. The apparatus of claim 12, wherein the indoor scene is associated with a first geographical location, and wherein the virtual outdoor conditions are associated with a second geographical location different from the first geographical location.

18. The apparatus of claim 12, wherein capturing an image of the indoor scene, and wherein rendering the indoor scene comprises rendering the image of the indoor scene based on the lighting model of the indoor scene.

19. The apparatus of claim 18, wherein lighting effects are removed from the image before the lighting model of the indoor scene is applied.

20. The apparatus of claim 18, wherein the virtual object inserted into the rendered image is illuminated according to the lighting model of the indoor scene.

21. The apparatus of claim 18, wherein at least one region of the image is not modeled in the geometric model, and wherein the at least one region is color-corrected based on the lighting model.

22. The apparatus of claim 13, wherein the opacity of the opening is configurable via a user interface.

Images