Physical-based transmitter estimation for indoor scenes

Abstract

Systems and techniques are provided for physical-based light estimation for inverse rendering of indoor scenes. For example, a computing device can obtain an estimated scene geometry based on a multi-view observation of a scene. The computing device can also obtain a light emission mask based on the multi-view observation of the scene. The computing device can also obtain an emission radiation field based on the multi-view observation of the scene. The computing device can then determine a geometry of at least one light source of the estimated scene geometry based on the light emission mask and the emission radiation field.

Description

Technical Field

[0001] Aspects of the present disclosure generally relate to rendering indoor scenes given single-view or multi-view input.For example, aspects of the present disclosure include systems and techniques for performing physically based emitter estimation for indoor scenes. Background Art

[0002] Inverse rendering of indoor scenes has always been a challenging task in computer vision. These efforts aim to estimate the intrinsic properties of indoor scenes, including geometry, materials, and spatially varying illumination, from single-view or multi-view observations. In single-view inverse rendering, given the limited information about scene appearance and the inherent ambiguity between geometry, materials, and illumination, most methods either rely heavily on heuristic priors and regularization, or rely on priors learned from a large collection of indoor scenes. Multi-view inverse rendering assumes that multi-view observations of a scene are usually coupled with a given camera pose, which greatly alleviates the appearance ambiguity and can therefore be addressed using optimization-based methods using a view synthesis objective. Summary of the Invention

[0003] The following presents a simplified summary of one or more aspects disclosed herein. Therefore, the following summary should neither be considered an exhaustive overview of all contemplated aspects nor be considered to identify key or critical elements related to all contemplated aspects or to delineate the scope associated with any particular aspect. Therefore, the sole purpose of the following summary is to present certain concepts related to one or more aspects of the mechanisms disclosed herein in a simplified form prior to the detailed description presented below.

[0004] Systems and techniques for implementing a physically based light estimation approach for inverse rendering of interior scenes are disclosed. According to an illustrative example, an apparatus for performing light source estimation is provided. The apparatus includes at least one memory and at least one processor, the at least one processor being coupled to the at least one memory and configured to: obtain an estimated scene geometry based on multi-view observations of a scene; obtain a light emission mask based on the multi-view observations of the scene; obtain an emitted radiation field based on the multi-view observations of the scene; and determine the geometry of at least one light source of the estimated scene geometry based on the light emission mask and the emitted radiation field.

[0005] In another illustrative example, a method for performing light source estimation is provided. The method includes: obtaining an estimated scene geometry based on a multi-view observation of a scene; obtaining a light emission mask based on the multi-view observation of the scene; obtaining an emitted radiation field based on the multi-view observation of the scene; and determining a geometry of at least one light source of the estimated scene geometry based on the light emission mask and the emitted radiation field.

[0006] In another illustrative example, a non-transitory computer-readable medium is provided having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: obtain an estimated scene geometry based on a multi-view observation of a scene; obtain a light emission mask based on the multi-view observation of the scene; obtain an emitted radiation field based on the multi-view observation of the scene; and determine the geometry of at least one light source of the estimated scene geometry based on the light emission mask and the emitted radiation field.

[0007] In another illustrative example, an apparatus for performing partial segmentation is provided. The apparatus includes: means for obtaining an estimated scene geometry based on a multi-view observation of a scene; means for obtaining a light emission mask based on the multi-view observation of the scene; means for obtaining an emitted radiation field based on the multi-view observation of the scene; and means for determining a geometry of at least one light source of the estimated scene geometry based on the light emission mask and the emitted radiation field.

[0008] In some aspects, one or more of the devices described herein are and / or include and / or are part of: an extended reality (XR) device or system (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a mobile device (e.g., a mobile phone or other mobile device), a wearable device, a wireless communication device, a camera, a personal computer, a laptop computer, a vehicle or a computing device or component of a vehicle, a server computer or server device (e.g., an edge or cloud-based server, a personal computer acting as a server device, a mobile device such as a mobile phone acting as a server device, an XR device acting as a server device, a vehicle acting as a server device, a network router, or other device acting as a server device), another device, or a combination thereof. In some aspects, the device includes one or more cameras for capturing one or more images. In some aspects, the device also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the devices described above may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyrometers, one or more accelerometers, any combination thereof, and / or other sensors).

[0009] The features and technical advantages of the examples according to the present disclosure have been outlined quite broadly above so that the detailed description that follows may be better understood. Additional features and advantages will be described below. The concepts and specific examples disclosed may be readily used as a basis for modifying or designing other structures for achieving the same purposes of the present disclosure. Such equivalent constructions do not depart from the scope of the appended claims. The characteristics of the concepts disclosed herein, both in terms of their organization and method of operation, and the associated advantages will be better understood by considering the following description in conjunction with the accompanying drawings. Each of the figures in the accompanying drawings is provided for the purpose of illustration and description and not as a definition of limitations to the claims.

[0010] Although various aspects are described in this disclosure by illustrating some examples, it will be understood by those skilled in the art that such aspects can be implemented in many different arrangements and scenarios. The technology described herein can be implemented using different platform types, devices, systems, shapes, sizes and / or packaging arrangements. For example, some aspects can be implemented via integrated chip implementations or other devices based on non-module components (e.g., end-user devices, vehicles, communication equipment, computing equipment, industrial equipment, retail / shopping equipment, medical equipment and / or artificial intelligence devices). Various aspects can be implemented in chip-level components, modular components, non-modular components, non-chip-level components, device-level components and / or system-level components. The equipment incorporating the various aspects and features described may include additional components and features for implementing and practicing the various aspects claimed and described. The various aspects described herein are intended to be practiced in various devices, components, systems, distributed arrangements and / or end-user devices of various sizes, shapes and compositions.

[0011] Other objects and advantages associated with the various aspects disclosed herein will be apparent to those skilled in the art based on the drawings and detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. This subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all of the drawings, and each claim.

[0012] The foregoing and other features and aspects will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are presented to aid in describing various aspects of the present disclosure and are provided solely for illustration and not limitation of the aspects.

[0014] Figure 1 illustrates an example single-view inverse rendering of an indoor scene according to some examples;

[0015] Figure 2illustrates multi-view inverse rendering of an indoor scene using an optimization-based approach according to some examples;

[0016] Figure 3 illustrates examples of outputs of applications of emitter estimation approaches according to some examples;

[0017] Figure 4 A block diagram illustrating a method for obtaining scene geometry, material, and emission estimates according to some examples is illustrated;

[0018] Figure 5 illustrates images showing emitter representation and initialization according to some examples;

[0019] Figure 6 illustrates estimated scene geometry and luminaire geometry masked by an emission mask according to some examples;

[0020] Figure 7 illustrates an estimated environment map obtained by scattering observations through a window in all views onto a global environment map according to some examples;

[0021] Figure 8 illustrates an estimated outdoor environment map according to some examples, where the sun is captured and ground truth is used;

[0022] Figure 9A illustrates an initial environment map estimate, a fitted environment map, and a ground truth map according to some examples;

[0023] Figure 9B illustrates a bidirectional reflectance distribution function with albedo values, roughness values, and metallic values ​​according to some examples;

[0024] Figure 10 Rendering equations according to some examples are illustrated;

[0025] Figure 11 illustrates scenes with various approximations of rendering equations using a multilayer perceptron according to some examples;

[0026] Figure 12 illustrates scenes with various approximations of rendering equations using mesh functions according to some examples;

[0027] Figure 13 is a flowchart illustrating an example method of performing light estimation according to some examples; and

[0028] Figure 14 is a block diagram illustrating an example of a computing system according to some examples. DETAILED DESCRIPTION

[0029] For illustrative purposes, certain aspects of the present disclosure are provided below. Without departing from the scope of the present disclosure, alternative aspects may be designed. Additionally, well-known elements of the present disclosure will not be described in detail or will be omitted to avoid making the relevant details of the present disclosure difficult to understand. Some aspects described herein can be applied independently, and some of them can be applied in combination, which will be apparent to those skilled in the art. In the following description, specific details are set forth for explanation purposes to provide a thorough understanding of various aspects of the application. However, it will be apparent that various aspects can be implemented without these specific details. Each drawing and description is not intended to be restrictive.

[0030] The following description provides only exemplary aspects and is not intended to limit the scope, applicability, or configuration of the present disclosure. Instead, the following description of the exemplary aspects will provide those skilled in the art with a description that can be used to implement the exemplary aspects. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the scope of the present application as set forth in the appended claims.

[0031] As mentioned above, inverse rendering of indoor scenes is a challenging task in computer vision. The goal of inverse rendering is to estimate the intrinsic properties of indoor scenes, including geometry, materials, and spatially varying lighting, from single-view (e.g., a single image) or multi-view (e.g., multiple images) observations. Figure 1 An example of a series of images 100, including images (a) through (h), is illustrated. Given a single image of an indoor scene (e.g., image (a)), the system can recover its diffuse albedo (e.g., as shown in image (b)). Albedo is the fraction of light reflected by a surface. For example, if all light is reflected from a corresponding surface or at a corresponding point on a surface, the albedo is equal to one. In another example, if 30% of the light is reflected, then the albedo for that specific surface or location on the surface is 0.3.

[0032] The system or device can recover data associated with the scene, such as normal values ​​(e.g., Figure 1 (c), specular roughness values ​​(e.g., as shown in image (d)), depth values ​​(e.g., as shown in image (e)), and spatially varying lighting (e.g., as shown in image (f)). The method also enables downstream applications such as object insertion, even for specular objects (e.g., as shown in image (c)). Figure 1 (g) and in a real image (e.g., as shown in image (h)). Obtaining an accurate and usable representation of the scene can enable further uses, such as interpolating images or enabling a physical robot to move around a room and accurately perform tasks.

[0033] This paper describes systems and techniques for performing multi-view inverse rendering for indoor scenes. Generally speaking, multi-view inverse rendering for indoor scenes is an emerging task in recent years, and most of them adopt optimization-based methods that estimate dense properties of all surfaces, including emitted radiance and emission masks. However, existing methods do not model emitters as parameterized models (e.g., a single semantic object with emission characteristics including global radiance), making it difficult to independently and heuristically control each emitter (e.g., turn a light fixture on / off or change a light fixture from white to red). In addition, those methods rely on expensive path tracing to optimize dense emission properties, which may take hours to days to converge in terms of training neural networks and may produce uneven emission across the emitter surface.

[0034] Furthermore, recent years have seen the emergence of methods that use neural radiance fields for single-object inverse rendering tasks or intrinsic decomposition of indoor scenes. These methods either focus on single-object settings where lighting and emitters are assumed to be global, or do not attempt to recover emission data at all. One existing approach attempts to model lighting as a spatially varying neural radiance field, but does not model parameterized emitters and does not incorporate physically based light transport constraints to ensure the physical plausibility of the estimated lighting.

[0035] Figure 2 2 is a diagram illustrating an approach to scene lighting and emitter estimation, wherein an optimization-based method is used to perform multi-view inverse rendering. Dense surface properties including material (e.g., albedo and roughness) and emission (e.g., emission values ​​and masks) may be estimated from the scene's geometry and a target view 202, wherein inverse path tracing 204 is applied to generate a rendering 206 having values ​​associated with the material and emission properties.

[0036] The emission mask may include a value of "1" for objects that emit light, such as Figure 2 , which is a flat screen TV. An emission mask 210 identifies the emissive portion of the image (e.g., region 208) and the non-emissive portion of the image (e.g., region 212). The emission mask will identify such emissive and non-emissive regions for each light source in the scene.

[0037] Lighting estimation for indoor scenes is valuable for downstream tasks including light editing and virtual object insertion. Early algorithms modeled indoor lighting as a global environment map, which treated the environment map as a single emitter surrounding the interior of the scene. Later research progressed on dense spatially varying lighting, but did not model globally consistent or semantic emitters, and thus was not suitable for heuristic lighting editing. Some approaches modeled lighting as dense emitted radiance and emission masks on scene surfaces, also failing to identify emitters. On the other hand, some approaches adopt a parameterized representation of emitters as semantic objects in the scene, while other approaches model emitters as distant point lights, and more recent approaches model both area lights (e.g., luminaires as area lights of geometric shapes and a single global radiance) and outdoor environment maps. This approach is in Figure 3 , which shows an example of the output and application of this approach to emitter estimation 300, where two types of parameterized emitters are considered. For example, this approach allows turning off lamps, inserting objects, inserting virtual light sources, and opening virtual windows. An example of this approach is found in Li, Zhengqin et al., “Physically-Based Editing of IndoorScene Lighting from a Single Image,” Computer Vision - ECCV 20-22; 17th European Conference on Computer Vision, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Section VI, which is incorporated herein by reference.

[0038] Due to the limited generalization capabilities of the above approach, this method simplifies the outdoor environment mapping using three spherical Gaussian lobes, sacrificing high-frequency outdoor details and resulting in artifacts when rendering windows and strong highlights or shadows cast through windows. By back-projecting the depth map into three-dimensional space, luminaires are modeled using fixed geometry, which often has artifacts and cannot be optimized in subsequent stages. For example, this approach generates the geometry of a first light source (which can be of different types, such as indoor light sources such as luminaires or outdoor light sources such as the sun or other light outside a building) to be differentially optimized after initialization.

[0039] The systems and techniques described herein provide a pipeline for end-to-end light estimation and light editing that can model the complete 3D scene geometry and lighting based on Neural Radiance Fields (NeRF) and can take a sequence of scene observations as input. These systems and techniques explicitly estimate parameterized physical light sources and globally consistent light transport in an end-to-end trainable pipeline that can be optimized on a scene. The differentiable neural renderer shown below is able to efficiently render the scene.

[0040] The systems and techniques described in this article are applicable to a variety of applications, including augmented reality (AR), indoor navigation, etc., in which understanding the lighting in indoor environments is critical for light editing, material editing, virtual object insertion, and home robot manipulation under challenging light conditions. These systems and techniques are applicable to a wide range of applications and products, including scene editing and light editing for AR (mobile phones, glasses), home robot navigation, etc.

[0041] The systems and techniques described herein can use light estimation to better distinguish between emission and reflection in a scene and provide an emitter initialization process from a stored emission radiance field. These systems and techniques also provide emitter optimization using differentiable rendering. An example apparatus for performing light source estimation to solve the above problems (e.g., Figure 14 The system 1400 may include: at least one memory; and at least one processor coupled to the at least one memory and configured to: obtain an estimated scene geometry based on a multi-view observation of a scene; obtain a light emission mask based on the multi-view observation of the scene; obtain an emitted radiation field based on the multi-view observation of the scene; and determine a geometry of at least one light source of the estimated scene geometry based on the light emission mask and the emitted radiation field. The geometry may be optimized and configured to enable scene editing and separate light editing over the previous process.

[0042] Figure 4 A block diagram 400 is illustrated of an approach to obtaining scene geometry, material, and emission estimates according to some examples. The data obtained can be used to initialize an initialization engine 414 in novel ways that enable new capabilities, such as the ability to independently and heuristically control each emitter in the scene in the sense of seeking faster solutions or the ability to independently control emitters. In this approach, a multi-view observation of an indoor scene is given. An algorithm (e.g., such as MonoSDF (single signed distance function)) 402 can be used to obtain scene geometry 404. In one example using the MonoSDF approach, the scene geometry 404 can be represented as a signed distance function (SDF). The signed distance function is a continuous function f that, for a given three-dimensional (3D) point, returns the distance from the point to the nearest surface:

[0043]

[0044] In equation (1), x is a 3D point and s represents the corresponding SDF value. The system can parameterize the SDF function with learnable parameters θ and explore several different design choices for representing the function: explicitly as a dense grid of learnable SDF values, implicitly as a single multi-layer perceptron (MLP), or using a hybrid of an MLP in combination with a single-resolution or multi-resolution feature grid. Details of this example approach for obtaining or generating scene geometry 404 can be found in Yu, Zehao et al. “Monosdf: Exploring monocular geometric cues for neural implicit surfacereconstruction”, arXiv preprint arXiv:2206.00665(2022), which is incorporated herein by reference. Other algorithms or approaches for obtaining or generating scene geometry 404 are also contemplated. For example, any three-dimensional reconstruction method such as multi-view stereo or neural radiance field can be used. In general, algorithm 402 produces scene geometry 404 represented using an SDF. Using a given geometry, the system (e.g., Figure 4 The system 1400) may estimate material data 412, emitted radiation field 410, and emission mask 408. Data for emission mask 408, emitted radiation field 410, and material data 412 may be determined or obtained offline as part of a pre-processing pipeline.

[0045] Other data may be used to start the emitter via the initialization engine 414. For example, the algorithm 406 may be used to obtain one or more of the emission mask 408, the emission radiation field 410, and / or data regarding the material data 412. The data may be an estimate based on the operation of the algorithm 406. In one aspect, the emission mask 408 may be represented as: α = NN emit (x), where α∈[0,1] is the probability of a surface point being emitting. If α is equal to 1, the corresponding point in the scene is emitting. If α is equal to 0, the point is not emitting, but it may be reflecting.

[0046] Initialization of the emitter via the initialization engine 414 may utilize the scene geometry 404, the emission mask 408, and the emission radiation field 410. The geometry and physics of the scene may be used to initialize the emitter (or multiple emitters in the scene) via the initialization engine 414. Figure 14 The system 1400 may include a Figure 4 The initialization engine or module of the initialization engine 414 block in the scene initializes the position of each emitter (or light source), the geometry of each emitter in the scene, and the physical intensity of the emitter. The device can also use differentiable rendering to optimize the emitter.

[0047] In one aspect, the scene geometry 404 may be assumed to be given from a three-dimensional reconstruction of a real scene or from the ground truth geometry of a synthetic scene.The scene geometry may be represented as a collection of triangles or a triangle mesh.

[0048] Figure 5

[0046] An image 500 illustrates exemplary emitter representation and initialization according to some examples. A first image 502 illustrates initializing the geometry of a window. A second image 504 illustrates initializing a lamp or other indoor light source. These initializations are performed based on a given estimated scene geometry 404 and / or emission mask 408.

[0049] Given an emission mask 408, area lights (e.g., lamps or other light sources such as ceiling lights or wall sconces) and windows through which light from an outdoor environment map is projected into the room are masked on the estimated scene geometry 404. Utilizing the scene geometry 404 and the availability of vertices including corresponding triangles on the triangle mesh, there is an emission mask 408 that can be defined at each surface point. Where the emission mask 408 has a value of "1" for a given surface location, then the associated vertex can be selected to identify the emitter. The emission mask 408 can be defined on the triangle mesh, including material data 412. This process generates or produces lamps (or other interior lights) as separate shapes, and windows as holes carved out in the wall, as shown in image 502.

[0050] Each luminaire can be represented as a semantic object with implicit geometry (e.g., a signed distance function or signed distance field (SDF)) and physical properties (e.g., global radiance). The SDF is a way of representing geometry. The input to the process can include positions, and the output can be an SDF value. If the point or position in the scene is outside the surface, the value is positive, and if the point is inside the surface, the value is negative. Positions on the surface have a value of zero. The input is any spatial position, and the output is the SDF to the surface. This can be used as a representation of geometry.

[0051] This representation allows for hybrid geometry representations, where implicit SDF parameters can be converted to a triangle mesh using marching cubes, allowing the luminaire geometry to be optimized differently after initialization. The marching cubes step may or may not be differentiable. For example, neural implicit evolution methods can be applied to optimize the scene SDF, a process that does not require the marching cubes step to be differentiable. An example of a "marching cubes" algorithm is an algorithm that creates a triangular model of a constant-density surface from 3D data. Using a divide-and-conquer approach to generate inter-slice connectivity, a case table defining the triangle topology is created. The algorithm processes the 3D data in scanline order and computes the triangle vertices using linear interpolation. The gradients of the original data can be normalized and used as the basis for shading the model. The detail in the image produced from the generated surface model is the result of maintaining the inter-slice connectivity, surface data, and gradient information present in the original 3D data. The "marching cubes" approach is just one of many different approaches that can be applied to convert scene geometry parameters into a collection of triangles or a triangle mesh. This process enables each emitter (e.g., a luminaire or other indoor light source) to be optimized individually and independently after initialization. Representing emitters with implicit geometry in SDF makes it easier to optimize the data, as described in this paper.

[0052] On the other hand, indoor light sources (e.g. Figure 6 The luminaires 606, 610 shown in image 602 may be physically modeled as global radiance values, which indicates that the light source emits light omnidirectionally and uniformly from all surfaces of the light source. Figure 6 An image 600 is illustrated that includes an estimated scene geometry and a first image 602 of the luminaire geometry masked by an emission mask. The geometry of the luminaire is carved out of the estimated scene geometry 404 using the emission mask 408. Initial radiance values ​​can be obtained by sampling the emitted radiation field 410 at the luminaire surface location and taking the median of the emitted radiation field 410. When there is a set of triangles or a triangle mesh available for the geometry, for each triangle of the triangle mesh, the system can sample the radiation from the corresponding triangle. Sampling the radiation can be done using two types of data, including the position and direction within the corresponding triangle (e.g., the direction of the normal to the surface or point on the corresponding triangle). The system can pass this data to the radiation field to query the radiation. For example, as described in more detail below, the system can sample multiple points x on each triangle. i Sampling is performed, and for the corresponding triangle, it can be determined whether the average value of the sampled data is greater than a threshold. If the average value of the triangle is greater than the threshold, the system can classify the triangle as an emitter. If the average value of the sampled data is equal to or lower than the threshold value for the corresponding triangle, the system can classify the corresponding triangle as a non-emitter.

[0053] exist Figure 6 In image 602, there are three light sources, including luminaire 606, ceiling light 608, and luminaire 610. The initialized emitter radiance can be visualized as a line (not shown) on the surface of each corresponding luminaire, where the length of the line corresponds to the amplitude. The radiance can be determined as a scalar value and can be omnidirectional and uniform in one aspect. Image 604 represents the ground truth scene and luminaire geometry with ground truth luminaire radiance lines (not shown).

[0054] Figure 7 700 illustrates an estimated environment map 702 obtained by scattering observations through a window 704 in all views onto a global environment map 708. From the window 704, a given window can be carved out or represented as a hollow structure in the wall, as shown in image 706. Observations through the window (such as in image 706) can include a frame in which the sun (or other outdoor light source) is directly observed. The lighting can represent a global environment map 708, which can be imagined as the upper hemisphere and can depend on the angle and can be independent of the spatial position (for example, the spatial position can not be a factor because the dependence is based on the position of the observer looking or gazing). By definition, the global environment map is independent of position because it represents long-distance lighting. It can be assumed that the sun or outdoor light source is very far away from the observer in the scene or room. The system can sample rays and if the ray passes through the hollow structure, the system can obtain an observation of the environment map.

[0055] Figure 8 Image 800 is illustrated, including a first image of an estimated outdoor environment map associated with image 802 in which the sun is captured, and a second image of a ground truth environment shown in image 804. In one example, the sun or other outdoor light source or light emitter can be used, and at least one window or representation of a window can have a direct view of the outdoor light source. Image 802 shows an illustration of an input image and the estimated environment map. Image 804 illustrates the ground truth environment.

[0056] From image 802, the system can sample camera rays and obtain an environment map. Image 802 shows the set of all available views. However, image 802 shows some missing parts because it shows the input image and the estimated environment map, but other areas are blank. The system can be represented as Figure 7 The upper hemisphere parameterization of the global environment map 708 is shown. Using the two observations of the window, the system can select the view through the window and fill in the missing details of the image 802. MLP can be used to fill in the gaps. The camera rays can be sampled at the observations, and the corresponding image intensities can be converted from Figure 7 Representative points 712 in are scattered onto the global environment map 708 to make an initial estimate.

[0057] Figure 9A Image 900 is illustrated, which includes an initial environment map estimate shown in image 902, a fitted environment map shown in image 904, and a ground truth map shown in image 906. In order to allow the window radiance (e.g., the environment map estimate of image 902) to be optimized differentiably in a later stage, similar to the light fixture, the disclosed approach is to convert the initial partial environment map into an implicit representation using a multi-layer perceptron (MLP). Note that in the estimated environment map shown in image 902, there are missing parts of the image. The gaps need to be filled from the initial estimate. To do this, the system fits an MLP to the initial environment map, as shown in image 904, and optimizes using consistency in the valid region. The final representation of the outdoor environment map is an MLP that covers the entire hemisphere visible from the window, as shown in image 904. Image 906 represents the ground truth of the image.

[0058] Figure 9B An image 920 illustrates a bidirectional reflectance distribution function having albedo values, roughness values, and metallic values. For example, a bidirectional reflectance distribution function (BRDF) may illustrate albedo data as shown in image 922, metallic data as shown in image 924, and roughness data as shown in image 926. The material data may correspond to Figure 4 . Note that the albedo data in image 922 will typically be presented as different colors representing the fraction of light reflected at each point on the surface. The metal data shown in image 924 is entirely gray, indicating that there are no metal surfaces in the image. The roughness data in image 926 can be black and white, gray, or color to visually represent the roughness value or characteristics at each surface point in the scene. This data provides information about the properties of the materials in the scene.

[0059] In one aspect, the material data 412 can be represented as BRDF(a,m,σ)=NN brdf (x), where a∈[0,1] 3 albedo (base color); σ∈[0,1] roughness; m∈[0,1] metallic; and x∈R 3 The location of the surface point.

[0060] Figure 10 is a diagram 1000 illustrating a differentiable rendering technique (eg, utilizing a differentiable rendering equation) that may be used to describe the total amount of light emitted from a point x along a viewing direction, according to some examples. Figure 10 The technique illustrated in exemplifies a given function for incident light and BRDF.

[0061] In one example, the rendering equation is an integral equation where the equilibrium radiance leaving a point is given as the sum of the emitted radiance plus the reflected radiance under a geometric optics approximation. Various realistic rendering techniques in computer graphics attempt to solve this equation. The physical basis of the rendering equation is the law of conservation of energy. In one aspect, let L denote the radiance, and at each specific location and direction, the outgoing light (denoted as L o ) can be the emitted light (denoted as L e ) and the sum of the reflected light. The reflected light itself is the sum of all directions of the incident light (denoted as L i ) multiplied by the sum of the surface reflection and the cosine of the incident angle.

[0062] The rendering equation can be written as follows:

[0063] l0(x,w0,λ,t)=L e (x,w0,λ,t)+∫ Ω f r (x,w i ,w o ,λ,t)L i (x,w i ,λ,t)(w i ·n)dw i , (2)

[0064] Where L0(x,ω o ,λ,t) is the total spectral radiation of wavelength λ directed outward from a specific location x at time t along the direction, where x is the location in space, and ω o is the direction of the emitted light, λ is the specific wavelength of light, t is the time, L e (x,ω o ,λ,t) is the emitted spectral radiation, ∫ Ω …w i is the integral over Ω, which is the integral containing ω i All possible values ​​of the unit hemisphere centered on n,

[0065] where ω i* ,f r (x,ω i ,λ,t) is the bidirectional reflectance distribution function, at position x, time t and at wavelength λ from ω i Reflection to ω o The light ratio, ω i is the negative direction of the incident light, L i (x,ω i ,λ,t) is the time t from the direction ω i spectral radiation of wavelength λ incident inward toward x, where n is the surface normal at x, and ω i*n is the attenuation factor of the outward irradiance due to the angle of incidence, as the luminous flux is distributed over a surface larger than the projection area perpendicular to the ray.

[0066] Several features of equation (2) include its linearity (it consists only of multiplications and additions) and its spatial uniformity (it is the same at all positions and orientations). These features of equation (2) provide a wide range of factorizations, and rearrangements of the equation are possible. Equation (2) is a Fredholm integral equation of the second kind, similar to those that arise in quantum field theory.

[0067] In equation (2), the spectral and time dependence (L o ) can be sampled or integrated over a portion of the visible spectrum to obtain, for example, three primary color samples. By fixing t, the pixel values ​​of a single frame in an animation can be obtained; motion blur can be obtained by adjusting L at a given time interval. o The average is generated (e.g., by integrating over a time interval and dividing by the length of the interval).

[0068] In one example, the solution to the rendering equation may be the function L o Function L i Through ray tracing operation and L o For example, the incoming radiation from a certain direction at one point is the outgoing radiation at some other point in the opposite direction.

[0069] Figure 11 A scene 1100 is illustrated with various approximations of the above rendering equation using a multi-layer perceptron. Figure 12 A scene 1200 is illustrated having rendering equations similar to those used above and using various approximations of the mesh function.

[0070] According to the techniques described herein, the system can query N incident radiation samples at locations across the entire scene, regardless of the shape type of the sampling location (e.g., both emitters and non-emitting objects). i =mesh(SDF(Ω j ),δ j ;x i ,d i ), which is composed of the optimizable SDF parameter Ω j The parameterized explicit mesh is again a physical property (e.g., the universal single radiation intensity δ of a luminaire j), the system can query the emitter j model instead of the radiance cache MLP to obtain the incident radiance for sample i. Benefits of this approach may include that by distinguishing queries on emitters from queries on non-emitting surfaces, the rendered image will be a function of the emitter parameters, which allows the user to optimize the emitter parameters differentiably by minimizing the error on the rendered image. Furthermore, queries of emitter surfaces using the original radiance cache MLP may return varying intensities for a single emitter of an area light (e.g., a luminaire), whereas the result should ideally be uniform instead, equal to a single radiance intensity δ j .

[0071] However, using the techniques described herein, the system can achieve this by explicitly estimating the parameters of each emitter and ensuring that the radiation samples at one emitter are subject to a smaller set of parameters (e.g., emitter shape Ω j and physical properties δ j ) to better regularize the radiation space. Windows as emitters are parameterized differently from lamps, through L i =MLP(δ j ;d i ), because windows, as cutouts in the wall, have a known shape and therefore no geometric parameters to optimize, and they provide non-spatially varying illumination, since the outdoor light cast into the room from the window is assumed to originate from a global environment map in the distance (e.g., the sun and clouds), and the incident radiation depends only on the radiation direction d i It does not depend on the position x in the room i .

[0072] Once the rendering equation is formulated as a function of the BRDF parameters and the additional emitter parameters, the system is able to optimize both parameters at the expense of image rendering and obtain physically accurate parameterized emitters (e.g., a box-shaped light fixture with an intensity of 100 for each RGB channel), which is useful for downstream applications including intuitive light editing and relighting.

[0073] and Figure 10The associated differentiable rendering equations utilizing parameterized emitters use the emitter only for the direct lighting component (e.g., incident radiation that does not originate from the emitter is queried using the radiance cache MLP). However, there are other differentiable rendering techniques that allow only emitters to be used for both direct and indirect lighting. One such technique is inverse path tracing (IPT), where all sampled rays originating from a position x branch into ray paths that all terminate on the emitter. In this way, the pre-trained radiance cache MLP used by some approaches is no longer useful, and the present system will be able to optimize emitter parameters without a pre-trained component. However, IPT is known to be extremely inefficient because it requires a large number of path samples and has low stability and unbiasedness. IPT is a possible second stage that can be used for emitter optimization, well initialized by the aforementioned pipeline. In this way, IPT is expected to converge faster and be able to improve the global consistency of radiance for the entire scene.

[0074] Figure 13 is a flow chart illustrating an example of a process 1300 for performing initialization of emitters from a scene or for performing light estimation. The operations of process 1300 may be implemented as a process on one or more processors (e.g., Figure 14 software components executed and run on the processor 1410 and / or other processors).

[0075] At block 1302, process 1300 may include obtaining an estimated scene geometry (e.g., obtained or generated from, for example, MonoSDF algorithm 402 or other algorithms) based on a multi-view observation of the scene. Figure 4 The scene geometry of 404 may be an indoor scene. However, other scenes are also contemplated, such as partially indoor scenes or other scenes having similar characteristics to indoor scenes. The estimated scene geometry may be expressed as a signed distance function.

[0076] In one aspect, the signed distance function may comprise a continuous function f. For example, for a given 3D point, function f returns a value representing the distance from the given 3D point to the nearest surface in the scene. The estimated scene geometry may be generated based on one or more of depth fusion (e.g., truncated signed distance function or TSDF fusion), monocular depth estimation (e.g., using a single image), multi-view stereo, and / or neural radiance field.

[0077] At block 1304, process 1300 may include obtaining a light emission mask (eg, Figure 4

[0066] The light emission mask 408 may include information indicating a likelihood that corresponding surface points in the scene are associated with light emission. In one exemplary aspect, the light emission mask may include a first value for surface points associated with light emission and a second value for surface points not associated with light emission, the second value being different from the first value.

[0078] At block 1306, process 1300 may include obtaining an emitted radiation field (eg, Figure 4 of the emitted radiation field 410).

[0079] At block 1308, process 1300 may include determining a geometry of at least one light source that estimates scene geometry based on the light emission mask and the emitted radiation field (eg, via Figure 4 The initialization engine 414 in initializes one or more transmitters).

[0080] In one aspect, process 1300 may also include generating a separate shape for the at least one light source, or generating a hole defined in a wall in the scene that represents the at least one light source (e.g., Figure 5 The image 502 shows the geometry of the window as a hole or opening in the wall).

[0081] In one aspect, a first light source of the at least one light source may comprise an indoor light (e.g., Figure 6 606, 610 or ceiling light 608 in the images 602, 604), and a second light source in the at least one light source comprises a window (e.g., Figure 5 502). The first light source can be represented as having physical properties modeled as a global radiance value (e.g., a single scalar value). Process 1300 can include determining a global radiance value based on sampling the emitted radiation field at the surface location of the first light source to generate sampled data and determining an average or median of the sampled data. Not shown in image 604 are arrows or vectors projected from the surface of the emitters (e.g., luminaires 606, 610 and ceiling light 608), with the length of each line corresponding to the magnitude of the radiance.

[0082] In one aspect, process 1300 may include representing the first light source as a semantic object having an implicit geometry associated with a signed distance function (SDF) to generate implicit SDF parameters. To extract the emitter geometry from the scene geometry, in one aspect, process 1300 may directly assume that the geometry is a triangle mesh. The geometry reconstruction result can be an SDF or a mesh, but in one aspect, it does not matter as long as the method includes converting the result to a mesh. Process 1300 may also include converting the implicit SDF parameters into a triangle set or a triangle mesh. In some cases, it may be assumed that the emission of all points on each triangle is the same and that the emitter reflects zero light.

[0083] In some aspects, the process 1300 may sample a set of points of corresponding triangles in the set of triangles to generate sampled data. For example, the system may sample 100 points x on each triangle. i Sampling is performed to obtain α i =NN emit (x i ), i=1, ..., 100. The process 1300 can then determine whether the average value of the sampled data for the corresponding triangle is greater than a threshold. For example, if the average value ([x1, ..., x 100 ])>0.45, the corresponding triangle is classified as an emitter. In this regard, process 1300 can obtain emitter geometry. The threshold of 0.45 is an example for illustrative purposes, and other thresholds can be used. Process 1300 can classify the corresponding triangle as an emitter based on the average value of the sampled data being greater than the threshold value for the corresponding triangle. Process 1300 can classify the corresponding triangle as a non-emitter based on the average value of the sampled data being equal to or less than the threshold value for the corresponding triangle. Classifying a triangle as a non-emitter can indicate that the triangle is not a light source.

[0084] In some cases, for each training image view, if the corresponding triangle intersects the camera ray, process 1300 may obtain the corresponding red, green, and blue (RGB) values ​​to generate a set of pixel values ​​(e.g., RGB values), and then set the emission value n to the median of the pixel values ​​as part of the step of obtaining the emission value for the triangle or portion of the triangle. When using RGB values, the approach is based on the RGB color model, in which red, green, and blue are primary colors and are added together in various ways to reproduce a variety of different colors.

[0085] In another aspect, process 1300 may determine that a corresponding triangle intersects a camera ray. Based on the intersection of the corresponding triangle and the camera ray, process 1300 may obtain pixel values ​​corresponding to the corresponding triangle to generate a set of pixel values. Process 1300 may then set an emission value for the median of the set of pixel values.

[0086] In one aspect, a multi-view observation of a scene may include at least one input image where a primary light source can be seen through a window (e.g., Figure 7 706). Process 1300 may also include estimating an initial partial environment map based on the sampled camera rays and the corresponding image intensities scattered onto the global environment map. Process 1300 may use a multi-layer perceptron (MLP) to convert the initial partial environment map into an implicit representation. Process 1300 may use the MLP to fit observations having red, green, and blue (RGB) values ​​having camera rays that intersect the window geometry associated with the window to the initial partial environment map. In some cases, it may be assumed that a dominant light source (e.g., the sun or other outdoor light source) is visible in at least one of the input images. The approach may include using an MLP approach to fit the observations as follows: L i =MLP ENVMAP (ω). An observation can be the RGB value of where its camera ray intersects with the window geometry.

[0087] In another aspect, process 1300 may include generating spatially varying material properties associated with a scene (e.g., Figure 4 The spatially varying material properties may include at least one of an albedo value, a roughness value, or a metallic value associated with the scene (e.g., see Figure 9B 920 in the image). Spatially varying material properties can be represented via a bidirectional reflectance distribution function defined at each surface point in the scene. The bidirectional reflectance distribution function can represent an estimate of the albedo value, roughness value, and metallic value.

[0088] Figure 4 The optimization features 424 may include applying a differentiable rendering algorithm (e.g., Figure 4 In one aspect, the differentiable rendering algorithm renders pixel values ​​(e.g., Figure 4 The rendered image or pixel value 418) is compared to the input image pixel value (e.g., Figure 4 420 ) to generate a comparison value (e.g., Figure 4 422). A differentiable rendering algorithm (e.g., implemented by the differentiable renderer 416) may include, for example, one or more of an inverse path tracing process and / or a neural implicit evolution process. Other differentiable rendering algorithms may also be used.

[0089] The comparison 422 may be performed as follows: Where L(x,ω o ) is the rendered pixel value 418, and I(x,ω o ) are pixel values ​​from the training image 420, is a binary emission mask 408, and the emitter parameters can be emission values ​​η or parameters of the environment mapping MLP. Comparison 422 can generate a loss (via the forward pass of the neural network), which can be back-propagated as gradients back to the emitter parameterized model so that the emitter can be optimized. Back-propagation can tune the parameters of the neural network associated with the emitter initialization engine 414. The optimization can be a scalar value (global radiance value) or for the MLP envmap (ω). The system can use pre-computed terms to perform fast rendering for each view L o , the rendering loss can then be used to optimize the emitter parameters.

[0090] In one aspect, differentiable rendering is used to further refine the estimate of emitter parameters, where the estimated BRDF is fixed.

[0091] Process 1300 may include optimizing one or more of the first light source geometry and / or the second light source geometry based on the comparison value (e.g., Figure 4 Optimization features 424).

[0092] In some examples, the processes described herein (e.g., process 1300 and / or other processes described herein) can be performed by a computing device or apparatus (e.g., Figure 14 For example, a device for performing light source estimation (e.g., a computing system 1400) is used to perform light source estimation. Figure 14 The computing system 1400 may include at least one memory (e.g., Figure 14 1415) and at least one processor (e.g., Figure 14 The at least one processor is coupled to the at least one memory and configured to: obtain an estimated scene geometry based on a multi-view observation of the scene; obtain a light emission mask based on the multi-view observation of the scene; obtain an emitted radiation field based on the multi-view observation of the scene; and determine a geometry of at least one light source of the estimated scene geometry based on the light emission mask and the emitted radiation field.

[0093] In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, a computing device may include a display, one or more network interfaces configured to communicate and / or receive data, any combination thereof, and / or other components. The one or more network interfaces may be configured to communicate and / or receive wired and / or wireless data, including data according to 3G, 4G, 5G, and / or other cellular standards, data according to WiFi (802.11x), data according to Bluetooth, and / or other standards. TM Standard data, data according to the Internet Protocol (IP) standard and / or other types of data.

[0094] Components of a computing device may be implemented in circuitry. For example, a component may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0095] Process 1300 is illustrated as a logical flow diagram, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally speaking, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to implement the process.

[0096] Additionally, process 1300 and / or other processes described herein may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes together on one or more processors, implemented in hardware, or implemented by a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0097] Figure 14 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. Specifically, Figure 14 An example of a computing system 1400 is illustrated, which can be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any component thereof, wherein the components of the system communicate with each other using a connection 1405. Connection 1405 can be a physical connection using a bus, or a direct connection to processor 1410, such as in a chipset architecture. Connection 1405 can also be a virtual connection, a networked connection, or a logical connection.

[0098] In some aspects, computing system 1400 is a distributed system in which the functionality described herein can be distributed within a data center, multiple data centers, a peer-to-peer network, etc. In some aspects, one or more of the described system components represent a plurality of such components that each perform some or all of the functionality for which the component is described. In some aspects, a component can be a physical or virtual device.

[0099] Example system 1400 includes at least one processing unit (CPU or processor) 1410 and connections 1405 that communicatively couple various system components including system memory 1415, such as read-only memory (ROM) 1420 and random access memory (RAM) 1425, to processor 1410. Computing system 1400 may include cache 1415 of high-speed memory directly connected to, in close proximity to, or integrated as part of processor 1410.

[0100] Processor 1410 may include any general-purpose processor and hardware or software services, such as services 1432, 1434, and 1436 stored in storage device 1430, configured to control processor 1410 as well as a dedicated processor where software instructions are incorporated into the actual processor design. Processor 1410 may essentially be a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0101] To enable user interaction, computing system 1400 includes input device 1445, which can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. Computing system 1400 can also include output device 1435, which can be one or more of a plurality of output mechanisms. In some cases, a multimodal system can enable a user to provide multiple types of input / output to communicate with computing system 1400.

[0102] The computing system 1400 may include a communication interface 1440, which generally governs and manages user input and system output. The communication interface may perform or facilitate receiving and / or sending wired or wireless communications using wired and / or wireless transceivers, including using audio jacks / plugs, microphone jacks / plugs, Universal Serial Bus (USB) ports / plugs, Apple TM Lightning TM Ports / plugs, Ethernet ports / plugs, fiber optic ports / plugs, dedicated wired ports / plugs, 3G, 4G, 5G and / or other cellular data network wireless signal transmission, Bluetooth TM Wireless signal transmission, Bluetooth TM Low energy (BLE) wireless signal transmission, IBEACON TMThe communication interface 1440 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing system 1400 based on one or more signals received from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States' Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There is no restriction on operating on any particular hardware arrangement, and thus the base features herein may be readily substituted for improved hardware or firmware arrangements as they are developed.

[0103] The storage device 1430 may be a non-volatile and / or non-transitory and / or computer-readable memory device and may be a hard disk or other type of computer-readable medium that can store data accessible by a computer, such as a magnetic cassette, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette, a floppy disk, a floppy disk, a hard disk, a magnetic tape, a magnetic stripe / magnetic stripe, any other magnetic storage medium, flash memory, a memristor memory, any other solid-state memory, a compact disc read-only memory (CD-ROM) disc, a rewritable compact disc (CD) disc, a digital video disc (DVD) disc, a Blu-ray disc (BDD) disc, a holographic disc, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory stick card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASH EPROM), a cache memory (e.g., a level 1 (L1) cache, a level 2 (L2) cache, a level 3 (L3) cache, a level 4 (L4) cache, a level 5 (L5) cache, other (L#) cache), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin transfer torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof.

[0104] Storage device 1430 may include software services, servers, services, etc., which, when the code defining such software is executed by processor 1410, causes the system to perform a function. In some aspects, hardware services that perform a particular function may include software components stored in a computer-readable medium connected to the necessary hardware components (such as processor 1410, connection 1405, output device 1435, etc.) to perform the function. The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data may be stored and does not include carrier waves and / or transient electronic signals propagated wirelessly or via a wired connection. Examples of non-transitory media may include, but are not limited to, disks or tapes, optical storage media (such as compact discs (CDs) or digital versatile discs (DVDs)), flash memory, memory, or memory devices. Computer-readable media can store thereon code and / or machine-executable instructions that can represent a process, function, subroutine, program, routine, subroutine, module, software package, category, or any combination of instructions, data structures, or program statements. A code segment can be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. can be passed, forwarded, or sent via any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0105] Specific details are provided in the description above to provide a thorough understanding of the various aspects and examples provided herein, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although the illustrative aspects of the present application have been described in detail herein, it is to be understood that each inventive concept can be implemented and adopted in various other ways, and the appended claims are not intended to be interpreted as including these variations, unless limited by the prior art. The various features and aspects of the application described above can be used individually or in combination. In addition, without departing from the broader scope of the specification, the various aspects can be utilized in any number of environments and applications beyond those described herein. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For illustrative purposes, each method is described in a specific order. It should be understood that, in alternative aspects, each method can be performed in a different order than described.

[0106] For clarity of explanation, in some instances, the present technology can be presented as including separate functional blocks, which include devices, device components, steps or routines in the method embodied in software or a combination of hardware and software. Additional components other than those components shown in the drawings and / or described herein can be used. For example, circuits, systems, networks, processes and other components can be shown as components in block diagram form to avoid confusing these aspects in unnecessary details. In other cases, well-known circuits, processes, algorithms, structures and techniques can be shown without unnecessary details to avoid confusing various aspects.

[0107] In addition, it will be appreciated by those skilled in the art that the various exemplary logic blocks, modules, circuits, and algorithmic steps described in conjunction with the various aspects disclosed herein can be implemented as electronic hardware, computer software, or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed for the entire system. Those skilled in the art can implement the described functions in different ways for each specific application, but such specific implementation decisions should not be interpreted as resulting in departure from the scope of this disclosure.

[0108] Various aspects may be described above as processes or methods, which may be depicted as flow charts, flowcharts, data flow diagrams, structure diagrams, or block diagrams. Although a flow chart may describe operations as a sequential process, many of the operations may be performed in parallel or concurrently. Furthermore, the order of the operations may be rearranged. A process is terminated when its operations are completed, but a process may have additional steps not included in the accompanying figures. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, termination of the process may correspond to the function returning to the calling function or main function.

[0109] The processes and methods according to the examples described above can be implemented using stored computer-executable instructions or computer-executable instructions otherwise available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessible via a network. The computer-executable instructions may be, for example, binary, intermediate format instructions such as assembly language, firmware, source code. Examples of computer-readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, and the like.

[0110] In some aspects, computer-readable storage devices, media, and memories may include wired or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media specifically excludes media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.

[0111] Those skilled in the art will understand that information and signals can be represented using any of a variety of different techniques and methods. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be mentioned throughout the above description may, in some cases, be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or light particles, or any combination thereof, depending in part on the specific application, in part on the desired design, in part on the corresponding technology, etc.

[0112] The various illustrative logic blocks, modules, and circuits described in conjunction with the various aspects disclosed herein may be implemented or executed using hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., a computer program product) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Examples of form factors include: a laptop computer, a smartphone, a mobile phone, a tablet device, or other small form factor personal computer, a personal digital assistant, a rack-mounted device, a stand-alone device, and the like. The functionality described herein may also be embodied in a peripheral device or add-in card. By way of further example, such functionality may also be implemented on circuit boards in different chips or different processes executed on a single device.

[0113] Instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0114] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices, or integrated circuit devices with multiple uses, including applications in wireless communication devices and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a computer-readable data storage medium comprising program code, which includes instructions that, when executed, perform one or more of the methods, algorithms, and / or operations described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, or the like. Additionally or alternatively, the techniques may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0115] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Thus, the term "processor," as used herein, may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein.

[0116] It should be understood by those skilled in the art that the less than ("<") symbol and the greater than (">") symbol or term used herein may be replaced by the less than or equal to ("≤") symbol and the greater than or equal to ("≥") symbol, respectively, without departing from the scope of this specification.

[0117] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0118] The phrases “coupled to” or “communicatively coupled to” refer to any component being physically connected directly or indirectly to another component, and / or any component being in communication, directly or indirectly, with another component (e.g., connected to the other component via a wired or wireless connection and / or other suitable communication interface).

[0119] Claim language or other language reciting "at least one of" a set and / or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or any repetition is information or data (e.g., A and A, B and B, C and C, A and A and B, etc.), or any other ordering, repetition, or combination of A, B, and C. The language "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language stating "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0120] Illustrative aspects of the present disclosure include:

[0121] Aspect 1. A device for performing light source estimation, the device comprising: at least one memory; and at least one processor, the at least one processor being coupled to the at least one memory and configured to: obtain an estimated scene geometry based on multi-view observations of the scene; obtain a light emission mask based on the multi-view observations of the scene; obtain an emitted radiation field based on the multi-view observations of the scene; and determine the geometry of at least one light source of the estimated scene geometry based on the light emission mask and the emitted radiation field.

[0122] Aspect 2. The apparatus according to aspect 1, wherein the at least one processor is configured to: generate an individual shape for the at least one light source.

[0123] Aspect 3. The apparatus according to any one of aspects 1 to 2, wherein the at least one processor is configured to: generate a hole defined in a wall in the scene, wherein the hole represents the at least one light source.

[0124] Aspect 4. The apparatus of aspect 3, wherein the hole corresponds to a window in the wall.

[0125] Aspect 5. The apparatus according to any one of aspects 1 to 4, wherein the scene comprises an indoor scene.

[0126] Clause 6. The apparatus according to any one of clauses 1 to 5, wherein the estimated scene geometry is represented as a signed distance function.

[0127] Aspect 7. The apparatus according to aspect 6, wherein the signed distance function comprises a continuous function f, which, for a given three-dimensional point, returns a value representing the distance of the given three-dimensional point to the nearest surface in the scene.

[0128] Aspect 8. An apparatus according to any one of aspects 1 to 7, wherein the estimated scene geometry is generated based on at least one of depth fusion, monocular depth estimation, multi-view stereo, or neural radiance field.

[0129] Aspect 9. The apparatus of any one of aspects 1 to 8, wherein the light emission mask comprises information indicating a likelihood that a corresponding surface point in the scene is associated with a light emission.

[0130] Aspect 10. The apparatus of aspect 9, wherein the light emission mask comprises first values ​​for surface points associated with light emission and second values ​​for surface points not associated with light emission, the second values ​​being different from the first values.

[0131] Aspect 11. The apparatus of any one of aspects 1 to 10, wherein a first light source of the at least one light source comprises an indoor light and a second light source of the at least one light source comprises a window.

[0132] Aspect 12. The apparatus of aspect 11, wherein the first light source is represented as having physical properties modeled as global radiance values.

[0133] Aspect 13. An apparatus according to Aspect 12, wherein the at least one processor is configured to: determine the global radiation value based on sampling the emitted radiation field at the surface position of the first light source to generate sampled data and determining an average value of the sampled data.

[0134] Aspect 14. The apparatus of aspect 13, wherein the at least one processor is configured to: represent the first light source as a semantic object having an implicit geometry associated with a signed distance function (SDF) to generate implicit SDF parameters.

[0135] Aspect 15. The apparatus according to aspect 14, wherein the at least one processor is configured to convert the implicit SDF parameters into a triangle set.

[0136] Aspect 16. An apparatus according to Aspect 15, wherein the at least one processor is configured to: sample a set of points of a corresponding triangle in the set of triangles to generate sampling data; determine whether an average value of the sampling data is greater than a threshold for the corresponding triangle; classify the corresponding triangle as an emitter based on the average value of the sampling data being greater than the threshold for the corresponding triangle; and classify the corresponding triangle as a non-emitter based on the average value of the sampling data being equal to or less than the threshold for the corresponding triangle.

[0137] Aspect 17. An apparatus according to Aspect 16, wherein the at least one processor is configured to: determine that the corresponding triangle intersects the camera ray; obtain pixel values ​​corresponding to the corresponding triangle to generate a set of pixel values ​​based on the intersection of the corresponding triangle with the camera ray; and set an emission value for the median of the set of pixel values.

[0138] Aspect 18. An apparatus according to any one of Aspects 11 to 17, wherein the multi-view observation of the scene includes at least one input image in which a dominant light source is visible through the window, and wherein the at least one processor is configured to: estimate an initial partial environment map based on sampled camera rays and corresponding image intensities scattered onto a global environment map; convert the initial partial environment map into an implicit representation using a multi-layer perceptron (MLP); and fit observations having red, green, and blue (RGB) values ​​having camera rays that intersect a window geometry associated with the window to the initial partial environment map using the MLP.

[0139] Aspect 19. The apparatus according to any one of aspects 1 to 18, wherein the at least one processor is configured to generate spatially varying material properties associated with the scene.

[0140] Aspect 20. The apparatus of aspect 19, wherein the spatially varying material property comprises at least one of an albedo value, a roughness value, or a metallic value associated with the scene.

[0141] Aspect 21. The apparatus of aspect 20, wherein the spatially varying material properties are represented via a bidirectional reflectance distribution function defined at each surface point in the scene.

[0142] Aspect 22. The apparatus of aspect 21, wherein the bidirectional reflectance distribution function represents an estimate of the albedo value, the roughness value, and the metallic value.

[0143] Clause 23. The apparatus of clause 22, wherein the at least one processor is configured to apply a differentiable rendering algorithm to refine the estimate of emitter parameters.

[0144] Clause 24. The apparatus of clause 23, wherein the differentiable rendering algorithm compares rendered pixel values ​​with input image pixel values ​​to generate comparison values.

[0145] Aspect 25. The apparatus of aspect 24, wherein the at least one processor is configured to optimize at least one of the first light source geometry or the second light source geometry based on the comparison value.

[0146] Clause 26. The device of any one of clauses 24 or 25, wherein the differentiable rendering algorithm comprises one of inverse path tracing or neural implicit evolution.

[0147] Aspect 27. A method for performing light source estimation, the method comprising: obtaining an estimated scene geometry based on multi-view observations of a scene; obtaining a light emission mask based on the multi-view observations of the scene; obtaining an emitted radiation field based on the multi-view observations of the scene; and determining the geometry of at least one light source of the estimated scene geometry based on the light emission mask and the emitted radiation field.

[0148] Aspect 28. The method of aspect 27, wherein the at least one processor is configured to generate an individual shape for the at least one light source.

[0149] Aspect 29. The method according to any one of aspects 27 or 28, wherein the at least one processor is configured to: generate a hole defined in a wall in the scene, wherein the hole represents the at least one light source.

[0150] Aspect 30. The method of aspect 29, wherein the hole corresponds to a window in the wall.

[0151] Aspect 31. A method according to any one of aspects 27 to 30, wherein the scene comprises an indoor scene.

[0152] Aspect 32. A method according to any one of aspects 27 to 31, wherein the estimated scene geometry is represented as a signed distance function.

[0153] Aspect 33. The method according to aspect 32, wherein the signed distance function comprises a continuous function f, which, for a given three-dimensional point, returns a value representing the distance of the given three-dimensional point to the nearest surface in the scene.

[0154] Aspect 34. A method according to any one of aspects 27 to 33, wherein the estimated scene geometry is generated based on at least one of depth fusion, monocular depth estimation, multi-view stereo, or neural radiance field.

[0155] Aspect 35. The method of any one of aspects 27 to 34, wherein the light emission mask comprises information indicative of the likelihood that a corresponding surface point in the scene is associated with a light emission.

[0156] Aspect 36. The method of aspect 35, wherein the light emission mask comprises first values ​​for surface points associated with light emission and second values ​​for surface points not associated with light emission, the second values ​​being different from the first values.

[0157] Aspect 37. The method of any one of aspects 27 to 36, wherein a first light source of the at least one light source comprises an indoor light and a second light source of the at least one light source comprises a window.

[0158] Clause 38. The method of clause 37, wherein the first light source is represented as having physical properties modeled as global radiance values.

[0159] Aspect 39. A method according to Aspect 38, wherein the at least one processor is configured to: determine the global radiation value based on sampling the emitted radiation field at the surface position of the first light source to generate sampled data and determine an average value of the sampled data.

[0160] Clause 40. The method of clause 39, wherein the at least one processor is configured to represent the first light source as a semantic object having an implicit geometry associated with a signed distance function (SDF) to generate implicit SDF parameters.

[0161] Clause 41. The method according to clause 40, wherein the at least one processor is configured to convert the implicit SDF parameters into a triangle set.

[0162] Aspect 42. A method according to Aspect 41, wherein the at least one processor is configured to: sample a set of points of a corresponding triangle in the set of triangles to generate sampling data; determine whether an average value of the sampling data is greater than a threshold for the corresponding triangle; classify the corresponding triangle as an emitter based on the average value of the sampling data being greater than the threshold of the corresponding triangle; and classify the corresponding triangle as a non-emitter based on the average value of the sampling data being equal to or less than the threshold of the corresponding triangle.

[0163] Aspect 43. A method according to Aspect 42, wherein the at least one processor is configured to: determine that the corresponding triangle intersects with the camera ray; based on the intersection of the corresponding triangle with the camera ray, obtain pixel values ​​corresponding to the corresponding triangle to generate a set of pixel values; and set an emission value for the median of the set of pixel values.

[0164] Aspect 44. A method according to any one of Aspects 37 to 43, wherein the multi-view observation of the scene includes at least one input image in which a dominant light source is visible through the window, and wherein the at least one processor is configured to: estimate an initial partial environment map based on sampled camera rays and corresponding image intensities scattered onto a global environment map; convert the initial partial environment map into an implicit representation using a multi-layer perceptron (MLP); and fit observations having red, green, and blue (RGB) values ​​having camera rays that intersect a window geometry associated with the window to the initial partial environment map using the MLP.

[0165] Aspect 45. The method according to any one of aspects 27 to 44, wherein the at least one processor is configured to: generate spatially varying material properties associated with the scene.

[0166] Clause 46. The method of clause 45, wherein the spatially varying material property comprises at least one of an albedo value, a roughness value, or a metallic value associated with the scene.

[0167] Clause 47. The method according to clause 46, wherein the spatially varying material properties are represented via a bidirectional reflectance distribution function defined at each surface point in the scene.

[0168] Aspect 48. The method of aspect 47, wherein the bidirectional reflectance distribution function represents an estimate of the albedo value, the roughness value, and the metallic value.

[0169] Clause 49. The method of clause 48, wherein the at least one processor is configured to apply a differentiable rendering algorithm to refine the estimate of emitter parameters.

[0170] Clause 50. The method of clause 49, wherein the differentiable rendering algorithm compares rendered pixel values ​​with input image pixel values ​​to generate comparison values.

[0171] Aspect 51. The method of aspect 50, wherein the at least one processor is configured to optimize at least one of the first light source geometry or the second light source geometry based on the comparison value.

[0172] Aspect 52. A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform the operations of any one of aspects 27 to 51.

[0173] Aspect 53. An apparatus for performing light source estimation, the apparatus comprising one or more components for performing the operations according to any one of aspects 27 to 51.

Claims

1. An apparatus for performing light source estimation, the apparatus comprising: at least one memory; and at least one processor coupled to at least one memory and configured to: Obtaining estimated scene geometry based on multi-view observations of the scene; obtaining a light emission mask based on the multi-view observation of the scene; obtaining an emitted radiation field based on the multi-view observation of the scene; as well as The geometry of at least one light source of the estimated scene geometry is determined based on the light emission mask and the emitted radiation field.

2. The apparatus of claim 1 , wherein the at least one processor is configured to: An individual shape of the at least one light source is generated.

3. The apparatus of claim 1 , wherein the at least one processor is configured to: An aperture defined in a wall in the scene is generated, wherein the aperture represents the at least one light source.

4. The apparatus of claim 3, wherein the aperture corresponds to a window in the wall. The apparatus according to claim 1 , wherein the scene comprises an indoor scene. The apparatus of claim 1 , wherein the estimated scene geometry is represented as a signed distance function. 7 . The apparatus of claim 6 , wherein the signed distance function comprises a continuous function f that, for a given three-dimensional point, returns a value representing the distance of the given three-dimensional point to a nearest surface in the scene.

8. The apparatus of claim 1, wherein the estimated scene geometry is generated based on at least one of depth fusion, monocular depth estimation, multi-view stereo, or neural radiance field.

9. The apparatus of claim 1, wherein the light emission mask comprises information indicating a likelihood that a corresponding surface point in the scene is associated with a light emission.

10. The apparatus of claim 9, wherein the light emission mask comprises first values ​​for surface points associated with light emission and second values ​​for surface points not associated with light emission, the second values ​​being different from the first values.

11. The apparatus of claim 1, wherein a first light source of the at least one light source comprises a room light and a second light source of the at least one light source comprises a window.

12. The apparatus of claim 11, wherein the first light source is represented as having physical properties modeled as global radiance values.

13. The apparatus of claim 12, wherein the at least one processor is configured to: The global radiance value is determined based on sampling the emitted radiation field at surface locations of the first light source to generate sampled data and determining an average of the sampled data.

14. The apparatus of claim 13, wherein the at least one processor is configured to: The first light source is represented as a semantic object having an implicit geometry associated with a signed distance function (SDF) to generate implicit SDF parameters.

15. The apparatus of claim 14, wherein the at least one processor is configured to: Convert the implicit SDF parameters to a triangle set.

16. The apparatus of claim 15, wherein the at least one processor is configured to: Sampling a point set of a corresponding triangle in the triangle set to generate sampling data; determining, for the corresponding triangle, whether an average value of the sampled data is greater than a threshold; classifying the corresponding triangle as an emitter based on the average value of the sampled data being greater than the threshold value of the corresponding triangle; as well as The corresponding triangle is classified as a non-emitter based on the average value of the sampled data being equal to or less than the threshold value of the corresponding triangle.

17. The apparatus of claim 16, wherein the at least one processor is configured to: Determining the intersection of the corresponding triangle and the camera ray; Obtaining pixel values ​​corresponding to the corresponding triangle based on intersection of the corresponding triangle with the camera ray to generate a set of pixel values; and An emission value is set for the median of the set of pixel values.

18. The apparatus of claim 11 , wherein the multi-view observation of the scene comprises at least one input image in which a dominant light source is visible through the window, and wherein the at least one processor is configured to: Estimate an initial partial environment map based on the sampled camera rays and the corresponding image intensities scattered onto the global environment map; Converting the initial partial environment map into an implicit representation using a multi-layer perceptron (MLP); and Observations having red, green, and blue (RGB) values ​​with camera rays intersecting window geometry associated with the window are fit to the initial partial environment map using the MLP.

19. The apparatus of claim 1 , wherein the at least one processor is configured to: Spatially varying material properties associated with the scene are generated.

20. The apparatus of claim 19, wherein the spatially varying material property comprises at least one of an albedo value, a roughness value, or a metallic value associated with the scene.

21. The apparatus of claim 20, wherein the spatially varying material properties are represented via a bidirectional reflectance distribution function defined at each surface point in the scene.

22. The apparatus of claim 21, wherein the bidirectional reflectance distribution function represents an estimate of the albedo value, the roughness value, and the metallic value.

23. The apparatus of claim 22, wherein the at least one processor is configured to: A differentiable rendering algorithm is applied to refine the estimate of emitter parameters.

24. The apparatus of claim 23, wherein the differentiable rendering algorithm compares rendered pixel values ​​with input image pixel values ​​to generate comparison values.

25. The apparatus of claim 24, wherein the at least one processor is configured to: At least one of the first light source geometry or the second light source geometry is optimized based on the comparison value.

26. The device of claim 24, wherein the differentiable rendering algorithm comprises one of inverse path tracing or neural implicit evolution.

27. A method for performing light source estimation, the method comprising: Obtaining estimated scene geometry based on multi-view observations of the scene; obtaining a light emission mask based on the multi-view observation of the scene; obtaining an emitted radiation field based on the multi-view observation of the scene; as well as The geometry of at least one light source of the estimated scene geometry is determined based on the light emission mask and the emitted radiation field.

28. The method according to claim 27, further comprising: An individual shape of the at least one light source is generated.

29. The method of claim 27, wherein a first one of the at least one light source comprises a room light and a second one of the at least one light source comprises a window.

30. The method of claim 29, wherein the first light source is represented as having physical properties modeled as global radiance values.

Images