Systems and methods for generalized scene reconstruction
Through processors and machine learning models, process image data, generate three-dimensional models and perform optical transmission operations, the problem of complexity of light and matter interaction in the prior art is solved, and efficient reconstruction and accurate scene model of dynamic scenes are realized.
Patent Information
- Application Number
- CN202380026144.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-07
- Filing Date
- 2023-03-07
- Publication Date
- 2024-11-05
AI Technical Summary
The prior art is difficult to effectively deal with the complex interactions between light and matter when implementing 3D imaging and scene reconstruction, resulting in the robustness and efficiency of scene models, especially in the case of dynamic scenes and light field changes.
By using processors and machine learning models, image data is processed to generate three-dimensional models, including reconstruction of matter and light fields, using voxels and radiant elements to represent the medium and light fields in the scene, performing light transmission operations and iterative optimizations to improve Accuracy and efficiency of scene reconstruction.
It realizes more accurate reconstruction of dynamic scenes and more efficient modeling of light and material fields interactions, which improves the robustness and processing speed of scene models, and is suitable for complex real-world scenarios.
Smart Images

Figure CN118901083A_ABST
Abstract
Description
[0001] Related Applications
[0002] This application claims priority to US Provisional Application No. 63 / 317,330 filed on March 7, 2022, the contents of which are hereby incorporated by reference in their entirety. Technical Field
[0003] The present invention relates generally to the field of 3D imaging, and more specifically to tools for implementing various systems and methods related to generating 3D models from images (sometimes referred to as generalized scene reconstruction (GSR), stereo scene reconstruction (VSR), or everyday scene reconstruction (QSR)), as well as systems and methods for light field reconstruction (LFR), as further described herein. Background Art
[0004] There are countless uses for 3D models of real-world scenes. Applications include global sectors including defense, security, entertainment, education, healthcare, infrastructure, manufacturing, and mobility. In the metaverse, applications include virtual real estate creation, NFT creation, and avatar creation. Various methods for capturing 3D images have been proposed or developed, some of which are capable of providing digital 3D models of real-world scenes with varying fidelity and for various purposes including visualization and information extraction. Such 3D images can be acquired by 3D imagers, which are also known as 3D sensors, 3D cameras, 3D scanners, VR cameras, 360° cameras, RGBD cameras, and depth cameras.
[0005] Previous methods for extracting 3D information from scenes typically involve active light sources such as lasers and have limitations such as high power consumption and limited range. A more ideal approach would be to generate a detailed scene model using one or more images from inexpensive sensors, including cameras or devices that form images by sensing light fields using detectors. To improve the robustness of scene models extracted from images, improved modeling of the transport of light is needed. This includes the properties of light's interaction with matter, including transmission, reflection, refraction, scattering, etc. An in-depth analysis of this topic is presented in Jarosz's paper "Efficient MonteCarlo Methods for Light Transport in Scattering Media" (2008).
[0006] Early work has suggested a way to create 3D images and models using a process generally referred to as generalized scene reconstruction ("GSR"), which may also alternatively be referred to as stereoscopic scene reconstruction ("VSR") or daily scene reconstruction ("QSR"). For example, U.S. Patent No. 10,521,952 to Ackerson et al., U.S. Patent No. 11,508,115 to Ackerson et al., and U.S. Patent Publication No. 2021 / 0133929A1 to Ackerson et al. and U.S. Provisional Patent Application No. 63 / 317,330 to Ackerson et al. (each of which is incorporated herein by reference in its entirety) each variously describe systems and methods for implementing aspects of GSR. In some cases, GSR can be implemented using a scene reconstruction engine ("SRE") that is used to create a 3D scene model from a digital image using a process referred to as scene reconstruction. SRE can enable components of a class of devices that use scene reconstruction (DSR), such as 3D mobile phones, tablets, computers, virtual reality (VR) and augmented reality (AR) glasses and other devices, drones and other autonomous, semi-autonomous or controlled unmanned systems, and other digital handheld or non-handheld devices.
[0007] Certain advantages of GSR have been described in the patents and patent applications cited above. For example, U.S. Patent Publication No. 2021 / 0133929A1 Figure 4 B and Fig.10 is a graphical diagram representing a real-world scene, where the representation can be considered as an abstract scene model view of the data included in the plenoptic scene database. Figure 4 B focuses on the larger scene, while Fig.10 Focus on smaller scenes. The abstract representation of the scene model for these two different types of scenes contains a full light field consisting of the scene's matter field and light field. The light field interacts with any number of objects in the matter field as well as other objects, such as, for example, described objects, undescribed areas, opaque objects, fine structure objects, distant objects, emitting objects, highly reflective objects, featureless objects, or partially transmissive objects. U.S. Patent Publication No. 2021 / 0133929A1 teaches that an important aspect of GSR is to identify the matter field through scene reconstruction sufficient to distinguish multiple types of objects, wherein any individual type of object uniquely located in the model scene can then be further processed, for example, by using machine learning to perform object recognition and classification, changing various characteristics and properties to cause model rendering effects (such as changes in visualization, object enhancement and labeling, and even object removal).
[0008] In some embodiments, GSR can be implemented using a codec. Various codecs are well known in the art and are generally devices or programs that compress data to achieve faster transmission and decompress the received data. Exemplary types of codecs include video (e.g., MPEG, H.264), audio (e.g., MP3, ACC), image (e.g., JPEG, PNG), and data (e.g., PKZIP), where the codec type encapsulates the data type and is strongly coupled to the data type. In many traditional applications, strong coupling necessarily leads to a limited end-user experience. Codecs are often implemented in an essentially "file-based" manner, where the file is a data representation of some real or synthetic pre-captured sensory experience, and where the file (such as a movie, song, or book) necessarily limits the user experience to the experience path selected by the file creator. Therefore, users can watch movies, listen to songs, and read books in a substantially ordered experience defined by the creator.
[0009] In the context of GSR, the use of codecs requires an increase in the types of data handled by such codecs (and specifically, performing GSR), where sensors such as cameras and range-finding devices create scene models of real-world scenes. Challenges in implementing GSR include adequately representing and organizing representations to describe the complexity of real-world matter and light fields in an efficiently controllable and highly scalable manner, and where there are challenges in managing the distribution of active (even real-time) scene models across multiple interactive clients, each of which may request any of a nearly unlimited number of scene perspectives, details, and data types.
[0010] In addition, machine learning (ML) and artificial intelligence (AI) systems have made great progress in recent years and have become useful and effective in many application areas. Many such systems are used for object recognition and other useful purposes in scenes. ML and AI systems are typically based on processing 2D images from camera systems. Such images are typically arrays of red, green, blue (RGB) values. This sensed information consists of samples of the light field that enters the camera lens and converges at the viewpoint. These light samples are the result of a series of complex interactions between light and matter in the scene and are governed by the laws of physics. Although the "real" features of an object (such as actual color and reflective properties) may be important for determining the type or nature of matter in the scene, this information cannot usually be determined from a conventional photograph.
[0011] The interaction of light from light sources inside and outside the scene reflected and blocked by other materials in the scene forms a complex scene light field, which effectively obscures basic information about the materials in the scene. In addition to obvious examples such as shadows, visible color changes occur when light reflected from another object affects the light reflected from the object. Lambertian surfaces reflect light approximately uniformly in all directions. When the surface of an object is non-Lambertian, significant additional difficulties arise. Such surfaces have complex reflective properties that cannot be easily resolved in conventional systems. The difficulties include subsurface scattering, specular reflections, transparency, etc. For example, subsurface scattering is a major component of the visual appearance of human skin.
[0012] Due to the difficulty in determining fundamental properties of materials from reflected light in a scene captured by a camera, image-based supervised ML systems typically require large training sets to reasonably represent the light interactions that might be expected during operation. Such image training sets are typically classified manually by labeling the images with identification information for each object of interest (OOI) in each image. Depending on the application, the training set may identify objects as "good" (in the case where the image or a portion of an image contains an OOI) or "bad" (the image does not contain an OOI). Typically, there are approximately equal numbers of good and bad objects.
[0013] In some cases, such as anomaly detection, only good or mostly good training examples are needed. If a production image is not identified as an OOI, then the anomaly has been detected. If there are errors (even subtle errors) in the training set, the quality of the results may be affected. For example, if the mix of good and bad training cases is significantly unfavorable, there is a risk of overfitting, which is a situation where the system will correctly identify the training objects but cannot reliably identify the objects in the production images.
[0014] In most cases, a properly, error-free labeled dataset for training and testing is the most important part of a machine learning system. Compiling such a dataset typically requires thousands or even hundreds of manually labeled images, which is a nontrivial expense and a significant barrier to widespread use. If the effects of light field interactions in the scene and the effects of disentanglement in camera images could be modeled, such a system could be more efficient and easier to train.
[0015] Various patents, patent applications, and other publications have considered how to perform GSR, other forms of 3D imaging or scene reconstruction, or component processes or systems for such activities. For example, the following references cite various aspects of GSR and aspects of segmenting three-dimensional space according to medium properties: Lef fingwell, J. et al., “Generalized Scene Reconstruction,” arXiv:1803.08496, May 24, 2018; Kutulakos, K. et al., “A Theory of Shape by Space Carving,” University of Rochester, 2000; Bonfort, T. and Sturm, P., “Voxel Carving for Specular Surfaces,” Proceedings of the 9th IEEE International Conference on Computer Vision (ICCV), 2003; Broadhurst, A. et al., “A Probabilistic Framework for Space Carving,” Proceedings of the International Conference on Computer Vision, I, pp. 282-291, 2001; Broadhurst, A. and Cipolla, R., “A Statistical Consistency Check for the Space Carving Algorithm, Proceedings of the 11th British Conference on Machine Vision, pp. 282–291, 2000; Gaillard, M. et al., “Voxel Carving Based 3D Reconstruction of Sorghum Identities Generic Determinants of Ration Interception Efficiency,” bioRxiv preprint https: / / doi.org / 10.1101 / 2020.04.06.028605, April 7, 2020; Sainz, M. et al., “Hardware Accelerated Voxel Carving,” ResearchGate, open 228917433; Scharr, H.et al., “Fast High Resolution Volume Carving for 3D Plant Shoot Reconstruction”, Frontiers in Plant Science, September 28, 2017; Seitz, S. and Dyer, C. et al., “Photorealistic Scene Reconstruction by Voxel Coloring”, Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 1067-1073, 1997; Culbertson, W., Malzbender, T. and Slabaugh, G. et al., “Generalized Voxel Coloring”, 7th International Conference on Computer Vision, September 1999; Dyer, C. et al., “Volumetric Scene Reconstruction from Multiple Views (Stereo Scene Reconstruction from Multiple Views), Foundations of Image Analysis, LSDavis ed., Chapter 1, 2001; Seitz, S. and Kutulakos, K. et al., “Plenoptic Image Editing”, 6th International Conference on Computer Vision (IEEE Cat. No. 98CH36271), 1998; Troccoli, A. and Allen, P. et al., “Relighting Acquired Models of Outdoor Scenes”, Proceedings of the 5th International Conference on 3D Digital Imaging and Modeling, 2005; Singh, R. et al., “3D convolutional neural network for object recognition: a review”, Multimedia Tools and Applications, 2018; Riegle r, G. et al., “OctNetFusion: Learning Depth Fusion from Data (OctNetFusion)” tNetFusion: Learning Deep Fusion from Data”, arXiv:1704.01047v3, October 31, 2017; Riegler, G. et al., “OctNet: Learning Deep 3D Representations at High Resolutions”, arXiv:1611.05009v4, April 10, 2017; Meka, A. et al., “Deep Rel ightable Textures”, ACM Trans. Graph., Vol. 39, No. 6, Article 259, 2020; Liu, J. et al., “RocNet: Recursive Octree Network for Efficient 3D Deep Representation”, arXiv:2008.03875v1, August 10, 2020; Lei, H. et al., “Octree guided CNN with Spherical Kernels for 3D Point Clouds”, Computer Vision Foundations, pp. 9631-40; Bi, S. et al., “Deep Relightable Appearance Models for Animatable Faces (Deep Relightable Appearance Model for Animatable Faces)", ACM Trans. Graph., Vol. 40, No. 4, Article 89, August 2021; Wang, P. et al. "O-CNN: Octree-based Convolutional Neural Networks for 3D Shape Analysis", ACM Transactions on Graphics, Vol. 36, No. 4, Article 72, July 2017; Wang, P. et al. "Adaptive O-CNN: A Patch-based Deep Representation of 3D Shapes", arXiv:1809.07917v1, September 21, 2018; Wang, P. et al. "Deep Octree-based CNNs with Output-Guided Skip Connections for 3D Shape and Scene Completion (Deep Octree-based CNNs with Output-guided Skip Connections for 3D Shape and Scene Completion), Computer Vision Foundations, 2020; U.S. Patent No. 4,694,404; U.S. Patent No. 5,123,084; U.S. Patent No.6,123,733; U.S. Patent No. 6,980,935; U.S. Patent No. 6,831,641; U.S. Patent No. 7,843,449; U.S. Patent No. 8,432,435; U.S. Patent No. 8,547,374; U.S. Patent No. 8,749,620; U.S. Patent No. 8,749,694; U.S. Patent No. 9,179,126; U.S. Patent No. 9,857,470; U.S. Patent No. 10,169,910; U.S. Patent No. 10,50 9,153; U.S. Patent No. 10,893,262; U.S. Patent No. 11,164,368; U.S. Patent Publication No. 20080068372; U.S. Patent Publication No. 20110128412; U.S. Patent Publication No. 20130038696; U.S. Patent Publication No. 20130156297; U.S. Patent Publication No. 20140184749; U.S. Patent Publication No. 20140201022; U.S. Patent Publication No. 20150146032; U.S. Patent Publication N o.20150305612; U.S. Patent Publication No.20150373320; U.S. Patent Publication No.20160028935; U.S. Patent Publication No.20180113200; U.S. Patent Publication No.20180144540; U.S. Patent Publication No.20180149791; U.S. Patent Publication No.20180227568; U.S. Patent Publication No.20190011621; U.S. Patent Publication No.20190072897; U.S. Patent Publication No.2019 0155835; U.S. Patent Publication No. 20220058854; British Patent No. GB2535475B; European Patent Application No. EP3144887A1; PCT Publication No. WO2011066275A2; PCT Publication No. 2018200316; PCT Publication No. WO2018200316; PCT Publication No. WO2019213450A1; New Zealand Patent Publication No. NZ743841A; China Patent Publication No. CN111796255A. The following references refer to various aspects of GSR and aspects of nonparametric modeling: Freeman, H., "On the encoding of arbitrary geometric configurations", IRE Transactions on Electronic Computers EC-10, pp. 260-268; Samet, H."The Design and Analysis of Spatial Data Structures", Addison-Wesley Computer Science Series, 1989; Marschner, S., Shirley, P., et al., "Fundamentals of Computer Graphics", CRC Press, 2016; Varma, M. and Zisserman, A., "A Statistical Approach to Texture Classification from single Images", International Journal of Computer Vision 62(1 / 2), 61-81m, 2005. The following papers cite various aspects of GSR and aspects of integral rendering: Mildenhall, B. et al., “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis” arXiv:2003.08934v4, August 3, 2020 (video: https: / / www.matthewtancik.com / nerf); Yu, Alex et al., “Plenoxels: Radiance Fields without Neural Networks” arXiv:2103.14024 (2021); Yu, A. et al., “Plenoxels: Radiance Fieldswithout Neural Networks” arXiv:2112.05131v1 (2021); EyeCue Vision Tech’s “Qlone 3D Qlone 3D Scanner, Apple App Store, Version 4.6.0 (2022) (available at https: / / apps.apple.com / us / app / qlone-3d-scanner / id1229460906); J. Paul Morrison, Flow-based Programming: A New Approach to Application Development, 2nd ed., JPMorrison Enterprises, 2010; Karras, T. et al., “A Style-Based Generator Architecture for Generative Adversarial Networks,” CoRR 2018, vol abs / 1812.04948 (available at https: / / arxiv.org / abs / 1812.04948); R. Martin-Brualla et al., “NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections,” 2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 7206-7215 (2020); Zhang, X. et al., “NeRFactor: Neural Factorization of Shape and Reflectance Under an ImageNet Framework,” in Proceedings of the 13th IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 117-119 (2020); UnknownIllumination (NeRFactor: Neural Decomposition of Shape and Reflection under Unknown Illumination), ACM | SIGGRAPH Asia 2021 Technical Papers (2021) (available at https: / / dspace.mit.edu / handle / 1721.1 / 146375). Each of the foregoing documents and the disclosures therein are incorporated herein by reference in their entirety.
[0016] Therefore, there is a need to overcome the shortcomings and deficiencies in the art by providing various systems and methods for implementing GSR and components thereof, as well as the many needs and opportunities of the marketplace. Summary of the invention
[0017] The following simplified summary of the invention can provide a basic initial understanding of some aspects of the systems and / or methods discussed herein. The summary of the invention is not an extensive overview of the systems and / or methods discussed herein. It is not intended to identify all key / important elements or to describe the entire scope of such systems and / or methods. Its sole purpose is to present some concepts in a simplified form as a preface to a more detailed description presented later.
[0018] In some embodiments, a processor for processing digital scene data and an interface for receiving input related to a scene to be captured may be used to reconstruct one or more objects in the scene. In this embodiment, (i) the input includes digital scene data in the form of image data, the image data representing the scene from a viewpoint, (ii) the processor processes the digital scene data and the input to generate a three-dimensional model of at least a portion of the scene, the scene including matter containing an interactive medium, (iii) the processor processes the image data by accessing one or more voxels in the matter field represented by the image data, and (iv) the processor processes the image data by determining whether the matter represented in each of the one or more voxels includes an interactive medium. The image data may be captured by a camera and may be data related to electromagnetic radiation, such as visible light, infrared light, polarized light or unpolarized light and / or a radiation value of a radar. The orientation may include a pose of the camera, and in some embodiments may include more than one pose or orientation. The three-dimensional model may be represented by a data structure. In some embodiments, the three-dimensional model is represented by a combination of a first data structure and a second data structure, the first data structure storing plenoptic data and the second data structure including the orientation of the digital scene data. Some embodiments of the present invention may also store information related to the light field in the scene in the first data structure or the third data structure. In some embodiments, the processor processes the image data from at least two orientations sequentially. In some embodiments, the matter represented in the voxel is represented by a mediel, and the data related to the mediel may be stored in the plenoptic data structure. The data related to the mediel may include the outgoing light field and / or the incoming light field, and such data may be represented by a radiel.
[0019] In some embodiments, scene reconstruction may include processing image data by assuming an orientation of digital scene data. Processing the image data may include: (i) assuming the presence of a medium in a voxel; (ii) assuming one or more of a surface normal, light interaction properties, an outgoing radiation vector, an incident light field, and other properties of the medium; (iii) calculating a cost for the presence of the medium in the voxel based on one or more of the surface normal, light interaction properties (e.g., refractive index, roughness, polarization diffusion coefficient, non-polarization diffusion coefficient, or extinction coefficient) of the assumed medium, an outgoing radiation vector, and an incident light field; (iv) comparing the cost to a cost threshold; and (iv) accepting the presence of the medium at the voxel when the cost is below the cost threshold. In some embodiments, when the system has accepted the presence of a medium at a voxel, the medium remains in the scene during subsequent processing of the scene. Certain embodiments may update assumptions about the light field of one or more other voxels based on the accepted presence of the medium. The system may iteratively perform this process for more than one voxel and / or more than one image data set. In some embodiments, the results of the processing may be stored in a data structure, including in a hierarchical data structure. Processing may be accomplished by hierarchically traversing the data structure from a coarser level to a finer level, and the finer levels of detail may be stored in the data structure by subdividing the data structure.
[0020] Certain embodiments of the present invention include a method for training a machine learning model, comprising: providing image data to the machine learning model, wherein the image data includes one or more objects of interest; processing the image data to generate a model, wherein such processing includes analyzing the image data to generate one or more of a light field model of the scene or a reconstruction of one or more matter fields in the scene; selecting objects of interest in the model of the scene; extracting objects of interest in the model of the scene; and outputting a re-illuminable matter field model of the objects of interest in the scene. The image data may include re-illuminable matter field data. In some embodiments, the image data includes one or more objects of interest in multiple scenes and objects of interest under various conditions. The re-illuminable matter field may be constructed from multiple two-dimensional or higher dimensional images. The re-illuminable matter field model may include one or more of shape information, bidirectional light interaction function (BLIF) information, an emitted light field (if present; for example, a light source in the scene itself), and an incident and / or response light field caused by the emitted light field. In addition, the light field information may be used to calculate light interaction characteristics of locations in the matter field. In some embodiments, the method may also include: changing the BLIF and / or geometric information of the model; inputting the model with the changed BLIF information into the machine learning model; and performing one or more of the aforementioned steps on the model with the changed BLIF information to further train the machine learning model.
[0021] Some embodiments of the invention include a method of using a machine learning model, comprising: identifying one or more objects of interest in a model of a scene; accessing a re-illuminable matter field of the scene; selecting a portion of the matter field to be processed; processing the selected portion of the matter field to extract at least a portion of the re-illuminable matter field; and outputting the extracted portion of the re-illuminable matter field. The method may also include testing the utility of the portion of the re-illuminable matter field output by the machine learning model.
[0022] In some embodiments, the present invention includes: using a trained machine learning model to identify one or more objects or features of interest in a scene, and using such identification to provide an initial hypothesis for light field and / or matter field reconstruction. In such embodiments, the present invention can provide faster processing of image data to perform reconstruction of a scene or a portion thereof. In some embodiments, the output of the trained machine learning model includes one or more of the size, shape, and / or position of a medium in the scene and / or light interaction properties of a medium in the scene.
[0023] Some embodiments of the present invention may use additional input reconstructions of light field and / or material field properties as input to the scene reconstruction process. For example, embodiments of the present invention may use a point cloud provided by LiD AR and / or another material field or light field reconstruction provided by other technologies (e.g., multi-view stereo, photometry, infrared, radar, etc.) to provide an initial or updated hypothesis of the properties of the medium in the scene. Then, embodiments of the present invention may perform the scene reconstruction process described herein to reconstruct the light field and / or material field in the scene.
[0024] In some embodiments, the present invention can reconstruct one or more objects in a scene by a processor for processing digital scene data and an interface for receiving input related to a scene to be captured, wherein the processor processes the digital scene data and the input to generate a three-dimensional model of at least a portion of the scene; wherein the processor input indicates at least a portion of the processing of the digital scene data; and wherein the processor provides an output including a three-dimensional model of at least a portion of the scene. The input may include at least one of the following items: an approximation of at least a portion of the light field in the scene, an approximation of at least a portion of the material field in the scene, one or more shapes present in the scene, one or more objects in the scene, or information related to one or more light sources in the scene. The input may control one or more sensing devices that provide digital scene data. In some embodiments, the system may provide feedback related to one or more objects to be reconstructed within the scene, and the feedback may include a preview of the one or more objects to be reconstructed within the scene. When reconstructing one or more objects using the results of such reconstruction, the system may update the preview. The preview may also include one or more indications related to one or more parameters of the reconstruction. The preview may include one or more masks that represent data related to the generated model and information received from the digital scene data capture device. The feedback may include one or more of information related to the rate of capture of digital scene data, the location for capturing digital scene data, the sensor angle for capturing digital scene data, aspects of the light field in the scene, or aspects of the material field in the scene. In some embodiments, the input is data that allows the digital scene data to be aligned with the newly received digital scene data. In some embodiments, the system may also include an instruction set for achieving one or more goals of generating a three-dimensional model, wherein the one or more goals include a desired resolution of the light field, a desired resolution of the material field, a desired certainty threshold for reconstruction, a threshold for eliminating gaps in the captured digital scene information, and one or more of triggers for events encountered during the capture of the digital scene information. In some embodiments, the trigger includes one or more of a specified material field structure, a specified light field structure, the passage of time, and a change in the uncertainty level in the model. The system may be configured to take action in response to the trigger, and the response may include changing a display configuration, adding an overlay to the display, providing an audio prompt, providing a visual prompt, changing a reconstruction target, and changing one or more of the settings of a device connected to the system.
[0025] Certain embodiments of the present invention may be configured to change one or more features of a scene model. For example, the change may include one or more of the following: editing a light field reconstruction; editing a matter field reconstruction; transforming the model; deforming the model; relighting all or any portion of the model; changing one or more light interaction properties of the BLIF; assigning one or more BLIFs to different regions of the matter field; manipulating the model by dragging anchor points, by typing keyboard shortcuts, or by sculpting and painting on the model using a brush tool; inserting a new matter field; inserting a new light field; relighting one or more matter fields (all or part); deleting a light field in whole or in part; and deleting a matter field in whole or in part. In some embodiments, the system may be configured to search the model spatially using a search query including one or more parameters. Such a spatial search may include obtaining one or more of a count, a selection, or a group of light field structures, or obtaining one or more of a count, a selection, or a group of matter field structures that match one or more parameters of the search query. The search query may be provided as a selected light region, a selected matter region, and / or a descriptive term based on machine learning to generate a response. These parameters may include one or more of matter field shape, light field structure, radiation intensity, size, and BLIF.
[0026] In some embodiments, the system also includes a display for capturing digital scene information, wherein, during the capture of information from multiple sources, spatially interlaced layers are displayed in three or more adjacent areas of the display. An area of the display may include a real-time reconstruction preview, and all layers on the display may be substantially aligned with the same viewpoint. In addition, all layers on the display may contain information about the scene. In some embodiments, one of the layers on the display is a pre-scene rendering (e.g., a priori scene and / or a partially or fully initialized scene model) that is aligned to substantially the same viewpoint as the other layers. During capture, the display may be used to indicate how many angles around a certain area of the scene have been captured, and the indication may be provided by displaying a spherical or hemispherical overlay centered on a selected mesosome including the BLIF. At least one portion of the spherical overlay may change in response to viewing the mesosome from various angles relative to the corresponding position of the mesosome in real space, and the change of at least one portion of the spherical overlay may include disappearing, changing color, or undergoing one or more of other visible changes.
[0027] Although the example embodiments are expressed in the form of systems or methods, those skilled in the art will recognize that these examples can be modified to include structures at least similar to: (A) a machine device on which the claimed functionality resides, (B) execution of method steps that perform the processes described by the system, and / or (C) a non-volatile computer program storage medium containing executable program instructions that, when executed on a compatible digital processor, provide the functionality of the system or method.
[0028] Additional features and advantages of example embodiments are described below. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] These and other features and advantages will be better and more completely understood by reference to the following detailed description of example non-limiting illustrative embodiments taken in conjunction with the following drawings.
[0030] Figures 1A to 1E An exemplary structure of a system using generalized scene reconstruction (GSR), an exemplary configuration for capturing image data, and an exemplary scene including a matter field and a light field are shown.
[0031] Figure 2A and Figure 2B Examples of surfels and mogels are shown.
[0032] Figures 3A to 3G An exemplary method for reconstructing a scene is shown.
[0033] Figure 4 An exemplary type hierarchy of mesosomes is shown.
[0034] Figure 5 An exemplary scene including various intermediaries and other elements is shown.
[0035] Figure 6 An end view of a panel representing a curve is shown.
[0036] Figure 7 A side view of a panel representing a curve is shown.
[0037] Figure 8 The bins representing the angles are shown.
[0038] Fig. 9 An exemplary method for generating a trained machine learning model (TMLM) is shown.
[0039] Fig.10 An exemplary method using a trained machine learning model (TMLM) is shown.
[0040] Fig.11 An example of a point grid showing a reconstruction preview.
[0041] Fig. 12A and Fig. 12B An example of a video feed interleaved with a reconstructed preview is shown.
[0042] Fig.13 Scene capture guidance is shown.
[0043] FIG. 14A to FIG. 14C An exemplary process for determining the presence and interaction of media within an area of a scene is shown.
[0044] Fig.15 is a graphic representation of a machine learning model.
[0045] Fig.16 is a diagram of a physical-information neural network (PINN).
[0046] Fig.17 is an illustration of a neural network architecture with physical constraints.
[0047] Fig.18 is a graphic illustration of incorporating physical priors into the loss function.
[0048] Fig.19 is a graphical representation of residual modeling.
[0049] Fig. 20 is a graphic illustration of the combination of physics-based methods and neural networks.
[0050] Fig.21 is an illustration of a reconstruction performed using the method described herein combined with a reconstruction created using another method. DETAILED DESCRIPTION
[0051] As described in various embodiments herein, one object of the present invention is to provide systems and methods for performing scene reconstruction (and in particular, performing generalized scene reconstruction (GSR)). In some embodiments, the results of a GSR process or system may result in a reconstruction of a light field, a matter field (including a re-illuminable matter field), a characterization of a camera pose, or any combination of the foregoing. The results of the GSR process may generate a model representing a scene based solely and separately on the reconstructed light field or the matter field (including a re-illuminable matter field), or on both, depending on the circumstances. As used herein, a scene may refer to the entire extent of a light field and / or matter field represented in an image, any portion thereof, or any medium therein. Although the terms "sub-scene," "portion of a scene," "region of interest," "object of interest," and other similar terms may be used to refer to a portion of a larger scene, each of the foregoing terms is itself a scene.
[0052] In some embodiments, the present invention may be configured to create a model of a scene using static data (i.e., captured data of a scene where the contents of the scene do not move) or a dynamic scene (i.e., captured data of a scene where the contents of the scene move relative to each other and / or relative to an image capture device). Similarly, the model may be configured to represent a scene, a portion of a scene, or one or more objects in a scene in a static configuration (i.e., the reconstruction depicts a scene where the contents of the scene do not move) or a dynamic configuration (i.e., some or all of the contents of the scene are in motion). In the case of a dynamic configuration, the model may be configured to represent dynamics in the matter field, the light field, or both.
[0053] The invention described herein may provide advantages over conventional representations of dynamic scenes. For example, in some known systems for representing scenes (e.g., where the representation primarily involves the light field of the scene rather than the material field of the scene), representing dynamics may be challenging because the associated light properties are directly associated with the medium in the scene, resulting in the need to reinitialize and / or retrain large portions of the scene model for each time step at which the material field has changed configuration (e.g., changed shape or motion). In some embodiments of the invention, when reconstructing a dynamic scene, the invention described herein may compute the interaction of the light field in the scene with the portions of the scene in motion, thereby allowing for a better understanding of the medium including such objects. Similarly, when using a model to represent a dynamic scene, embodiments of the invention described herein may more accurately render portions of the scene in motion by understanding how such portions will interact with the light in the modeled scene. In some embodiments, a subscene represented as a re-illuminable material field may use a motion model with optional deformations to represent dynamics in a real material field. Whether for rendering purposes or for other purposes, the effects of dynamics on the light field may be more directly computed using the light transport operations described herein.
[0054] As described herein, a system using the GSR 100 may include Figure 1A Specifically, the system 100 may include: application software 101 for interfacing with the system; a scene solver 102 configured to perform certain GSR functions; a plenoptic scene database 103 configured to store information related to the reconstruction of the scene; and a scene codec 104 for encoding and / or decoding information related to the reconstruction of the scene.
[0055] Figure 1C to Figure 1E Various exemplary embodiments of scene models are depicted in FIG. Scene model 110 may include matter field 120 and light field 130, which may be combined in a single model (eg, Figure 1C ) or may be separate (as shown Figure 1D (matter field) and Figure 1E (light field) shown). The scene can have external lighting 112, which flows into the scene and provides a light source in the scene. The scene can also be a single scene, in which there is no light 112 flowing into the scene. The scene can have a boundary 115, which can be optionally defined by the system during reconstruction of the scene, by physical boundaries in the scene, by a user or other input, by some combination of the foregoing, or otherwise defined. Information outside the boundary of the scene can be considered a boundary 117 and may not be represented in the scene. However, in some embodiments, the boundary 115 can include a window boundary 111 in whole or in part. The window boundary 111 can be a portion of the scene boundary 115 through which incident light 112 can flow into the scene and outgoing light 116 can flow out of the scene. In some embodiments, a portion of the boundary 117 may be at least partially represented at the window boundary 111. For example, the window boundary 111 may be defined based on a physical feature in the scene (e.g., a window or skylight in a wall or ceiling through which light can enter the scene), based on scene parallax (e.g., a boundary based on distance or insufficient resolution of the image data, such as looking at the sky for an outdoor night scene, where the range of the field of view is very far), some combination of the two, or some other factor. The scene may include one or more objects, including a responsive object 113 and an emitting object 114. An emitting object 114 may emit light independently of light incident on the object, while a responsive object may interact with the incident light without emitting light itself.
[0056] The systems and processes described herein may use image data. Image data may provide one or more characteristics of a light field at a certain moment (e.g., for a still image or a frame of a video) or a series of moments (e.g., for a video or other data that provides image information over time). In various embodiments, the image data may be two-dimensional, three-dimensional, or higher dimensional. The image data may optionally include information about the distance or position associated with the medium in the scene, one or more measurements or characterizations of depth and / or range, polarization data, infrared data, hyperspectral data, or other data related to radiation characteristics. The image data may include previously captured image data, image data captured from one or more cameras or other imaging devices simultaneously with the processes discussed herein, synthetic or computer-generated image data, or any combination of the foregoing. In addition, the systems and processes described herein may use other types of data when performing the GSR process.
[0057] like Figure 1BAs shown, embodiments of the present invention can obtain one or more images of the scene, which can be any of the images captured by a camera or other image sensing device 105, previously stored images, or other image data representing the scene. In some embodiments, the image data may include data related to light (i.e., electromagnetic radiation), including but not limited to radiation values of visible light, infrared, radar, and / or polarized light or non-polarized light. Such data can be represented based on pixels or in other ways. Each image or image data set can preferably represent the incident light field at the point where the image data is being or was captured. In some embodiments, the present invention can select an image (which may be the first image taken) to define the origin and orientation of the scene.
[0058] Certain embodiments of the invention provide for using information from a scene (including, for example, image information that may be represented in digital form) to create one or more models of the scene, a region of interest within the scene, or the entire scene.
[0059] In some embodiments, a scene or a portion thereof may be represented by one or more plenoptic elements or primitives that can be stored in a data structure. In some embodiments of the invention, the spatial information in the scene is separated into plenoptic information and analytical information. In this embodiment, a plenoptic element may represent the scene in the model, and preferably may represent elements in the scene more realistically than a solution element. Some embodiments of the invention use at least one plenoptic element, one or more of which may be contained within a voxel, and / or a sael, or a solid angle element.
[0060] A scene may contain one or more voxels, each of which may be of the same size and shape, or may be selected from a range of sizes and / or shapes determined by the user or the system. A voxel may contain a medium or dielectric element, which may represent all or part of the medium sampled in the voxel. A medium is a three-dimensional region that includes some substance through which light flows or does not include substance through which light flows. A medium may be homogeneous or heterogeneous. Examples of homogeneous media include: empty space, air, and water. Examples of heterogeneous media include three-dimensional regions including the surface of a mirror (part air and part shard glass), the surface of a sheet of glass (part air and part transmissive glass), and the branches of a pine tree (part air and part organic material). Light flows in a medium through phenomena including absorption, reflection, transmission, and scattering. Examples of partially transmissive media include the branches of a pine tree and a sheet of glass.
[0061] A Szaeel may contain radiative elements or radiation elements, which may represent all or a portion of light flowing in one or more directions. Light includes electromagnetic waves at frequencies including visible light, infrared light, and ultraviolet light. Certain embodiments of the present invention may use digital images, digital art, other processes, or some combination of the foregoing to create, calculate, and / or store the mediators and / or radiation elements contained by a plenoptic element. Thus, in certain embodiments, a plenoptic element may be used to sample light and matter in a spatial scene, representing the three dimensions of a matter field and the two dimensions (5D) of light flowing in a scene in a manner that may be similar to how a pixel element may be used to sample light at a specific location in a scene. Analysis elements may include geometric entities such as points, lines, planes, and CAD models.
[0062] A plenoptic element may have one or more features, or sets of characteristics, such as length, color, and / or shape. In some embodiments, features may be identified within and / or between segments in a scene. Features may have one or more of a description and an example.
[0063] Some plenoptic elements may include an interposer 201 that includes a surface element or facet 202. Such an element may represent an abrupt interface between two regions of homogeneous but different media. Figure 2A The facet 202 is exemplarily depicted in FIG. Figure 2A A planar panel is depicted including vectors 204 and 205 as axes of the plane and a normal direction 203 extending perpendicularly from the plane.
[0064] Some media may include a homogenous element or "module" 210, which represents a medium having a uniform composition throughout its bounding voxels. Figure 2B An exemplary module 210 is depicted in Figure 2B A coordinate frame is depicted with vectors 213 and 214 representing directional information about the contained medium. Element 210 may be used to define material gradients, such as 3D "textures." If a plenoptic element is heterogeneous (i.e., does not have a uniform composition throughout its bounding voxels), it may be referred to as a hybrid element or "hybrid element."
[0065] Another type of intermediary can include a medium between one or more other types of media (usually but not always homogeneous media) sandwiched in an intermediary or "sandel". When solving one or more faces, a sandel appears afterwards, and the system determines that the intermediary contains multiple facets of opposite orientation or partially opposite orientation. An example of a sandel is an intermediary containing all or part of the side of a glass plate. In the aforementioned example, the surface of the glass represents the surface element in the sandel, and the air at either side of the glass represents the homogeneous medium at each side of the "sandel" glass facet. Sandel can provide data, power or processing savings opportunities. For example, compared with using only other types of intermediary, using sandel can allow processing to be completed with a rougher intermediary size. This saving can be achieved during scene reconstruction by specifying multiple surfaces in the intermediary in a single intermediary, rather than subdividing the intermediary into separate facets for each surface. Sandel can also allow the use of lower bandwidth, power or processing during the reconstruction of the scene in an intermediate size output and / or presentation of a similar rougher degree. For example, if the thickness of the exemplary glass is 0.25 inches, a cube-shaped Sander with dimensions of 0.5 inches can represent both one side of the glass, the glass and the air on both surfaces, and the air at the other side of the glass. If the system is configured to use only surface elements, moduli, and hybrid elements, the system may need to subdivide the element at least once additionally, thereby creating at least two additional elements to represent the surface and the homogenous medium.
[0066] Figure 4 and Figure 5 Three types of mediators and examples of their use are shown in FIG. Figure 4 A type hierarchy of exemplary elements 401 is depicted, where the elements may include facets 402, model elements 403, and hybrid elements 404. Various embodiments of the present invention may include all, any combination, or none of the foregoing elements. Figure 5 An exemplary glass sheet 501 is depicted represented by a set of voxels including primitives 502, which are shown as boxes in the figure. For visual clarity, the figure shows only a small number of primitives in a small number of voxels. In a typical digital model of a real-world scene, primitives will be densely present throughout the scene and will appear at several different levels of resolution in the data structure storing information related to the matter field.
[0067] like Figure 5As shown, facet 503 can contain more than one type of material. In this figure, facet 503 contains both glass and air, with a surface separating the two; module 504 contains only glass; and mixed element 505 represents the corner of the plate and therefore contains multiple surfaces. In general, elements can contain various forms of property information. For example, facets and modules can contain BLIF values or other property information that can be used for relighting. In some cases, mixed elements can contain information that makes them relightable.
[0068] The characteristics of BLIF are described elsewhere, such as in U.S. Pat. No. 10,521,952 to Ackerson et al. Fig.10 BLIF can characterize the incident light field, the emitted light field, the response light field and / or the outgoing light field. Fig.10 An exemplary model is depicted that can be used to represent the interactions occurring at a single cell, which consists of voxels and associated BLIFs. A radiation element of an incident light field enters the cell. The BLIF operates on the incident light field and generates a response light field that leaves the cell. The total exiting light field is the union of the response light field and the (optional) emission light field. The emission light field is emitted by the cell, independent of the stimulus of the incident light.
[0069] In some embodiments, the invention described herein may use non-plenoptic primitives, which may, for example, contain analytical information. Such non-plenoptic primitives may represent elements in a scene other than mediators and radiants, and generally do not contain information related to the interaction between light and matter in the scene. Examples of such non-plenoptic primitives include, but are not limited to, computer-aided drawing (CAD) or similar structures representing spheres, cones, or other shapes that may have been adapted for local facets, computer vision, or other scale-invariant feature transform (SIFT) style features or other information formed from patterns of pixels in an image.
[0070] Each of the foregoing elements or parameters can optionally be configured to be expandable into multiple finer parameters and / or foldable or combinable into a single parameter, a smaller set of parameters and / or a coarser parameter. This configuration optionally applies to all types of elements or parameters, including plenoptic, analytical, sampled, and learned parameters and elements. For example, voxels and / or Saels can be subdivided, or multiple voxels and / or Saels can be combined. Similarly, the overall diffuse reflectance can be subdivided into polarized diffuse reflectance and non-polarized diffuse reflectance. Another example is that the Phong reflectance model can be expanded into a set of sampled BLIF coefficients (e.g., ratios) that are stored in a hierarchical Sael data structure of the ratio of outgoing radiation to incident radiation for directional Sael pairs. Reference is made to Fig. 20Examples are further discussed where the analytical BLIF can be extended to a coarse analysis coupled with a fine neural network to more accurately predict the outgoing radiation.
[0071] refer to Figure 3A , some embodiments of the present invention may be operable to reconstruct a full light field, including using an incremental process, where the full light field may represent an entire scene, a portion of a scene, or a specific object or region of interest in the scene. In some embodiments, the system may first determine settings for reconstructing the scene 301. For example, the system may access or set a working resolution, an initial size, a target accuracy, a relightable property, or other properties. In some embodiments of the present invention, the system may assign an initial size to the scene. The size of the scene may be, for example, the size of a human living space for an indoor scene, a different size for an outdoor scene, or another size defined by the system, a user, or other factors that can be determined to be acceptable or advantageous. In some embodiments, including Figure 1B In the exemplary embodiment shown, a first camera 105 or image data set may define the origin of a scene, and subsequent camera images captured by camera 105, a second camera, or image sensing device 106, or otherwise, may be added to the scene and processed.
[0072] Some embodiments of the present invention may then initialize a data structure for storing scene 302, which in some embodiments may include a full light field and is referred to herein as Figure 3B and are further described elsewhere. Some embodiments of the invention may store data in a data structure starting at a coarse subdivision level. Some embodiments of the invention may store further data related to the scene in the data structure, including at iteratively finer levels of detail. Some embodiments of the invention may also be configured to compute or refine characteristics of the scene 303, which may include computing or refining characteristics of the full light field, and may be referred to herein as Figure 3C and further described elsewhere. In some embodiments, the system can be configured to use termination criteria, computational budget, or other factors to guide reconstruction activities 304. In embodiments with such termination criteria 304, if the criteria are met, processing can end, otherwise the system can determine whether any new image data is available 305. If new image data is available, the system can be configured to merge the new data 306, which is referred to herein as Figure 3D and are further described elsewhere.After incorporating the new image data 306, or if no new image data is available 305, the system may repeat the process starting at step 303 until termination.
[0073] refer to Figure 3B, some embodiments of the system may store the matter field and / or light field associated with the scene in a data structure. The data structure may take any number of forms known in the art, including, in some embodiments, one or more of a hierarchical, multi-resolution, and / or spatially ordered data structure. Exemplary data structures include a boundary volume hierarchy, a tree structure, a binary space partitioning, or other structure capable of storing image data in an accessible manner. In some embodiments, the data structure may be configured such that if the scene is divided into one or more plenoptic elements discussed herein. In addition, the data structure may be configured such that information associated with one aspect of the data structure (e.g., the matter field) may be associated with one or more other aspects of the data structure (e.g., one or more of a camera pose, features of the light field, or segments).
[0074] In some embodiments (specifically, the present invention is configured to reconstruct and / or store matter fields), the present invention may be configured to initialize a data structure to store the matter field in scene 311. Initializing the matter field may include preparing a data structure to store one or more of the size, shape, position, and / or light interaction properties associated with the matter in the scene. In embodiments where the matter field is divided into one or more voxels or elements, the data structure may be configured to store information associated with each voxel or element. In some embodiments, the data structure may be initialized to assume a specific type of medium associated with the element, which may be some homogeneous medium (e.g., air, water, fog, turbid water, or other homogeneous medium). Certain embodiments of the present invention may access some prior information related to the matter field, wherein the information may include one or more of the following: information describing the geometry of the scene or objects therein (e.g., an OBJ file representing a room and its contents), values of parameters in a low-dimensional parameter BLIF, values of parameters and / or coefficients in a high-dimensional sampling BLIF, and / or any combination of the geometry (position and / or orientation) of part or all of the scene and / or BLIF information. In embodiments where the present invention is not configured to reconstruct and / or store matter fields, these processes may not be necessary.
[0075] Certain embodiments of the invention may also be configured to initialize a data structure to store information related to one or more camera poses 312. In some embodiments, the data structure may store information related to assumed or known positions of one or more images of the scene, and may be related to other aspects of the data structure, such as one or more of voxels or mesels.
[0076] Some embodiments of the invention may also be configured to initialize a data structure to store information related to the light field in the scene 313. The data structure may initially be configured to store information about both incoming and outgoing light associated with various points, locations, or voxels in space, including information about portions of the data structure related to the matter field. Such information may be represented as one or more radiation elements associated with each location or voxel / element. Certain embodiments of the present invention may have access to some prior information related to light fields, where such information may include one or more of quantified information describing the following light fields: an incident light field for a point in a position space (e.g., a panoramic "environment map"); a surface light field that quantifies the incident light field and / or the exit light field in one or more directions at one or more points in the position space (e.g., a 4D light field), possibly at a touchable physical surface; a surface light field that quantifies the isotropic (or nearly isotropic) incident light field and / or exit light field at one or more points in the position space (e.g., a 2D light field), possibly at a touchable physical surface; and / or any combination of incident light field and / or exit light field information for part or all of the full light space of the scene.
[0077] Embodiments of the invention may also be configured to initialize a data structure to store information associated with one or more segments in the scene 314. A segment may represent one or more groups of media in the scene. In some embodiments, a segment may represent media with a specified likelihood of association (e.g., a suitably high confidence of association). For example, if the scene includes a vase with a flower, a segment may represent a leaf or petal of the flower, the entire flower, the vase, a vase containing the flower, etc.
[0078] In some embodiments, the present invention may also be configured to initialize a data structure to store other data related to scene 315. For example, such data may include non-plenoptic information, which may represent analytical information, for example.
[0079] Although the foregoing steps may be performed in any order, certain embodiments of the present invention may perform the steps in the order described herein. For example, initializing the data structure to store the matter field first may help associate the radiating element with the more relevant (or most relevant) belonging element. Similarly, initiating the data structure to store information related to one or more camera poses before initiating the data structure to store information related to the light field may allow the radiating element to be initiated with more relevant (or most relevant) position and / or direction information.
[0080] refer to Figure 3C, some embodiments of the invention may provide for computing or refining one or more characteristics of a scene (e.g., a full light field). Certain embodiments of the invention may process camera images or other image data sequentially, in parallel, or in some combination of the two. In some embodiments, the system may compute the light field in the scene 321 based on the image data, which is referred to herein as Figure 3E and described elsewhere.
[0081] The system may calculate or refine information about one or more gestures associated with the image data 322, such as reference Figure 3F and elsewhere. In some embodiments, the system can determine whether the light field containing the voxel has been changed 322 for one or more camera or image data viewpoints, which can optionally be determined based on some significance threshold that can be preset or calculated by the system. The determination can be based in part on other information that the system assumes or has that indicates that a camera image or image data set exists at the voxel 201 in the data structure, such as Figure 2A In such an embodiment, for each hypothesized position, the system may hypothesize an orientation in a coarse orientation space.
[0082] In some embodiments (specifically, the system is configured to reconstruct and / or store matter fields), the system can be configured to access and test one or more (or all) voxels whose light fields containing voxels have been altered 323. In some embodiments, if the light field associated with the voxel has changed by a certain amount (including a threshold set in the system, or a threshold set by a user, or a threshold calculated by the system), the system can be configured to calculate or refine the voxel, referring to Figure 3G and described elsewhere. In some embodiments, the system may be configured to compute one or more segments 324 of the scene. Some embodiments of the system may be configured to compute or refine other data associated with the scene 325, such as non-plenoptical and / or analytical information. In embodiments where the invention is not configured to reconstruct and / or store matter fields, these processes may not be necessary.
[0083] The system can also be configured to include specific termination criteria, computational budgets, or other thresholds 326, including with respect to computing or refining the full light field. In such embodiments, the system can determine whether the termination criteria, computational budget, or other threshold has been exceeded, as discussed elsewhere herein. If the threshold has not been exceeded, the system can be configured to repeat the process, e.g., starting from step 321. If the threshold has been exceeded, the system can complete the process.
[0084] refer to Figure 3E, the system may be configured to calculate and / or refine the light field in the scene. Some embodiments of the present invention may perform light transport operations 341 to calculate the light field. For example, the system may calculate the propagation of radiation through the default medium and / or pass the incident radiation through the BLIF to produce outgoing radiation. The light transport operation may optionally be limited to radiation elements that exceed a threshold change from a previous state. The light transport operation may be limited to or depend on the path length. The path length may be automatically determined, for example, by a confidence level or a change in confidence level, and may be run in some combination of a downstream direction and an upstream direction. The light transport operation may include any combination of downstream (forward in time) radiation propagation through the default medium, upstream (backward in time) radiation propagation through the default medium, incident radiation through the BLIF to produce outgoing radiation (BLIF operation in the downstream direction), and / or emitted radiation through the BLIF to produce incident radiation (BLIF operation in the upstream direction).
[0085] For example, light transmission may be performed in a downstream direction relative to a face element of semi-gloss paint. In this case, the incident radiation element may have an updated, higher confidence radiation value in at least one of its color bands. The higher confidence radiation value may prompt a downstream (forward in time) BLIF interaction, which may produce one or more new radiation values in one or more radiation elements from the face element. As another example, light transmission may be performed in an upstream direction relative to a face element of shiny chrome. In this case, the outgoing radiation element may have an updated, higher confidence value (e.g., a radiation value in at least one of its color bands). This may occur when a new camera viewpoint is added that directly sees the chrome face element. The new outgoing radiation may prompt an upstream (backward in time) BLIF interaction, which produces new radiation values for one or more radiation elements incident to the face element. In other embodiments, such as after providing image data representing a new image at a viewpoint in a room, light transmission may occur in two time directions. The pixels of the new image may be resampled to high confidence incident radiation elements at the voxels containing the viewpoint. The incoming radiance may be propagated upstream to modify the lower confidence outgoing radiance at a surfel in the scene (e.g., on a wall that the camera sees within its field of view). Additionally, the incoming radiance at the camera viewpoint voxel may optionally be computed to make it the antipodal outgoing radiance, which may then be propagated downstream to modify the lower confidence incoming radiance at a surfel in the scene (such as on a surfel on a wall behind the camera).
[0086] Although the light field (radiance element) and matter field (mediation element) calculations, refinements and / or updates may be separate steps, this configuration may optionally be modified. In some embodiments, this structure may result in an undesirable avoidance of finding the global best (deepest) solution in the overall cost function space of the (sub) scene being reconstructed. For example, certain calculations may reach a certain degree of solution accuracy, but are no longer close to the optimal solution (e.g., the global best solution). This may occur, for example, when iterating between "light field only" and "matter field only" search directions (which may avoid the optimal parameter stepping direction in the multi-dimensional cost function space where both light field and matter field parameters are changed simultaneously). An exemplary solution to this problem, optionally implemented by the system, is to use the identification of the assumed scene model revisiting the same state in a limit cycle (which may occur at any sub-scene level within the entire scene). Upon identifying the presence of a limit cycle, the system may revert to an earlier and / or coarser visited state and proceed after changing one or more control parameters of the search (e.g., the next region of the parameter space to be visited, and / or the step size for each or a specific parameter dimension). Thereafter, the system can follow any known method to escape from the limit cycle.
[0087] Such operations may optionally use actual values of radiation and / or changes in values, other radiation meta-characteristics, and / or confidence (consistency) of radiation meta-characteristics to determine when to terminate the sequence of operations. For example, the system may also be configured to include specific termination criteria, computational budgets, or other thresholds 342, including with respect to light transmission depths that reflect iterative and / or recursive calculation sets. In such embodiments, the system may determine whether the termination criteria, computational budget, or other threshold has been exceeded, as discussed elsewhere herein. If the threshold has not been exceeded, the system may be configured to repeat the process, for example, starting from step 341. If the threshold has been exceeded, the system may complete the process.
[0088] refer to Figure 3F , the system can be configured to calculate or refine a camera pose associated with particular image data. In some embodiments, the system can be configured to create a trial copy 351 of the scene or a relevant portion thereof. The system can determine an assumed camera pose 352. In some embodiments, the determination of the assumed camera pose can be performed in a coarse to fine order in a parameter space that defines potential camera poses. The system can determine whether the assumed camera pose is outside the scene boundary 353. If the assumed pose is outside the scene boundary, the system can increase the size of the trial copy of the scene to accommodate the assumed camera pose 354.
[0089] If the assumed pose is within the scene bounds, or after the scene bounds have been augmented, the system can, for example, use a reference Figure 3C Step 322 and the processes described elsewhere are used to compute or refine a trial copy of the scene 355. This computation can be performed using a modest computational budget and / or recursive camera pose refinement can be skipped.
[0090] In some embodiments, the accuracy of the camera pose can have a huge impact on scene accuracy. The system can be configured to represent the camera pose with higher accuracy than the discrete spatial elements containing the camera viewpoint. For example, the camera pose at a given viewpoint can be represented as a floating point number, rather than by subdividing the containing voxels of the viewpoint into many finer levels in a position hierarchy. Similarly, the orientation of the camera can be represented using floating points containing parameters such as yaw, pitch, and roll, rather than using discrete Sauer data structures to represent these features.
[0091] The system can also be configured to include a specific termination criterion, computational budget, or other threshold 356, including information related to the camera pose. In such an embodiment, the system can determine whether the termination criterion, computational budget, or other threshold has been exceeded, as discussed elsewhere herein. If the threshold has been exceeded, the system can replace the full light field or a relevant portion thereof with a trial copy 357, complete the process, or both. If the threshold has not been exceeded, the system can be configured to repeat the process, for example, starting from step 352.
[0092] In some embodiments, the system may perform calculations to determine if there is a measurable parallax. The measurable parallax calculation may be calculated based on changes in received light, and may further depend on the camera or image data resolution and the positional separation between viewpoints. In some embodiments, the first two quantities may set the actual parallax boundary distances outward in different directions from the envelope of the camera / image data working space or camera viewpoints. The parallax boundary is typically directional, meaning that the boundary may be a different distance in different directions depending on the shape of the camera / image data working space. For example, a wide separation between viewpoints in a given direction may push the parallax boundary further outward in a plane in a direction perpendicular to that viewpoint separation vector.
[0093] In some embodiments, the system can use the disparity boundary to set the size of the scene (e.g., the outer boundary of the scene). In some embodiments, the disparity boundary can be a boundary voxel of an envelope that limits the directional disparity boundary distance. In some embodiments, for example, when new image data is accessed, the camera / image data workspace can expand and push the disparity boundary outward. The system can increase the size of the scene and / or the associated full light field in response to the extension of the disparity boundary.
[0094] In some embodiments, the system can represent scene information outside the parallax boundary in two-dimensional properties. For example, such information can be represented as a two-dimensional map and / or a multi-resolution grid volume. As an illustration, the night sky can be represented as a two-dimensional light field. Depending on the size of the scene, an airplane flying at high altitude can also exist in a two-dimensional layer outside the parallax boundary. Information outside the parallax boundary does not necessarily contain only light, but can also have other associated properties. For example, in the case of the night sky, the moon can be represented as moon dust in a two-dimensional material field with an associated BLIF. In addition, any number of layers can be stacked in a certain order based on known distances or other priorities. In some embodiments, the light field and / or material field information or data in such a layer can be dynamic in time in the same way that the information within the parallax boundary is dynamic in time.
[0095] about Figure 3G , the system can be configured to compute properties of one or more intermediaries. The system can first use the image data to compute intermediary properties 361. In some embodiments, the system can be configured to compute intermediary properties so that adjustments are made in the direction of improving an overall confidence metric for the intermediary computation (e.g., an associated radiance element with a lower current confidence can be updated by a radiance element with a higher confidence). For example, the overall confidence of an intermediary can be computed by obtaining new predicted outgoing radiance in one or more directions of interest (e.g., toward a direct camera viewpoint), subtracting pre-existing reference and / or observed outgoing radiance from the predicted radiance, and computing some variation of the average of the predicted radiance minus the reference radiance deviation to produce a scalar cost and / or error. In such embodiments, the confidence can be considered to be some inverse of the cost (e.g., 1-cost or 1 / cost).
[0096] In some embodiments, the system may assume or otherwise determine that a surface exists within an element. An element containing a surface may be referred to as a surface element or a facet. In such embodiments, the system may be configured to search for geometric parameters associated with the surface. For example, the system may calculate a surface normal vector, a BLIF, or other dimensions or parameters associated with a facet.
[0097] In embodiments where the system tests whether a voxel in a data structure is a facet, the system may compute the outgoing light at the voxel based on one or more of cameras 105 and 106 or other image data including voxels in the field of view captured by the camera or represented in the data. In some embodiments, each camera may observe the voxel, or each image data set may represent a voxel from a different angle, and may observe specific radiation emanating from that location in space in a specific direction. The system may use one or more of these observations to determine the observed outgoing light field for that location or voxel in space. Similarly, the system may compute the incoming light field for a location or voxel in space based on one or more cameras or image data sets that observe light entering that point or voxel in space.
[0098] In some embodiments, the system may be configured to compute properties associated with the light field of an element, such as the directional resolution of the light field 362 containing the voxel. For example, if the directional resolution of the light field containing the voxel is calculated and / or adjusted, the system may split and / or merge the radiation elements associated with the voxel or neighboring voxels. Certain embodiments of the present invention may also adaptively sample the light field associated with an element. For example, the system may adaptively sample the incident plenoptic light field at the element using one or more data sets, such as an assumed BLIF, an exit direction of interest (e.g., a camera viewpoint), or other data associated with the measured element. Some embodiments of the present invention may perform such calculations based on a target for exiting plenoptic light field confidence (e.g., based on the observed light at the element) or a computational budget (e.g., a maximum number of radiation elements (or radiation elements) that may be associated with the element). In some embodiments, the system may be configured to represent the directional resolution using spherical harmonics. For example, in an embodiment where the system characterizes the light field associated with a glossy paint facet, such facet may have a highly specular behavior in a real scene. Early in processing, a search for candidate BLIF properties may indicate that the facet is likely to be more specular than diffuse. The system may then be configured to instantiate higher-order spherical harmonic coefficients to produce tighter specular lobes in the direction of high incident radiation from other scene regions. Spherical harmonics may be defined in a coordinate system that is rotated with an assumed normal vector. For materials with a granular appearance (e.g., brushed metal), the system may be configured with material grain tangent vectors associated with an anisotropic BLIF.
[0099] Some embodiments of the present invention may assume that a surface exists within one or more elements, such as Figure 2AAs shown. By assuming the presence of a surface within the voxel, the system can also assume a particular surface normal 203, the displacement of the surface within the voxel, and / or the light interaction properties of the surface. The system can then calculate a set of predicted outgoing radiation vectors, including based on the assumed surface normal and / or light interaction properties, which may include one or more of a refractive index, roughness, polarized diffusion coefficient, non-polarized diffusion coefficient, and / or extinction coefficient, the latter of which may be particularly applicable to metallic materials. In some embodiments, the system can be configured to search for one or more of the aforementioned properties in a serial manner (e.g., by following a "waterfall" test of correctness from most likely to least likely based on the applicable image data).
[0100] Some embodiments of the present invention may calculate a "cost" for the presence of a surface. In some embodiments, the cost of the presence of a surface having the calculated properties may be expressed as the difference between the predicted or calculated (i) surface normal, (ii) light interaction value, (iii) outgoing radiation vector, and / or (iv) other properties and the corresponding observed values. In some embodiments, the system may have a specified or specifyable cost threshold, wherein when the cost is below the threshold, the face element is accepted to exist at the voxel. In such embodiments, when a voxel is determined to be material and / or contains a surface, the surface may be retained in the scene for subsequent iterations. In some embodiments, the surface normal may be searched in a hierarchical manner that matches the data structure storing the sael or radiation element. In the case of a cube-shaped voxel, the system may perform calculations for each of the six faces. In addition, the system may be configured to divide the voxel into eight cube-shaped sub-voxels, resulting in the need to calculate surface normals for a total of 24 outward-facing faces and 96 overall sub-radiation elements. For each direction, the system may be configured to calculate a predicted outgoing radiation vector and an associated surface normal and / or light interaction property. This processing can be done in a variety of ways, including in a highly parallel or multithreaded manner, using GPUs, AI and / or ML, binary tree hierarchies, or other configurations to accelerate processing. In some embodiments, the system can determine the most likely, lowest cost, highest confidence, or other set of parent hypotheses and use these hypotheses as a starting point for processing the segmentation.
[0101] In some embodiments, the solution for the existence of the light field and / or surface at voxel 201 may use the light field computed for one or more other voxels to compute hypothesized incident light, outgoing light, and / or other radiative properties for voxel 201. Such computations may be in addition to, or computed using, the incident light field represented by a camera image or other image data set. Thus, certain embodiments of the present invention may update hypotheses for the light field of one or more voxels 201 using one or more radiative elements or radiative units of a projection emitted from one or more other voxels (by tracking the radiative unit's impact on the scene and / or its interaction with the medium in other voxels).
[0102] In some embodiments, the system may first compute a light field associated with a scene, and provide information about the light field to inform the process to detect the presence of a facet. When computing the presence of a medium in the scene represented by the facet, the system may use the presence as a factor in recomputing the light field associated with the scene. The process may be performed in any order, and may be performed iteratively to increase confidence or reduce costs associated with the light field and / or matter field in the scene. In some embodiments, the system may perform the process before subdividing a facet into multiple subfacets. The system may also be configured to perform similar tests of the light field and / or matter field after subdividing the facet (which may be based in part on the parent facet), and then perform the same processing as described above for the parent facet.
[0103] In some embodiments, the foregoing process may continue until the system achieves certain specific termination criteria, computational budgets, or other thresholds 363, including light fields and / or matter fields associated with the mesocell. In such embodiments, the system may determine whether the termination criteria, computational budgets, or other thresholds have been exceeded, as discussed elsewhere herein. If the threshold has not yet been exceeded, the system may be configured to iteratively and / or recursively repeat the process, for example starting from step 361. If the threshold has been exceeded, the system may determine whether one or more mesocell properties have exceeded a confidence threshold 364. If the confidence threshold has been exceeded, the system may complete the process. If the confidence threshold has not yet been achieved, the system may optionally subdivide the mesocell into N sub-mesocells 365 as described below, unless a resolution budget or limit has been reached.
[0104] Some embodiments of the invention may then use a confidence threshold or other metric to guide processing and calculate an associated confidence or other metric associated with each intermediary or other volumetric element within the scene. If a confidence threshold is used, the system may check one or more intermediaries for a confidence level below the confidence threshold. In some embodiments, if the confidence level is below the threshold, the system may compare the characteristics of the intermediary to various known light interaction characteristics, such as bidirectional light interaction functions (or BLIFs) associated with different types of media. For example, in Fig.14A In the example shown, if the confidence threshold is 75, the system can be configured to perform further calculations on each of the four elements 1403 shown because the associated confidence is below 75. Some embodiments can use a waterfall or sequential comparison order based on the most likely candidate characteristics (e.g., most likely candidate BLIF) that the system has calculated for a particular element. For example, for a particular element, the system can first test whether the element contains air, then test general dielectric media, then test general metallic media, etc.
[0105] In some embodiments, such as Fig.14A As shown, the scene of interest can contain both homogeneous transmissive media and opaque media. In embodiments where the region of interest includes empty space (e.g., air or other homogeneous space that does not contain a medium that substantially interacts with light), the system can specify within the data structure that the scene consists of elements that include empty space (e.g., air). Empty elements or elements that include air and other homogeneous elements can be referred to as elements. In some embodiments, even if element 1401 contains an opaque surface 1402, the system can initially specify that element 1401 consists of one or more elements 1403 that contain empty space or air (or air elements), wherein such initialization will allow the system to allow light to flow through element 1401 and element 1403 rather than assuming that it is blocked by an interacting medium (such as 1402). Some embodiments of the present invention can specify a low confidence 1405 associated with each air element, which can help the system later determine the presence of other media within each air element. In Fig.14A , the assumed content 1404 and confidence 1405 are depicted, where the content "A" 1404 represents the initial assumption that the module 1403 contains air, and the confidence "10" 1405 represents the assumed confidence value associated with the assumption.
[0106] The system can determine the confidence (or cost) associated with a particular cell in a variety of ways. For example, for a cell that is assumed to be empty or air, the system can expect that the difference between radiation elements entering the cell should be substantially equal to the radiation elements leaving the cell in the antipodal direction. Similarly, if the cell is a surface element, the system can expect a specific relationship between the light field incident on the cell and the light field exiting the cell based on the specific medium properties including the surface element. Therefore, the system can be configured to determine the confidence associated with a particular cell by calculating the error between the expected difference between the incident light and the exiting light and the actual difference (e.g., the antipodal error of the radiation element of the surface element assumed to be air or the air surface element).
[0107] In some embodiments, the present invention may utilize machine learning (ML) and / or artificial intelligence components as described elsewhere herein to assist in determining the confidence (or cost) associated with the radiation element difference. For the example of the air element, the AI / ML model can be configured to determine the confidence or cost that the element is an air element by comparing the calculated results with the corresponding radiation element differences. In various embodiments, the AI / ML model can be configured to compare the average, median, minimum, maximum, and / or other differences between the various calculated radiation elements. In some embodiments, the AI / ML model can be configured to discard a selected or variable number or percentage of radiation elements (e.g., a specific percentage of the most inconsistent radiation elements) when performing confidence or cost determinations.
[0108] Some embodiments of the present invention may perform the aforementioned calculations in an iterative and / or recursive manner. For example, in some embodiments, the system may calculate the scene data described herein for a particular camera pose or image data set, including finding any facets 202 therein. Thereafter, the system may process subsequent camera images or image data sets in a similar manner. In the case where such iterative processing results in finding more facets 202 in the scene, the system may more accurately determine the orientation of subsequent camera poses and / or image data sets. Some embodiments of the present invention may also update information related to the orientation of one or more previously determined camera poses and / or image data sets to better fit new observations. In some embodiments, when locating new camera images and / or accessing image data from alternative viewpoints, and updating the orientation of existing camera image positions and / or image data sets, the camera data structure may be hierarchically traversed from a coarse level to a finer level. When more camera images and / or image data sets become available, the system may resolve the material field present in the scene to a finer level of detail. In some embodiments, the present invention may store such finer level of detail in a material field data structure, including by subdividing the material field data structure. The result in some embodiments of the invention is a set of camera images, camera poses, other image data and / or information related to the orientation of any of the foregoing, a matter field computed by the system, and a light field associated therewith. The foregoing results may optionally be data that best describes a given image through a light transport calculation, and may be referred to as a reconstructed scene.
[0109] In some embodiments, the present invention may perform the aforementioned calculations for other elements (or all elements) within the region or scene of interest. The system may then compare the results for one or more elements to a confidence threshold or other metric, for example, based on the predicted radiation characteristics minus the observed characteristics associated with the outgoing radiation element of the element. For elements that do not meet the confidence threshold or other metric, the system may be configured to perform further processing related to such elements. For example, Fig. 14B Depicted is a case where the system has determined that the lower right module 1403 of the element 1401 does not meet an appropriate threshold. In some embodiments, the system may subdivide the element that does not meet the threshold or other metric into two or more sub-elements, such as subdividing a cube-shaped element into eight cube-shaped sub-elements. Fig. 14B In FIG. 1 , the system subdivides the module 1403 into four sub-modules 1406, each sub-module 1406 having an associated content hypothesis 1407 and a confidence level 1408. Fig. 14BIn the illustrated embodiment, the system has now assumed that subcell 1409 contains a surface, such as an opaque dielectric surface (represented by "S") that can be represented by a bin, as shown at 1413, with a confidence of 50. The remaining subcells are still assumed to contain air with varying confidences.
[0110] After such subdivision, the system can be configured to perform the aforementioned processing to determine the BLIF or other characteristics associated with the region or scene of interest until a confidence threshold, other metric, or maximum computational threshold is reached. In some embodiments, the system can determine that the confidence of the element does not change significantly after subdivision. In this case, the system can be configured to determine a local minimum threshold, for example, based on an asymptotic determination that can be determined in a traditional cost function minimization problem. For example, referring to Fig. 14C , under the assumption that the confidence threshold is set to 75, the system has further subdivided sub-meshes 1409, 1410, and 1411 because these sub-meshes do not exceed the threshold, but has not further subdivided meso 1412 because the confidence exceeds the threshold. Fig. 14C After the processing shown, all sub-elements have satisfied the associated thresholds except for sub-element 1413. Specifically, all sub-elements within sub-elements 1410, 1411, and 1412 have exceeded the confidence threshold for being air. All sub-elements within sub-element 1409 have exceeded the threshold for containing surfaces, but the system has determined that sub-element 1413 cannot be further resolved because the material within sub-element 1413 has been completely obscured by the remaining sub-elements within sub-element 1409.
[0111] In some embodiments, the system can be configured to assume a higher confidence or lower cost for a particular cell based on the confidence or cost associated with neighboring cells. For example, the system can be configured to assume a higher confidence or lower cost associated with the presence of a surface within a cell if the neighboring cell contains a surface, an even higher confidence or lower cost if both neighboring cells contain surfaces, etc. Similar assumptions can be made for other types of media (e.g., a cell includes a particular type of media in neighboring cells, or one or more empty or air cells adjacent to the cell in question).
[0112] refer to Figure 3D , the system can be configured to merge new image data. In some embodiments, the system can initialize one or more new camera poses 331, which can be referenced, for example Figure 3B and Figure 3F Then, some embodiments of the present invention may place one or more new radiation elements at the voxels in the scene that contain one or more new viewpoints 332 .
[0113] Some embodiments of the present invention may choose to provide the position and orientation of the hypothesized facets for the best pose of the incoming camera in this iteration of the algorithm. For example, the system may assign confidence to a particular position or orientation (or conversely, assign a cost associated with the lack of confidence). Such confidence may optionally be associated with the presence or absence of a surface, a radiation characteristic such as an incident light field or an outgoing light field, or another characteristic at the position or orientation. The system may determine the confidence or cost based on a variety of factors, some of which include the proximity or non-proximity of the position to the camera pose orientation, the consistency of the observations with data from other camera poses, or other information. For example, in an embodiment where the information is a radiation element associated with a position in space, the confidence assigned may be higher or the cost assigned may be lower for a position directly associated with an observation from a camera. Similarly, for a position that is located not too close to a position observed by a camera or a position depicted in image data, the confidence assigned may be lower or the cost assigned may be higher. Some embodiments of the present invention may use confidence or cost as a weighting factor. In this way, the system may be able to determine position, orientation, or other information in scenarios where there is high consistency and / or low cost, low consistency and high cost, or some scenario in between.
[0114] In some embodiments, the result of the foregoing processing may result in one or more local regions within a data structure storing information related to a matter field and / or a light field being subdivided into finer or deeper resolutions. Such fine subdivisions may be triggered by local contrast (such as a sharp change and / or gradient in voxel occupancy, intermediary type, BLIF, geometry, or other characteristics). Local regions of finer and / or deeper resolution may appear in the matter field (e.g., at a specific location, voxel, location in the data structure) and / or in the light field (e.g., at a specific direction, sael, location in the data structure). For example, such local regions may occur in a matter field in the middle of a large solid-colored wall, where small dots may be represented by finely subdivided facets. As a second example, such local regions may occur in a light field, where direct sunlight is represented by finely subdivided radiation elements in a light field incident on a facet of shiny chrome. After reflecting off the chrome facet, the resulting outgoing light field may have finely subdivided facets in the specular reflection direction relative to the incident sunlight.
[0115] In some embodiments, there may be local areas (e.g., sub-scenes) where additional computational energy can be applied in order to reconstruct certain scene features with higher accuracy than the rest of the scene. For example, the system may be applied to reconstruct an entire room, but there may be a single object in the room that is of particular interest. Additional computational energy may be applied to the object of interest. Some embodiments of the invention may apply additional computational energy in areas that are the same or similar to areas where the data structure is subdivided into finer and / or much deeper resolutions. For example, in the case of a solid colored wall with small dots, it may be advantageous to reconstruct the location of the wall facets with 1 mm accuracy in a direction perpendicular to the wall, but the wall surface may have a homogeneous BLIF and may be represented as facets that are 10 cm wide.
[0116] Those skilled in the art will recognize that the processes described herein can be used in conjunction with, as part of, or to enhance various scene reconstruction techniques known in the art. For example, various methods for performing scene reconstruction are taught in U.S. Patent No. 10,521,952 to Ackerson et al. Figures 3A to 3G and FIG. 14A to FIG. 14C The described process may be incorporated in whole or in part at various points in the process described in U.S. Pat. No. 10,521,952, including as referenced with respect to FIG. 3 thereof (e.g., at steps 309, 311, and / or 313), FIG. 14 thereof (e.g., at step 1403), and FIG. Fig.16 (e.g., at step 1609), Fig.18 A (e.g., at steps 1811, 1813, 1815, and / or 1819), Fig.18 B (e.g., regarding step 1819) and its Fig.18 D (e.g., at step 1880). Accordingly, some embodiments of the invention described herein are intended to be fully compatible with the processes taught in U.S. Patent No. 10,521,952 and other scene reconstruction techniques.
[0117] Additionally, in some embodiments of the invention, reconstructions of opaque external structures of an object or scene may be combined with reconstructions of internal structures of the same object or scene (including internal reconstructions created using different methods, such as X-ray imaging or MRI scans), such as Fig.21 As shown. The internal structure can be nested within the external structure to form a more complete model of the object or scene. In some embodiments, if the method for reconstructing the internal structure lacks BLIF information, the BLIF information can be automatically generated based on the BLIF of the external structure using methods such as machine learning.
[0118] Certain embodiments of the present invention may be used to represent tubular structures, 3D corners, or other surfaces. In some embodiments, a surface may be determined to be curved based on prior knowledge or a posteriori segment (region) data, each of which may optionally be represented by a face element. In some embodiments, a face element may have a map aligned with a tangent vector. Such a map may optionally represent various properties, such as the roughness (bump map), color (texture map), material, and / or other properties of the surface. In some embodiments, the material gradient along the normal may be a step function, or may be more complex (e.g., a "fuzzy" surface or a multi-layer surface, such as a clear-coated car), or represented in other ways.
[0119] Figure 6 and Figure 7 A solid tubular structure (e.g., a tree branch) represented using a panel is exemplarily depicted. Figure 6 As shown, Figure 6 A graph along a curve with some bins 601 is shown, which are shown as planes within voxels 603 but are stored as normal and tangent vectors. A tubular boundary 602 may be represented by an analytical curve 604. Figure 7 Depicted is a side view of an analysis curve 701 of a curved object with representative surfels 703 depicted in voxels 702. In this manner, surfels may be identified as planes, but are not necessarily identified as planes. Figure 8 The use of bins 801 within voxels 802 to represent corners 803 is exemplarily depicted. Corner 803 may be a single point-like feature, which may be a typical analysis feature that can be found during the beam initialization process.
[0120] Some embodiments of the present invention may retain data associated with a facet or mesotope in a priority order designed to optimize the performance of the system. For example, certain embodiments of the present invention may retain information in a descending priority order of image or other digital imaging information, point features and / or oriented features, dielectric primitives, outgoing light fields and incoming light fields (which may themselves be exact in the viewing direction, interpolated, interpolated only in the non-viewing direction), geometry, bump maps, and texture. Other embodiments of the present invention may use an alternative priority order, omit one or more categories of the foregoing information, and / or include one or more other categories of information. In some embodiments, the present invention may retain all of the foregoing information, may discard higher level information where lower level information is available or illustrative, or some combination of the two.
[0121] In certain embodiments of the invention, the system may be capable of capturing, receiving, processing, using and / or representing certain analysis primitives within a scene. The analysis primitives may optionally include one or more of the following types of data or other types of information obtained from or provided about the scene: points, vectors, lines, planes, spheres, rectangles, parallelepipeds, grids, other CAD-like models or features (including constructive solid geometry (CSG) and / or boundary representations (B-re p)), and / or other information.
[0122] In the context of generalized scene reconstruction, plenoptic information and analytical information may be processed independently, together, or in some combination of the two. In some embodiments of the present invention, plenoptic information and analytical information may be processed in a common workspace (preferably in an "as needed" manner) to achieve reconstruction of the scene or another goal. The present invention includes means for inspecting, processing, storing, and using such information, including, for example, spatial elements, data structures, and related processing functions. In some embodiments, some of the spatial, plenoptic, and other processing operations may be performed with improved efficiency, optionally by using parallel computing elements, dedicated processors, and the like (including arrays of the foregoing). An example of such improved efficiency is the transmission of light field radiation values between elements in a scene. For example, the system may use a set of FPGA cores, CPUs, CPU cores, or hardware acceleration (HWA) using one or more graphics processing units (GPUs), neural processing units (NPUs), tensor processing units (TPUs), and / or other specialized processing units (including the use of HWA managed by one or more CPUs or other computing devices) to process incoming radiation elements and / or outgoing radiation elements. When processing many intermediaries and / or incident radiation elements, an example FPGA-based embodiment can run light transport calculations for tens, hundreds, thousands, or more radiation elements in parallel. In some embodiments, if a scene is divided into segments or sub-scenes, the system can provide parallel processing of radiation elements, intermediaries, or groups thereof within each of one or more sub-scenes.
[0123] In some embodiments, the present invention may use a segment or sub-scene, which may include a collection of one or more plenoptic elements, each of which may contain one or more associated mediators and radiators. A segment may have: a sub-segment, which may include a subset of one or more plenoptic elements in a segment; and a super-segment, which may include one or more plenoptic elements from one or more segments.
[0124] Some scenes may include one or more objects (which in some embodiments represent one or more segments represented by a human or computer), which are substances present in the scene (e.g., a basketball, a bird, or a person). Although a lawn or even blades of grass may not be colloquially referred to as an object, such substances may be represented as segments and are referred to as segments or, in the context of some embodiments of the invention, as objects.
[0125] In some embodiments, generalized scene reconstruction can be implemented as a non-parametric process. In some embodiments, non-parametric modeling means that the modeling of the structure of the segment is not completely predetermined. Instead, in such embodiments, at least some of the parameter information used to represent the segment is derived from the sensory data itself.
[0126] Some embodiments of the invention may use spatially ordered, layered, and / or multi-resolution plenoptic elements, or any combination of the foregoing. In some embodiments of the invention, local curvature constraints (e.g., B-splines) may be used to regularize panels or surface elements in one or more segments, subsegments, or supersegments. Such a process may be used to specify a priori or discover a posteriori locations within a scene where discontinuous derivatives exist.
[0127] Some embodiments of the present invention allow for the distinction of different segments or sets of segments (super segments), which in some embodiments may represent different objects, materials, or other characteristics of a scene. In some embodiments, this distinction may be made at least in part using certain information attached to plenoptic elements, which are collectively referred to as properties and may be represented in one or more nodes or regions within a data structure. Such information may include, but is not limited to, properties such as color, normal, gradient or tangent vectors, material, associated bidirectional light interaction functions, density, transparency, radiance, and / or other factors. In embodiments where plenoptic data sets are spatially ordered, the present invention may be implemented in a computationally efficient manner by, for example, simultaneously traversing an aligned data structure to access the same or corresponding stereoscopic or directional regions of space in the structure. In some embodiments, the ability to spatially order data sets may facilitate the maintenance of multiple data sets representing different information in the same scene and allow for the collocation of properties. In this manner, exemplary embodiments of the present invention may maintain one or more features of a scene in multiple structures and make available and / or process a subset thereof that is relevant to a particular operation.
[0128] Certain embodiments of the present invention may use a data structure to represent segments of plenoptic information. Subsets of plenoptic data may be represented in a variety of ways, depending on the requirements of a particular application. In some embodiments, a plenoptic data structure may include a segment identifier, such as part of a core structure for identifying the type or nature of a segment attached to the data structure. Such an implementation may be particularly advantageous where it is necessary or desirable to represent a small number of segments.
[0129] In embodiments where data may be included in multiple segments, the identifier may preferably provide multiple segment memberships. In some embodiments, this identification may be implemented directly using properties, in other embodiments, by using table pointer properties and membership in a table, and in other embodiments, a combination of the foregoing or alternative methods may be used.
[0130] In some embodiments, a segment may be implicitly represented based on type, property, or other characteristic or variable. For example, a segment may be defined as a portion of a plenoptic data structure that matches a certain set of inclusion or exclusion properties (eg, density within specified limits).
[0131] In some embodiments, a separate shadow plenoptic data structure may be defined. A shadow plenoptic data structure may be a plenoptic data structure that represents a portion of the data of at least a subset of another plenoptic data structure or structures but represents alternative information (e.g., membership in a selection set or property deviations). A shadow plenoptic data structure may be advantageous when it is necessary or desirable to represent a large number of segments. For example, a binary data structure (e.g., data marked as included or not included in a selection set) may represent data in another plenoptic data structure belonging to a particular segment. In such an embodiment, multiple such segments may be combined using set operations performed simultaneously with the associated plenoptic data structures so that a super segment may be created. Such a super segment may have overlapping data.
[0132] In some embodiments of the present invention, it may be desirable to replicate a structure at one or more locations within a scene. This preference may be particularly useful for analysis projects. A graphical structure may be used to eliminate duplication of identical data, avoid the need to make any changes at multiple locations, some combination of the two, or for other purposes. In certain embodiments, a graphic may be evaluated or "flattened" into a simpler data structure before being used for operations such as displaying a scene or a portion thereof, wherein all transformations applicable to individual elements are combined into a single transformation. As a non-limiting example of using a graphical structure in a scene, consider an assembly including a plurality of bolts. The foregoing process may be used to change the bolt type, position, orientation, or other features. Thus, the present invention may be used to implement the transformation of all relevant bolts in the scene by a single transformation or a reduced number of transformations based on set or determined parameters.
[0133] In some embodiments, the system can provide sampling to determine a function that can represent a surface. In some embodiments, the sampling can include sampling of polarization data so that the function can represent the intensity on the surface, polarization information on the surface, a light source, an outgoing light field or output, or any combination of the foregoing. When the system assigns a function on a surface, the resulting function can provide a data set that can represent the surface. For example, the function can provide a voxel fitting model that allows determination of how the voxel will respond under different conditions. Such a function can be used to determine the response in a specific light field or to determine how the response in a single light field can be transformed into other light fields. For example, such a representation can also be used where the system has not (or has not yet) separated the material field from the light field.
[0134] In certain embodiments of the present invention, the aforementioned plenoptic representation may be used to perform generalized scene reconstruction. For example, some of the aforementioned plenoptic representations may be space-filling (space-filling of 3D stereoscopic space or directional space) and may support more powerful processing operations than the exemplary surface, line, or point representations described in detail herein. Certain embodiments of the present invention may combine, process, analyze, or perform other operations to implement and facilitate scene reconstruction with functions that are difficult or impossible to implement using low-dimensional representations. Exemplary categories of operations that may be performed as part of scene reconstruction or other processes described herein include, but are not limited to: threshold processing based on one or more properties; calculation of connectivity of one or more elements or objects; calculation of mass properties (volume, mass, center of mass, etc.); identifier tagging (e.g., identifying regions of stereoscopic space or directional space); performing set operations, transformations, and / or morphological operations (e.g., dilation, erosion, etc.); calculating directional projections and / or visibility; determining spatial and / or directional masking; determining internal filling; and performing conflict and / or collision analysis; etc. In some embodiments, multiple operations may be used together to implement one or more composite operations, such as isolating disjoint portions, region selection, and / or determining nearest neighbors (either spatially or directionally).
[0135] With reference to the various embodiments described herein, certain embodiments of the present invention may utilize one or more of the modeling methods and processing tools described herein to perform reconstruction tasks and related operations to support a variety of uses.
[0136] As a first example, the embodiments described herein can be used to reconstruct a scene including a tree and / or to view a tree from a distance. In this example, one or more images containing a static tree are taken at a great distance. In this example, a single pixel in the image may include one or more leaves, a background, or some combination of the two. In some embodiments of the present invention, the information contained in the pixel may be represented by a Sahel, where the origin is at the viewpoint and surrounds a plane that intersects the edge of the pixel. The information attached to the Sahel may include a color and a calculated normal vector. This calculated normal vector may be calculated based on a polarization analysis of the data, based on a combination of images, or in other ways. The color may be a combined color of the leaves, branches, and background represented by the pixel. In this example, the normal vector is the result of many subsurface reflections. Therefore, for the example of a tree viewed at such a distance, the image information will not be able to reach a confidence level sufficient to indicate that the information comes from a single surface.
[0137] In some cases, there may be multiple correlated pixels in a region of an image. If the number of correlated pixels is sufficient, certain embodiments of the present invention may perform a statistical analysis of the texture. Such a statistical analysis may involve applying a set of one or more filters to the region, and will preferably include clustering of responses to the one or more filters combined into a texture signature. In this example, the computed texture signature may then be added as a property to a scene model, and later used to insert a synthetically generated texture into a rendering to provide a realistic view.
[0138] Continuing with the example of an image of a tree taken at a certain distance, higher level Sael data may be calculated from individual pixels. Due to the relatively low resolution and / or high degree of object represented in a single pixel, there is not enough information to begin to build a spatial representation of the tree. Therefore, in certain embodiments where the system may be implemented to reconstruct a 3D scene for 3D viewing, the image of the tree and anything within such an image will be assumed to be beyond the parallax distance (i.e., the reconstructible scene boundary) and used as a background in some embodiments.
[0139] If the proposed example system receives one or more additional images or other scene data of the same tree from different viewpoints, the system can extract certain landmark points or ray information from the original image and the newly received image or data, and in a preferred embodiment, try to match such landmark points or ray information. If the system successfully matches such landmark points or ray information, the system can estimate the 3D position of these points. At this point, some embodiments of the present invention can initiate a 3D model, and the spatial area around the matching point can be assigned the color value of the pixel from the two images or other characteristics of the point. The system can perform similar operations on the area around the matching point. If additional images from a closer range are available, the system can create a higher resolution spatial area, and the system can optionally re-evaluate the relevant higher and lower resolution areas. This process can be performed on an as-needed basis, so that high-resolution information is processed only when needed to achieve an immediate goal. The goal can be, for example, to reconstruct a specific object of interest, whereby the system can retain most or all of the imaging data (which will be retained for later use), but not processed to a high level of detail unless needed. In some embodiments, the higher resolution information can be processed immediately and / or stored for later processing. The system can then use all or a subset of the available data to construct a 3D model.In some embodiments, the highest resolution of the spatial model will roughly correspond to the projected size of a pixel.
[0140] The system described herein can also generate a lower resolution region of a spatial model using color information calculated according to a lower level. In some embodiments, the color information contained in the higher resolution region of the processing data structure is used to generate one or more colors represented in the parent node. Then, the system can calculate an estimate of the fraction of the area occupied by the calculated color in the lower resolution representation of the image based on the corresponding higher resolution information. Then, the system can use this calculation to calculate the transparency estimate of the spatial region of multiple resolution levels, wherein the transparency estimate can be the fraction of the spatial region that is counted as contributing color to the associated region in the image. In some embodiments, the remaining one or more colors are assumed to be substances from a larger distance (e.g., in the background of the image). In some embodiments, the system can continue to represent spatial regions of increasing size by including additional colors and transparency values, which can lead to increased complexity in some cases. The colors representing different items in the scene can be separated into other spatial regions, thereby limiting the needs in each node. Therefore, the composite color and transparency value of the region can be calculated as a single color and transparency value of the region based on the sub-value.
[0141] In certain embodiments of the invention, the system may allow for display from new viewpoints. In this case, the plenoptic representation may optionally be traversed from the viewpoint in a front-to-back order, with pixel information accumulated from a spatial region roughly corresponding to the size of the projected pixel, where the spatial region may increase with distance. In such embodiments, the system may accumulate a composite color value for the pixel based on the color weighted by the transparency value encountered, which may continue when a node of appropriate size is encountered. In some embodiments, the system may include a threshold for the pixel, whereby when the accumulated transparency weight exceeds the threshold for the pixel, the color is determined and the plenoptic traversal is terminated.
[0142] As a second example, the embodiments described herein may be used to reconstruct a scene that includes a vase containing one or more flowers and one flower removed from a base. In this example, a plenoptic model of a vase with flowers has been obtained from an image. The system may generate a data structure for characterizing segments using one or more 3D segmentation tools, such as described herein. The system may then provide one or more segment identification numbers to one or more regions of the model, wherein the segment identification number may be based on an estimated similarity of the elements between the one or more segments. For example, the system may perform segment identification based on the spatial smoothness of the outer surface of the vase, indicating that the elements are related and belong to the same category together. In some embodiments, prior information (perhaps AI-assisted) may be employed to guide segmentation. Certain embodiments may connect the segments that belong to an identifiable structure, such as a vase, flower, or other object in the scene. Such exemplary embodiments may be implemented to store information that may optionally associate the segments that form the flower. Exemplary embodiments may also include a high-level data structure that may connect various representations of the various parts of the flower, such as petals, stems, and leaves. Exemplary embodiments may also include an even higher-level data structure that may represent an entire vase with flowers.
[0143] To extract a particular flower, exemplary embodiments of the present invention may perform one or more operations, such as transformations. In doing so, the system may then transform one or more associated segments, thereby manipulating one or more associated segments and manipulating the segments away from the vase. The system may also perform collision analysis, which may guide the trajectory of the flower in the scene to avoid spatial intersections. In some embodiments, depending on the level of recognition and modeling achieved, occluded and / or unreconstructed portions of the flower model may be interpolated, or inset using an analytical model.
[0144] As a third example, the embodiments described herein can be used to reconstruct scenes including water, objects immersed or partially immersed in water, one or more water drops entering a body of water (such as a swimming pool), or objects immersed in water or another liquid. In one such example, multiple water drops and nearby bodies of water can be reconstructed. In certain embodiments, water drops moving into and entering a body of water can be modeled according to the laws of physics or other properties that the system can provide or know. In some embodiments, the water drops can be represented in three dimensions, which provides a basis for the system to calculate the mass properties of each water drop using the known mass properties of water. The system can then model the trajectory of each water drop falling into the water based on the mass and / or center of mass of the water drop in whole, in part, or otherwise. In some embodiments, the system can optionally include an advanced modeling system that can support deformation of one or more water drops or swimming pools.
[0145] In some embodiments, the motion of a water droplet may be modeled at discrete time instances. At the point in time when a water droplet first enters a larger segment representing a body of water, an operation may be performed to determine the amount of water that is common between the pool and the water droplet. The system may then use the results of this operation to compensate for the increase in volume of the larger segment, which may optionally be accomplished using a morphological dilation operation. Following this operation, one or more volumetric elements on the surface of the larger segment (the pool) that interfaces with the movable material (the water droplet) may be gradually extended to compensate for the amount of water displaced, and may be further modified to account for the dynamic response of the segment surface to interactions with the movable material. The system may use this and similar tools to implement more advanced displacement models. In some embodiments, the entire process may continue with additional water displacements until the water droplet is fully integrated into the body of water.
[0146] As a fourth example, the embodiments described herein may be used to reconstruct a scene including a satellite, a spacecraft, or other object orbiting the Earth or another celestial body. Using a spacecraft as an exemplary object for reconstruction, an imaging device (such as a camera) may capture one or more images of the spacecraft. The digital imaging device may be provided as part of the spacecraft itself (e.g., disposed on a boom, arm, or other structure to allow the spacecraft to inspect itself), or may be disposed on a separate spacecraft or object (which may, for example, be operable to inspect the spacecraft). In some embodiments, the systems described herein may provide for creating a plenoptic model of a spacecraft based on image data that has been acquired. In some embodiments, the systems described herein may provide for creating a plenoptic model of a spacecraft based on captured image data or a combination of previously acquired data and captured data. Such a model may be used to assess the condition of the spacecraft, whether there is damage to all or part of the spacecraft, materials present on the spacecraft, instruments present on the spacecraft, or other information that can assist in assessing the spacecraft, or any combination of the foregoing.
[0147] In some embodiments, the system may store one or more models in a database or other data structure. In addition to providing storage for models, the database may also provide access to one or more models. In some embodiments, access may be provided by searching for one or more features of the model, by indexing, by a user interface (which provides browsing of one or more categories of models), etc. Certain embodiments of the present invention may provide a data service that allows access to one or more models. Such a data service may be personal (i.e., on a user-by-user basis, allowing the user to access a subset of models associated with the user), available to a group of users (e.g., a group, company, enterprise, business, or other group with limited access to such a group), or even available to the public. Some embodiments may provide data services as an interface with other systems or applications. For example, the system may provide access to a model or information about a model to an application or system that can use the model for other purposes (e.g., a third-party metaverse application may use one or more furniture models provided by the system to recreate a house). In some embodiments, the system may store models created by the system itself, models created by a third-party model creation system or software, or models created by a combination of the two.
[0148] The system may use one or more 3D segmentation tools to generate a representative data structure of the segments, such as described herein. The system may then provide one or more segment identification numbers to one or more regions of the model, wherein the segment identification number may be based on an estimated similarity of the elements between the one or more segments. For example, the system may identify segments based on the spatial smoothness of the outer surface of the spacecraft or the materials present in the model (e.g., using segment identification numbers to classify materials in the solar array of a spacecraft), thereby indicating that the elements are related and belong to the same category together. In some embodiments, prior information (perhaps AI-assisted) may be used to guide the segmentation. Certain embodiments may use one or more high-level data structures to connect the segments belonging to identifiable structures (such as solar arrays, thermal management systems, propulsion systems, communication systems, or other aspects of a spacecraft). Such exemplary embodiments may be implemented using a separate high-level data structure that may optionally associate segments forming a specific system or subsystem of a spacecraft. Exemplary embodiments may also include even higher-level data structures that may connect various separate high-level data structures representing various parts of a spacecraft, for example, subsystems including a wider range of systems that may be observed on a spacecraft. The exemplary embodiment may also include even higher-level data structures capable of representing the entire spacecraft.
[0149] As a fifth example, the embodiments described herein can be used to reconstruct scenes including parts of the human body. As an example, the embodiments of the present invention disclosed herein can be used in dental applications to reconstruct teeth, gums or other soft tissues, dental implants or products, or other objects in a person's mouth, to reconstruct all or part of a human eye, or for other medical-related applications. For example, the system can be implemented to perform dental virtualization in a dentist's or other provider's office, a surgical center or hospital, or even a patient's home. In various embodiments, imaging can be performed using a handheld commercial device (e.g., a mobile phone, a tablet computer, or a camera) or a dedicated medical or dental device (e.g., a dental range of other ranges known in the art). In some embodiments, the system can process the captured images to virtualize a scene of interest in the patient's mouth, including optionally providing a representation of the size / dimensions, color, reflectivity, translucency, and / or other optical properties of an object (e.g., a tooth) in the scene. Such a virtualized scene can include a model (such as a full-light model) that can be used for a variety of applications (such as the design and size determination of braces or alignment devices, dental implants or other appliances, mouthguards, retainers, etc.). Although the system is provided in the context of dental-related applications, the system can be used in a similar manner for medical-related applications (e.g., diagnostics, surgery and operating rooms, therapy, etc.), analysis of body size and / or structure for athletic training, sizing of clothing, and many other applications. For example, the system can create a model that can be used for the following purposes: verifying the accuracy and integrity of surgical equipment, medications, or other objects before entering a surgical environment to perform a procedure; modeling a region of interest in or on a human body before, during, and / or after surgery; for training purposes; or many other applications.
[0150] In addition to the foregoing examples, the systems described herein can also be used in a variety of other contexts, including situations where the reconstruction of light and matter in a scene has advantages over existing systems. Such situations include, but are not limited to, advanced driver support systems, traffic solutions (e.g., speeding or license plate detection), human scanning (e.g., for the health, medical, dental, and / or fashion industries), object classification and / or recognition, drone-enabled inspections for area access, battery inspection, robotics (e.g., box picking or indoor mapping), logistics (e.g., packing trucks or measuring the size of objects to be packaged), automotive applications (e.g., body inspection), sorting applications (e.g., food sorting or recycling), or connection to infrared scanners (long wave or short wave).
[0151] One of the advantages of the embodiments of the invention disclosed herein is the ability to configure the embodiments of the invention to jointly or separately reconstruct the light field, the matter field (which may be a re-illuminable matter field), or both. Those skilled in the art will recognize that various applications of embodiments of the invention may require only one or both of the light field and matter field reconstruction, and furthermore, the re-illuminable matter field may have advantages in certain circumstances, while it may not be necessary in other circumstances. In addition, the embodiments of the invention described herein may be configured to perform certain of the aforementioned reconstruction techniques for the entirety of a scene, or alternatively to perform these techniques in various configurations for different regions or objects within the scene. In addition, the aforementioned reconstruction techniques may be paired with other techniques for characterizing a scene (e.g., photogrammetry, NeRF, and other techniques described herein) to reconstruct all or the same region of a scene, or by using different techniques to characterize different regions or objects of interest in the scene. For example, embodiments of the invention may be configured to determine which technique may provide the fastest, most computationally efficient, least power, etc. alternative to reconstruct a scene (or some or all of its individual portions), and combine various reconstruction techniques.
[0152] Reference is made to the related discussion in U.S. Patent Publication No. 2021 / 0133929A1 of Ackerson et al. and FIG. 73 for illustrating several advantages of light field and matter field reconstruction by way of example. FIG. 73 highlights the fact that the representation of matter and light fields and their interaction in producing images can be complex and difficult to analyze and understand, particularly when attempting to glean such understanding from the images themselves. In some embodiments, the invention described herein can customize the information displayed to the immediate needs of the view, for example by specifying the type and viewing characteristics of scene elements (e.g., zoom factor) and how the elements are rendered (e.g., wireframe vs. shaded).
[0153] refer to Figure 1C to Figure 1E , in some embodiments, the invention disclosed herein can be configured to allow customization of the nature of the re-illuminatable properties present in the model. For example, a user, a calling process, a higher-level reconstruction goal (manually or automatically determined), or other features can specify the desired re-illuminatable properties of any given scene. In some embodiments, a scene can be considered re-illuminatable if the elements within the scene have relevant properties (e.g., one or more BLIFs) that are able to predict the responsive radiation in one or more (or any) outgoing directions of interest given an incident light field. In some embodiments, a scene can be considered fully re-illuminatable if all elements within the scene have the aforementioned properties and the model has had external illumination removed (i.e., except for the case where the model is reconstructed using a specified incident light field, the responsive radiation is based solely on the emitted light within the scene). For example, Figure 1DA matter field 120 is depicted in which all external illumination (e.g., incoming light 112) has been removed. A fully re-illuminable matter field may be configured to respond to an emitted light field from an emitting object 114 and / or a windowed light field 112. A windowed light field 112 may represent light incident on the scene from its larger enclosing environment (boundary 117). Having a windowed light field 112 may be desirable to illuminate or re-illuminate a scene under the conditions of the original light field that existed during capture or measurement from associated image data and / or to characterize the scene under alternative lighting conditions (e.g., if the model is a room, a windowed light field may allow characterization of the room under both day and night conditions). Characterizing the original aperture light field 112 may be less important if it is not desirable to reconstruct the properties of the original light field, although some embodiments may reconstruct a close equivalent in the process of reducing the scene to a form represented primarily by the physics of light interactions in the full light element field. For example, if the reconstruction goal is to obtain the size and shape of a foot for a shoe order, a windowed light field may be less important. In this case, the reconstruction target is the intrinsic matter field of the foot element, and the light field is too important.
[0154] In some embodiments, the present invention can coordinate or balance the incident light field and the outgoing light field at the window boundary 111 between sub-scenes or regions. For example, at the window boundary element 111, the radiation calculated as incident should match the radiation calculated as outgoing along the given direction at the window boundary 111 of another sub-scene. This configuration can allow the system to be configured to implement different reconstruction techniques between sub-scenes or regions in the scene. For example, a single scene can contain separate sub-scenes reconstructed using each of the various techniques described herein (e.g., using physically based light transport, NeRF, etc.), and each sub-scene can coexist in the entire reconstructed scene. By querying the predicted outgoing light at each sub-scene window boundary element 111 in each sub-scene in a specified query direction, the sub-scenes can be coordinated or balanced with each other, and optionally querying the predicted incident light may also be important. Having this balance can allow for varying degrees of re-lighting capabilities between sub-scenes or regions by providing transitions at boundaries, which can be thought of as similar, though perhaps coarser, to the way incoming and outgoing light are balanced via intervening BLIFs in sub-scenes.
[0155] Figure 1C to Figure 1E Various embodiments of scene models are shown. The scene model can be an external lighting scene model, such as Figure 1C and 1E As shown, this means that incident light flow 112 exists at the aperture boundary 111, thereby providing light to the scene. The scene model 110 may optionally include one or more emission objects 114, such as Figure 1C, where the emitting object emits light flow to the scene independently of the incident light flow 112. Thus, the model can represent the medium in the scene based on both the incident light flow 112 and the emitted light flow from the emitting object 114 (if applicable). The scene model 110 can optionally include one or more response objects 113, such as Figure 1C As shown, the responsive object provides a responsive optical flow or an outgoing optical flow in response to an incident optical flow.
[0156] For example, if there is no incident light flow represented at the hole boundary 111 and there is no emitted light in the scene, the scene model can be a single scene model or the matter field 120 can be re-illuminated, such as Figure 1D Thus, the single scene model can represent the scene without external incident light. When the single model 120 is completely without light, such as Figure 1D As shown, the system must provide a window light source or an emitted light source to illuminate the scene (e.g., when it is desired to use it for purposes such as rendering for predicting the emitted light). In such an embodiment, Figure 1D As shown, the single model 120 will represent the light interaction properties of the scene without inherent lighting, so that the final rendering represents the interaction with the provided light source. In some embodiments, when the single model 120 includes an emissive object 114, there are potential light sources in the scene itself. In such embodiments, the single scene model can be reconstructed using only the emissive light source, or a combination of the emissive light source and another light source can be used to reconstruct the single scene model.
[0157] In some embodiments, a single scene model 120 may be considered fully relightable (e.g., if the model can eliminate the effects of incident light flow 112 present in the original image data from which the model is constructed), wherein such relightability may be facilitated by determining a relightable matter field that characterizes the stereo, radiometric, and light interaction properties of matter in the scene. Given a given light field (e.g., lighting conditions), the model may reconstruct the matter field as it would appear in such a light field. In some embodiments, a scene model 110 may be considered non-relightable (e.g., if incident light 112 cannot be separated from the model's matter field), partially relightable (e.g., if incident light 112 can be partially separated from the model's matter field), or fully relightable (e.g., if incident light 112 can be completely separated from the model's matter field). The degree of relightability may be affected by performing light field and / or matter field reconstruction according to various embodiments disclosed herein, including by performing such reconstruction to a particular level of detail or resolution based on need or preference.
[0158] The scene model may also be a light field model 130, such as Figure 1EAs shown. Light field model 130 can characterize the flow of light within the scene. For example, light field model 130 can characterize the flow of light entering scene 112 and / or the flow of light flowing out of scene 116. In addition, light field model 130 can characterize the interaction and flow of light within the scene, including, for example, responsive light flow from the interaction of light with the responsive medium and emitting light flow from emitting objects.
[0159] Although the embodiments of the invention described herein can be applied to a variety of situations to reconstruct both the light field and the matter field in a scene and output these reconstructions together, certain applications of the embodiments of the invention described herein may only require the reconstruction of the light field in the scene. Reconstruction of the scene light field can allow viewing of scenes without requiring specific properties of the matter in the scene. For example, if the process described herein is used to detect traffic signals by an autonomous vehicle, the main goal of the application would be to determine the state of the traffic light under specific lighting conditions. However, the specific properties of the matter field (e.g., the size and shape of the traffic light itself) may be irrelevant to the final application. Of course, there are many applications that can independently illustrate the light field reconstructed according to the embodiments of the invention described herein to achieve the desired results.
[0160] Similarly, applications of some embodiments of the present invention may require only reconstruction of the matter field in the scene, without reconstructing the corresponding light field. For example, consider a case where the results of reconstruction using the process herein can be used for reverse engineering or additive manufacturing of a part or assembly. The application may require only information related to the matter field, and specifically, only models of the size and shape of one or more objects in the scene or the scene itself. Of course, there are many applications that can independently illustrate the matter field reconstructed according to the embodiments of the present invention described herein to achieve desired results.
[0161] In addition, the application of some embodiments may be enhanced by reconstructing a re-illuminable matter field. The term "re-illuminable" may be understood to provide certain light interaction properties of the matter in the scene, non-limiting examples of which include properties related to transparency, refractive index, roughness, polarization diffusion properties, non-polarization diffusion properties, and / or extinction coefficient, etc. Characterization of these and / or other properties may allow modeling of how the matter will interact with light fields other than those present in the image data from which the model is reconstructed. In addition, the light interaction properties of the re-illuminable matter field may be used in conjunction with the embodiments of the invention described herein to characterize the composition or materiality of the matter in the scene. For example, for applications where the reconstruction techniques described herein are implemented to create a model for a metaverse, an accurate and usable model may preferably be able to respond to simulated lighting conditions in a metaverse scene, but the original light field may be irrelevant. Of course, there are many applications that can independently illustrate the re-illuminable matter field reconstructed according to the embodiments of the invention described herein to achieve desired results. The embodiments of the invention described herein may also be configured to provide multiple or different degrees of re-illuminating capabilities within a single scene. For example, in some embodiments, it may be desirable for certain portions of a scene to have a higher degree of re-lightability (e.g., the reflective cans shown in FIG. 73 of U.S. Patent Publication 2021 / 0133929A1 to Ackerson et al.), while other areas of the scene may require only a lower degree of re-lightability (e.g., walls, trees, or other features closer to a parallax or border of the scene shown in FIG. 73 of U.S. Patent Publication 2021 / 0133929A1 to Ackerson et al.).
[0162] Fig. 9 and Fig.10 The use of ML in generalized scene reconstruction (GSR) is exemplarily shown in . Fig. 9A process for generating a fully trained machine learning system, which may be referred to as a trained machine learning model or TMLM, is shown. The first step 901 is to provide a new scene image for training. The images of the object of interest for training may be taken in many scenes and / or under various conditions. The exemplary GSR system may analyze this information and / or other information in step 902 to generate a light field model for each scene and / or reconstruct one or more material fields of the scene in step 903. In step 904, the object of interest may be extracted from the scene using a material field selector function. In some embodiments, in step 905, this processing may generate one or more re-illuminable material field (RMF) models including geometry (e.g., shape) and BLIF information. In step 906, such models may be used as examples in the training of the machine learning system. Although not shown, in some embodiments, the BLIF parameters may be varied to create multiple training examples from a single RMF model. The result of step 907 is a TMLM.
[0163] Fig.10 An illustration of an exemplary production use of the TMLM is shown. In step 1001, an image of a new scene can be provided. In step 1002, the GSR system can process the image and create a new light field model, and in step 1003, a new RMF is reconstructed. In some embodiments, in steps 1004 and 1005, a matter field selector function can be used to extract the portion of the matter field to be processed (such as for recognition). Finally, in steps 1006 and 1007, the new RMF can then be tested using a previously trained TMLM to generate useful output.
[0164] In some embodiments, the system may train and use the TMLM using one or more of the following: one or more material fields, one or more source light fields, one or more window light sources, and one or more captured images, each captured image may optionally have associated posture information. Such training may be accomplished as described herein (including by using one or more neural networks) to compute the light field of a scene or sub-scene. In some embodiments, the TMLM may be trained using one or more models, wherein the models include neural reconstructions of the full light field, color, intensity, or other radiation information associated with the scene, or models containing other information. Some embodiments of the TMLM may apply physically based modeling methods to perform light field reconstruction alone or in combination with one or more neural networks. The foregoing embodiments may allow the TMLM to create, reconstruct, or otherwise generate one or more images based on the input to the TMLM (which, in some embodiments, includes postures associated with one or more images).
[0165] In some embodiments, a multi-dimensional scene may be stored using a voxel grid or a polygonal grid, but other embodiments may avoid using either of the foregoing grids. Specifically, voxels may be expensive to store in terms of data size or required processing, and polygonal grids can generally only represent hard surfaces. The system may use one or more fields that define a set of spatial and / or temporal coordinates. In embodiments where a neural network is used to model one or more fields, these fields may be referred to as neural fields, or in the case of modeling a 3D space, as neural graphics primitives.
[0166] In some embodiments, the light field physics module can be used to model the interaction between one or more elements and radiation elements entering and / or leaving one or more of these elements. Some embodiments of the present invention can use neural networks instead of or in combination with parametric functions to represent light interactions. In some embodiments, the system can use a sine representation network (SIREN) to incorporate high frequencies (texture layout).
[0167] Although the system described herein may use parameterized functions to represent light interactions in a scene, the system may be configured in alternative ways. Light interaction functions may be complex and the parameter space is typically high dimensional. The system may optionally select an ML model that attempts to learn and approximate the light interaction function. One aspect that can improve the successful training of such an ML model is to ensure compliance with the physical laws related to light interactions. Certain embodiments of the present invention may use physically informed neural networks to ensure such compliance. The method may utilize prior knowledge of the system to modify the loss function and use neural networks to model unknown physical components or properties. These neural networks may be configured to use nonlinear activation functions to increase the representational power of the model. By using prior information (such as the known physics of light interactions discussed elsewhere herein), the system can incorporate additional constraints based on the laws of physics.
[0168] There are multiple approaches for incorporating physics-based modeling into neural networks, including, for example, physically-informed neural networks (PINNs), neural network architectures with physical constraints, incorporating physical priors into loss functions, hybrid modeling, and residual modeling.
[0169] refer to Fig.16In an embodiment using a PINN architecture, the ML model may include two main parts: a fully connected neural network 1602 and a residual layer 1604, both of which may be designed to satisfy the underlying physical equations of the system being modeled. In some embodiments, the input to the PINN may be a spatial and / or temporal coordinate 1601, which may be normalized between 0 and 1. The fully connected neural network 1602 may process the input to output a predicted solution 1603 for the system being modeled. The PINN may consist of several hidden layers with nonlinear activation functions.
[0170] A residual layer 1604 may be applied to the predicted solution. The residual layer is optionally designed to ensure that the predicted solution satisfies the governing physics equations of the system. The residual layer may calculate partial derivatives of the predicted solution from the fully connected neural network with respect to input coordinates and time, and enforce the physics equations governing the predicted solution 1603. The output of the residual layer 1605 may then be combined with a loss function, which may include one or both of data constraints (such as known boundary conditions or initial conditions) and physical constraints (such as conservation laws or other governing equations). The loss function may be used to train a neural network to minimize the difference between the predicted solution and the observed data while still satisfying the underlying physics.
[0171] Some embodiments of the present invention may use PINN to enforce physical constraints such as object stiffness, object continuity, and / or object motion invariance during reconstruction. Incorporating these constraints into a neural network can generate a more accurate and / or more robust output model. In some embodiments, the present invention may use PINN to improve the efficiency of the scene reconstruction process. For example, PINN can replace or combine with computationally expensive algorithms (e.g., algorithms for solving partial differential equations (PDEs)). PINN can approximate the solution of one or more PDEs with a neural network, which can significantly reduce the computation time required for scene reconstruction.
[0172] refer to Fig. 20 , some embodiments of the present invention may combine a physics-based approach (e.g., computing a BLIF and / or its parameters) with a neural network capable of performing residual modeling. In an exemplary embodiment, the input to the model is incident data (e.g., light) 2001. The system may use a physics-based model 2002, such as computing a BLIF, to predict the outgoing light intensity 2003. The physics-based calculations may be configured to calculate to a certain accuracy, which may reduce processing and / or power requirements. The results of the incident light field 2001 and / or the physics-based calculations 2002 may be provided to a neural network 2004 to further refine the output in order to improve the predictive performance and / or resolution of the predicted light interactions 2005. In some embodiments, the use of energy conservation constraints (e.g., incorporated into the loss function) may improve neural network training.
[0173] refer to Fig.17 , some neural network architectures have been developed with built-in physical constraints. For example, the Neural Hamiltonian Network (NHN) architecture is designed to obey Hamiltonian dynamics and conserve energy in the system. The inputs to the NHN 1701 may be the state variables (q, p) of the system, which may represent, for example, the position and momentum of a particle. The neural network may be configured to predict the time derivatives of the state variables based on the input state variables 1702. The neural network may be a fully connected neural network with several hidden layers.
[0174] The output of the neural network can be passed through a Hamiltonian layer 1703, which can be configured to compute the Hamiltonian dynamics of the system based on the predicted derivatives. The Hamiltonian layer can compute the dot product of the predicted derivatives with the Jacobian matrix representing the underlying physics of the system. This configuration will ensure that the predictions of the NHN are consistent with the underlying physics of the system and that energy is conserved over time.
[0175] The output of Hamiltonian layer 1704 may be the predicted state variables at the next time step. These predicted state variables may be used to generate physically plausible trajectories for the system or other downstream tasks such as control or optimization.
[0176] refer to Fig.18 , the system can also be configured to incorporate physical priors into the loss function. The loss function of the neural network 1804 can be modified to include physical priors or constraints. For example, in a processing application where the system is configured to receive an input 1801, process the input using a neural network 1802, and generate a predicted output 1803, the system can incorporate constraints on the physics of the imaging system (such as a point spread function) into the loss function.
[0177] refer to Fig.19 , another approach for addressing the imperfections of physics-based models is residual modeling. Where the system is configured to use residual modeling, the ML model may learn to predict errors 1905 or residuals produced by a physics-based model 1902. Some embodiments may provide input data 1901 to a physical model 1902 and a data model 1903. The system may be configured to learn the deviations of the physical model 1902 and its output 1904 relative to observations, and to correct the predictions of the physical model using the predicted deviations 1905. Because this approach models errors rather than physical quantities in a physics-based problem, residual modeling may not enforce physics-based constraints. Therefore, it may be advantageous to combine residual modeling with another form of modeling to ensure consistency with the laws of physics.
[0178] In some embodiments, the system can use a combination of physics-based modeling and data-dependent modeling, or hybrid modeling. Hybrid models can take advantage of the advantages of both approaches to provide accurate predictions.
[0179] In some embodiments, the above neural network can be designed by adding one or more known differential equations directly to the loss function when training the neural network. In some embodiments, the training can be done by sampling a set of input training positions and passing the samples to the network. The network can calculate one or more gradients of the network's output relative to its input at one or more sampled positions. In some embodiments, the gradients can be calculated using automatic differentiation or similar features, which exist in many ML libraries such as PyTorch and TensorFlow. The system can then use one or more gradients to calculate the residual of the underlying differential equation, which can be added to the loss function as an additional term.
[0180] Some embodiments may use a similar approach to predict one or more surface normals in a scene, including based at least in part on the incident light field. Using a trained ML model may be able to overcome sources of error present in the model. For example, even for volumes with air or empty space, it is difficult to model errors in the model, and this problem may be even more complex for volumes with complex or inhomogeneous media. In this case, the system may be configured to approximate the loss function using a neural network and then add additional constraints in the loss function using known photophysical properties.
[0181] In some embodiments, the invention described herein may utilize structure from motion (SfM) techniques. SfM is a technique for 3D scene reconstruction that estimates the 3D structure of a scene from a set of 2D images. An SfM system can match points in an image and then use that information to estimate data such as camera pose and / or 3D scene structure. SfM can be used to reconstruct scenes from a variety of sources, including photographs, videos, and LiDAR data.
[0182] Some embodiments of the present invention may utilize multi-view stereo (MVS) technology. MVS is a technology for 3D scene reconstruction that can estimate the depth of a scene based on multiple 2D images. An MVS system can find correspondences between points in different images and use this information to estimate the 3D structure of the scene. MVS can be used to reconstruct scenes from photos and videos.
[0183] Those skilled in the art will appreciate that SfM and MVS are sometimes referred to as photogrammetry.Some embodiments of the present invention may be configured to reconstruct scenes from aerial photographs, satellite images, and ground photographs using photogrammetry.
[0184] Certain embodiments of the present invention may utilize data from a LiDAR system. LiDAR is a technique for 3D scene reconstruction in which laser pulses are emitted and can be used to estimate the position of objects in the scene by calculating the time it takes for light to reflect back to the source from the objects. By measuring the flight time and angle of the laser pulses, LiDAR can generate a point cloud of the scene, which can be used to reconstruct the 3D structure of the scene.
[0185] In some embodiments, the present invention may use other data to initialize data structures before making initial assumptions, where such image data may be 2D information and / or 3D information. In some embodiments, the present invention may use 3D data sets, such as data sets from 3D sensing components (such as Kinect RGB-D (RGB and depth) camera systems, structured light, multi-view stereo, stereo cameras, LiDAR, radar and / or infrared sensors, photometry software, laser scanners, and other devices that provide 3D image information, simultaneous localization and mapping (SLAM), and other technologies, software and techniques). Such embodiments may receive 3D input information, where, for example, in addition to color information, depth information may also determine the 3D position of each pixel relative to the camera. Such information may be represented as a 3D point cloud, voxel array, and / or other data storage structure.
[0186] In such an embodiment, the system may use previously captured and / or processed data to provide an initial hypothesis about the relative position of the medium in the scene and associate a corresponding confidence value with such data. Providing such data may provide advantageous processing results by reducing the initial processing associated with populating the scene, perhaps providing faster performance. For example, the system may operate in conjunction with photometry or other techniques to provide a sparse grid model of an object, subscene, or scene at a relatively low processing cost, and then perform the full light processing techniques described herein. The combination of embodiments of the present invention with such a system may allow for additional visualization of the relative position of light and / or medium in the scene. In some embodiments, the ML model may be trained on light field information (such as incident and / or outgoing radiation element trees) as a means for accelerating the identification of media and / or surfaces within a scene. As a non-limiting example, certain outgoing light fields of a Lambertian surface may have a disk-shaped artifact. The artifact may represent a cosine decay around the surface normal of a particular surface, which may accelerate the identification of applicable surface normals.
[0187] In certain embodiments, the present invention may be combined with, used in parallel with, supplemented by, or otherwise implemented in whole or in part using artificial intelligence (AI), machine learning (ML), and neural networks, including neural radiance networks such as neural radiance fields (or NeRF), stereoscopic scene methods such as PlenOctrees or Plenoxels, deep signed distance functions (SDF), and neural volumes, or other techniques. This approach may be used for scene reconstruction, new view synthesis (NVS), and other uses to model radiance, density, or other information as a continuous function in a 3D space such as a stereo space using a multi-layer perceptron (MLP) or voxels such as a voxel array. For example, given a position in space with 3D coordinates (x, y, and z) and a viewing direction, the representation will return the color (red, green, and blue) and density at that position. A deep SDF system may be configured to learn a signed distance function in a 3D space whose zero level set represents a 2D surface. A neural volume system may be configured as a neural graphics primitive, which may be parameterized by a fully connected neural network. A NeRF system may be configured to model the color and density of a scene. Other embodiments operate using alternative input and return information. In some embodiments, the density returned may be a differential opacity value that includes, in part or in whole, an estimate of the radiance and other information such as the color that can be accumulated by a ray in a specified direction through a specified point.
[0188] In some embodiments, such representations may be initialized using random values. At first, any specified point and direction may return meaningless values. The exemplary system may then be trained using calibration images from various known viewpoints (e.g., hundreds of images from random positions on a hemisphere above a real or simulated scene) or other image-related information. In some embodiments, the process may be initiated by selecting one or a group of pixels in a training image. For each pixel, the network may emit a ray from the viewpoint into the scene. The network may then perform a query or other process on a certain number of points along the ray (e.g., 50, 100, 200 points, or any other number of points selected for the query). These points may be selected in various ways. In some embodiments, the network or a portion thereof may perform an "integrated rendering" operation to calculate the returned color or other values along the projected ray, and integrate these values in some way to calculate the estimated color or other information for the pixel. In some embodiments, for example, when the network is initialized using random values, such color values will be independent of the true color. In some embodiments, the network may calculate a representation of the difference between the estimated color and the true color (e.g., the sum of the squares of the color component differences). This difference can be used to modify the MLP weights or stereo information using back-propagation.In some embodiments, the foregoing process can be iterated to allow for increasingly accurate estimation of color or other values.
[0189] In some embodiments, the foregoing process can generate new viewpoints with a high degree of realism after a certain degree of "learning". For example, this can be achieved through AI (such as converging estimated color values within the scene). In certain processes known in the art, using a neural radiation network or a stereo representation to generate new viewpoint images may require a lot of processing and / or time. Some queries may require 500,000 to 1,000,000 multiplications and / or other operations for each point on the ray. Some previous systems may take 30 seconds or more to generate a single 800 pixel × 800 pixel image on a powerful graphics processing unit ("GPU") (such as Nvidia V100).
[0190] Several methods have been developed to reduce the time, computation, cost, and power required to generate images using these methods. In some embodiments, a partially trained system can be used to generate an octree called a PlenOctree, which is a data structure different from a "full light octree". In such an embodiment, the system can generate pixel values by sampling the PlenOctree at points along the ray rather than by using an MLP. Such an embodiment can improve performance by two, five, or more orders of magnitude. The present invention can be implemented to further reduce the computation and hardware required to generate pixel values from an octree data structure while improving performance.
[0191] In some NeRF architectures, a single neural network can be used to predict the color and density of 3D points in the scene. In some architectures, separate networks can be used to predict different properties. For example, separate networks can be used to learn material properties such as reflectivity, roughness, or transparency. These networks can be configured to predict material properties for different parts of the scene, and optionally can be combined with the color and density predictions to render an image of the scene. In some embodiments, the system can be configured using a two-stage approach, where separate networks are used to predict the shape and material properties of the scene, which can be combined to generate the final rendering.
[0192] However, NeRF-based architectures may have certain limitations. For example, such networks may have: limited scalability, requiring large amounts of training data to capture variations in the appearance and lighting of real-world scenes; limited generalization, where the approach may rely heavily on the quality and diversity of the training data and may not generalize well to scenes that differ significantly from the training data; limited accuracy, where errors and / or artifacts may be present, particularly in occluded or poorly illuminated areas of the scene; and limited control over the output due to the implicit nature of NeRF, which may make it difficult to control specific properties of the output, such as the precise position or orientation of objects in the scene. Using NeRF-based approaches in conjunction with the systems described herein may address one or more of these limitations.
[0193] In some embodiments, the present invention can generate a surface normal vector by fitting a plane to adjacent surface elements. Such a surface normal vector can be used for processing operations, optionally with spatial and color information.
[0194] Certain embodiments of the present invention may use ML to reconstruct the light field in a scene, including in some cases constructing a physical model of the interaction of the light field and the material field in the scene. In such embodiments, the system may decouple components that can contribute to the light sensed by a camera pixel or another imaging device. The data may be used to determine the properties of the material and objects in the scene, including non-Lambertian surfaces (e.g., human skin, cloth, mirrors, glass, and water). In some embodiments, certain surface information may be represented as a bidirectional light interaction function (BLIF) of one or more sensing locations on an object (and optionally, all sensing locations on an object). The sensing locations may include locations captured by individual pixels of a camera or imaging device. The present invention may use BLIF and BLIF-based modeling to extend concepts such as bidirectional reflectance distribution functions (BDRFs) and / or cosine lobe reflectance models to develop higher levels of complexity by including light / matter interactions involving color, material, roughness, polarization, and the like.
[0195] In some embodiments, the processing can be used in conjunction with other data about the relative positions of substances in the scene. For example, a variety of techniques are known in the art for providing coarse, fine, or high-precision three-dimensional information, including photometry (e.g., through software packages such as COLMAP or Metashape), structured light, multi-view stereo, LiDAR, radar, infrared, laser scanners, simultaneous localization and mapping (SLAM), and other techniques and techniques. The ML model can be configured to use a combination of image data and other data (such as information that is not easily visible from the image data alone or better information than that available from the image data alone) to make assumptions about the properties of the medium in the scene. This assumption can then be used to accelerate the processing of the full light field of the scene. For example, this assumption can allow the system to predict light interaction characteristics associated with the medium in the scene (wherein this assumption is provided as an assumption to the processor before performing the reconstruction of the light in the scene), provide an updated set of assumptions based on which the light interaction is based, or reorder the processing workflow to match the predicted medium in the scene.
[0196] In some embodiments, the system can be configured to classify 3D scenes / objects using raw point cloud data (such as can be provided by LiDAR). For example, the system can use methods such as PointNet (global features) or PointNet++ (local features) that use raw Cartesian point data to perform classification and segmentation tasks. The system can be configured to use an MLP layer for each point and use symmetric functions to handle permutation variance. The system can also be configured to use re-illuminable material field (RMF) data, including using re-illuminable material field (RMF) data in conjunction with global and / or local feature extractors.
[0197] Some embodiments of the present invention may implement NeRF Self-Supervised Object Segmentation (NeRF SOS) or similar processing schemes to use the latent representation to perform downstream object recognition, object segmentation, and / or other tasks. In some embodiments, RMF data may be used as a latent representation of a scene and used for downstream tasks such as object recognition and / or segmentation.
[0198] Certain embodiments of the present invention may utilize AI and / or ML to perform activities such as object classification. Prior art systems typically perform object classification based on images as training input, but the performance of such systems often depends on the quality and scope of the training data. For example, a traffic signal detection model trained under normal conditions may not be robust enough to resolve difficult lighting conditions (e.g., the sun is behind or in front of the traffic light) and / or adverse weather conditions (e.g., snow, rain, or fog).
[0199] Certain embodiments of the invention may include the ability to extract BLIF parameters (either direct measurements or mathematical models) of a sensed surface. Such extracted BLIF parameters may be used to construct a new light field model for a new situation, which in some embodiments may allow the system to model the interaction of light with a reconstructed matter field representing objects or materials in a new scene with different illumination. Thus, in certain embodiments, the system may include the ability to generate realistic renderings (which is referred to as "relighting") of a matter field under a variety of lighting conditions (and preferably, under nearly any lighting condition). In such embodiments, the reconstructed matter field becomes a re-illuminable matter field (RMF), which may have substantially improved representation robustness and / or support an improved level of realism in an application.
[0200] Some embodiments of the present invention may use sensed material "signatures" in the form of BLIF at viewing locations on the surfaces of objects in a scene in place of and / or in conjunction with sensed color information in a 3D ML system (convolutional neural network (CNN), NeRF, etc.). In such embodiments, a training model for the ML system may contain information about the basic surface materials and properties of viewing objects with and / or without the interaction of scene lighting. In applying such training models, certain embodiments of the present invention may allow for a reduction in the number of lighting and scene conditions that must be obtained from the real world for effective training, and / or allow for the synthetic generation of new training models by varying one or more BLIF parameters in a single object model to account for a variety of properties (e.g., color and / or surface conditions) that an object can encounter when in production use. In some embodiments of the present invention, such signatures may simplify the training and use of 2D ML systems. By obtaining a 3D model of a real-world object that includes BLIF information, a 2D training data set may be synthetically generated by rendering the model from various viewpoints and / or varying the lighting and BLIF parameters appropriately. In such embodiments, the system may be used to provide a large number of training or synthetic data sets to a TMLM.
[0201] In some embodiments, the system can use the re-illuminable material field data for object type classification. In an exemplary configuration, the system can use a CNN-based architecture and / or a transformer-based architecture. The nodes of the CNN can be viewed as a sequence and fed into a self-attention-based model, such as Fig.15 As shown. Fig.15 In the illustrated embodiment, a model of a re-illuminable matter field 1501 may be used as an input to a model 1502. The model 1502 may include a deep learning based model f θ, which can be trained for parameter θ. Model 1502 can be sequential, convolutional based, or a multi-layer perceptron. Model 1502 can be configured to create output 1503 as a classification, shape completion (e.g., via ShapeNet), or another output parameter. For embodiments where there are different sizes, a transformer based model can be an effective approach (e.g., by dividing the submission into multiple voxels and treating the voxels as a sequence). For embodiments where the system is trained to perform shape completion (which can involve predicting the volume of one or more objects in a scene), architectures based on variational autoencoders (VAEs) and / or generative adversarial networks (GANs) can also be utilized.
[0202] In some embodiments, the system can perform image painting and / or image painting. Image painting can predict the state of damaged, occluded or missing parts of the image. Image painting is the generation of new image content beyond the boundaries of the original image. The system can use any number of known methods associated with these techniques.
[0203] In some embodiments, the system can use a generative adversarial network (GAN), including for image painting tasks. GANs can learn the underlying distribution of image data and generate new images. GANs can use generative and discriminative networks to achieve good visual effects. Generators can be trained to fill in missing pixels, and discriminators can be trained to distinguish generated images from real images.
[0204] Some embodiments of the present invention may use an autoencoder. The autoencoder may include an encoder network and / or a decoder network. The encoder network may be configured to compress an input image into a low-dimensional representation. The decoder network may be configured to reconstruct an image from the compressed representation. In image rendering, the encoder network may be trained to encode a damaged image, and the decoder network may be trained to fill in missing pixels in the encoded representation.
[0205] Certain embodiments of the present invention may use a deep image prior. The deep image prior may use random values to initialize the weights of a deep neural network, and optimize the weights to minimize the reconstruction loss between the generated image and the input image data. By optimizing the weights, the network may learn to generate reasonable image completions that are consistent with the input image data.
[0206] In some embodiments, the system can perform various tasks to reconstruct missing or damaged parts of a 3D point cloud. For example, in a manner similar to that used with images, a GAN can be trained to learn the distribution of point cloud data. One such method is PCGAN, which uses a conditional GAN to generate missing points in an input point cloud. In other embodiments, an encoder-decoder architecture such as U-Net or PointCNN can be used. The U-Net or PointCNN architecture may include an encoder that maps the input point cloud to a low-dimensional feature space and a decoder that maps the features back to the reconstructed point cloud. Some methods may also incorporate attention mechanisms into the encoder-decoder architecture to better capture local and global structures. Some embodiments may use conditional reversible neural networks or PI-NeRF (e.g., by filling in missing points in the input view using a point cloud drawing network, and then reconstructing the scene from the completed view using NeRF).
[0207] In cases where the missing portion of the scene is not obvious in the input image data, it may be more feasible to directly paint in the generated 3D scene. Some embodiments may be configured to learn the distribution of voxels in the scene and then predict occupancy values for the missing / corrupted portions. For example, a 3D-GAN may paint in missing voxels by training a conditional GAN to generate a complete voxel-based 3D scene given an incomplete or corrupted input.
[0208] In some embodiments, the system can predict the material properties and optical properties of the missing or damaged portion.The system can be configured to learn two distribution functions: a distribution function for the matter field and another distribution function for the light field.
[0209] Some embodiments of the present invention may use alternating optimization, i.e., an iterative process for jointly optimizing certain functions for all parameters by alternating restricted optimization of various subsets of parameters. For example, as described elsewhere herein, the system may be configured to calculate parameters related to BLIF and / or surface normals. These calculations may be non-convex in nature, meaning that convergence is not guaranteed and there may be a risk of effectively minimizing certain local minima. In cases where joint optimization is not effective or it is advantageous not to do so, the system may use alternative optimization to perform light field and / or material field reconstruction. Alternative optimization may improve optimization time in some cases compared to joint optimization, and may also be better at bypassing local optima in some cases.
[0210] One problem with some ML models is that such ML models may process or make decisions based on learning statistics or patterns present in the training data. Since the training and reasoning of ML models are highly data-dependent, inconsistent or inaccurate data may adversely affect the output generation. In some embodiments, the system can be configured to impose certain restrictions, formulas, or other constraints on one or more ML models. For example, the system can impose some formulas for the laws of light field physics that always follow the laws of nature. Alternatively, the system can configure the ML model so that the system can perform parametric modeling using conventional light physics in combination with processing via a neural network.
[0211] In some embodiments, the system can incorporate physical constraints into the ML model to improve the explainability of the model. For example, if the model is designed to predict the trajectory of a ball in conjunction with physical laws of motion, it can assist in constraining the model's predictions in a way that helps make the system more explainable. A similar approach can be used with the models described herein by including constraints related to or characterizing the laws of light physics.
[0212] Certain embodiments of the present invention may use physics-based models to illustrate ML models. For example, physics-based models may be used to explain the output of a neural network and / or explain why the neural network makes certain predictions. The physics-based models may help identify key features or inputs that drive the model's predictions. In the context of scene or light field reconstruction, physics-based models may assist in identifying and providing characteristics of specific radiometric elements, nodes, or other data that contribute to a particular reconstruction output or result.
[0213] Some embodiments of the invention may use physics-based simulations to validate ML models. Such embodiments may compare the predictions of the ML model with the results of the physics-based simulations. These comparisons may assist in identifying areas where the model may be inaccurate or biased and / or improve the accuracy and / or reliability of the model. For example, the system may be configured to use the output of a reconstruction method that is not assisted by ML or is only partially assisted by ML as a comparison to the output of a machine learning-based method.
[0214] In some embodiments, the system can combine the physics-based model with the ML model. The combination can produce one or more hybrid models that are easier to interpret. For example, the physics-based model can be used to generate initial conditions or constraints for the ML model, or the ML model can be used to refine the predictions of the physics-based model.
[0215] One advantage of the present invention is that it can be configured in a way that can be used to reverse engineer an existing model. Although conventional techniques such as NeRF can be used to reconstruct a scene, the reconstruction is not necessarily a "deconstructable" and / or "reconstructable" technique, as such techniques cannot be used to reverse engineer an already existing 3D model.
[0216] In some embodiments, the present invention may incorporate one or more non-photophysics-based techniques (e.g., LiDAR, MVS, SfM, photometry, etc.) in conjunction with physics-based methods to achieve better deconstructibility and / or reconstructibility. In an example of an ML-based approach, the goal may be to learn the material field and / or light field in a scene. The system may be configured to generate high-resolution images with fine-grained control over various aspects of the image, such as the subject's pose, expression, and appearance. For example, some embodiments of the present invention may be configured to use a Style GAN, which is configured to use a "style" vector. Such a vector may control various properties of the generated image, may be learned during the training process, and may be manipulated to generate new images with different styles to enrich the generation capabilities of relighting the scene. In a similar manner, the present system may be configured to generate latent variables of light and material properties, which may help improve the deconstructibility and / or reconstructibility of the scene.
[0217] The system described herein can be configured to reconstruct a scene in a deconstructible and / or reconstructible manner. For example, the system can be configured to decompose a scene (or sub-scene, object, etc.) into parts, which can be done manually or automatically using techniques such as segmentation or clustering. In some embodiments, one or more parts can be reconstructed individually using techniques such as stereoscopic vision, motion recovery structure, or other techniques. Parts can be merged together to form a complete reconstruction. Some embodiments of the system can inject physical properties to enhance the robustness of the process. For example, the system can identify the core individual parts of an object and attempt to characterize the basic properties of the material comprising these parts. After building a model of each part, each part may still retain the deconstructible material properties of the object.
[0218] Some embodiments of the present invention may use a hierarchical neural network (HNN) to achieve different levels of accuracy. The HNN may have a stack of multiple layers of neural networks. Each layer may be configured to learn increasingly complex features and patterns from the input data, allowing the HNN to achieve higher accuracy. The initial layer may be used to learn low-level features. Subsequent layers may use different stacks depending on the desired accuracy.
[0219] Another approach to HNNs is to use a tree structured network, where each node in the tree corresponds to a different level of abstraction. This configuration can allow the network to selectively activate different parts of the network based on the complexity of the input data. This structure can improve efficiency and / or reduce the amount of data required for training. Each node in the network can correspond to a different level of a hierarchical data structure, where lower level nodes represent smaller voxels and higher level nodes represent larger voxels. The network can learn to selectively activate different parts of the data structure based on the complexity of the input data.
[0220] The system can also be configured to calculate a distance function to evaluate surface models, including, for example, polarized surface models. Such distance functions can have a variety of uses, including for testing models (i.e., reconstructions of scenes or sub-scenes) with other model configurations (e.g., other models known in the industry, such as those provided by Unreal Engine). 5) Compatibility. The system can be configured to use statistical comparison methods to determine the consistency between the model and another model configuration.
[0221] In some embodiments, the distance function can also be used to develop one or more segments or groups of media and / or light interactions modeled as described elsewhere herein. The distance function can also be used to compare multiple (e.g., two) different parameter spaces or the same parameter space, and can be done in a Euclidean manner (e.g., in the same parameter space), a non-Euclidean manner (e.g., in different parameter spaces), or some combination of the two. For example, the distance function can be used for a planar form space (e.g., using Unreal Engine 5) to map between that space and a different parameter space. For example, if a model is created using a plenoptic parameter space that represents both the light field and the matter field in a scene, a distance function can facilitate conversion to a planar form space by providing the distances between relative points within each model.
[0222] In some embodiments, the invention described herein may be operable to achieve certain goals, which may be set in the system, specified by the user, determined by the system during processing, made in response to conditions or some other factors, including any combination of the foregoing. An illustrative embodiment is when the invention is operated to characterize the state of a traffic light, as described elsewhere herein. In the context of an autonomous vehicle or an advanced driver assistance system, such characterization may require both identification of a traffic signal and knowledge of the state of the signal, all of which need to be done in advance enough so that the system makes a decision about slowing down, stopping, or not slowing down, not stopping, which is a challenge. Those skilled in the art may recognize certain challenges associated with characterizing the state of a traffic light. For example, there are few standards for traffic lights, which results in significant differences in stacking and orientation (e.g., horizontal, vertical, side by side, and other variations), color (e.g., variations in the colors used), types of bulbs (e.g., incandescent and LED), and other features of the light (e.g., red lights are not always located at the top of the signal). Color detection may be more challenging because red and yellow are close in the color spectrum, while green is in a narrow band. Environmental factors can also cause problems (e.g., if the sun is behind the traffic light, the traffic light is in shadow and causes over-capture; if the sun is directly behind the observer, the light in the traffic light is washed out). The challenge can become more complex when considering the structure, light color, additional components, and the order of lights that exist in other countries around the world (e.g., the United States typically uses the order of green, yellow, red, and then back to green, while the United Kingdom typically uses the order of green, yellow, red, yellow, and then back to green).
[0223] Those skilled in the art may recognize certain known approaches, such as narrowing the problem space by first analyzing one of the most common configurations in the United States (e.g., a three-tier stacked traffic light with red, yellow, and green lights from top to bottom). The analysis can then continue to expand the dataset used to build the ML model to include as many angle variations, states of repair, light color ranges, bulb types (e.g., an incandescent light may be white in the model and then an appropriate color outside the model), and environmental variations as described above. The dataset can be augmented with synthetically generated imagery to simulate different sun positions and address imbalances found when captured in nature (e.g., there are fewer yellow lights because yellow is typically the shortest duration signal). Performing the foregoing operations can result in the generation of large amounts of data and / or high costs associated with creating a balanced set of appropriate variations.
[0224] In some embodiments, the invention described herein may provide a model of a traffic light, for example as a voxel field having a boundary around a traffic light object. Such a model may include a re-illuminable matter field such that the model has removed at least some portion of the incident light field associated with the light. In some embodiments, the only light associated with the model may be light emitted by the light itself. Such a model may optionally be used in conjunction with a supervised training method that employs a full light model labeled by the state of emitted light. The system may be configured to be operable to capture and reconstruct an unilluminated scene, including a possible traffic light in a certain state, which may be used to make inferences about the model.
[0225] In some embodiments, the invention described herein may use image-based machine learning to build a machine learning model. Image-based machine learning may include collecting examples of a subject and capturing as many images of variations as possible (e.g., for a traffic light, the variations may include each of the lighting state, type, and environmental conditions (such as lighting, weather, quality, location, etc.)). Embodiments of the present invention may be configured to build a model of a subject (e.g., a traffic light) in a re-illuminable manner using a full light scene reconstruction of the subject itself, eliminating any external influences so that the only light in the scene may be light emitted from the subject itself (e.g., the illumination light in the traffic light). For example, the model may be a single sub-scene in which a voxel boundary surrounds the traffic light and no light enters the sub-scene. In this case, the only light in the sub-scene may be the light emitted by the traffic light and the downstream response light generated due to the emitted light.
[0226] The system can collect various sets of captured plenphotonic scene subjects (e.g., traffic lights) to allow the ML model to learn the shape and type of the subject (e.g., it is a traffic light, its light properties, its shape (three-layer stack, five-layer stack, etc.), the type and color of the traffic light lens, the emission state of each bulb, and other characteristics). In some embodiments, it can be advantageous to have a balanced number of subject objects in each possible state. Some embodiments of the present invention may be operable to generate additional plenphotonic scene representations, for example by including changes in surface quality, positional aspects, and emission properties.
[0227] Some embodiments of the present invention may use supervised training of machine learning models by providing one or more labels associated with each type of subject and its possible states. In the case of a traffic light, the labels may characterize objects that emit red light, objects that emit yellow light, objects that emit green light, and so on, in various arrangements. Labels may be done manually, or may be generated at capture or generation time. The machine learning model may be constructed after all scene sub-reconstruction objects are labeled. Alternatively, some embodiments may use a self-supervised training approach, in which, instead of generating labels, the machine learning training method discovers the correct category or state of the subject. Self-supervised training may be more likely to be achieved using a full light scene recapture approach. For example, in the case of a traffic light, the color lens and which traffic light is emitted from are easier to calculate than traditional methods based on the image source.
[0228] Some embodiments may obtain a new subject, process the subject as a plenoptic scene object (e.g., a re-illuminable matter field), and apply the plenoptic scene object to a machine learning model. Such application may have a variety of purposes, such as inferring or determining the class or state that the subject is in. In the case of a traffic light, given a machine learning model built from re-illuminable matter field training data, a new re-illuminable matter field representing a traffic light may be classified according to which light is illuminated.
[0229] There are several potential advantages to this approach, including, for example, addressing adverse ambient lighting issues (e.g., by computing a re-illuminable matter field that removes the effects of external / incident light), better characterizing the quality of the emitted light source (e.g., by providing a model of the emitted light quality of the signal to more closely match real-world observations), detecting or characterizing traffic light lens materials (e.g., determining the color associated with the lens), detecting bulb emissions (e.g., emissions behind an optical filter, which can help determine the color), resolving the source of the emission (e.g., determining the type of bulb among incandescent, LED, halogen, etc.), removing weather effects (e.g., water or ice on a signal, precipitation causing imaging difficulties), and avoiding separate model classification categories (e.g., by resolving whether the potential signal being detected is an actual signal or an artifact such as a reflection).
[0230] More specifically, a potential advantage of the methods described herein is to account for different light sources present in a subject (e.g., the type of light bulb in a traffic light). In the case of a traffic light, the light source may be an incandescent light bulb behind a colored and / or textured lens, an LED light source optionally behind a lens, a halogen light source behind a lens, etc. In some cases, the light may have one or both of an initial filter (which diffuses the light source) and a smooth or textured lens (for protection and / or better light emission). In the aforementioned types of light bulbs, the frequency and quality of the emitted colors can have a wide range due to the degradation of the quality of the light source, lens, or other variations. For example, in the case of an incandescent light bulb behind a colored lens, the light emitted in the center tends to be whiter, while the frequency of the light emitted outside of the center varies. This variation can result in difficulty in distinguishing between a red traffic light and a yellow traffic light. This variation may require the development of larger machine learning models and / or additional light type categories to handle accurate inference. Embodiments of the invention described herein can address these shortcomings by directly handling the lens, any additional filtering, the varying light source, and the varying color range and emission quality using all-photon scene reconstruction. Embodiments of the invention described herein can use plenoptic matter and light field reconstruction to understand and / or process the lens and any filtering. For example, embodiments described herein can understand how and why the color of the emitted light can vary from the center, for example, by encoding the physics of the emitted light through light field processing.
[0231] More specifically, one potential advantage of the methods described herein is distinguishing traffic lights from reflections of traffic lights (such as reflections from windows of buildings or other vehicles) or from shiny surfaces (such as the sides) of metal truck trailers. In known approaches to this problem, reflections may require new categories to be added to the machine learning model for all the different states of traffic lights reflecting in various materials, with the goal of filtering reflections from actual signals. This approach can be problematic and / or cost-prohibitive. The methods described herein can substantially solve these problems by understanding glass or other reflective materials as reflective surfaces. All-optical reconstruction can allow the system to understand the materials in the scene, such as glass on a building or shiny surfaces of a vehicle, etc., eliminating the need to add categories to the machine learning model to handle reflections.
[0232] Additionally, embodiments described herein that provide a relightable model may be operable to create and / or provide a synthetic image dataset for training, including as a light-independent model. Such a synthetic dataset may be useful for distinguishing an incandescent light bulb behind a red or yellow glass or plastic cover (which may have similar light properties in some cases).
[0233] Some embodiments of the invention may optionally include a human-machine interface for performing generalized scene reconstruction (GSR) and / or other functions of the system.
[0234] In some embodiments of the present invention, reconstruction of a scene (including using GSR) may be performed via a graphical user interface, command line interface operation, audio input (including voice recognition), or other input on a computing device, which may include a portable computing device, a personal computer, a server, a mobile phone, an augmented reality or virtual reality device, an unmanned aerial vehicle or other vehicle, or other digital device. In some embodiments, the interface may provide the ability to import or build an approximation of the light field and the material field (hereinafter referred to as a pre-scene) for later reconstruction. In such embodiments, the starting point may increase the speed of the reconstruction process and / or avoid errors. Some embodiments of the present invention may provide primitive shapes, common objects, and digitally generated light or otherwise make them accessible to build a pre-scene. When using a mobile device, some embodiments of the present invention may provide the user with the option of placing light and matter at a location corresponding to the physical location of the device while physically walking through the scene to be reconstructed as part of the pre-scene. In some embodiments, the pre-scene may also consist of previous reconstructions, which may optionally be updated with new scene data. For example, the pre-scene may be updated completely, updated only in the light field, updated only in the material field, updated only in a specified area, or any combination of the foregoing.
[0235] In certain embodiments of the present invention, the human-machine interface may provide control over the sensing device used to obtain the image. For example, the user may be allowed to change device settings and / or view potential image inputs in a preview video feed before capture begins. In some embodiments, the human-machine interface may display an overlay of a selected pre-scene and allow the user to start capturing only after the pre-scene is roughly aligned with the preview video feed. In such embodiments, the system may be configured to capture the reconstructed preview spatially. During capture, the human-machine interface may also display a continuous video feed from each input device. If input from a sensing device is paused, the human-machine interface may require and / or assist in alignment between the preview video feed and the existing portion of the reconstructed preview before input resumes.
[0236] In some embodiments of the invention, the human-machine interface may provide a live preview of the reconstruction during capture, updated with each incoming image or video frame. If a pre-scene is being used, the reconstruction preview may include the pre-scene. The video feed may be displayed in one or more areas of the screen, while the live preview may be displayed in one or more other areas of the screen, allowing comparison of the video feed and the reconstruction preview. The live preview may display the entire light field and matter field, such as Fig. 12AAs shown, or only a portion of the light field and / or matter field (such as a BLIF in a small area) is displayed, as shown Fig. 12B Analysis information may be superimposed on the camera feed display and / or on the live preview display, including false color associated with setup parameters such as resolution and / or information about the BLIF associated with one or more regions on the screen. In certain embodiments of the invention, the reconstructed live preview may be optionally navigated by rotation, translation, and / or zooming.
[0237] Some embodiments of the present invention may arrange the video feed 1101 and the reconstructed preview 1102 in one or more configurations. In some embodiments, the video feed 1101 and the preview 1102 may include a clipping mask of dots, a checkerboard, or other shapes, optionally with adjustable size and optionally arranged in a regular grid, with the reconstructed preview displayed above a background layer displaying the video feed, wherein the two are spatially aligned and rendered to the same viewing angle, as in the exemplary embodiment. Fig.11 and as shown in Figure 12. In some embodiments, the video feed and preview may include a resizable irregularly shaped clipping mask, which is optionally randomly arranged, displaying all or a portion of the reconstructed preview and / or above a background layer displaying the video feed, wherein the mask and layer are spatially aligned and rendered to the same viewing angle. In some embodiments, the video feed and preview may include an adaptive clipping mask that displays specified features of the reconstructed preview above a background layer displaying the video feed, wherein the mask and background layer are aligned as much as possible. In some embodiments, the video feed and preview may include a rectangular window that displays the reconstructed preview above a background window displaying the video feed, wherein the preview and background windows are aligned to the greatest extent possible.
[0238] In some embodiments, Fig. 12A and Fig. 12BAs shown, the reconstruction preview and video feed can be designed to allow the user to navigate the reconstruction process without the need for previous training with 3D software. Certain embodiments of the present invention can make the difference between the reconstruction preview (optionally including the pre-scene) and the current camera feed very obvious, such as by using a checkerboard, dot pattern, or other staggered pattern between the preview and the real-time capture. In some embodiments, the implementation can make reconstruction errors and lags more obvious. In such embodiments, the user may be able to notice areas in the scene that have changed since a previous reconstruction, where the previous reconstruction may include or contain the pre-scene. In some embodiments, the present invention can provide information about errors and lags, which can provide the user with feedback related to the rate at which all or a portion of the scene is captured, view areas of the scene from multiple angles, or perform another action to obtain additional information about all or a portion of the scene. In some embodiments, the system can facilitate the user to understand whether the reconstruction is accurate enough before completing the capture process. For example, as Fig. 12A and Fig. 12B As shown, the preview can be designed such that as the fidelity of the reconstruction increases during capture, the difference between the real-time capture and the preview decreases, including to the point where there is little or no noticeable difference between the real-time capture and the preview.
[0239] In certain embodiments of the invention, a user may receive feedback to guide 1301 the process for refining the bidirectional light interaction function or BLIF reconstruction by capturing selected interferometers from as many angles as possible. Fig.13 An exemplary embodiment of such a BLIF capture guide 1301 is described in . In some embodiments of the present invention, the BLIF capture guide 1301 will display a spherical augmented reality overlay on the video feed and / or reconstruction preview. As the user moves around the corresponding position of the intermediary in real space and views portions of the spherical overlay from various angles, portions of the spherical overlay may optionally change (e.g., by disappearing, changing color, or making other visible changes). Thus, the overlay can assist the user in determining the number of angles that have been viewed.
[0240] In certain embodiments of the present invention, the human-machine interface may provide a means for creating a program to be read during capture, which may optionally guide the behavior of the user and / or automated device. For example, the program may include a goal for the reconstruction process, such as the desired resolution of the light field and / or the material field, the desired certainty threshold of the reconstruction, and / or the goal of eliminating gaps in the captured information. The program may also include functions for responding to triggers encountered during input capture. Triggers may include specified material field and / or light field structures, the passage of time, and / or changes in the uncertainty level in the model generated by the incoming information. When these triggers are encountered, the software may change its display configuration, add an overlay that can be positioned using augmented reality to its display, play audio prompts or verbal instructions, change the reconstruction target, and / or change the settings of any connected devices. In some embodiments of the present invention, the user can link the trigger to its corresponding function using a node-based program editor in a graphical user interface. In some embodiments of the present invention, the user can also create a path to be used to guide the future capture process by drawing a line in a pre-scene or by moving the capture device along a desired path in physical space. In some embodiments, the trigger can be designed to enhance the usability or accessibility of the system to the user.
[0241] In certain embodiments of the present invention, the human-machine interface is capable of rendering and displaying the completed reconstruction. In some embodiments, in addition to the real view, the display may also include analytical visualization. In some embodiments, one or more intermediaries, radiation elements, voxels, and saers may be rendered as smaller primitive shapes centered at one or more locations. False colors may be applied according to any properties that may vary with intermediaries or radiation elements (which may include z-depth, ambient occlusion, and / or segmentation). The BLIF may be rendered according to the generated outgoing light, or may be replaced with a default BLIF to provide a unified view of specific geometry in the scene. In some embodiments, the path of the radiation element may be traced back for a specified number of reflections, and may optionally be visualized as an overlay. In some embodiments, the user may move the viewpoint of the rendering camera in the scene in a manner corresponding to their device, such as scrolling and clicking on a desktop or walking with a VR headset.
[0242] In certain embodiments of the present invention, all or part of the data captured or reconstructed by the system may be completely hidden from the user and / or automatically deleted. These processes may include data that may reveal the user's location or other personal information, sensitive information, images captured to perform the reconstruction, or the reconstruction data itself. These data may include all or part of the light field or the matter field or both. For example, in some embodiments, the remaining matter field, complete with all light interaction properties, can be rendered and displayed in the general light field and / or a new light field selected by the user. Alternatively, the matter field discovered during reconstruction can be completely hidden from the user and / or automatically deleted. The remaining light field can be used to illuminate the general matter field and / or a new matter field selected by the user. Alternatively, the light field and matter field discovered during reconstruction can be completely hidden from the user and / or automatically deleted. The remaining light interaction properties can be applied to the general matter field and / or a new matter field selected by the user, which can be rendered and displayed in the general light field and / or a new light field selected by the user. In some embodiments, the system can use AI / ML to identify and delete information of concern or information otherwise identified for deletion or hiding.
[0243] In certain embodiments of the invention, a human-machine interface will allow a user to edit light field and matter field reconstructions. In some embodiments, a user can transform, deform, or relight all or any portion of a reconstruction. In some embodiments, a user can change the light interaction properties of a BLIF, and assign one or more BLIFs to different regions of a matter field. In certain embodiments of the invention, a user can manipulate the scene by dragging on anchor points, by typing keyboard shortcuts, or by sculpting and painting on the reconstruction using a brush tool. In some embodiments of the invention, a user can insert new matter fields and light fields into the reconstruction, and / or relight matter fields (in whole or in part) to match a specified light field. In some embodiments, a user can delete light fields and matter fields in whole or in part.
[0244] In some embodiments of the invention, the human-machine interface may allow the user to select intermediaries and radiators. For example, segments may be selected, intermediaries and radiators may be selected individually or together, or machine learning may be used to create selections based on semantic descriptors. In some embodiments, the user may group intermediaries and radiators, and / or may select groups. In some embodiments, groups and segments may be hidden and / or displayed.
[0245] In certain embodiments of the invention, a human-machine interface may allow a user to perform metrological analysis after the reconstruction. For example, a user may make one or more measurements of a matter field structure, and optionally use all or a portion of such measurements to calculate geometric properties, such as volume. In some embodiments, the measurements and calculations may be saved and exported. In some embodiments, the invention may allow a user to spatially search the scene to obtain one or more of a count, selection, or group of light fields and / or matter field structures that sufficiently match the query. In certain embodiments of the invention, the query may be provided as a selected area of light and / or matter, presented as descriptive words that generate responses based on machine learning, or some combination of the two. Characteristics evaluated to determine a match may include matter field shape, light field structure, radiation intensity, size, and BLIF.
[0246] In some embodiments, the system can be designed to use information (including information related to the light field and / or matter field in the scene) to support procedural generation activities. For example, the system can be used to create a repetition or extension of the reconstructed light field and / or matter field to change the size or shape of the reconstruction. In addition, the system can adjust the lighting conditions associated with the generated data based on the original lighting conditions or the calculated light field in the scene.
[0247] Further aspects and embodiments of the present invention will become apparent from the following list of embodiments:
[0248] 1. A method for reconstructing a scene, comprising: accessing image data representing light in the scene; processing the image data to provide a light field model and a matter field model, wherein the matter field model includes a representation of a medium in the matter field, which includes functions representing the interaction of the medium with light at locations in the matter field; storing the light field model and the matter field model in a data structure, wherein data representing the light field model is separately accessible and data representing the matter field model is separately accessible and configured to be reconstructed using functions having characteristics of light represented in the light field model and having different characteristics from the light represented by the light field model; and outputting at least a portion of the matter field model.
[0249] 2. The method of embodiment 1, further comprising outputting at least a portion of the light field model.
[0250] 3. The method of embodiment 1, wherein the function is a bidirectional light interaction function.
[0251] 4. The method of embodiment 1, wherein the function characterizes absorptivity, transmittance, reflectivity, scattering, refractive index, roughness, polarization diffusion coefficient, non-polarization diffusion coefficient, and extinction coefficient associated with the medium.
[0252] 5. A method according to embodiment 1, wherein the material field model includes a full light field, which is represented by at least three dimensions characterizing the position, size and shape of the medium in the scene and at least two dimensions characterizing the interaction between the medium and light.
[0253] 6. A method according to embodiment 1, wherein the processing includes using a machine learning model to perform at least part of the processing.
[0254] 7. A method according to Example 6, wherein the machine learning model includes one or more of a physical information neural network (PINN), a neural network architecture with physical constraints, incorporating physical priors into the loss function, hybrid modeling, and residual modeling.
[0255] 8. The method of embodiment 6, wherein the machine learning model provides additional information characterizing the size, shape, position, or interaction of the medium with light.
[0256] 9. The method of embodiment 6, wherein the processing includes using a neural network and a non-neural network to find parameters of the light field model, the matter field model, or both.
[0257] 10. The method of embodiment 1, further comprising accessing data providing additional information related to the light field and / or the medium field in the scene, and wherein the processing further comprises using the additional information to inform the processing.
[0258] 11. The method of embodiment 10, wherein the additional information includes one or more information characterizing the size, shape, relative position, or light interaction characteristics of the medium in the scene.
[0259] 12. The method of embodiment 1, wherein the model is one or more of decomposable, recombinable, and interpretable.
[0260] 13. The method of embodiment 1, wherein the matter field model is configured to allow extraction of one or more segments of a medium in the scene.
[0261] 14. The method according to embodiment 13 further includes placing the extracted segment at a different position in the scene or a different scene.
[0262] 15. The method of embodiment 1, wherein the processing allows for local areas of higher resolution or application of additional computational energy to reconstruct certain scene features with higher accuracy than other areas.
[0263] 16. A method of using a neural network to represent surface light fields in various regions of a scene, wherein the scene model includes at least one full light field.
[0264] 17. The method of embodiment 16, wherein the method is implemented to support autostereoscopic screens or mesoscopic spaces.
[0265] 18. A method of using a neural network to represent entities in a scene, comprising one or more bidirectional light interaction functions, a two-dimensional light field, a four-dimensional light field, or both a two-dimensional light field and a four-dimensional light field, and a re-illuminable matter field.
[0266] 19. A method for constructing a re-illuminable material field (RMF) from a light field model.
[0267] 20. A method for simultaneously optimizing neural network parameters and conventional equations, wherein the conventional equations can be understood in the field of physics or other fields.
[0268] 21. An optimization method comprising simultaneously finding parameters of a neural network and a non-neural network.
[0269] 22. A method for reconstructing a scene, comprising: accessing image data of one or more images including the scene; using a data structure in a memory representing a plurality of stereoscopic elements, the plurality of stereoscopic elements representing the scene; using the image data to reconstruct radiation characteristics of each of the plurality of stereoscopic elements, wherein the radiation characteristics characterize a light field associated with each stereoscopic element and an interaction of the one or more light fields with a medium in the stereoscopic element, and wherein a set of reconstructed radiation characteristics is sufficient to reconstruct a viewpoint of the scene from viewpoints other than one or more viewpoints; using the reconstructed radiation characteristics to populate a data structure; and outputting at least a portion of the data structure representing a viewpoint of the scene from a viewpoint other than the one or more viewpoints.
[0270] 23. A method according to embodiment 22, wherein the radiation characteristics include one or more of outgoing light from the three-dimensional element, intensity of color at the three-dimensional element, transparency associated with the three-dimensional element, transmittance associated with the three-dimensional element, or opacity associated with the three-dimensional element.
[0271] 24. A method according to embodiment 22, wherein the data structure includes a hierarchical, multi-resolution, spatially ordered data structure, and filling the data structure includes filling one or more of an empty data structure, a partially filled data structure, or a data structure filled with previously calculated radiation characteristics.
[0272] 25. The method of embodiment 22, wherein reconstructing the radiation characteristics comprises using the image data by selecting a viewpoint and calculating the radiation characteristics associated with each stereoscopic element along one or more corridors extending from the viewpoint.
[0273] 26. The method of embodiment 25, wherein one or more corridors are represented by rays extending outward from the viewpoint, and one or more volumetric elements passed by the rays are determined.
[0274] 27. A method according to embodiment 22, wherein the light field represents one or more of light flowing into and out of a three-dimensional element, and the interaction with the medium is represented by calculating one or more of transmittance, opacity, transparency associated with the three-dimensional element and surfaces present in one or more three-dimensional elements.
[0275] 28. The method of embodiment 27 further comprises using the interaction of the medium to relight all or a portion of the scene.
[0276] 29. The method of embodiment 22, wherein reconstructing the radiation characteristics uses one or more of spherical harmonics, interpolation, machine learning, or machine intelligence.
[0277] 30. The method of embodiment 22, wherein reconstructing the radiation characteristic comprises optimizing the reconstructed radiation characteristic by iteratively performing reconstruction until the reconstructed radiation characteristic exceeds a threshold level of accuracy, certainty, confidence, or another factor.
[0278] 31. A system for reconstructing a scene, comprising: a storage medium configured to store image data from one or more viewpoints of the scene and a model of the radiation characteristics of the scene; a processor; and an output circuit configured to output the model, the processor being configured to: access at least a portion of the image data; create a model including radiation characteristics associated with multiple stereo elements; use the image data to reconstruct the radiation characteristics of each stereo element of the scene, wherein the set of reconstructed radiation characteristics is sufficient to allow viewing of the scene from viewpoints other than one or more viewpoints associated with the image data; populate the model using the reconstructed radiation characteristics; and at least temporarily store the model in the storage medium.
[0279] 32. A system according to embodiment 31, wherein the radiation characteristics include one or more of outgoing light from the three-dimensional element, intensity of color at the three-dimensional element, transparency associated with the three-dimensional element, transmittance associated with the three-dimensional element, or opacity associated with the three-dimensional element.
[0280] 33. The system of embodiment 31, wherein the storage medium comprises a hierarchical, multi-resolution, spatially ordered data structure, and the processor is configured to at least temporarily store the model in the data structure.
[0281] 34. The system of embodiment 31, wherein the processor is configured to create the model from one or more of an empty model, a partially populated model, or a model populated with previously calculated radiation data.
[0282] 35. A system according to embodiment 31, wherein the processor is configured to reconstruct the radiation characteristics using the image data by selecting a viewpoint and calculating the radiation characteristics associated with each stereoscopic element along one or more corridors extending from the viewpoint.
[0283] 36. The system of embodiment 35, wherein the one or more corridors are formed by extending rays outward from the viewpoint and determining one or more volumetric elements through which the rays pass.
[0284] 37. The system of embodiment 31, wherein the reconstructed radiance characteristics comprise a reconstruction of a light field in the scene, wherein the light field represents light flowing into and out of one or more volumetric elements.
[0285] 38. A system according to embodiment 37, wherein the model also includes a reconstruction of a matter field in the scene, wherein the matter field represents one or more surfaces present in one or more solid elements.
[0286] 39. A system according to embodiment 38, wherein the matter field is represented by one or more of a refractive index, polarization properties, the presence of one or more holes, transmittance, opacity, or transparency associated with the three-dimensional element.
[0287] 40. The system of embodiment 38, wherein the matter field is a re-illuminable matter field.
[0288] 41. The system of embodiment 31, wherein the processor is further configured to reconstruct the radiation characteristics using one or more of spherical harmonics, machine learning, or machine intelligence.
[0289] 42. The system of embodiment 31, wherein the processor iteratively performs the reconstruction until the processor determines that the model has exceeded a threshold level of accuracy.
[0290] 43. A method for training a machine learning model, comprising: providing image data to the machine learning model, wherein the image data includes information characterizing light in a scene; processing the image data to generate a re-lightable model of the scene, wherein such processing includes dividing the scene into multiple stereo elements, analyzing the image data to generate a model of the light field in at least a portion of the stereo elements, predicting the interaction between the light in the scene and a medium in one or more stereo elements, and providing information for characterizing the appearance of the medium under lighting conditions other than the light in the image data; and outputting the model.
[0291] 44. The method of embodiment 43, wherein the scene includes one or more objects of interest.
[0292] 45. The method according to embodiment 44 also includes extracting one or more objects of interest from the model and inserting the one or more extracted objects of interest into the second scene.
[0293] 46. A method according to embodiment 43, wherein the processing also includes determining shape information associated with a medium in the scene, and wherein the prediction includes calculating a bidirectional light interaction function associated with the medium.
[0294] 47. The method of embodiment 46, wherein the processing further comprises generating a model of the medium in the scene as a matter field that characterizes the medium in the scene in at least three dimensions.
[0295] 48. The method of embodiment 43, wherein the image data includes one or more relightable models.
[0296] 49. The method according to embodiment 48 also includes: changing the bidirectional optical interaction function associated with the output model to create a modified model; inputting the modified model into the machine learning model; and repeating the process to further train the machine learning model.
[0297] 50. The method according to embodiment 43 also includes: receiving a second image data set; using the trained machine learning model to generate a second model by processing the second image data set; and outputting the second model.
[0298] 51. A system for reconstructing one or more objects in a scene, comprising: a processor for processing digital scene data; an interface for receiving input related to a scene to be captured, wherein the input includes digital scene data in the form of image data, which represents the scene from a certain orientation, wherein the processor processes the digital scene data and the input to generate a three-dimensional model of at least a portion of the scene, the scene including matter including at least one surface, wherein the processor processes the image data by accessing one or more volume elements in a matter field represented by the image data, and wherein the processor processes the image data by determining whether the matter represented in each of the one or more volume elements includes a surface.
[0299] 52. The system of embodiment 51, wherein the image data is captured by a camera.
[0300] 53. A system according to embodiment 51, wherein the orientation is the posture of the camera.
[0301] 54. The system of embodiment 51, wherein the image data includes data related to electromagnetic radiation.
[0302] 55. A system according to embodiment 54, wherein the data related to electromagnetic radiation includes one or more of radiation values of visible light, infrared light and / or polarized light or unpolarized light and / or radar.
[0303] 56. A system according to embodiment 51, wherein the digital scene data includes image data representing the scene from at least two orientations.
[0304] 57. A system according to claim 56, wherein the processor processes image data from at least two orientations sequentially.
[0305] 58. A system according to embodiment 51, wherein the substance represented in the solid element is represented by a face element.
[0306] 59. A system according to embodiment 58, wherein the data related to the element includes one or more of the outgoing light field and the incident light field.
[0307] 60. The system of embodiment 51, wherein the processor processes the image data by assuming an orientation of the digital scene data.
[0308] 61. A system according to embodiment 51, wherein the processor processes image data in the following manner: assuming the existence of a surface in a solid element; assuming one or more of a surface normal, light interaction properties, an outgoing radiation vector, and an incident light field of the surface; calculating a cost for the surface to exist in the solid element based on the assumed surface normal, light interaction properties, an outgoing radiation vector, and an incident light field of the surface; comparing the cost to a cost threshold; and when the cost is lower than the cost threshold, accepting that the face element exists in the solid element.
[0309] 62. A system according to embodiment 61, wherein, when the system has accepted that a face element exists in a solid element, the surface remains in the scene during subsequent processing of the scene.
[0310] 63. The system of embodiment 61 further comprises updating assumptions about the light field of one or more other volumetric elements based on the accepted existence of the facet.
[0311] 64. The system of embodiment 61, wherein the processor iteratively performs the process for more than one volume element.
[0312] 65. A system according to embodiment 61, wherein the processor iteratively performs the process for more than one image data set.
[0313] 66. The system of embodiment 63, wherein the light field is not transmitted to the user.
[0314] 67. The system of embodiment 63, wherein the matter field is not transmitted to the user.
[0315] 68. The system of embodiment 63, wherein the light field and the matter field are not transmitted to the user, and the matter field-light interaction properties are not transmitted to the user.
[0316] 69. A method for training a machine learning model, comprising: providing image data to the machine learning model, wherein the image data includes one or more objects of interest; processing the image data to generate a model, wherein such processing includes analyzing the image data to generate one or more of a light field model of the scene or a reconstruction of one or more matter fields in the scene; selecting objects of interest in the model of the scene; extracting objects of interest in the model of the scene; and outputting a re-illuminable matter field model of the objects of interest in the scene.
[0317] 70. The method of embodiment 69, wherein the image data includes re-illuminable matter field data.
[0318] 71. The method of embodiment 69, wherein the image data includes one or more of objects of interest in multiple scenes and objects of interest under multiple conditions.
[0319] 72. The method of embodiment 69, wherein the re-illuminable matter field is comprised of a plurality of two-dimensional or higher-dimensional images.
[0320] 73. The method of embodiment 69, wherein the re-illuminable matter field model comprises one or more of shape information and bidirectional light interaction function (BLIF) information.
[0321] 74. The method of embodiment 69, wherein light field information is used to calculate light reflection characteristics of locations in the material field.
[0322] 75. The method of embodiment 69, wherein the light field is not transmitted to the user.
[0323] 76. The method of embodiment 69, wherein the matter field is not transmitted to the user.
[0324] 77. The method of embodiment 69, wherein the light field and the matter field are not transmitted to the user, and the matter field-light interaction properties are transmitted to the user.
[0325] 78. The method according to embodiment 69 also includes: changing the BLIF information of the model; inputting the model with the changed BLIF information into the machine learning model; and performing one or more of the aforementioned steps on the model with the changed BLIF information to further train the machine learning model.
[0326] 79. A method for using a machine learning model, comprising: identifying one or more objects of interest in a model of a scene; accessing a re-illuminable matter field of the scene; selecting a portion of the matter field to be processed; processing the selected portion of the matter field to extract at least a portion of the re-illuminable matter field; and outputting the extracted portion of the re-illuminable matter field.
[0327] 80. The method of embodiment 79 further includes testing the effectiveness of re-illuminating a portion of the matter field output by the machine learning model.
[0328] 81. A system for reconstructing one or more objects in a scene, comprising: a processor for processing digital scene data; an interface for receiving input related to a scene to be captured, wherein the processor processes the digital scene data and the input to generate a three-dimensional model of at least a portion of the scene, wherein the processor input guides at least a portion of the processing of the digital scene data, and wherein the processor provides output including the three-dimensional model of at least a portion of the scene.
[0329] 82. A system according to embodiment 81, wherein the input includes at least one of the following items: an approximation of at least a portion of the light field in the scene, an approximation of at least a portion of the matter field in the scene, one or more shapes present in the scene, one or more objects in the scene, or information related to one or more light sources in the scene.
[0330] 83. A system according to embodiment 81, wherein the input controls one or more sensing devices that provide digital scene data.
[0331] 84. A system according to embodiment 81, wherein the system provides feedback related to one or more objects to be reconstructed within the scene.
[0332] 85. A system according to embodiment 84, wherein the feedback includes a preview of one or more objects to be reconstructed within the scene.
[0333] 86. A system according to embodiment 85, wherein the system can update the preview when the results of such reconstruction are used to reconstruct one or more objects.
[0334] 87. The system of embodiment 86, wherein the preview further comprises one or more indications related to one or more parameters of the reconstruction.
[0335] 88. The system of embodiment 85, wherein the preview includes one or more masks representing data associated with the generated model and information received from a digital scene data capture device.
[0336] 89. A system according to embodiment 84, wherein the feedback includes one or more of information related to the rate of capture of digital scene data, the location used to capture the digital scene data, the sensor angle used to capture the digital scene data, aspects of the light field in the scene, or aspects of the matter field in the scene.
[0337] 90. The system of embodiment 81, wherein the input is data that allows the digital scene data to be aligned with newly received digital scene data.
[0338] 91. The system of embodiment 81, wherein the system further comprises a set of instructions for implementing one or more objectives for generating a three-dimensional model.
[0339] 92. A system according to embodiment 91, wherein the one or more objectives include one or more of the following: a desired resolution of the light field, a desired resolution of the matter field, a desired certainty threshold for reconstruction, a threshold for eliminating gaps in the captured digital scene information, and triggers for events encountered during the capture of the digital scene information.
[0340] 93. The system of embodiment 92, wherein the trigger comprises one or more of a specified matter field structure, a specified light field structure, the passage of time, and a change in the level of uncertainty in the model.
[0341] 94. The system of embodiment 92, wherein the system is configured to take action in response to the trigger.
[0342] 95. The system of embodiment 94, wherein the response comprises one or more of: changing a display configuration, adding an overlay to a display, providing an audio prompt, providing a visual prompt, changing a reconstruction target, and changing a setting of a device connected to the system.
[0343] 96. The system of embodiment 81, wherein the system is configured to change one or more features of the model.
[0344] 97. A system according to embodiment 96, wherein the change includes one or more of the following: editing a light field reconstruction; editing a matter field reconstruction; transforming the model; deforming the model; relighting all or any portion of the model; changing one or more light interaction properties of the BLIF; assigning one or more BLIFs to different areas of the matter field; manipulating the model by dragging anchor points, by typing keyboard shortcuts, or by sculpting and painting on the model with a brush tool; inserting a new matter field; inserting a new light field; relighting one or more matter fields (in whole or in part); deleting a light field in whole or in part; and deleting a matter field in whole or in part.
[0345] 98. The system of embodiment 81, wherein the system is configured to spatially search the model using a search query comprising one or more parameters.
[0346] 99. A system according to embodiment 98, wherein the spatial search includes: obtaining one or more of a count, a selection, or a group of light field structures, or obtaining one or more of a count, a selection, or a group of matter field structures that match one or more parameters of the search query.
[0347] 100. The system of embodiment 98, wherein the search query is provided as a selected light region.
[0348] 101. The system of embodiment 98, wherein the search query is provided as a selected region of matter.
[0349] 102. The system of embodiment 98, wherein the search query is presented as descriptive terms that generate responses based on machine learning.
[0350] 103. The system of embodiment 98, wherein the one or more parameters include one or more of matter field shape, light field structure, radiation intensity, size, and BLIF.
[0351] 104. The system of embodiment 81 further comprises a display for capturing digital scene information, wherein during the capture of information from multiple sources, spatially interleaved layers are displayed in three or more adjacent areas of the display.
[0352] 105. The system of embodiment 104, wherein at least one of the regions is a real-time reconstruction preview.
[0353] 106. A system according to embodiment 104, wherein all layers on the display are substantially aligned with the same viewpoint.
[0354] 107. The system of embodiment 104, wherein all layers on the display contain information about the scene.
[0355] 108. A system according to embodiment 104, wherein one of the layers on the display is a pre-scene rendering that is aligned to substantially the same viewpoint as the other layers.
[0356] 109. The system of embodiment 81 further comprising a display for indicating during capture how many angles have been captured around a certain BLIF.
[0357] 110. The system of embodiment 109, wherein the indication is provided by displaying a spherical or hemispherical overlay centered about the selected medium including the BLIF.
[0358] 111. A system as described in embodiment 110, wherein at least a portion of the spherical covering changes in response to viewing the medium from various angles relative to a corresponding position of the medium in real space.
[0359] 112. The system of embodiment 111, wherein the change in at least a portion of the spherical covering comprises one or more of disappearing, changing color, or undergoing other visible changes.
[0360] 113. A method for reconstructing one or more objects in a scene, comprising: accessing digital scene data and input related to the scene; processing the digital scene data to generate a three-dimensional model of at least a portion of the scene, wherein the processing includes responding to the input to guide a manner of processing at least a portion of the digital scene data; and outputting the three-dimensional model of at least a portion of the scene.
[0361] 114. A method according to embodiment 113, wherein the input includes at least one of the following items: an approximation of at least a portion of the light field in the scene, an approximation of at least a portion of the matter field in the scene, one or more shapes present in the scene, one or more objects in the scene, or information related to one or more light sources in the scene.
[0362] 115. The method of embodiment 113 further comprises using the input to control one or more sensing devices that provide digital scene data.
[0363] 116. The method of embodiment 113 further includes providing feedback related to one or more objects to be reconstructed within the scene.
[0364] 117. A method according to embodiment 116, wherein providing the feedback includes providing a preview of one or more objects to be reconstructed within the scene.
[0365] 118. The method according to embodiment 117 also includes: updating the preview when the results of such reconstruction are used to reconstruct one or more objects.
[0366] 119. The method of embodiment 118 further includes providing one or more indications related to one or more parameters of the reconstruction for the preview.
[0367] 120. The method of embodiment 117 further comprises providing one or more masks for the preview that represent data associated with the generated model and information received from the digital scene data capture device.
[0368] 121. A method according to embodiment 116, wherein providing the feedback includes providing one or more of information related to the rate of capture of digital scene data, the location used to capture the digital scene data, the sensor angle used to capture the digital scene data, aspects of the light field in the scene, or aspects of the matter field in the scene.
[0369] 122. The method of embodiment 113 further comprises using the input to align the digital scene data with newly received digital scene data.
[0370] 123. The method according to embodiment 113 also includes accessing an instruction set and executing the set of instructions to achieve one or more goals for generating a three-dimensional model.
[0371] 124. A method according to embodiment 123, wherein the one or more objectives include one or more of the following items: a desired resolution of the light field, a desired resolution of the matter field, a desired certainty threshold for reconstruction, a threshold for eliminating gaps in the captured digital scene information, and triggers for events encountered during the capture of the digital scene information.
[0372] 125. A method according to embodiment 124, wherein the trigger includes one or more of a specified matter field structure, a specified light field structure, the passage of time, and a change in the uncertainty level in the model.
[0373] 126. The method according to embodiment 124 also includes taking action in response to the trigger.
[0374] 127. The method of embodiment 126, wherein taking action comprises one or more of: changing a display configuration, adding an overlay to a display, providing an audio prompt, providing a visual prompt, changing a reconstruction target, and changing a setting of a device connected to the system.
[0375] 128. The method of embodiment 113 further includes changing one or more features of the model based on the input.
[0376] 129. A method according to embodiment 128, wherein the change includes one or more of the following: editing a light field reconstruction; editing a matter field reconstruction; transforming the model; deforming the model; re-illuminating all or any part of the model; changing one or more light interaction properties of the BLIF; assigning one or more BLIFs to different areas of the matter field; manipulating the model by dragging anchor points, by typing keyboard shortcuts, or by sculpting and painting on the model with a brush tool; inserting a new matter field; inserting a new light field; re-illuminating one or more matter fields (in whole or in part); deleting a light field in whole or in part; and deleting a matter field in whole or in part.
[0377] 130. The method of embodiment 113 further comprises searching the model spatially using a search query comprising one or more parameters.
[0378] 131. A method according to embodiment 130, wherein the spatial search includes: obtaining one or more of a count, a selection or a group of light field structures, or obtaining one or more of a count, a selection or a group of matter field structures, matching one or more parameters of the search query.
[0379] 132. The method of embodiment 130, wherein the search query is provided as a selected light region.
[0380] 133. The method of embodiment 130, wherein the search query is provided as a selected area of matter.
[0381] 134. A method according to embodiment 130, wherein the search query is presented as descriptive terms that generate responses based on machine learning.
[0382] 135. The method of embodiment 130, wherein the one or more parameters include one or more of matter field shape, light field structure, radiation intensity, size, and BLIF.
[0383] 136. The method of embodiment 113 further comprises providing a display for capturing digital scene information, wherein during the capture of information from multiple sources, spatially interleaved layers are displayed in three or more adjacent areas of the display.
[0384] 137. A method according to embodiment 136, wherein at least one of the areas is a real-time reconstruction preview.
[0385] 138. A method according to embodiment 136, wherein all layers on the display are substantially aligned with the same viewpoint.
[0386] 139. A method according to embodiment 136, wherein all layers on the display contain information about the scene.
[0387] 140. The method of embodiment 136, wherein one of the layers is a pre-scene rendering that is aligned to substantially the same viewpoint as the other layers.
[0388] 141. The method of embodiment 113, further comprising providing a display and using the display during capturing of digital image data to indicate how many angles around a BLIF have been captured.
[0389] 142. The method of embodiment 141, wherein the indication is provided by displaying a spherical or hemispherical overlay centered about the selected medium including the BLIF.
[0390] 143. A method according to embodiment 142, wherein at least a portion of the spherical covering layer changes in response to viewing the medium from various angles relative to the corresponding position of the medium in real space.
[0391] 144. The method of embodiment 143, wherein the change in at least a portion of the spherical covering comprises one or more of disappearing, changing color, or undergoing other visible changes.
[0392] 145. A method according to the preceding embodiment, wherein the reconstruction of the opaque outer structure is combined with the reconstruction of the inner structure to form a more complete reconstruction.
[0393] 146. The method of the preceding embodiment, wherein the internal structure does not yet include BLIF information, and the BLIF information is automatically generated based on the external structure.
[0394] 147. A method for operating a machine learning model, comprising: creating a training set including models of objects, wherein the models include re-illuminable properties that allow the models to be reconstructed under incident lighting conditions other than lighting conditions associated with image data from which the models were created, wherein the re-illuminable properties include functions that characterize the interaction of a medium within the model with light at a location in a matter field; accessing the training set using the machine learning model; training the machine learning model using the training set, wherein the training includes configuring the machine learning model to perform one or more of object classification, surface resolution, light field reconstruction, matter field reconstruction, and material signature identification; and using the trained machine learning model to characterize new objects.
[0395] 148. The method of claim 147, wherein the relightable model includes data representing one or more emission solid angle elements at a plurality of solid elements.
[0396] 149. The method of claim 147, wherein the machine learning model comprises one or more of a physically-informed neural network (PINN), a neural network architecture with physical constraints, incorporating physical priors into the loss function, hybrid modeling, and residual modeling.
[0397] 150. The method of claim 147, wherein the function consists of one or more bidirectional light interaction functions (BLIFs).
[0398] 151. The method of claim 150, wherein one or more BLIFs are processed using a neural network or a sampled data function.
[0399] 152. The method of claim 150, wherein at least one of the BLIFs is spatially varying.
[0400] 153. The method of claim 150, wherein BLIF represents one or more light interaction phenomena including absorption, transmission, reflection, and scattering.
[0401] 154. The method of claim 147, wherein the function represents properties including refractive index, roughness, characterization of pores in the medium, polarized diffusion coefficient, non-polarized diffusion coefficient, and extinction coefficient.
[0402] 155. The method of claim 147, wherein the classification comprises characterizing one of the states of a traffic light.
[0403] In the examples described herein, for the purpose of illustration and non-limiting, a large number of specific details such as specific nodes, functional entities, technologies, protocols, standards, etc. are set forth to provide an understanding of the described technology. It is obvious to those skilled in the art that other embodiments can be practiced without the specific details disclosed below. In other examples, a detailed description of known methods, devices, technologies, etc. is omitted so as not to make the description unclear with unnecessary details. A separate functional block is shown in the figure. Those skilled in the art will appreciate that the functions of these blocks can be implemented using a separate hardware circuit, using a software program and data combined with a properly programmed microprocessor or general-purpose computer, using an application-specific integrated circuit (ASIC) and / or using one or more digital signal processors (DSP). Software program instructions and data can be stored in a computer-readable storage medium, and when instructions are executed by a computer or other suitable processor, the control computer or processor performs functions. Although the database is depicted as a table herein, other formats (including relational databases, object-based models, and / or distributed databases) can also be used to store and manipulate data.
[0404] Although process steps, algorithms, etc. can be described or claimed in a particular order, these processes can be configured to work in different orders. In other words, any order or sequence of steps that can be clearly described or claimed does not necessarily indicate the requirement of performing these steps in that order. The steps of the process described herein can be performed in any possible order. In addition, although some steps are described or implied as non-simultaneous occurrences (for example, because a step is described after another step), these steps can be performed simultaneously. In addition, illustrating the process by depicting the process in the accompanying drawings does not mean that the process shown excludes other variants and modifications thereto, does not mean that the process shown or any of its steps is necessary for the technology, and does not mean that the process shown is preferred. In addition, process steps, algorithms, etc. described as recursive processes can be iteratively performed, and process steps, algorithms, etc. described iteratively can be recursively performed.
[0405] The above-mentioned processor, memory, network interface, I / O interface and display are or include hardware devices (eg, electronic circuits or a combination of circuits) configured to perform various functions of the computing device.
[0406] In some embodiments, each or any of the processors is or includes, for example, a single-core or multi-core processor, a microprocessor (e.g., which may be referred to as a central processing unit or CPU), a digital signal processor (DSP), a microprocessor associated with a DSP core, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, or a system on a chip (SOC) (e.g., an integrated circuit including a CPU and other hardware components such as memory, network interface, etc.). In some embodiments, each or any of the processors uses an instruction set architecture such as x86 or Advanced RISC Machine (ARM).
[0407] In some embodiments, each or any of the memory devices is or includes a random access memory (RAM) such as a dynamic RAM (DRAM) or a static RAM (SRAM), a flash memory (based on, for example, NAND or NOR technology), a hard disk, a magneto-optical medium, an optical medium, a buffer memory, a register (e.g., which holds instructions), or other types of devices that perform volatile or non-volatile storage of data and / or instructions (e.g., software executed on or by a processor). The memory devices are examples of non-volatile computer-readable storage media.
[0408] In some embodiments, each or any of the network interface devices includes one or more circuits (such as a baseband processor and / or a wired or wireless transceiver) and implements layer 1, layer 2 and / or higher layers for one or more wired communication technologies (such as Ethernet (IEEE 802.3) and / or wireless communication technologies (such as Bluetooth, WiFi (IEEE 802.11), GSM, CDMA2000, UMTS, LTE, Advanced LTE (LTE-A), 5G and 5G New Radio (5G NR) (including but not limited to IEEE 1914.1 and IEEE1914.3), enhanced mobile broadband (eMBB), ultra-reliable low latency communication (URLL C), massive machine type communication (mMTC) and / or other short-range, medium-range and / or long-range wireless communication technologies). The transceiver may include circuits for a transmitter and a receiver. The transmitter and the receiver may share a common housing, and may share some or all of the circuits in the housing to perform transmission and reception. In some embodiments, the transmitter and the receiver of the transceiver may not share any common circuits, and / or may be in the same or separate housings.
[0409] In some embodiments, each or any of the I / O interfaces is or includes one or more circuits that receive data from a processor, generate corresponding image data based on the received data (e.g., via a discrete GPU, an integrated GPU, a CPU that performs graphics processing, etc.), and / or output the generated image data (e.g., via a High Definition Multimedia Interface (HDMI), a DisplayPort (DisplayPort) interface, a Video Graphics Array (VGA) interface, a Digital Video Interface (DVI), etc.) to a display device that displays the image data. Alternatively or additionally, in some embodiments, each or any of the display interfaces is or includes, for example, a video card, a video adapter, or a graphics processing unit (GPU).
[0410] In some embodiments, each or any of the user input adapters in the I / O interface is or includes one or more circuits that receive and process user input data from one or more user input devices that are included in the computing device, attached to the computing device, or otherwise communicate with the computing device and output data to the processor based on the received input data. Alternatively or additionally, in some embodiments, each or any of the user input adapters is or includes, for example, a PS / 2 interface, a USB interface, a touch screen controller, etc.; and / or the user input adapter facilitates input from a user input device (such as, for example, a keyboard, a mouse, a touch pad, a touch screen, etc.).
[0411] Various forms of computer-readable media / transmissions may be involved in carrying data (e.g., sequences of instructions) to a processor. For example, data may be (i) transferred from a memory to a processor; (ii) carried via any type of transmission medium (e.g., wired, wireless, optical, etc.); (iii) formatted and / or sent according to a variety of wired or wireless formats, standards, or protocols (such as Ethernet (or IEEE 802.3), ATP, Bluetooth, and TCP / IP, TDMA, CDMA, 3G, etc.); and / or (iv) encrypted in various ways known in the art to ensure privacy or prevent fraud.
[0412] It will be understood that as used herein, the terms system, subsystem, service, programmed logic circuit, etc. may be implemented as any suitable combination of software, hardware, firmware, etc. It will also be understood that the storage location herein may be any suitable combination of disk drive devices, memory locations, solid-state drives, CD-ROMs, DVDs, tape backups, storage area network (SAN) systems, and / or any other suitable tangible computer-readable storage media. It will also be understood that the techniques described herein may be implemented by causing a processor to execute instructions that may be tangibly stored on a computer-readable storage medium.
[0413] As used herein, the term "non-transitory computer-readable storage medium" includes registers, buffer memory, ROM, semiconductor memory devices (such as D-RAM, S-RAM or other RAM), magnetic media (such as flash memory), hard disks, magneto-optical media, optical media (such as CD-ROM, DVD or Blu-ray Disc), or other types of non-transitory electronic data storage devices. The term "non-transitory computer-readable storage medium" does not include transitory propagating electromagnetic signals.
[0414] When this document describes an action “may,” “could,” or “might” be performed, a feature or component “may,” “could,” or “might” be included in or applicable to a given context, a given item “may,” “could,” or “might” possess a given property, or whenever any similar phrases involving the terms “may,” “could,” or “might” are used, it should be understood that the given action, feature, component, property, etc. is present in at least one embodiment, but not necessarily in all embodiments.
[0415] While the invention has been described in conjunction with what is presently considered to be the most practical and preferred embodiments, it should be understood that the invention is not limited to the disclosed embodiments, but on the contrary is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.
Claims
1. A scene reconstruction and machine learning system, comprising: a storage medium configured to store image data, one or more scene models, one or more re-illuminable matter fields, information associated with a machine learning model, and an output of the machine learning model; input circuitry configured to receive image data representative of light in a scene, wherein the scene is occupied by matter including an object; The processor is configured as: reconstructing a scene model representing the scene using the image data, wherein the scene model represents a volumetric region of the scene occupied by matter that interacts with the light, extracting a re-illuminable matter field from the scene model representing the object, wherein the re-illuminable matter field characterizes light interactions with the object, storing the scene model and the re-lightable matter field representing the object in the storage medium, applying the re-illuminable matter field as an input to the machine learning model, and generating an output from the machine learning model after applying the re-illuminable matter field as an input; and The output circuit is configured to output the generated output.
2. The system according to claim 1, wherein: The re-illuminable matter field characterizes the light interaction using data representing parameters of a neural network.
3. The system according to claim 1, wherein: The processor is further configured to calculate a solid angle element of light in the exiting light field given a solid angle element of light in the incident light field.
4. The system according to claim 1, wherein: The re-illuminable matter field representation comprises properties of at least one of a refractive index, a roughness, an absorptivity, a transmittance, a reflectivity, a scattering rate, a characterization of holes in the medium, a polarized diffusion coefficient, a non-polarized diffusion coefficient, and an extinction coefficient.
5. The system according to claim 4, wherein: The property is expressed as at least one bidirectional light interaction function.
6. The system according to claim 5, wherein: The at least one bidirectional optical interaction function is spatially varying.
7. The system according to claim 1, wherein: The output is one or more of classification, regression, clustering, prediction, pattern recognition, determination of the state of a traffic light, detection of surface anomalies, characterization of features of an object, estimation of the cost for repairing a hail damaged object.
8. A method for using a machine learning model and re-illuminable matter field data to serve the purpose of: Accessing image data representing the light in a scene, where Said scene is occupied by matter, including objects; reconstructing a scene model representing the scene using the image data, wherein the scene model represents a volumetric region of the scene occupied by matter that interacts with the light; extracting a re-illuminable matter field from the scene model representing the object, wherein the re-illuminable matter field characterizes light interactions with the object; storing the scene model and the relightable matter field representing the object in a storage medium; applying the re-illuminable matter field as input to a machine learning model; and After applying the re-illuminable matter field as input, an output is generated from the machine learning model.
9. The method according to claim 8, wherein: The re-illuminable matter field characterizes the light interaction using data representing parameters of a neural network.
10. The method according to claim 8, wherein: The method also includes using data associated with the light interactions to calculate a solid angle element of light in an exiting light field given a solid angle element of light in an incident light field.
11. The method according to claim 8, wherein: The re-illuminable matter field representation comprises properties of at least one of a refractive index, a roughness, an absorptivity, a transmittance, a reflectivity, a scattering rate, a characterization of holes in the medium, a polarized diffusion coefficient, a non-polarized diffusion coefficient, and an extinction coefficient.
12. The method according to claim 11, wherein: The property is expressed as at least one bidirectional light interaction function.
13. The method according to claim 12, wherein: The at least one bidirectional optical interaction function is spatially varying.
14. The method of claim 8, further comprising using the output to perform one or more of classification, regression, clustering, prediction, pattern recognition, determining the state of a traffic light, detecting surface anomalies, characterizing features of an object, and estimating a cost for repairing a hail damaged object.
15. A machine learning system for use with re-illuminable matter field data, comprising: a storage medium configured to store re-illuminable matter field data, information associated with a machine learning model, and an output of the machine learning model; input circuitry for receiving re-illuminable matter field data representing one or more objects, wherein at least some of said re-illuminable matter field data characterizes light interactions with said objects; The processor is configured as: Using the data as a training set to train the machine learning model, receives as input the relightable matter field of the new object, and generating an output in response to the input; and The output circuit is configured to output the generated output.
16. The system of claim 15, wherein: The re-illuminable matter field data characterizes the interaction of light with properties of the object, including at least one of refractive index, roughness, absorptivity, transmittance, reflectivity, scattering, characterization of holes in the medium, polarized diffusion coefficient, non-polarized diffusion coefficient and extinction coefficient.
17. The system of claim 16, wherein: The light interaction properties represent parameters of the neural network.
18. The system of claim 17, wherein: The light interaction property is represented as at least one bidirectional light interaction function.
19. The system of claim 17, wherein: The at least one bidirectional optical interaction function is spatially varying.
20. The system of claim 15, wherein: The output is one or more of classification, regression, clustering, prediction, pattern recognition, determination of the state of a traffic light, detection of surface anomalies, characterization of features of an object, estimation of the cost for repairing a hail damaged object.
21. The system of claim 15, wherein: The storage medium is further configured to store one or more scene models; The input circuit is further configured to receive the one or more scene models, wherein the one or more scene models represent a volumetric region of the scene occupied by matter that interacts with light; and The processor is further configured to extract re-illuminable matter field data from the one or more scene models, wherein the re-illuminable matter field data represents an object, and wherein at least some of the re-illuminable matter field data characterizes a light field emerging from the object given a light field incident on the object.
22. A method for training a machine learning model using re-illuminable matter field data, comprising: collecting re-illuminable matter field data representing one or more objects, wherein at least some of the re-illuminable matter field data characterizes light interactions with the objects; and The machine learning model is trained using the re-illuminable matter field data as a training set, wherein the trained machine learning model is configured to receive the re-illuminable matter field of a new object as input, thereby generating an output in response to the input.
23. The method according to claim 22, wherein: The re-illuminable matter field data characterizes the interaction of light with properties of the object, including at least one of refractive index, roughness, absorptivity, transmittance, reflectivity, scattering, characterization of holes in the medium, polarized diffusion coefficient, non-polarized diffusion coefficient and extinction coefficient.
24. The method according to claim 23, wherein: The light interaction properties represent parameters of the neural network.
25. The method according to claim 23, wherein: The light interaction property is represented as at least one bidirectional light interaction function.
26. The method according to claim 25, wherein: The at least one bidirectional optical interaction function is spatially varying.
27. The method of claim 22, wherein: The output is used for one or more of classification, regression, clustering, prediction, pattern recognition, determining the state of a traffic light, detecting surface anomalies, characterizing features of an object, and estimating the cost for repairing a hail damaged object.
28. The method of claim 22, wherein: The collection also includes: accessing one or more scene models representing a scene, wherein the scene models represent a volumetric region of the scene occupied by matter that interacts with light; and A re-lightable matter field is extracted from the one or more scene models, wherein the re-lightable matter represents the object.
Citation Information
Cited By
Human-computer interaction method based on expression recognition, interactive intelligent toy and robot
CN120088834A
