Systems and methods for generalized scene reconstruction - Patents.com

Abstract

Various embodiments of the present disclosure relate to a scene reconstruction and machine learning system. In an embodiment, the system comprises a storage medium configured to store information related to image data, one or more scene models, one or more reilluminatable material fields, and a machine learning model. In one or more embodiments, the system comprises an input circuit configured to receive image data characterizing light in a scene. In an embodiment, the system comprises a processor. In an embodiment, the processor is configured to reconstruct a scene model representing the scene using the image data. In an embodiment, the processor is configured to extract the reilluminatable material field from a scene model representing an object, store the scene model and the reilluminatable material field representing the object in the storage medium, apply the reilluminatable material field as an input to the machine learning model, and generate an output from the machine learning model.

Description

[Technical field]

[0001] The present invention relates generally to the field of 3D imaging, and more particularly to various systems and methods for generating 3D models from images, referred to as generalized scene reconstruction (GSR), volumetric scene reconstruction (VSR), or quotidian scene reconstruction (QSR), as well as tools for implementing the light field reconstruction (LFR) systems and methods described further herein. [Background technology]

[0002] 3D models of real-world scenes have a myriad of applications. They are used in general areas such as defense, security, entertainment, education, healthcare, infrastructure, manufacturing, and mobile. In the metaverse, applications include virtual real estate creation, NFT creation, and avatar creation. Various methods have been proposed or developed to capture 3D images, some of which can provide digital 3D models of real-world scenes with various fidelity for various purposes such as visualization and information extraction. Such 3D images are acquired by 3D imagers, which are variously called 3D sensors, 3D cameras, 3D scanners, VR cameras, 360° cameras, RGBD cameras, and depth cameras.

[0003] Traditional methods used to extract 3D information from a scene often use active light sources such as lasers, which are limited by high power consumption and limited range. A more ideal method would be to generate a detailed scene model using one or more images from inexpensive sensors, such as cameras or devices that form an image by sensing a light field with a detector. To increase the robustness of the extraction of a scene model from images, improved modeling of light transport is required. This includes the properties of light's interaction with matter, such as transmission, reflection, refraction, and scattering. Jarosz's paper "Efficient Monte Carlo Methods for Light Transport in Scattering Media" (2008) provides a detailed analysis of the subject.

[0004] Previous work has proposed methods for creating 3D images or models using a technique commonly known as generalized scene reconstruction (GSR) (also called volumetric scene reconstruction (VSR) or quatidian scene reconstruction (QSR)). For example, U.S. Pat. No. 10,521,952 to Ackerson et al., U.S. Pat. No. 11,508,115 to Ackerson et al., U.S. Patent Application Publication No. 2021 / 0133929A1 to Ackerson et al., and U.S. Provisional Patent Application No. 63 / 317,330 to Ackerson et al., each of which is incorporated herein by reference in its entirety, variously describe systems and methods for implementing embodiments of GSR. In some circumstances, GSR can be implemented using a scene reconstruction engine (SRE) that creates a 3D scene model from a digital image using a process called scene reconstruction. SREs can embody items in the Devices Utilizing Scene Reconstruction (DSR) category, such as 3D mobile phones, tablets, computers, devices such as virtual reality (VR) and augmented reality (AR) glasses, autonomous, semi-autonomous, or controlled unmanned systems such as drones, and other digital portable or non-portable devices.

[0005] Some of the advantages of GSR are described in the patents and patent applications mentioned above. For example, Figures 4B and 10 of US Patent Application Publication No. 2021 / 0133929A1 are diagrams representing real-world scenes, and this representation can be considered an abstract scene model view of the data stored in the plenoptic scene database. Figure 4B focuses on a larger scene, and Figure 10 focuses on a smaller scene. The abstract representation of the scene model for the two different types of scenes includes a plenoptic field, which includes the matter field and the light field of the scene. The light field interacts with any number of objects in the matter field, and also with other objects, such as described objects, undescribed areas, opaque objects, finely structured objects, distant objects, luminous objects, highly reflective objects, featureless objects, or partially transparent objects. U.S. Patent Application Publication No. 2021 / 0133929A1 teaches that a key aspect of GSR is that a matter field is identified from the scene reconstruction sufficient to distinguish between multiple types of objects, and that individual types of objects uniquely located within the model scene can then be further processed, for example using machine learning to perform object recognition and classification, or to modify various characteristics and properties to cause model presentation effects such as visualization changes, object enhancement and tagging, or even object removal.

[0006] In one embodiment, GSR may be implemented by utilizing codecs. Various codecs are well known in the art and are generally devices or programs that compress data to allow faster transmission and decompress received data. Examples of codec types include video (e.g., MPEG, H.264), audio (e.g., MP3, ACC), image (e.g., JPEG, PNG), and data (e.g., PKZIP), where a codec type encapsulates data and is tightly bound to the type of data. In many legacy applications, this tight binding is inherent, limiting the end user's experience. Codecs are often implemented in a fundamentally "file-based" manner, where a file is a data representation of a pre-captured real or synthetic sensory experience, and a file (e.g., a movie, a song, a book, etc.) necessarily limits the user's experience to the experience-path selected by the file creator. Thus, the user watches a movie, listens to a song, or reads a book in a substantially ordered experience limited by the creator.

[0007] With respect to GSR, the use of codecs requires an increase in the types of data processed by such codecs, particularly in order to perform GSR where sensors such as cameras and range finders create scene models of real-world scenes. Challenges in achieving GSR include representing and organizing sufficient representations to represent the complexity of real-world material and light fields in an efficiently controllable and highly scalable manner, as well as distribution challenges in terms of managing active, possibly live, scene models to multiple interactive clients, each of which may require a virtually unlimited number of scene views, details, and data types.

[0008] Furthermore, machine learning (ML) and artificial intelligence (AI) systems have made great strides in recent years, making them useful and effective in many application areas. Many such systems are used for object identification in a scene and other useful purposes. ML and AI systems are often based on processing 2D images from a camera system. Such images are typically an array of red, green, and blue (RGB) values. Such detected information consists of samples of the light field that enters the camera lens and converges to the viewpoint. These light samples are the result of a complex series of interactions between light and materials in the scene, and are governed by the laws of physics. The "true" characteristics of objects, such as their actual color and reflective properties, are important for determining the type and nature of materials in a scene, but this information cannot generally be determined from traditional photography.

[0009] The interaction of light rays from light sources both inside and outside the scene, as they are reflected and blocked by other materials in the scene, contributes to a complex scene light field, which in fact obscures fundamental information about materials in the scene. In addition to obvious examples such as shadows, when light reflected from an object is affected by light reflected from other objects, it changes the perceived color. Lambertian surfaces reflect light approximately equally in all directions. Even greater challenges arise when the object's surface is non-Lambertian. Such surfaces have complex reflectance properties that are not easily solved by conventional systems. These challenges include subsurface scattering, specular reflection, transparency, etc. For example, subsurface scattering is a major component of the appearance of human skin.

[0010] Because it is difficult to determine fundamental properties of materials from reflected light in a scene captured by a camera, image-based supervised ML systems typically require large training sets to provide a reasonable representation of the light interaction conditions expected during operation. Such image training sets are typically manually classified by labeling images with the identity of objects of interest (OOI) in each image. Depending on the application, the training set may have objects identified as "good" (the image or part of the image contains OOI) or "bad" (the image does not contain OOI). In many cases, there are roughly equal numbers of good and bad objects.

[0011] In cases like anomaly detection, only good or nearly good training examples are needed. If the generated image is not identified as OOI, an anomaly has been detected. Errors in the training set, even subtle errors, can degrade the quality of the results. For example, if mixing good and bad training examples is substantially inconvenient, there is a risk of overfitting, where the system can accurately recognize the training objects but not reliably recognize the objects in the generated images.

[0012] What is often most important for machine learning systems is a correctly labeled and error-free dataset for training and testing. Creating such datasets often requires thousands or even millions of manually labeled images, which is cost prohibitive and a major obstacle to widespread adoption. Being able to model the effects of light field interactions in a scene and isolate their effects with camera images would make such systems more effective and easier to train.

[0013] Various patents, patent applications, and other publications disclose methods for implementing GSR, other forms of 3D imaging, scene reconstruction, or component processes or systems of such activities. For example, the following references touch upon various aspects of GSR and 3D space segmentation by media properties: Leffingwell, J. et al., "Generalized Scene Reconstruction", arXiv:1803.08496, May 24, 2018; Kutulakos, K. et al., "A Theory of Shape by Space Carving", U. of Rochester, 2000; Bonfort, T. and Sturm, P., "Voxel Carving for Specular Surfaces", Proceedings of the Ninth IEEE International Conference on Computer Vision (ICCV), 2003; Broadhurst, A. et al., "A Probabilistic Framework for Space Carving", Proc. of Int. Conference on Computer Vision, I, pp. 282-291, 2001; Broadhurst, A. and Cipolla, R., "A Statistical Consistency Check for the Space Carving" Algorithm”, Proceedings of the 11th British Machine Vision Conference, pp. 282-291, 2000;Gaillard, M. et al. “Voxel Carving Based 3D Reconstruction of Sorghum Identities Generic Determinants of Ration Interception Efficiency”, bioRxiv preprint, https: / / doi.org / 10.1101 / 2020.04.06.028605, 7 April 2020;Sainz, M."Hardware Accelerated Voxel Carving," Research Gate, publication 228917433;Scharr, H. et al., "Fast High Resolution Volume Carving for 3D Plant Shoot Reconstruction," Frontiers in Plant Science, September 28, 2017;Seitz, S. and Dyer, C., "Photorealistic Scene Reconstruction by Voxel Coloring," Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 1067-1073, 1997;Culbertson, W., Malzbender, T. and Slabaugh, G., "Generalized Voxel Coloring," Seventh International Conference on Computer Vision, September 1999;Dyer, C., "Volumetric Scene Reconstruction from Multiple Views," Foundations of Image Analysis, edited by L.S. Davis, Chapter 1, 2001;Seitz, S. and Kutulakos, K., "Plenoptic Image Editing," Sixth International Conference on Computer Vision (IEEE Cat. No. 98CH36271), 1998;Troccoli, A. and Allen, P., "Relighting Acquired Models of Outdoor Scenes," Proceedings of the 5th Int'l Conf. on 3-D Digital Imaging and Modeling, 2005;Singh, R.et al., "3D convolutional neural networks for object recognition: a review," Multimedia Tools and Applications, 2018;Riegler, G. et al., "OctNetFusion: Learning Depth Fusion from Data," arXiv:1704.01047v3, October 31, 2017;Riegler, G. et al., "OctNet: Learning Deep 3D Representations at High Resolutions," arXiv:1611.05009v4, April 10, 2017;Meka, A. et al., "Deep Relightable Textures," ACMTrans. Graph., Vol. 39, No. 6, Article 259, 2020;Liu, J. et al., "RocNet: Recursive Octree Network for Efficient 3D Deep Representation," arXiv:2008.03875v1, Aug. 10, 2020;Lei, H. et al., "Octree guided CNN with Spherical Kernels for 3D Point Clouds," Computer Vision Foundation, pp. 9631-40;Bi, S. et al., "Deep Relightable Appearance Models for Animatable Faces," ACM Trans. Graph., Vol. 40, No. 4, Article 89, Aug. 2021;Wang, P. et al., "O-CNN: Octree-based Convolutional Neural Networks for 3D Shape Analysis," ACM Transactions on Graphics, Vol. 36, No. 4, Article 72, Jul. 2017;Wang, P. et al., "Adaptive O-CNN: A Patch-based Deep Representation of 3D Shapes," arXiv:1809.07917v1, Sep. 21, 2018;Wang, P.et al., "Deep Octree-based CNNs with Output-Guided Skip Connections for 3D Shape and Scene Completion," Computer Vision Foundation, 2020; U.S. Patent No. 4,694,404; U.S. Patent No. 5,123,084; U.S. Patent No. 6,123,733; U.S. Patent No. 6,980,935; U.S. Patent No. 6,831,641; U.S. Patent No. 7,843,449; U.S. Patent No. 8,432,435; U.S. Patent No. 8,547,374; U.S. Patent No. 8,749,620; U.S. Patent No. 8,749,694; U.S. Patent No. 9,179,126; U.S. Patent No. 9,857,470; U.S. Patent No. No. 10,169,910;U.S. Patent No. 10,509,153;U.S. Patent No. 10,893,262;U.S. Patent No. 11,164,368;U.S. Patent Application Publication No. 20080068372;U.S. Patent Application Publication No. 20110128412;U.S. Patent Application Publication No. 20130038696;U.S. Patent Application Publication No. 20130156297;U.S. Patent Application Publication No. 20140184749;U.S. Patent Application Publication No. 20140201022;U.S. Patent Application Publication No. 20150146032;U.S. Patent Application Publication No. 20150305612;U.S. Patent Application Publication No. 20150373320;U.S. Patent Application Publication No. 20160028935;U.S. Patent Application Publication No. 20180113 200;U.S. Patent Application Publication No. 20180144540;U.S. Patent Application Publication No. 20180149791;U.S. Patent Application Publication No. 20180227568;U.S. Patent Application Publication No. 20190011621;U.S. Patent Application Publication No. 20190072897;U.S. Patent Application Publication No. 20190155835;U.S. Patent Application Publication No. 2022 0058854; UK Patent No. 2535475; European Patent Application No. 3144887; WO 2011066275; WO 2018200316; WO 2018200316; WO 2019213450; New Zealand Patent Application Publication No. 743841; China Patent Application Publication No. 111796255. The following references address various aspects of GSR and non-parametric modeling: Freeman, H."On the encoding of arbitrary geometric configurations", IRE Transactions on Electronic Computers EC-10, pages 260-268; Samet, H., "The Design and Analysis of Spatial Data Structures", Addison-Wesley Series in Computer Science, 1989; Marschner, S. and Shirley, P. et al., "Fundamentals of Computer Graphics", CRC Press, 2016; Varma, M. and Zisserman, A., "A Statistical Approach to Texture Classification from single Images", International Journal of Computer Vision 62(1 / 2), 61-81m, 2005. The following publications address various aspects of GSR and integral rendering: Mildenhall, B. et al., "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis", arxiv: 2003.08934v4, August 3, 2020 (video: https: / / www.matthewtancik.com / nerf); Yu, Alex et al., "PlenOctrees for real-time rendering of neural radiance fields", arXiv:2103.14024, 2021; Yu, A. et al., "Plenoxels: Radiance Fields without Neural Networks", arXiv:2112.05131v1, 2021; EyeCue Vision Tech.'s "Qlone 3D Scanner", Apple App Store, Version 4.6.0, 2022 (https: / / apps.apple.com / Available at: us / app / qlone-3d-scanner / id1229460906);J.Paul Morrison, "Flow-based Programming: A New Approach to Application Development", 2nd Edition, JP Morrison Enterprises, 2010;Karras, T. et al., "A Style-Based Generator Architecture for Generative Adversarial Networks", CoRR 2018, vol abs / 1812.04948 (available at https: / / arxiv.org / abs / 1812.04948);R. Martin-Brualla et al., "NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections", 2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 7206-7215, 2020;Zhang, X. et al., "NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination", ACM|SIGGRAPH Asia 2021 Technical Papers, 2021 (available at https: / / dspace.mit.edu / handle / 1721.1 / 146375). Each of the foregoing documents and disclosures is hereby incorporated by reference in its entirety.

[0014] Thus, there exists a need, and many needs and opportunities in the marketplace, to overcome the shortcomings and deficiencies in the art by providing various systems and methods for implementing GSR, and components thereof. Summary of the Invention

[0015] Overview of Exemplary Embodiments of the Invention The following simplified summary may provide a basic, introductory understanding of some aspects of the systems and / or methods described herein. This summary is not an extensive overview of the systems and / or methods described herein, nor is it intended to identify all key / critical elements or to delineate the entire scope of such systems and / or methods. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.

[0016] In an embodiment, a processor for processing digital scene data and an interface for receiving input related to the captured scene may be used to reconstruct one or more objects in the scene. In that embodiment, (i) the input includes digital scene data in the form of image data representing the scene from a viewpoint, (ii) the processor processes the digital scene data and the input to generate a three-dimensional model of at least a portion of the scene including materials constituting an interacting medium, (iii) the processor processes the image data by accessing one or more voxels in a material field represented by the image data, and (iv) the processor processes the image data by determining whether the material represented in each of the one or more voxels includes an interacting medium. The image data may be captured by a camera and may be data related to electromagnetic radiation such as visible light, infrared light, polarized or unpolarized radiance values, and / or radar. The orientation may include a camera pose and in an embodiment may include multiple poses or orientations. The three-dimensional model may be represented by a data structure. In an embodiment, the three-dimensional model is represented by a combination of a first data structure storing plenoptic data and a second data structure including the orientation of the digital scene data. Some embodiments of the present invention may also store information related to a light field in a scene in the first data structure or the third data structure. In some embodiments, the processor sequentially processes image data from at least two directions. In some embodiments, the material represented in the voxels is represented by a mediel, and data related to the mediel can be stored in the plenoptic data structure. Data related to the mediel may include an outgoing light field and / or an incoming light field, and such data may be represented by a radiel.

[0017] In an embodiment, the scene reconstruction may include processing the image data by assuming an orientation of the digital scene data. The processing of the image data may include (i) assuming the presence of a medium in the voxel, (ii) assuming one or more of a surface normal, a light interaction property, an outgoing radiance vector, and an incident light field of the medium, among other properties, (iii) calculating a cost for the presence of a medium in the voxel based on one or more of the assumed surface normal, the light interaction property (e.g., refractive index, roughness, polarized diffusion coefficient, unpolarized diffusion coefficient, or extinction coefficient), the outgoing radiance vector, and the incident light field of the medium, (iv) comparing the cost to a cost threshold, and (iv) accepting the voxel as having a medium present if the cost is less than the cost threshold. In an embodiment, when the system accepts the voxel as having a medium present, the medium remains in the scene in subsequent processing of the scene. Certain embodiments may update the light field assumptions for one or more other voxels based on the acceptance of the presence of the medium. The system may perform the processing iteratively on multiple voxels and / or multiple sets of image data. In some embodiments, the results of the processing may be stored in a data structure, including a hierarchical data structure, where the processing may be performed by hierarchically traversing the data structure from coarse to fine levels, and the fine levels of detail may be stored in the data structure by subdividing the data structure.

[0018] Certain embodiments of the present invention include a method of training a machine learning model, comprising providing image data including one or more objects of interest to the machine learning model, processing the image data to generate a model, where such processing includes analyzing the image data to generate a light field for one or more scenes or reconstruct one or more material fields in the scene, selecting an object of interest in a model of the scene, extracting the object of interest in the model of the scene, and outputting a reilluminable material field model of the object of interest in the scene. The image data may include reilluminable material field data. In an embodiment, the image data includes one or more objects of interest in multiple scenes and under various conditions. The reilluminable material field may be composed of multiple images in two or more dimensions. The reilluminable material field model may include one or more of shape information, two-way light interaction function (BLIF) information, an emissive light field (e.g., light sources of the scene itself, if present), and incident and / or responsive light fields resulting from the emissive light field. Furthermore, the light field information may be used to calculate light interaction properties of locations in the material field. In one embodiment, the method may further include varying BLIF and / or geometry information of the model to further train the machine learning model, inputting the model with the varied BLIF information to the machine learning model, and performing one or more of the above steps on the model with the varied BLIF information.

[0019] An embodiment of the invention comprises a method of using a machine learning model, comprising identifying one or more objects of interest in a model of a scene, accessing a reilluminatable matter field of the scene, selecting a portion of the matter field to be processed, processing the selected portion of the matter field to extract at least a portion of the reilluminatable matter field, and outputting the extracted portion of the reilluminatable matter field. The method may further comprise testing the usefulness of the portion of the reilluminatable matter field output by the machine learning model.

[0020] In some embodiments, the invention includes using a trained machine learning model to identify one or more objects or features of interest in a scene, and using such identification to provide initial hypotheses for light field and / or matter field reconstruction. In such embodiments, the invention can provide faster processing of image data to perform reconstruction of a scene or a portion thereof. In some embodiments, the output of the trained machine learning model includes one or more of the size, shape, and / or position of media in the scene, and / or light interaction properties of media in the scene.

[0021] Some embodiments of the invention can use a reconstruction of light field and / or material field properties as an additional input to the scene reconstruction process. For example, embodiments of the invention may use a point cloud provided by LiDAR, or another material field and / or light field reconstruction provided by other techniques (e.g., multi-view stereo, photogrammetry, infrared, radar, etc.) to provide an initial or updated assumption of the properties of the media in the scene. Embodiments of the invention may then perform the scene reconstruction process described herein to reconstruct the light field and / or material field in the scene.

[0022] In an embodiment, the invention can reconstruct one or more objects in a scene by a processor for processing digital scene data and an interface for receiving input related to the captured scene, the processor processes the digital scene data and the input to generate a three-dimensional model of at least a portion of the scene, the processor can direct the input to at least a portion of the processing of the digital scene data, and the processor can provide an output that includes the three-dimensional model of at least a portion of the scene. The input can include at least one of an approximation of at least a portion of a light field in the scene, an approximation of at least a portion of a matter field in the scene, one or more shapes present in the scene, one or more objects in the scene, and information related to one or more light sources in the scene. The input can control one or more sensing devices that provide the digital scene data. In an embodiment, the system can provide feedback regarding one or more objects to be reconstructed in the scene, the feedback can include a preview of one or more objects to be reconstructed in the scene. The system can update the preview as one or more objects are reconstructed using results from such reconstruction. The preview can further include one or more indications regarding one or more parameters of the reconstruction. The preview can include one or more masks representing data related to the generated model and information received from the digital scene data capture device. The feedback may include one or more of information related to a capture rate of the digital scene data, a position for capturing the digital scene data, a sensor angle for capturing the digital scene data, an aspect of a light field in the scene, and an aspect of a matter field in the scene. In one embodiment, the input is data that enables alignment of the digital scene data with newly received digital scene data.In an embodiment, the system may further include an instruction set for achieving one or more goals for the generation of the three-dimensional model, the one or more goals including one or more of a desired resolution of the light field, a desired resolution of the material field, a desired certainty threshold for reconstruction, a threshold for eliminating gaps in the captured digital scene information, and a trigger for an event encountered during capture of the digital scene information. In an embodiment, the trigger includes one or more of a specified material field structure, a specified light field structure, a time lapse, and a change in the level of uncertainty of the model. The system may be configured to operate in response to the trigger, and the response may include one or more of changing a display configuration, adding an overlay to the display, providing an audio cue, providing a visual cue, changing a reconstruction goal, and changing a setting of a device connected to the system.

[0023] Certain embodiments of the invention may be configured to modify one or more features of a scene model. For example, the modifications may include one or more of: editing the light field reconstruction, editing the matter field reconstruction, transforming the model, deforming the model, relighting all or a portion of the model, changing one or more light interaction properties of a BLIF, assigning one or more BLIFs to different regions of a matter field, manipulating the model by dragging on anchor points, entering keyboard shortcuts, or using a brush tool to sculpt and paint the model, inserting a new matter field, inserting a new light field, relighting one or more matter fields (all or a portion), removing all or a portion of a light field, and removing all or a portion of a matter field. In an embodiment, the system may be configured to spatially search the model using a search query that includes one or more parameters. Such spatial searching may include obtaining one or more of a count, selection, and group of light field structures or obtaining one or more of a count, selection, and group of matter field structures to match one or more parameters of the search query. The search query may be provided as a descriptive term that generates a response based on selected light regions, selected matter regions, and / or machine learning. The parameters may include one or more of the following: the shape of the matter field, the structure of the light field, the emission intensity, the size, and the BLIF.

[0024] In an embodiment, the system further comprises a display used to capture the digital scene information, during capture, information from multiple sources is spatially interlaced layers and is shown in three or more adjacent regions of the display. The regions of the display may include a live reconstruction preview, and all layers on the display may be aligned to substantially the same viewpoint. Furthermore, all layers on the display may include information about the scene. In an embodiment, one of the layers on the display is a pre-scene rendering (e.g., a priori scene and / or a partially or fully initialized scene model) aligned to substantially the same viewpoint as the other layers. The display may be used during capture to indicate how many angles have already been captured around a particular region of the scene, and the indication may be provided by displaying a spherical or hemispherical overlay centered on the selected medial containing the BLIF. At least one section of the spherical overlay may change in response to viewing the medial from various angles relative to a corresponding position of the medial in real space, and the change to at least one section of the spherical overlay may include one or more of disappearance, color change, or other visible change.

[0025] While the example embodiments are expressed in the form of a system or method, those skilled in the art will recognize that these examples may be modified to include similar configurations, such as at least: (A) a mechanical apparatus having the functionality recited in the claims; (B) execution of method steps for carrying out the process described by the system; and / or (C) a non-volatile computer program storage medium containing an executable program of instructions that, when executed on a compatible digital processor, provides the functionality of the described system or method.

[0026] Additional features and advantages of exemplary embodiments are described below. [Brief description of the drawings]

[0027] These and other features and advantages will be better and more completely understood by reference to the following detailed description of non-limiting illustrative embodiments taken in conjunction with the following drawings. [Fig. 1A-1E] FIG. 1 illustrates an example architecture of a system using generalized scene reconstruction (GSR), an example configuration for capturing image data, and an example scene including a material field and a light field. [Figure 2A-2B] FIG. 1 is a diagram showing an example of a surfel and a mogel. [Figure 3A-3G] FIG. 1 illustrates an exemplary method for reconstructing a scene. [Figure 4] FIG. 2 illustrates an exemplary type hierarchy of media. [Diagram 5] FIG. 1 illustrates an exemplary scene containing various media and other elements. [Figure 6] FIG. 2 is an end view of a surfel representing a curve. [Figure 7] FIG. 2 is a side view of a surfel representing a curve. [Figure 8] FIG. 2 illustrates a surface representing a corner. [Figure 9] FIG. 1 illustrates an exemplary method for generating a trained machine learning model (TMLM). [Figure 10] FIG. 1 illustrates an exemplary method for using a trained machine learning model (TMLM). [Figure 11] FIG. 13 is a diagram showing an example of a dot mesh of a reconstruction preview. [Figure 12A-12B] FIG. 1 illustrates an example of a reconstruction preview and an interlaced video feed. [Figure 13] FIG. 13 illustrates a scene capture guide. [Figures 14A-14C] FIG. 2 illustrates an exemplary process for determining the presence and interaction of a medium in an area of ​​a scene. [Figure 15] FIG. 1 is an explanatory diagram of a machine learning model. [Figure 16]FIG. 1 is an explanatory diagram of a physical information neural network (PINN). [Figure 17] FIG. 1 is an illustration of a neural network architecture with physical constraints. [Figure 18] FIG. 1 illustrates the incorporation of physical priors into a loss function. [Figure 19] FIG. 1 is an explanatory diagram of residual modeling. [Figure 20] FIG. 1 is an illustration of a combination of a physics-based approach and neural networks. [Figure 21] FIG. 1 is an illustration showing a combination of a reconstruction performed in a manner described herein with a reconstruction created in another manner. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] As described in various embodiments herein, it is an object of the present invention to provide a system and method for performing scene reconstruction, in particular generalized scene reconstruction (GSR). In an embodiment, the result of a GSR process or system may result in the reconstruction of a light field, a material field (including relightable material fields), a characterization of the camera pose, or any combination of the foregoing. The result of a GSR process may result in a model representing the scene based on the reconstructed light field or material field (including relightable material fields), either individually or together, as desired under the circumstances. As used herein, a scene may refer to the entire extent of the light field and / or material field represented in an image, any portion thereof, or any medium therein. Sub-scene, part of a scene, region of interest, object of interest, and other similar terms may be used to refer to a portion of a larger scene, although each of the foregoing is a scene in its own right.

[0029] In some embodiments, the invention can be configured to create a model of a scene using static data (i.e., data capturing a scene in which the contents of the scene are not moving) or a dynamic scene (i.e., data capturing a scene in which the contents of the scene are moving relative to each other and / or the image capture device). Similarly, the model may be configured to represent a scene, a portion of a scene, or one or more objects within a scene in a static configuration (i.e., the reconstruction depicts a scene in which the contents of the scene are not moving) or a dynamic configuration (i.e., some or all of the content of the scene is moving). In the case of a dynamic configuration, the model may be configured to represent dynamic properties in the matter field, the light field, or both.

[0030] The invention described herein may provide advantages over conventional representations of dynamic scenes. For example, some known systems for representing scenes (e.g., where the representation emphasizes the light field of a scene over the material field of the scene) may have challenges in representing dynamic properties because the relevant light properties directly contribute to the media in the scene, potentially resulting in the need to reinitialize and / or relearn large parts of the scene model at every time step where the material field changes configuration (e.g., changes shape or motion). In some embodiments herein, when reconstructing a dynamic scene, the invention described herein may use parts of the scene that are in motion to calculate interactions with the light field in the scene to better understand the media containing such objects. Similarly, when using models to represent dynamic scenes, embodiments of the invention described herein may more accurately present parts of the scene that are in motion by understanding how such parts interact with light in the modeled scene. In some embodiments, sub-scenes represented as reilluminatable material fields may use kinematic models with arbitrary deformations to represent dynamic properties in the actual material field. The effect of dynamic properties on the light field, whether for rendering or for another purpose, may be more easily calculated using the light transport process described herein.

[0031] A system using the GSR 100 described herein may include the components shown in Figure 1A. Specifically, the system 100 may include application software 101 for interfacing with the system, a scene solver 102 configured to perform specific GSR functions, a plenoptic scene database configured to store information related to the reconstruction of a scene, and a scene codec for encoding and / or decoding information related to the reconstruction of a scene.

[0032] Various exemplary embodiments of a scene model are shown in Figures 1C-1E. The scene model 110 may include a matter field 120 and a light field 130 in a single model as illustrated in Figure 1C, or in separate models as illustrated in Figures 1D (matter field) and 1E (light field). The scene may have external lighting 112 that flows into the scene and provides a light source for the scene. The scene may be a single scene with no light flowing into the scene 112. The scene may have a boundary 115, which may be optionally defined by the system during reconstruction of the scene, by a physical boundary in the scene, by user or other input, by any combination thereof, or by other methods. Information beyond the boundary of the scene may be considered a frontier 117 and not represented in the scene. However, in some embodiments, the boundary 115 may include a fenestral boundary 111 in whole or in part. The window-like boundary 111 may be a portion of the scene boundary 115 through which incoming light 112 can enter the scene and outgoing light 116 can exit the scene. In some embodiments, a portion of the frontier 117 may be at least partially represented in the window-like boundary 111. As an example, the window-like boundary 111 may be determined based on physical features in the scene (e.g., a window or skylight in a wall or ceiling through which light can enter the scene), scene parallax (e.g., a boundary based on distance or lack of resolution of image data, such as in the case of an outdoor night scene looking at the sky with a very long distance in the field of view), a combination of the two, or other factors. The scene may include one or more objects, including a responsive object 113 and an emitting object 114. The emitting object 114 may emit light independent of the light incident on it, while the emitting object may not emit light itself and may interact with the incident light.

[0033] The systems and processes described herein may use image data. The image data may provide one or more characteristics of the light field at one instant in time (e.g., a still image or a frame of a video) or a series of instants in time (e.g., a video or other data providing image information over time). The image data may be two-dimensional, three-dimensional, or higher dimensional in various embodiments. The image data may include information regarding one or more measurements or characteristics of distance or position, depth and / or range associated with media in the scene, polarization data, infrared data, hyperspectral data, or other data related to radiation characteristics. The image data may include previously captured image data, image data captured from one or more cameras or other imaging devices simultaneously with the processes described herein, synthetic image data, or computer-generated image data, or any combination thereof. Additionally, the systems and processes described herein may use other types of data in performing the GSR process.

[0034] As illustrated in FIG. 1B, embodiments of the invention may acquire one or more images of a scene, either in the form of an image captured by a camera or other image sensing device 105, a previously stored image, or other image data representative of the scene. In some embodiments, the image data may include data related to light, i.e., electromagnetic radiation, including but not limited to visible light, infrared, radar, and / or polarized or unpolarized radiance values. Such data may be expressed in pixels or other units. Each image or set of image data may preferably represent the incident light field at the point where the image data is or was captured. In some embodiments, the invention may select an image, possibly the first image taken, to define the origin and orientation of the scene.

[0035] Some embodiments of the invention use information from a scene, including, for example, image information that may be represented in digital form, to create a model of one or more of the scene, a region of interest within the scene, and the entire scene.

[0036] In some embodiments, a scene or a portion thereof may be represented by one or more plenoptic elements or primitives, which may be stored in a data structure. In some embodiments of the invention, spatial information in a scene is separated into plenoptic and analytical information. In this embodiment, the plenoptic elements may represent the scene in the model, and preferably represent elements in the scene more realistically than the analytical elements. Some embodiments of the invention use at least one plenoptic element or solid angle element, one or more of which may be contained within a voxel and / or sael.

[0037] A scene may contain one or more voxels, each of which may be the same size and shape or may be selected from a range of sizes and / or shapes determined by the user or the system. A voxel may contain media or media elements that represent all or part of the medium sampled within the voxel. A medium is a volumetric region that contains some or no matter through which light propagates. Media can be homogeneous or inhomogeneous. Examples of homogeneous media include empty space, air, and water. Examples of inhomogeneous media include a volumetric region that contains a mirror surface (part air and part silver mirror glass), a glass surface (part air and part transparent glass), and a pine tree branch (part air and part organic matter). Light propagates through a medium by phenomena including absorption, reflection, transmission, and scattering. Examples of partially transparent media include a pine tree branch and a glass pane.

[0038] The sear may include radial or radiometric elements that represent all or a portion of the light propagating in one or more directions. Light includes electromagnetic waves at frequencies including visible, infrared, and ultraviolet bands. Some embodiments of the invention may use digital imaging, digital art, other processes, or any combination thereof to create, calculate, and / or store the medial and / or radial contained by the plenoptic elements. Thus, in some embodiments, plenoptic elements may be used to represent the three dimensions of the matter field and the two dimensions (5D) of the light propagating in the scene in such a way that light and matter in a spatial scene can be sampled, and pixel elements may be used to sample light at specific locations in the scene. Analytical elements may include geometric elements such as points, lines, surfaces, and CAD models.

[0039] A plenoptic element may have one or more characteristics or a combination of characteristics, e.g., length, color, and / or shape. In some embodiments, the characteristics can be distinguished within and / or between segments in a scene. The characteristics may have one or more descriptions or examples.

[0040] A plenoptic element may have a medium 201 that contains surface elements or surfels 202. Such elements may represent an abrupt interface between two regions of homogeneous but different media. Surfels 202 are illustratively depicted in Figure 2, which illustrates a planar surfel with vectors 204 and 205 as the axes of the plane and a normal direction 203 that extends perpendicularly from the plane.

[0041] Some medials may contain homogenous elements or "mogels" 210 that represent media of uniform composition across their bounding voxels. An exemplary mogel 210 is shown in FIG. 2B, which illustrates a coordinate frame with vectors 213 and 214 that represent directional information about the contained media. Mogels 210 may be used to define material gradients such as 3D "textures." Plenoptic elements that are not uniform across their bounding voxels, but are of non-uniform composition, are sometimes referred to as mixed elements or "mixels."

[0042] Yet another type of medial may include media sandwiched between one or more other types of media (usually, but not necessarily, homogeneous media) within the medial, i.e., a "sandel." A sandel occurs when the system determines that the medial contains multiple surfels in opposing or partially opposing directions after decomposing one or more surfels. An example of a sandel is a medial that includes all or part of a side of glass. In the previous example, the surface of the glass represents the interior surface element in the sandel, and the air on either side of the glass represents the homogeneous media on either side of the "sandeled" glass surfel. Sandels can provide opportunities for savings in data, power, or processing. For example, the use of sandels may allow processing to be completed with a coarser medial size than would be possible using only other types of medials. Such savings can be achieved by specifying multiple surfaces within the medial within a single medial, rather than subdividing the medial into separate surfels for each surface during scene reconstruction. Sandels can also allow for lower bandwidth, power, or processing during output and / or presentation of the scene reconstruction by similarly coarsening the medial size. For example, if an example glass is 0.25 inches thick, a cubic sandel 0.5 inches in size can represent the air on one side of the glass, the glass and both surfaces, and the air on the other side of the glass. If instead the system is configured to use only surfels, mogels, and mixels, it may be necessary to subdivide the medial at least once more to create at least two additional medials to represent the surfaces and homogenous media.

[0043] Three types of medials and examples of their use are shown in Figures 4 and 5. Figure 4 shows an example type hierarchy of medials 401, which can have surfels 402, mogels 403, and mixels 404. Various embodiments of the invention may include all or any combination of these elements, or none of them. Figure 5 illustrates an example sheet of glass 501 represented by a set of voxels containing medials 502, depicted as multiple boxes in the figure. For visual clarity, the figure shows only a few primitives in a few voxels. In a typical digital model of a real-world scene, primitives are densely populated throughout the scene and occur at several different levels of resolution in data structures that store information related to matter fields.

[0044] As illustrated in Figure 5, surfels 503 may contain multiple types of materials. In the figure, surfels 503 contain both glass and air with a surface separating them, mogels 504 contain only glass, and mixels 505 contain multiple surfaces by representing the corners of a plate. Medials in general can contain property information in various forms. For example, surfels and mogels can contain BLIF values ​​or other property information that can be used for relighting. In some cases, mixels may contain information to make them relightable.

[0045] The properties of BLIF are described elsewhere, such as in FIG. 10 of U.S. Patent No. 10,521,952 to Ackerson et al. BLIF can characterize incident, emissive, responsive, and / or outgoing light fields. FIG. 10 of U.S. Patent No. 10,521,952 shows an exemplary model that can be used to represent interactions occurring in a single medium, where the medium is composed of voxels and associated BLIF. The medium is struck by radials of the incident light field. The BLIF acts on the incident light field, resulting in a responsive light field that emanates from the medium. The total outgoing light field is the sum of the responsive light field and the (optional) outgoing light field. The emissive light field is emitted from the medium independent of stimulation by incident light.

[0046] In some embodiments, the inventions described herein can use non-plenoptic primitives that include, for example, analytical information. Such non-plenoptic primitives can represent elements of a scene other than medial and radial, and typically do not include information related to the interaction of light and matter in the scene. Examples of such non-plenoptic primitives include, but are not limited to, computer-aided drawing (CAD) or similar structures that represent spheres, cones, or other shapes fitted to local groups of surfels, computer vision or other scale-invariant feature transform (SIFT) style features formed by patterns of pixels in an image, or other information.

[0047] Each of the aforementioned elements or parameters can be optionally configured to be expandable into multiple finer parameters and / or collapsible or combinable into a single parameter, a smaller set of parameters, and / or coarser parameters. This configuration is optionally applicable to all types of elements or parameters, including plenoptic, analytical, sampled, and learned parameters and elements. For example, voxels and / or surfaces may be subdivided or multiple voxels and / or surfaces may be combined. Similarly, the overall diffuse reflectance may be subdivided into polarized and unpolarized diffuse reflectance. As another example, the Phong reflectance model may be extended to a set of sampled BLIF coefficients (e.g., ratios) stored in a hierarchical surface data structure of outgoing-to-incoming radiance ratios for pairs of directional surfaces. This example is further described herein with respect to FIG. 20, where the analytical BLIF can be extended to a coarse analytical plus fine neural network for greater accuracy in predicting outgoing radiance.

[0048] With reference to FIG. 3A, an embodiment of the present invention is operable to reconstruct a plenoptic field, including using an incremental process, where the plenoptic field can represent the entire scene, a portion of the scene, or a particular object or region of interest within the scene. In an embodiment, the system can first determine (301) settings for the reconstruction of the scene. For example, the system may access or set a working resolution, an initial size, a target accuracy, relighting characteristics, or other characteristics. In an embodiment of the present invention, the system may give the scene an initial size. The size of the scene may be, for example, the scale of a human living space for an indoor scene, a different size for an outdoor scene, or another size defined by the system, the user, or other factors that may be determined to be acceptable or advantageous. In an embodiment, including the exemplary embodiment shown in FIG. 1B, a first camera 105 or set of image data defines the origin of the scene, and subsequent camera images captured by either the camera 105, the second camera or image sensing device 106, or other may be added to the scene or processed.

[0049] An embodiment of the invention may then initialize (302) a data structure for storing the scene, which in an embodiment includes a plenoptic field, as further described herein with reference to, e.g., FIG. 3B. An embodiment of the invention may begin storing data in the data structure at a coarse level of granularity. An embodiment of the invention may store further data related to the scene in the data structure, including finer levels of detail iteratively. An embodiment of the invention may also be configured to calculate or refine (303) characteristics of the scene, which may include calculating or refining characteristics of the plenoptic field, as further described herein with reference to, e.g., FIG. 3C. In certain embodiments, the system may be configured to use (304) a termination criterion, computational budget, or other factor to guide reconstruction activities. In embodiments having such a termination criterion (304), if the criterion is met, processing may terminate, otherwise the system may determine (305) whether new image data is available. If new image data is available, the system may be configured to incorporate (306) the new data, as further described herein with reference to, e.g., FIG. 3D. After incorporating the new image data (306), or if no new image data is available (305), the system may repeat the process beginning at step 303 until finished.

[0050] With reference to FIG. 3B, an embodiment of the system can store the material field and / or light field associated with the scene in a data structure. In an embodiment, the data structure can take a form known in the art, including hierarchical, multi-resolution, and / or spatially sorted data structures. Exemplary data structures include bounding volume hierarchies, tree structures, binary space partitioning, or other structures capable of storing image data in an accessible manner. In an embodiment, if the scene is partitioned into one or more plenoptic elements as described herein, the data structure may be so configured. Furthermore, the data structure may be configured such that information related to one aspect of the data structure (e.g., the material field) is associated with one or more other aspects of the data structure (e.g., one or more of the camera pose, light field characteristics, and segments).

[0051] In particular in embodiments where the invention is configured to reconstruct and / or store a matter field, the invention may be configured to initialize (311) a data structure to store the matter field in the scene. Initializing the matter field may include preparing the data structure to store one or more of size, shape, position, and / or light interaction properties for the matter in the scene. In embodiments where the matter field is divided into one or more voxels or medials, the data structure may be configured to store information associated with each of the voxels or medials. In some embodiments, the data structure may be initialized to assume a particular type of medium associated with the medium, which may be any of a number of homogeneous media (e.g., air, water, fog, turbid water, or other homogeneous media). Certain embodiments of the invention have access to prior information related to the matter field, which may include one or more of information describing the geometry of the scene or objects therein (e.g., an OBJ file characterizing a room and its contents), values ​​of parameters in low dimensional parametric BLIF, values ​​of parameters and / or coefficients in high dimensional sampled BLIF, and / or any combination of geometry (position and / or orientation) and / or BLIF information for some or all of the scene. In embodiments in which the invention is not configured to reconstruct and / or store matter fields, these processes may not be necessary.

[0052] Certain embodiments of the invention may also be configured to initialize a data structure to store information regarding one or more camera poses (312). In some embodiments, the data structure may store information regarding assumed or known positions of one or more images of the scene, which may be correlated with other aspects of the data structure, such as one or more voxels or medials.

[0053] Some embodiments of the invention may be configured to initialize a data structure to store 313 information about the light field in a scene. The data structure may be configured to initially store information about both incoming and outgoing light associated with various points, locations, or voxels in space, including their relationship to portions of the data structure associated with the matter field. Such information may be represented as one or more radials associated with each location or voxel / medial. Certain embodiments of the present invention may have access to prior information related to light fields, which may include information describing a quantification of the incident light field at a point in position space (e.g., a panoramic "environment map"); surface light fields (e.g., a 4D light field) that quantify the incident and / or outgoing light fields in one or more directions at one or more points in position space, presumably at a distinct physical surface; surface light fields (e.g., a 2D light field) that quantify isotropic (or nearly isotropic) incident and / or outgoing light fields at one or more points in position space, presumably at a distinct physical surface; and / or any combination of incident and / or outgoing light field information for some or all of the plenoptic space of the scene.

[0054] Embodiments of the invention may also be configured to initialize a data structure to store 314 information about one or more segments in a scene. A segment may represent one or more groups of media in a scene. In some embodiments, a segment may represent media that has a particular relevance probability (e.g., a suitably high relevance confidence). For example, if a scene includes a vase of flowers, a segment may represent the leaves or petals of the flower, the entire flower, the vase, the vase containing the flowers, etc.

[0055] In some embodiments, the invention may be configured to initialize 315 a data structure to store other data related to the scene, such as non-plenoptic information representing analytical information, etc.

[0056] While the steps described above may be performed in any order, certain embodiments of the present invention perform the steps in the order described herein. For example, initializing a data structure to store a material field first helps to associate a radiel with a more relevant (or most relevant) home medial. Similarly, initializing a data structure to store information related to one or more camera poses before initializing a data structure to store information related to a light field can enable initializing a radiel with more relevant (or most relevant) positional and / or orientation information.

[0057] With reference to Figure 3C, some embodiments of the invention may provide for calculating or refining one or more characteristics of a scene (e.g., the plenoptic field). Some embodiments of the invention may process camera images or other image data in a sequential, parallel, or combination of the two manners. In some embodiments, the system may calculate (321) a light field in the scene based on the image data, as described herein with reference to Figure 3E, etc.

[0058] The system may calculate or refine information about one or more poses associated with the image data (322), such as described with reference to FIG. 3F. In some embodiments, the system may determine whether the viewpoint of one or more cameras or image data, including the light field of the voxel, has changed (322), which may be determined based on a significance threshold that may be preset or calculated by the system. This determination may be based, in part, on the system assuming, or having other information indicating, that a camera image or set of image data exists at the voxel 201 in the data structure as shown in FIG. 2. In such an embodiment, for each assumed position, the system may assume an orientation in a coarse orientation space.

[0059] In particular in embodiments where the system is configured to reconstruct and / or store a matter field, the system may be configured to access and test (323) one or more (or all) medial, including the light field of the changed voxel. In some embodiments, if the light field associated with the medial has changed by an amount (including a threshold set in the system, set by a user, or calculated by the system), the system may be configured to calculate or refine the medial, such as described with respect to FIG. 3G. In some embodiments, the system may be configured to calculate (324) one or more segments in the scene. Some embodiments of the system may be configured to calculate or refine (325) other data associated with the scene, such as non-plenoptic and / or analytical information. In embodiments where the invention is not configured to reconstruct and / or store a matter field, these processes may not be necessary.

[0060] The system may be configured to include (326) certain termination criteria, computational budgets, or other thresholds, including those related to the calculation or refinement of the plenoptic field. In such embodiments, the system may determine whether the termination criteria, computational budget, or other thresholds have been exceeded, as described elsewhere herein. If the thresholds have not been exceeded, the system may be configured to repeat the process, for example, beginning with step 321. If the thresholds have been exceeded, the system may complete the process.

[0061] With reference to FIG. 3E, a system may be configured to compute and / or refine a light field in a scene. Some embodiments of the invention may perform a light transport process (341) to compute the light field. For example, the system may compute radiance propagation through a default medium and / or pass the incident radiance through a BLIF to obtain the outgoing radiance. The light transport process may be optionally limited to radii that exceed a threshold change from a previous state. The light transport process may be limited to a path length at most. The path length may be determined automatically, for example, by a confidence or change in confidence, and may be performed in any combination of downstream and upstream directions. The light transport process may include any combination of downstream (forward in time) radiance propagation through a default medium, upstream (backward in time) radiance propagation through a default medium, incident radiance passing through a BLIF to obtain the outgoing radiance (downstream BLIF operation), and / or outgoing radiance passing through a BLIF to obtain the incident radiance (upstream BLIF operation).

[0062] For example, light transport may act in a downstream direction on a semi-gloss paint surface. In such a case, the incident radiel may have an updated, more reliable radiance value in at least one of its color wavebands. The more reliable radiance value may prompt downstream (advancing in time) BLIF interactions, which result in one or more new radiance values ​​in one or more radiels exiting the surfel. As another example, light transport may act in an upstream direction on a shiny chrome surfel. In such a case, the outgoing radiel may have an updated, more reliable value (e.g., radiance value in at least one of its color wavebands). Such a situation may occur when a new camera viewpoint is added that looks directly at a chrome surfel. The new outgoing radiance may prompt upstream (advancing in time) BLIF interactions, which result in new radiance values ​​for one or more radiels incident on the surfel. In other embodiments, light transport may occur in both time directions, such as after providing image data representing a new image at a viewpoint of a room. Pixels of the new image may be resampled to the reliable incoming radiel in the voxel that contains the viewpoint. That incoming radiance can propagate upstream to alter less reliable outgoing radiances at surfels in the scene (e.g., at the wall the camera is looking at in its field of view). Additionally, the incoming radiance at the camera's viewpoint voxel may optionally be computed to be an antipodal outgoing radiance, which may then propagate downstream to alter less reliable incoming radiances at surfels in the scene, e.g., at surfels on the wall behind the camera.

[0063] The calculation, refinement, and / or update of the light field (radial) and the matter field (medial) may be separate steps, although this configuration can be changed at will. In some embodiments, such a configuration may result in an unwanted avoidance of the globally optimal (deepest) solution in the overall cost function space of the (sub)scene to be reconstructed. For example, a particular calculation may reach a solution of some accuracy, but no longer approach the optimal solution (e.g., the globally optimal solution). Such a situation may arise, for example, when iterating between "light field only" and "matter field only" search directions in a multidimensional cost function space, which may avoid the optimal parameter step direction, where both the light field and the matter field parameters change simultaneously. An exemplary solution to this problem, which is optionally performed by the system, is to use the recognition that the assumed scene model is revisiting the same state in a limit cycle, which may occur at the level of a subscene within the entire scene. Upon recognizing the presence of a limit cycle, the system may revert to a faster and / or coarser search regime and proceed after modifying one or more control parameters of the search, such as the next region of parameter space to search and / or the step size in each or a particular parameter dimension, etc. The system may then follow known methods for escaping the limit cycle.

[0064] Such operations may optionally use actual values ​​and / or changes in values ​​of radiance, other radial properties, and / or confidence (consistency) of radial properties to determine when to terminate the sequence of operations. For example, the system may be configured to include (342) certain termination criteria, computational budgets, or other thresholds, including those related to light transport depth, that reflect an iterative and / or recursive sequence of calculations. In such an embodiment, the system may determine whether the termination criteria, computational budget, or other thresholds have been exceeded, as described elsewhere herein. If the thresholds have not been exceeded, the system may be configured to repeat the process, for example, beginning with step 341. If the thresholds have been exceeded, the system may complete the process.

[0065] With reference to FIG. 3F, the system may be configured to calculate or refine a camera pose associated with a particular image data. In one embodiment, the system may be configured to create (351) a trial copy of the scene or a relevant portion thereof. The system may determine (352) a hypothetical camera pose. In one embodiment, the determination of the hypothetical camera pose may be from a coarse to fine order in a parameter space that defines the possible camera poses. The system may determine (353) whether the hypothetical camera pose is outside the scene boundary. If the hypothetical pose is outside the scene boundary, the system may increase (354) the size of the trial copy of the scene to accommodate the hypothetical camera pose.

[0066] If the hypothetical pose is within the scene bounds, or after the scene bounds have been expanded, the system may compute or refine (355) a trial copy of the scene, such as using the process described with reference to Figure 3C, such as in step 322. Such computations may be performed with a low-cost computational budget and / or recursive camera pose refinement may be skipped.

[0067] In some embodiments, the accuracy of the camera pose can have a significant impact on the accuracy of the scene. The system may be configured to analytically represent the camera pose with greater accuracy than the individual spatial elements that comprise the camera's viewpoint. For example, the camera pose at a given viewpoint can be represented as a floating point number, rather than splitting the voxels that comprise the viewpoint into several levels in a position hierarchy. Similarly, the camera's orientation can be represented using a floating point number that includes parameters such as yaw, pitch, and roll, rather than using a separate surface data structure to represent these features.

[0068] The system may be configured to include (356) certain termination criteria, computational budgets, or other thresholds, including information related to the camera pose. In such embodiments, the system may determine whether the termination criteria, computational budget, or other thresholds have been exceeded, as described elsewhere herein. If the thresholds have been exceeded, the system may replace (357) the plenoptic field or relevant portions thereof with a trial copy, complete processing, or both. The system may be configured to repeat the process, for example beginning with step 352, if the thresholds have not been exceeded.

[0069] In an embodiment, the system can perform a calculation to determine whether there is a measurable disparity. The calculation of measurable disparity may be calculated based on the change in received light, and may also be based on the resolution of the camera or image data and the positional separation between the viewpoints. In an embodiment, the two quantities mentioned above can set realistic disparity boundary distances in different directions outward from the camera / image data workspace or the envelope of the camera viewpoints. The disparity boundary is often directional, meaning that the boundary may be different distances in different directions depending on the shape of the camera / image data workspace. For example, if the distance between the viewpoints in one direction is long, the disparity boundary may be pushed further outward in the plane perpendicular to that viewpoint separation vector.

[0070] In some embodiments, the system can use the disparity boundary to set the size of the scene (e.g., the outer boundary of the scene). In some embodiments, the disparity boundary can be a bounding voxel that encloses an envelope of the directional disparity boundary distance. In some embodiments, the camera / image data workspace can grow, pushing the disparity boundary outward, e.g., as new image data is accessed. The system can increase the size of the scene and / or associated plenoptic field in response to the expansion of the disparity boundary.

[0071] In some embodiments, the system may represent scene information beyond the disparity boundary in a two-dimensional nature. For example, such information may be represented as a two-dimensional map and / or multi-resolution grid of volumes. As an example, a night sky may be represented as a two-dimensional light field. Also, depending on the size of the scene, there may be an airplane high above the ground in a two-dimensional layer beyond the disparity boundary. Information beyond the disparity boundary need not only include light, but may have other associated properties. For example, in the night sky example, the moon may be represented as moon dust in a two-dimensional matter field with associated BLIF. Furthermore, any number of layers may be stacked in any order based on known distance or other priority. In some embodiments, the light field and / or matter field information or data in such layers may be time dynamic in the same way that information within the disparity boundary is time dynamic.

[0072] With reference to FIG. 3G, the system may be configured to calculate one or more medial attributes. The system may first use the image data to calculate 361 medial attributes. In some embodiments, the system may be configured to calculate medial attributes such that adjustments are made to increase the overall confidence metric calculated for the medial (e.g., associated medial with a low current confidence may be updated with a medial with a higher confidence). For example, the overall confidence metric for a medial may be calculated by taking a newly predicted outgoing radiance in one or more directions of interest (e.g., directly towards the camera viewpoint), subtracting the existing baseline and / or observed outgoing radiance from the predicted radiance, and calculating some variance of the average over the radiance deviation between the predicted and baseline values ​​to calculate a scalar cost and / or error. In such embodiments, the confidence may be considered the inverse of the cost (e.g., 1-cost or 1 / cost).

[0073] In some embodiments, the system may assume or otherwise determine that a surface exists in the media. Media containing surfaces may be referred to as surface elements or surfels. In such embodiments, the system may be configured to retrieve geometric parameters associated with the surfaces. For example, the system may calculate surface normal vectors, BLIFs, or other dimensions or parameters associated with surfels.

[0074] In embodiments in which the system tests whether a voxel in the data structure is a surfel, the system may calculate the outgoing light at the voxel from one or more of the cameras 105 and 106 or other image data that includes the voxel in a field of view captured by the cameras or represented in the data. In an embodiment, each camera may observe the voxel, or each set of image data may represent the voxel from a different angle and observe a particular radiance emanating from that position in space in a particular direction. The system may use one or more of these observations to determine the observed outgoing light field for that position in space or voxel. Similarly, the system may calculate the incident light field at that position in space or voxel from one or more of the cameras or sets of image data that observe light incident on that position in space or voxel.

[0075] In some embodiments, the system may be configured to calculate (362) properties associated with the light field of the medial, such as the directional resolution of the light field of the containing voxel. For example, when the directional resolution of the light field of the containing voxel is calculated and / or adjusted, the system may split and / or combine radii associated with the voxel or neighboring voxels. Certain embodiments of the invention may adaptively sample the light field associated with the medial. For example, the system may adaptively sample the plenoptic light field at the medial using one or more data sets, such as a hypothesized BLIF, an exit direction of interest (e.g., a camera viewpoint), or other data associated with the medial under test. Some embodiments of the invention may perform such calculations based on a confidence target for the exit plenoptic light field (e.g., based on observed light present in the medial) or a computational budget (e.g., a maximum number of radiometric elements (or radii) associated with the medial). In some embodiments, the system may be configured to use spherical harmonics to represent the directional resolution. For example, in embodiments in which the system characterizes a light field associated with glossy painted surfels, such surfels may behave very specularly in real scenes. In initial processing, a search for candidate BLIF properties indicates that surfels are more likely to be specular than diffuse. The system may then be configured to instantiate higher order spherical harmonic coefficients to generate tighter specular lobes in directions of high incident radiance originating from other scene regions. The spherical harmonics may be defined in a coordinate frame that rotates with an assumed normal vector. For materials with a grainy appearance (e.g., brushed metals), the system may be configured with material grain tangent vectors associated with anisotropic BLIF.

[0076] An embodiment of the present invention may assume that a surface exists in one or more media, as illustrated in FIG. 2. By assuming that a surface exists in a voxel, the system may further assume a particular surface normal 203, a displacement of the surface in the media, and / or light interaction properties of the surface. The system may then calculate a set of predicted exit radiance vectors based on the assumed surface normal and / or light interaction properties, which may include one or more of refractive index, roughness, polarized diffusion coefficient, unpolarized diffusion coefficient, and / or extinction coefficient, with the light interaction properties being particularly applicable to metallic materials. In an embodiment, the system may be configured to search for one or more of the aforementioned properties in a serial manner (e.g., by following a "waterfall" of tests based on most to least likely correctness based on the applicable image data).

[0077] An embodiment of the invention may calculate a "cost" for the presence of a surface. In an embodiment, the cost for the presence of a surface with a calculated property may be expressed as the difference between the predicted or calculated (i) surface normal, (ii) light interaction value, (iii) outgoing radiance vector, and / or (iv) other property and the corresponding observed value. In an embodiment, the system may have a specified or specifiable cost threshold, and if the cost is below the threshold, the surfel is accepted as being present in the voxel. In such an embodiment, when a voxel is determined to be material and / or to contain a surface, the surface may remain in the scene for subsequent iterations. In an embodiment, the surface normal may be searched in a hierarchical manner that matches a data structure that stores the surface normals or radials. For a cubic shaped voxel, the system may perform the calculation for each of the six faces. The system may further be configured to divide the voxel into eight cubic shaped sub-voxels, resulting in the need to calculate surface normals for a total of 24 exterior facing faces, and 96 overall sub-radials. For each direction, the system may be configured to calculate a predicted exit radiance vector and associated surface normals and / or light interaction properties. Such processing can be accomplished in many ways, including in highly parallel or multi-threaded ways using GPUs, AI, and / or ML, binary tree hierarchies, or other constructs to speed up processing. In an embodiment, the system may determine the most likely, lowest cost, most confident, or other population assumptions and use those assumptions as a starting point for processing during subdivision.

[0078] In some embodiments, the solution of the light field at the voxel 201 and / or the presence of a surface may use the light fields calculated for one or more other voxels to calculate hypothetical incoming light to the voxel 201, outgoing light from the voxel 201, and / or other radiance properties of the voxel 201. Such calculations may be added to or calculated together with the incoming light field represented by a camera image or other set of image data. Thus, certain embodiments of the invention may update the light field assumptions for one or more voxels 201 by using one or more projected radiometric elements or radii emanating from one or more other voxels by tracing the influence of the radii through the scene and / or the interaction of the radii with the medium in the other voxels.

[0079] In some embodiments, the system may first compute a light field associated with the scene and provide information about the light field to inform the process for detecting the presence of surfels. In computing the presence of media in the scene represented by surfels, the system may use the presence as a factor in recomputing the light field associated with the scene. This process may be performed in any order and may be performed iteratively to improve reliability or reduce costs associated with the light field and / or material field of the scene. In some embodiments, the system may perform this process prior to subdivision of a medial into multiple sub-medials. Furthermore, upon subdivision of a medial, the system may be configured to perform similar tests of the light field and / or material field, which may be based in part on the parent medial, and then perform similar processing as described above with respect to the parent.

[0080] In some embodiments, the above process may continue until the system reaches a particular termination criterion, computational budget, or other threshold, including the light field and / or matter field associated with the medial (363). In such embodiments, the system may determine whether the termination criterion, computational budget, or other threshold has been exceeded, as described elsewhere herein. The system may be configured to repeat the process, e.g., beginning with step 361, iteratively and / or recursively, if the threshold has not been exceeded. If the threshold has been exceeded, the system may determine (364) whether one or more medial attributes exceed a confidence threshold. If the confidence threshold has been exceeded, the system may complete processing. If the confidence threshold has not been achieved, the system may optionally subdivide (365) the medial into N children, as described below, unless a resolution budget or limit has been reached.

[0081] An embodiment of the present invention may then use a confidence threshold or other metric to guide processing and calculate a confidence or other metric associated with each medium or other volumetric element in the scene. If a confidence threshold is used, the system may examine one or more media whose confidence is below the confidence threshold. In an embodiment, if the confidence is below the threshold, the system may then compare the properties of the medium to various known light interaction properties, such as two-way light interaction functions (or BLIFs) associated with different types of media. For example, in the example illustrated in FIG. 14A, if the confidence threshold is 75, the system may be configured to perform further calculations for each of the four illustrated mogels 1403 because the associated confidence is below 75. An embodiment may use a waterfall, or sequential, comparison order based on what the system calculates to be the most likely candidate properties (e.g., the most likely candidate BLIFs) for a particular medium. For example, for a particular medium, the system may first test whether the medium contains air, then test whether it contains a common dielectric medium, followed by testing whether it contains a common metallic medium, and so on.

[0082] As illustrated in FIG. 14A, in some embodiments, the region of interest may include both homogeneous transmissive media and opaque media. In some embodiments, where the region of interest includes free space (e.g., air or other homogeneous space that does not include a medium that substantially interacts with light), the system may specify in the data structure that the scene is composed of a medium that includes free space (e.g., air). An empty medium or a medium that includes air and other homogeneous elements may be referred to as a mogel. In some embodiments, even if the medium 1401 includes an opaque surface 1402, the system may initially specify that the medium 1401 is composed of one or more mogels 1403 that include free space or air (or air mogels), and such an initial setting may allow the system to allow light to flow through the medium 1401 and the mogels 1403, rather than being assumed to be blocked by an interacting medium such as 1402. Some embodiments of the present invention may specify a low confidence level 1405 associated with each air mogel, which may facilitate the system to later determine the presence of other media within each air mogel. In FIG. 14A, hypothesis content 1404 and confidence 1405 are shown, where content "A" 1404 represents an initial hypothesis of an airy mogel 1403, and confidence "10" 1405 represents a hypothetical confidence value for that hypothesis.

[0083] The system can determine the confidence (or cost) associated with a particular medium in a number of ways. For example, for a medium assumed to be empty or air, the system can predict that the difference in radians incident on the medium should be substantially equal to the radians exiting in the opposite direction from the medium. Similarly, if the medium is a surfel, the system can predict a particular relationship between the light field incident on the medium and the light field exiting from the medium based on the characteristics of the particular medium containing the surfels. Thus, the system may be configured to determine the confidence associated with a particular medium by calculating the error between the predicted difference between the incident light and the actual difference between the exit light (e.g., the polar error of the radians for a mogel assumed to be air, or for a mogel of air).

[0084] In some embodiments, the invention may utilize machine learning (ML) and / or artificial intelligence components to assist in determining the confidence (or cost) associated with the difference in radiels, as described elsewhere herein. In the example of an air mogel, the AI / ML model may be configured to determine the confidence or cost that a medial is an air mogel by comparing the calculated results to the difference in the opposite radiels. In various embodiments, the AI / ML model may be configured to compare the mean, median, minimum, maximum, and / or other differences of the various calculated radiels. In some embodiments, the AI / ML model may be configured to discard a selected or configurable number or percentage of radiels (e.g., a certain percentage of the most inconsistent radiels) when making the confidence or cost determination.

[0085] An embodiment of the invention may perform the aforementioned calculations in an iterative and / or recursive manner. For example, in an embodiment, the system may calculate the scene data as described herein for a particular camera pose or set of image data, including finding any surfels 202 therein. The system may then process subsequent camera images or sets of image data in a similar manner. If such an iterative process finds more surfels 202 in the scene, the system may more accurately determine the orientation of subsequent camera poses and / or sets of image data. In an embodiment of the invention, information related to one or more of the pre-determined camera poses and / or orientations of the sets of image data may also be updated to better accommodate new observations. In an embodiment, the camera data structure may be hierarchically traversed from coarse to fine levels as new camera images are positioned and / or image data from alternative viewpoints is accessed and existing camera image positions and / or orientations of the sets of image data are updated. As more camera images and / or sets of image data become available, the system may resolve the material fields present in the scene to greater levels of detail. In some embodiments, the present invention allows for finer levels of detail to be stored in the material field data structure, including subdividing the material field data structure. The result in some embodiments of the present invention is a set of camera images, camera pose, other image data, and / or information regarding the orientation of any of the foregoing, a material field calculated by the system, and an associated light field. Such a result is preferably data that best describes the image provided by the light transport calculations, and may be referred to as a reconstructed scene.

[0086] In some embodiments, the invention may perform the above calculations for other medials (or all medials) within the region or scene of interest. The system may then compare the results for one or more medials to a confidence threshold or other metric, for example based on the predicted radiometric characteristics minus the observed characteristics associated with the medial's outgoing radiometric characteristics. For medials that do not reach the confidence threshold or other metric, the system may be configured to perform further processing associated with such medials. For example, FIG. 14B illustrates a situation where the system determines that the mogel 1403 at the bottom right of the medial 1401 does not meet the appropriate threshold. In some embodiments, the system may subdivide such medials that do not meet the threshold or other metric into two or more child medials, such as splitting a cube-shaped medial into eight cube-shaped child medials. In FIG. 14B, the system has subdivided the mogel 1403 into four child medials 1406, each with an associated content hypothesis 1407 and confidence 1408. In the embodiment illustrated in Figure 14B, the system assumes the sub-medium 1409 to contain a surface, such as an opaque dielectric surface represented by surfels, as shown at 1413, labeled with an "S" with confidence level of 50. The remaining sub-medium remains assumed to contain air with varying confidence levels.

[0087] During such subdivision, the system may be configured to perform the above-described process to determine BLIF or other characteristics associated with the region or scene of interest until a confidence threshold, other metric, or maximum computation threshold is reached. In an embodiment, the system may determine that the confidence level of the medial has not changed substantially during subdivision. In such a case, the system may be configured to determine a local minimum threshold based on an asymptotic determination, which may be determined, for example, in a traditional cost function minimization problem. For example, referring to FIG. 14, in a hypothetical situation where the confidence threshold was set to 75, the system further subdivided sub-medials 1409, 1410, and 1411 because they did not exceed the threshold, but did not further subdivide medial 1412 because its confidence exceeded the threshold. Upon completion of the process illustrated in FIG. 14C, all sub-medials except sub-medial 1413 met their associated thresholds. Specifically, all sub-medials within sub-medials 1410, 1411, and 1412 exceeded the confidence threshold as air. All of the sub-media within sub-medium 1409 exceeded the threshold as containing a surface, except that the system determined that it could not further decompose sub-medium 1413 because the material within sub-medium 1413 was completely occluded by the remaining sub-medium within sub-medium 1409.

[0088] In some embodiments, the system can be configured to assume a higher confidence or lower cost for a particular medium based on the confidence or cost associated with adjacent mediums. For example, the system can be configured to assume a higher confidence or lower cost associated with the presence of a surface in a medium if one adjacent medium contains a surface, and an even higher confidence or lower cost if two adjacent mediums contain a surface. Similar assumptions can be made for other types of media (e.g., mogels that contain a particular type of media in adjacent mediums or in one or more empty or air media adjacent to the medium in question).

[0089] With reference to Figure 3D, the system may be configured to capture new image data. In one embodiment, the system may initialize (331) one or more new camera poses, which may be performed, for example, as described with reference to Figures 3B and 3F. In one embodiment of the invention, one or more new radials may then be placed (332) into the scene at voxels that include one or more new viewpoints.

[0090] An embodiment of the invention may select the location and orientation that results in the lowest total cost for the hypothetical surfels to the best pose of the input camera in this iteration of the algorithm. For example, the system may assign a level of confidence (or conversely, a cost for lack of confidence) to a particular location or orientation. Such a level of confidence may be optionally related to radiometric properties such as the presence or absence of a surface, the incoming or outgoing light field, or another property of the location or orientation. The system may determine the confidence or cost based on a number of factors, including the proximity or lack of a location to the camera pose orientation, the consistency of the observation with data from other camera poses, or other information. In an embodiment in which the information is a radial relating to a location in space as an example, a higher confidence or a lower cost may be assigned to a location that is directly related to the observation from the camera. Similarly, a lower confidence or a higher cost may be assigned to a location that is not closer to the location observed by the camera or depicted in the image data. In an embodiment of the invention, the confidence or cost may be used as a weighting factor. In such a manner, the system may be able to determine position, orientation, or other information in scenes where there is high consistency and / or low cost, low consistency and high cost, or something in between.

[0091] In some embodiments, the above processing may result in one or more local regions in a data structure storing information related to the matter field and / or light field being subdivided to a much finer or deeper resolution. Such finer subdivision may be triggered by local contrast such as abrupt changes and / or gradients in voxel occupancy, media type, BLIF, geometry, or other characteristics. Local regions of finer and / or deeper resolution may occur in the matter field (e.g., at specific positions, voxels, locations in the data structure) and / or light field (e.g., at specific directions, surfaces, locations in the data structure). As an example, such a local region may occur in the matter field in the middle of a large monochromatic wall represented by small dots of finely subdivided surfels. As a second example, such a local region may occur in a light field where direct sunlight is represented by finely subdivided radials in a light field incident on shiny chrome surfels. After reflecting off the chrome surfels, the resulting outgoing light field may have finely subdivided surfels in the specular direction relative to the incoming sunlight.

[0092] In some embodiments, there may be localized regions (e.g., sub-scenes) where extra computational energy is applied to reconstruct certain scene characteristics with higher accuracy than the rest of the scene. For example, the system may be applied to reconstruct an entire room, but there may be one object in the room that is of particular interest. Extra computational energy may be applied to the object of interest. Some embodiments of the invention may apply extra computational energy to regions that are the same or similar to regions where the data structure has been subdivided to a much finer and / or deeper resolution. For example, for a monochromatic wall with small dots, it may be advantageous to reconstruct the location of the wall surfels with an accuracy of 1 mm in the direction perpendicular to the wall, but the wall surface may have a homogenous BLIF and be represented as 10 cm wide surfels.

[0093] Those skilled in the art will recognize that the processes described herein can be used in combination with, as part of, or to enhance various scene reconstruction techniques known in the art. For example, various methods for performing scene reconstruction are taught in U.S. Patent No. 10,521,952 to Ackerson et al. For example, the processes described herein with respect to Figures 3A-3G and 14A-14C may be incorporated in whole or in part at various points in the processes described in U.S. Patent No. 10,521,952, including portions of the processes described with respect to Figure 3 (e.g., at steps 309, 311, and / or 313), Figure 14 (e.g., at step 1403), Figure 16 (e.g., at step 1609), Figure 18A (e.g., at steps 1811, 1813, 1815, and / or 1819), Figure 18B (e.g., with respect to step 1819), and Figure 18D (e.g., at step 1880). Accordingly, certain embodiments of the invention described herein are intended to be fully compatible with the processes taught in US Pat. No. 10,521,952 and other scene reconstruction techniques.

[0094] Additionally, in some embodiments of the present invention, reconstruction of the opaque exterior structure of an object or scene can be combined with reconstruction of the interior structure of the same object or scene (including interior reconstructions created with different methods, such as X-ray images or MRI scans), as shown in Figure 21. By nesting the interior structure within the exterior structure, a more complete model of the object or scene can be formed. In some embodiments, if the method used to reconstruct the interior structure lacks BLIF information, the BLIF information can be generated automatically using methods such as machine learning based on the BLIF of the exterior structure.

[0095] Certain embodiments of the invention can be used to represent tubular structures, 3D corners, or other surfaces. In some embodiments, a surface may be determined to be curved based on prior knowledge or a posteriori segment (region) data, each of which may optionally be represented by surfels. In some embodiments, surfels may have maps aligned to tangent vectors. Such maps may optionally represent various properties of the surface, such as roughness (bump map), color (texture map), material, and / or other properties. In some embodiments, the material gradient along the normal may be a step function, or may be more complex (e.g., "fuzzy" surfaces, or multi-layered surfaces like a clear-coated car), or represented in other ways.

[0096] 6 and 7 exemplarily show a solid tubular structure (e.g., a tree branch) represented using surfels. As illustrated in FIG. 6, which shows a view along a curve with several surfels 601, the surfels 601 in a voxel 603 are shown as planes, but are stored as normal and tangent vectors. The boundary 602 of the tube can be represented by an analytical curve 604. FIG. 7 shows a side view of an analytical curve 701 of a curved object with representative surfels 703 illustrated in a voxel 702. In such an embodiment, the surfels can be, but are not necessarily, recognized as planes. FIG. 8 exemplarily shows the use of a surfel 801 in a voxel 802 to represent a corner 803. The corner 803 can be a single point-like feature, which can be a typical analytical feature that can be discovered during the bundle initialization process.

[0097] Some embodiments of the invention may retain data relating to surfels or medial in a priority order designed to optimize system performance. For example, certain embodiments of the invention may retain information in descending priority order of image or other digital image information, point-like and / or orientation features, medial primitives, outgoing and incoming light fields (which themselves may be accurate in observed directions, interpolated, or only interpolated in non-observed directions), as well as geometry, bump maps, and textures. Other embodiments of the invention may use alternative priorities, omit one or more categories of the foregoing information, and / or include one or more other categories of information. In some embodiments, the invention may retain all of the foregoing information, discard higher level information when lower level information is available or explainable, or a combination of the two.

[0098] In certain embodiments of the invention, the system can capture, receive, process, use, and / or display certain analysis primitives within a scene. Analysis primitives can optionally include one or more of the following types of data or other types of information obtained from or provided about a scene: points, vectors, lines, surfaces, spheres, rectangles, parallelepipeds, meshes, other CAD-like models or features including constructive solid geometry (CSG) and / or boundary representations (B-rep), and / or other information.

[0099] In the context of generalized scene reconstruction, the plenoptic and analytical information may be processed independently, together, or some combination of the two. In some embodiments of the invention, the plenoptic and analytical information may be processed in a common workspace, preferably in an "on-demand" manner, to reconstruct a scene or achieve another goal. The invention includes methods for inspecting, processing, storing, and using such information, including, for example, spatial elements, data structures, and associated processing functions. In some embodiments, some of the spatial, plenoptic, and other processing operations may be selectively performed with improved efficiency using parallel computing elements, including arrays of the aforementioned, specialized processors, and the like. One example of such improved efficiency is the transfer of light field radiance values ​​between media in a scene. For example, the system may process incoming and / or outgoing radials using an FPGA core, a CPU, a group of CPU cores, or using hardware acceleration (HWA) with one or more of a graphics processing unit (GPU), a neural processing unit (NPU), a tensor processing unit (TPU), and / or other specialized processing units, including HWAs managed by one or more CPUs or other computing devices. When processing a large number of medials and / or incoming radials, an exemplary FPGA-based embodiment may perform light transport calculations for tens, hundreds, thousands, or more radials in parallel. In an embodiment, when a scene is divided into segments or subscenes, the system may perform parallel processing of radials, medials, or groups thereof within each of one or more subscenes.

[0100] In certain embodiments, the invention may use segments or sub-scenes, which have a set of one or more plenoptic elements, each having one or more associated medials and radials. A segment may have sub-segments, which have a subset of one or more plenoptic elements within the segment, and super-segments, which have one or more plenoptic elements from one or more segments.

[0101] A particular scene may include one or more objects, which in some embodiments represent physical things (e.g., a basketball, a bird, or a person) that are present in the scene and have one or more segments that can be characterized by a human or computer. Although a blade of grass or a blade of grass may not be referred to as an object in everyday conversation, such material may be represented as a segment and referred to as such, and may be referred to as an object in aspects of some embodiments of the present invention.

[0102] In certain embodiments, the generalized scene reconstruction can be implemented as a non-parametric process. In certain embodiments, non-parametric modeling means that the modeling of the structure of the segments is not completely pre-determined. Rather, in such embodiments, at least a portion of the parametric information used to represent the segments is derived from the detected data itself.

[0103] Some embodiments of the invention may use spatially permuted hierarchical and / or multi-resolution plenoptic elements, or any combination thereof. Some embodiments of the invention may use local curvature constraints, e.g., B-splines, to regularize surfels or surface elements in one or more segments, subsegments, or supersegments. Such a process may be used a priori to identify or a posteriori to discover where discontinuous derivatives exist in the scene.

[0104] In some embodiments of the invention, it is possible to distinguish between different segments or collections of segments (supersegments), which in some embodiments may represent different objects, materials, or other characteristics of a scene. In some embodiments, such distinctions can at least in part use specific information attached to the plenoptic elements, collectively referred to as properties and represented in one or more nodes or regions in the data structure. Such information includes, but is not limited to, properties such as color, normal, gradient, tangent vector, material, associated two-way light interaction function, density, transparency, radiance, and / or other factors. In embodiments in which the plenoptic data sets are spatially sorted, the invention can be implemented in a computationally efficient manner, for example, by simultaneously scanning the aligned data structures to search for identical or corresponding volumetric or directional regions of space within the structures. In some embodiments, the ability to spatially sort the data sets facilitates the maintenance of multiple data sets representing disparate information in the same scene, allowing for the co-location of properties. In such aspects, an exemplary implementation of the invention can maintain one or more properties for a scene in multiple structures and make available and / or process a subset thereof relevant to a particular operation.

[0105] In certain embodiments of the present invention, a data structure can be used to represent a segment of plenoptic information. The subset of plenoptic data can be represented in a number of ways depending on the requirements of a particular application. In some embodiments, the plenoptic data structure may include a segment identifier, for example as part of a core structure that is used to identify a type or property attached to a segment within the data structure. Such an implementation is particularly beneficial when there is a need or desire to represent a small number of segments.

[0106] In embodiments where the data is contained in multiple segments, the identifier is preferably capable of providing multiple segment membership, in some embodiments such identification is made directly using the property, in other embodiments by using a table pointer property that contains the status of membership in a table, and in other embodiments by using a combination or alternative of the foregoing methods.

[0107] In some embodiments, a segment may be represented implicitly based on a type, property, or other characteristic or variable, for example, a segment may be defined as a portion of a plenoptic data structure that matches a set of inclusion or exclusion properties (e.g., density within a specified range).

[0108] In certain embodiments, a separate shadow plenoptic data structure may be defined. A shadow plenoptic data structure is a plenoptic data structure that represents at least a subset of data portions of one or more other plenoptic data structures, but may represent alternative information (e.g., membership in a selection set or property deviations). A shadow plenoptic data structure may be advantageous when a larger number of segments are needed or desired to be represented. For example, a data structure that is binary (e.g., data is marked as being included or not included in a selection set) may represent data of another plenoptic data structure that belongs to a particular segment. In such embodiments, multiple such segments may be combined in a set operation simultaneously with the associated plenoptic data structure to create a supersegment. Such a supersegment may have overlapping data.

[0109] In some embodiments of the invention, it may be desirable to replicate a structure in one or more locations in a scene. This desire may be particularly useful in analytical terms. The graph structure may be used to eliminate duplication of identical data, to avoid the need to make changes in multiple locations, a combination of the two, or for other purposes. In certain embodiments, before being used in an operation such as displaying a scene or a portion thereof, the graph may be evaluated or "flattened" into a simpler data structure in which all transformations applied to individual elements are combined into a single transformation. As a non-limiting example of the use of the graph structure in a scene, consider an assembly that includes multiple bolts. The process described above may be used to change the type, position, orientation, or other characteristics of the bolts. The present invention may thereby be used to achieve the transformation of all relevant bolts in a scene with a single transformation or a reduced number of transformations based on set or determined parameters.

[0110] In some embodiments, the system may provide sampling to determine a function that can represent the surface. In certain embodiments, the sampling may include sampling of polarization data, such that the function can represent the intensity on the surface, the polarization information on the surface, the light source, the outgoing light field or power, or a combination thereof. When the system assigns the function on the surface, the resulting function can provide a data set that can represent the surface. For example, the function may provide a fitting model of a voxel, allowing one to determine how that voxel responds under different conditions. Such a function can be used to determine the response in a particular light field, or to determine how the response in a single light field is projected to another light field. Such a representation can be used, for example, in situations where the system does not (or has not yet) separate the material field and the light field.

[0111] In certain embodiments of the present invention, such plenoptic representations can be used to perform generalized scene reconstruction. For example, such plenoptic representations may be space-filling (in 3D volumetric or directional space) and may support more powerful processing operations than the exemplary surface, line, or point representations described in detail herein. Certain embodiments of the present invention may combine, process, analyze, or perform other operations to enable and facilitate scene reconstruction with capabilities that are difficult or impossible with lower dimensional representations. Exemplary categories of operations that may be performed as part of scene reconstruction or other processing described herein include, but are not limited to, thresholding based on one or more properties, calculating connectivity of one or more elements or objects, calculating mass properties (volume, mass, center of mass, etc.), marking identifiers (e.g., identifying regions in volumetric or directional space), performing set operations, transformations, and / or morphological processing (e.g., dilation, shrinkage, etc.), calculating directional projections and / or visibility, determining spatial and / or directional masking, determining interior filling, performing interference and / or collision analysis, etc. In some embodiments, multiple operations may be used together to perform one or more compound operations, such as separating discontinuities, selecting regions, and / or determining nearest neighbors (spatial or directional).

[0112] With reference to the various embodiments described herein, certain embodiments of the invention may utilize one or more of the modeling methods and processing tools described herein to perform reconstruction tasks and related operations to support multiple applications.

[0113] As a first example, the embodiments described herein can be used to reconstruct a scene including a tree and / or to view the tree from a distance. In this example, one or more images including a stationary tree are taken from a distance. In this example, a single pixel from the image may include one or many leaves, a background, or a combination of these. In an embodiment of the invention, the information contained in this pixel can be represented by a surface that has the origin at the viewpoint and encompasses a plane that intersects the edge of the pixel. The information attached to the surface may include a color and a calculated normal vector. Such a calculated normal vector may be calculated from a polarimetric analysis of the data, from a combination of images, or otherwise. The color may be the composite color of the leaves, branches, and background represented by the pixel. In this example, the normal vector is the result of many surface reflections. Thus, in such a tree-at-a-distance example, the image information cannot achieve sufficient confidence to indicate that it is from a single surface.

[0114] In some situations, there may be multiple relevant pixels in a region of an image. If the number of relevant pixels is sufficient, certain embodiments of the present invention may perform a statistical analysis of the texture. Such statistical analysis may include application of a set of one or more filters to the region, preferably with a cluster of responses to the one or more filters assembled into a texture signature. In this example, the calculated texture signature may then be added as a property to a scene model, and then used to insert synthetically generated textures into a rendering to provide a realistic view.

[0115] Continuing with the example of an image of a tree taken at a distance, high-level surface data can be calculated from each individual pixel. Due to the relatively low resolution and / or the high degree of object representation in a single pixel, there is insufficient information to begin to construct a spatial representation of the tree. Therefore, in certain embodiments where the system is implemented to reconstruct a 3D scene for 3D display, the image of the tree and everything within such an image is assumed to be beyond the parallax distance (i.e., the reconstructable scene boundary) and is used as the background in some embodiments.

[0116] If the proposed exemplary system receives one or more additional images or other scene data of the same tree from different viewpoints, the system may extract specific landmark points or radiographic information from the original image and the newly received image or data, and in a preferred embodiment, attempt to match such landmark points or radiographic information. If the system is successful in matching such landmark points or radiographic information, the system may estimate the 3D location of the point. At this point, an embodiment of the present invention may initiate a 3D model and assign color values ​​or other characteristics of the point from the pixels of the two images to the spatial region around the matched point. The system may perform similar operations on the region around the matched point. If additional images from closer ranges become available, the system may create spatial regions with higher resolution, and may optionally re-evaluate associated upper lower resolution regions. This process may be performed on an on-demand basis, such that high resolution information is processed only as needed to achieve an immediate goal. This goal may be, for example, to reconstruct a particular object of interest, whereby the system retains most or all of the image data for later use, but does not process to a high level of detail unless required. In an embodiment, the high resolution information is processed immediately and / or stored for later processing. The system can then construct a 3D model using all or a subset of the available data. In one embodiment, the full resolution of the spatial model corresponds approximately to the projected size of a pixel.

[0117] The systems described herein may further generate lower resolution regions of the spatial model using color information calculated from lower levels. In an embodiment, color information contained in the high resolution region of the data structure is processed to generate one or more colors represented in the parent node. The system may then calculate an estimate of the percentage of the area that the calculated color occupies in the low resolution representation of the image based on the corresponding high resolution information. The system may then use this calculation to calculate transparency estimates for spatial regions at multiple levels of resolution, where the transparency estimate may be the percentage of the spatial region that is estimated to contribute color to the associated region in the image. In an embodiment, the remaining color or colors are assumed to be from more distant materials, for example materials in the background of the image. In an embodiment, the system may continue to represent spatial regions of increasing size by including additional color and transparency values, which may in some cases result in increased complexity. Colors representing different items in the scene may be separated into other spatial regions to limit the need for individual nodes. Thus, a composite color and transparency value for a region may be calculated as a single color and transparency value for that region based on the values ​​of its children.

[0118] In certain embodiments of the present invention, the system may permit viewing from novel viewpoints. In such circumstances, the plenoptic representation may be arbitrarily scanned in a front-to-back order from the viewpoint, with accumulated pixel information from a spatial region that corresponds approximately to the projected pixel size, which may increase with distance. In such embodiments, the system may accumulate composite color values ​​for pixels based on colors weighted by the encountered transparency values, which may continue as nodes of appropriate size are encountered. In some embodiments, the system may include a threshold value for a pixel, such that when the accumulated transparency weight exceeds the threshold value for the pixel, a color is determined and the plenoptic scan terminates.

[0119] As a second example, the embodiments described herein can be used to reconstruct a scene that includes a vase with one or more flowers and remove the flowers from the base. In this example, a plenoptic model of the vase with flowers has already been obtained from the image. The system can use one or more 3D segmentation tools to generate a data structure that characterizes the segments as described herein. The system can then assign one or more segment identification numbers to one or more regions of the model, which may be based on an estimated similarity of media between one or more segments. For example, the system may perform segment identification based on the spatial smoothness of the exterior surface of the vase, which indicates that individual media are related and belong together. In some embodiments, prior information, often assisted by AI, can be utilized to guide the segmentation. In certain embodiments, individual segments that belong to identifiable structures such as vases, flowers, or other objects in the scene can be connected. Such exemplary embodiments may be implemented to store information that optionally associates segments that form a flower. Exemplary embodiments may further include a high-level data structure that can connect various parts of a flower, such as various representations of petals, stems, and leaves. Exemplary embodiments may further include higher level data structures that may represent the entire vase of flowers.

[0120] To extract a particular flower, an exemplary embodiment of the invention may then perform one or more operations, e.g., a transformation, whereby the system may subsequently subject one or more associated segments to the transformation, thereby manipulating one or more associated segments to move such segments away from the vase. The system may further perform collision analysis and may guide the movement of the flower within the scene for a trajectory that avoids spatial intersections. In some embodiments, depending on the level of recognition and modeling achieved, sections of the flower model that are occluded and / or otherwise not reconstructed may be interpolated or analytical models inserted.

[0121] As a third example, the embodiments described herein can be used to reconstruct a scene that includes water, objects submerged or partially submerged in water, one or more droplets entering a body of water such as a swimming pool, or objects submerged in water or another liquid. In such an example, multiple droplets and nearby bodies of water may be reconstructed. In certain embodiments, droplets can be modeled to move into and enter a body of water according to physical laws or other characteristics provided to or known by the system. In some embodiments, droplets can be represented volumetrically, which provides the system with a basis for calculating mass properties of each droplet using known mass properties of water. The system can then model the trajectory of each such droplet into the water based on whole, partial, or other aspects of the droplet's mass and / or center of mass. In some embodiments, the system can optionally include an advanced modeling system, which can support deformation of one or more droplets or pools.

[0122] In some embodiments, the movement of the droplet can be modeled at discrete moments in time. At the time the droplet first enters a larger segment representing a body of water, an operation can be performed to determine the volume of water common between the pool and the droplet. The system can then use the results of such an operation to correct for the increase in the volume of the larger segment, which can optionally be achieved using a morphological dilation operation. During such an operation, one or more volume elements on the larger segment surface (swimming pool) that interfaces with the mobile material (droplet) may be incrementally dilated to compensate for the displaced water volume and may be further modified to account for the dynamic response of the segment surface to interaction with the mobile material. Using such and similar tools, the system can implement more advanced displacement models. In some embodiments, the overall process can be continued for further water displacements until the droplet is fully submerged in water.

[0123] As a fourth example, the embodiments described herein can be used to reconstruct a scene including a satellite, spacecraft, or other object orbiting the Earth or other celestial body. Using a spacecraft as an example target for reconstruction, one or more images of the spacecraft can be captured by an imaging device such as a camera. The digital imaging device can be provided as part of the spacecraft itself (e.g., on a boom, arm, or other structure to allow the spacecraft to inspect itself), or can be provided on another spacecraft or object operable to inspect the spacecraft, for example. In some embodiments, the systems described herein can create a plenoptic model of the spacecraft from previously acquired image data. In some embodiments, the systems described herein can create a plenoptic model of the spacecraft from as-captured image data, or from a combination of previously acquired and as-captured data. Such a model can be used to assess the condition of the spacecraft, the presence or absence of damage to all or part of the spacecraft, materials present on the spacecraft, equipment present on the spacecraft, other information useful in assessing the spacecraft, or a combination thereof.

[0124] In some embodiments, the system can store one or more models in a database or other data structure. In addition to storing the models, the database can provide access to one or more models. In some embodiments, access is provided by a user interface that allows searching, indexing, or browsing one or more categories of models, among other things. Certain embodiments of the invention can provide data services that allow access to one or more models. Such data services can be personal (i.e., per user, allowing a user to access a subset of models relevant to that user), available to a group of users (e.g., a group, company, business, or other group with limited access to such group), or available to the general public. Some embodiments can provide data services as an interface to other systems or applications. For example, the system can provide access to or information about the models to applications or systems that can use the models for other purposes (e.g., a third-party metaverse application can use one or more models of furniture provided by the system to recreate a home). In some embodiments, the system can store models created by the system itself, by a third-party model creation system or software, or a combination of the two.

[0125] The system can use one or more 3D segmentation tools to generate a representative data structure of segments as described herein. The system can then assign one or more segment identification numbers to one or more regions of the model, which may be based on estimated similarity of media between one or more segments. For example, the system can base segment identification on the spatial smoothness of the spacecraft's exterior surface, or the spatial smoothness of materials present in the model (e.g., classifying the materials of the spacecraft's solar arrays with segment identification numbers), indicating that individual media are related and belong together. In some embodiments, segmentation can be guided by employing prior information, often assisted by AI. In certain embodiments, one or more high-level data structures can be used to connect individual segments belonging to identifiable structures, such as solar arrays, thermal management systems, propulsion systems, communication systems, or other aspects of the spacecraft. Such exemplary embodiments can be implemented with individual high-level data structures optionally associated with segments forming particular systems or subsystems of the spacecraft. Exemplary embodiments may include higher level data structures that may connect various individual higher level data structures that represent various portions of the spacecraft, such as subsystems that include broader systems observable on the spacecraft. Exemplary embodiments may also include a higher level data structure that represents the entire spacecraft.

[0126] As a fifth example, the embodiments described herein can be used to reconstruct a scene that includes a portion of the human body. As an example, the embodiments of the invention disclosed herein can be used in dental applications to reconstruct teeth, gums, other soft tissues, dental implants or products, or other objects in an individual's mouth, to reconstruct all or part of a human eye, or other medical related applications. For example, the system can perform virtualization of teeth in a dentist's or other provider's office, a surgical center or hospital, or even in the patient's home. In various embodiments, imaging can be performed using handheld commercially available devices (e.g., cell phones, tablets, or cameras) or using specialized medical or dental equipment (e.g., dental scopes or other scopes known in the art). In some embodiments, the system can process the captured images to virtualize the scene of interest in the patient's oral cavity, optionally including characterizing the size / dimensions, color, reflectance, translucency, and / or other optical properties of objects (e.g., teeth) in the scene. Such virtualized scenes may include models, such as plenoptic models, that are useful in numerous applications, such as design and sizing for orthodontic or alignment devices, dental implants or other appliances, mouth guards, and retainers, etc. Although described in a dental-related application, the system may be used in a similar manner for medical-related applications (e.g., diagnosis, surgery and operating rooms, treatment, etc.), analyzing body size and / or composition for athletic training, apparel sizing, and numerous other applications. For example, the system may create models that can be used to verify the accuracy and completeness of surgical instruments, medications, or other objects prior to entering a surgical environment to perform surgery, to model regions of interest within or on the human body before, during, and / or after surgery, for training purposes, or for numerous other applications.

[0127] In addition to the aforementioned examples, the systems described herein can be used in multiple other situations, including situations where reconstruction of both light and matter in a scene is advantageous compared to existing systems, including, but not limited to, advanced driver assistance systems, traffic solutions (e.g., speeding or license plate detection), human body scanning (e.g., for the health, medical, dental, and / or fashion industries), object classification and / or identification, inspection where UAVs are used for area access, battery inspection, robotics (e.g., bin picking and indoor mapping), logistics (e.g., truck loading and packaged object dimensional measurement), automotive applications (e.g., vehicle body inspection), sorting applications (e.g., food sorting and recycling), or in combination with infrared scanners (long wave or short wave).

[0128] One advantage of the embodiments of the invention disclosed herein is that they can be configured to reconstruct light fields, matter fields (which may be reilluminable matter fields), or both, together or separately. Those skilled in the art will recognize that various applications of the embodiments of the invention may require only one or both of the reconstruction of light fields and matter fields, and furthermore, reilluminable matter fields may have advantages in certain situations and not be necessary in others. In addition, the embodiments of the invention described herein may be configured to perform some of the aforementioned reconstruction techniques for all of the scene, or to perform the techniques in various configurations for different regions or objects in the scene. Furthermore, the aforementioned reconstruction techniques may be combined with other techniques for characterizing the scene (e.g., photogrammetry, NeFR, and other techniques described herein) to reconstruct all or the same regions of the scene, or to characterize different regions or objects of interest in the scene using different techniques. For example, embodiments of the invention may be configured to determine which technique provides the fastest, most computationally efficient, lowest power, etc. alternatives for reconstructing the scene (or some or all distinct parts thereof) and combine various reconstruction techniques.

[0129] Some advantages of light field and matter field reconstruction are described, for example, in Figure 73 and related description of U.S. Patent Application Publication No. 2021 / 0133929 to Ackerson et al., which highlights situations where the representations of matter and light fields, and their resulting interactions in an image, are complex and difficult to analyze and understand, especially when trying to glean understanding from the image itself. In some embodiments, the invention described herein can tailor the displayed information to the immediate needs of the view, for example, by specifying the type and display characteristics (e.g., scale factor) of scene elements and how the elements are rendered (e.g., wireframe vs. shaded).

[0130] 1C-1E, in some embodiments, the invention disclosed herein can be configured to customise the nature of the relightable properties present in the model. For example, a user, an invocation process, a higher level reconstruction goal (manually or automatically determined), or other function can specify the desired relightable properties of any scene. In some embodiments, a scene is considered relightable if media in the scene have associated properties (e.g., one or more BLIFs) that can predict the response radiance in one or more (or any) outgoing directions of interest given an incident light field. In some embodiments, a scene can be considered fully relightable if all media in the scene have the aforementioned properties and the model has removed external illumination (i.e., the response radiance is based only on emitted light in the scene, except when the model is reconstructed with a particular incident light field). For example, FIG. 1D shows a matter field 120 with all external illumination (e.g., incoming light 112) removed. A fully relightable matter field can be configured to respond to emissive light fields from emissive objects 114 and / or windowed light fields 112. The windowed light field 112 can represent light entering the scene from a larger surrounding environment (frontier 117). Having a windowed light field 112 may be desirable to illuminate or relight the scene under the original light field present during capture or measurement from the associated image data, and / or to characterize the scene in alternative lighting conditions (e.g., if the model is of a room, the windowed light field can allow characterizing the room in daylight and nighttime conditions). If the characteristics of the original light field are not desired for reconstruction, it may be less important to characterize the original windowed light field 112, but an embodiment may reconstruct an approximate equivalent in the process of reducing the scene to a form that is primarily dictated by the physical laws of light interaction in a field of plenoptic elements. An example of a situation where the windowed light field is less important is when the goal of reconstruction is to obtain the size and shape of a foot for shoe ordering.In such a situation, the goal of reconstruction is the intrinsic matter field of the toe surfels, and the light field is less important.

[0131] In an embodiment, the invention can adjust or balance the incoming and outgoing light fields at the window-like boundary 111 between subscenes or regions. For example, the radiance calculated to be incoming at the window-like boundary element 111 should match the radiance calculated to be outgoing at the window-like boundary 111 of another subscene in that given direction. With such configuration, the system can be configured to implement various techniques for reconstruction between subscenes or regions within a scene. For example, a single scene can contain separate subscenes reconstructed using each of the various techniques described herein (e.g., using physically based light transport, NeRF, etc.), each of which can coexist in the entirely reconstructed scene. The various subscenes can be matched or balanced against each other by querying each subscene for its predicted outgoing light in a specified query direction at the window-like boundary element 111 of each subscene, and it is important to also optionally query for the predicted incoming light. Having such a balance allows for varying degrees of re-illumination between sub-scenes or regions by providing transitions at the boundaries, which can be thought of as similar, though potentially coarser, to the way incoming and outgoing light is balanced via media BLIF within a sub-scene.

[0132] 1C-1E show various embodiments of a scene model. As illustrated in FIG. 1C and FIG. 1E, the scene model may be an externally illuminated scene model, meaning that an incident light flow 112 exists at a window-like boundary 111 and provides light to the scene. As illustrated in FIG. 1C, the scene model 110 may optionally include one or more light-emitting objects 114, which emit light flows into the scene independent of the incident light flow 112. Thus, the model may represent a medium in the scene based on both the incident light flow 112 and the light-emitting flow from the light-emitting objects 114, as applicable. As illustrated in FIG. 1C, the scene model 110 may optionally include one or more response objects 113, which provide a response light flow, i.e., an outgoing light flow, in response to the incident light flow.

[0133] As illustrated in FIG. 1D, if there is no incoming light flow, e.g., represented by window-like boundaries 111, and no luminescence in the scene, the scene model may be a single scene model, i.e., a reilluminable matter field 120. The single scene model may thus represent the scene in the absence of incoming light from outside. If light is completely absent in the single model 120, as illustrated in FIG. 1D, then either window-like or emissive light sources must be provided by the system to illuminate the scene (e.g., when desired for purposes of predicting outgoing light, such as rendering). As illustrated in FIG. 1D, in such an embodiment, the single model 120 will represent the light interaction properties of the scene without intrinsic illumination, such that the final rendering represents the interaction with the provided light sources. In some embodiments where the single model 120 includes emissive objects 114, there are potential light sources in the scene itself. In such an embodiment, the single scene model may be reconstructed with only emissive light sources or a combination of emissive light sources and other light sources.

[0134] In some embodiments, the single scene model 120 may be considered fully reilluminatable (e.g., if the model can remove the effects of the incident light flow 112 present in the original image data from which the model was constructed), and such reilluminability may be facilitated by determining a reilluminatable matter field that characterizes the volumetric, emissive, and light-interaction properties of materials in the scene. Given a given light field (e.g., lighting conditions), the model may reconstruct the matter field to appear in such light field. In some embodiments, the scene model 110 may be considered non-reilluminatable (e.g., if the incident light 112 is inseparable from the model's matter field), partially reilluminatable (e.g., if the incident light 112 is partially separable from the model's matter field), or fully reilluminatable (e.g., if the incident light 112 is fully separable from the model's matter field). The degree of reilluminability may be affected by performing light field and / or matter field reconstruction according to various embodiments disclosed herein, including performing reconstruction at a particular level of detail or resolution based on need or preference.

[0135] 1E, the scene model may be a light field model 130. The light field model 130 may characterize the flow of light within a scene. For example, the light field model 130 may characterize the flow of light into the scene 112 and / or the flow of light from the scene 116. Additionally, the light field model 130 may characterize the interactions and flow of light within the scene, including, for example, responsive light flow from light interaction with responsive media and luminous flow from luminous objects.

[0136] Although the embodiments of the invention described herein can be applied in a variety of contexts to reconstruct both light and matter fields in a scene and output the reconstructions together, certain applications of the embodiments of the invention described herein may require only the reconstruction of the light field in a scene. Reconstructing the light field of a scene may enable a view of the scene where specific characteristics of the materials in the scene are not required. For example, if the process described herein is used for traffic light detection by an autonomous vehicle, the primary objective of the application would be to determine the state of the traffic light in a particular lighting condition. However, the specific characteristics of the material field (e.g., the size and shape of the traffic light itself) may not be relevant to the final application. Of course, there are numerous applications in which a light field reconstructed according to the embodiments of the invention described herein can be used independently to achieve a desired result.

[0137] Similarly, an application of an embodiment of the present invention may require only the reconstruction of a matter field in a scene without a corresponding light field. Consider, for example, a situation in which the results of a reconstruction using the processes herein are used for reverse engineering or additive manufacturing of a part or component. The application may require information related to the matter field, specifically a model of the size and shape of one or more objects in the scene, or just the scene itself. Of course, there are numerous applications in which a matter field reconstructed according to an embodiment of the invention described herein can be used independently to achieve a desired result.

[0138] Additionally, applications of certain embodiments can be enhanced by the reconstruction of reilluminable matter fields. The term reilluminable can be understood to provide specific light interaction properties of materials in a scene, non-limiting examples of which include properties related to transparency, refractive index, roughness, polarized diffusion properties, non-polarized diffusion properties, and / or extinction coefficient, among others. Characterization of these and / or other properties allows modeling of how materials interact with light fields other than those present in the image data from which the model was reconstructed. Additionally, the light interaction properties of the reilluminable matter field can be used in combination with the inventive embodiments described herein to characterize the composition or substantiality of materials in a scene. For example, for applications in which the reconstruction techniques described herein are implemented to create models for use in the Metaverse, an accurate and usable model can preferably respond to simulated lighting conditions in the Metaverse scene, but the original light field may not be relevant. Of course, there are numerous applications in which a reilluminable matter field reconstructed according to the inventive embodiments described herein can be used independently to achieve a desired result. The inventive embodiments described herein can be configured to provide multiple or varying degrees of reilluminability within a single scene. For example, in certain embodiments, it may be desirable for certain portions of a scene to have a higher relighting potential (e.g., the reflective pots illustrated in FIG. 73 of Ackerson et al., U.S. Patent Application Publication No. 2021 / 0133929 A1), while other areas of the scene may require a lower relighting potential (e.g., walls, trees, or other features near the parallax or boundary of the scene illustrated in FIG. 73 of Ackerson et al., U.S. Patent Application Publication No. 2021 / 0133929 A1).

[0139] The use of ML in generalized scene reconstruction (GSR) is illustrated in an exemplary manner in FIGS. 9 and 10. FIG. 9 illustrates a process for generating a fully trained machine learning system, sometimes referred to as a trained machine learning model or TMLM. A first step 901 provides new scene images for training. The training images may capture objects of interest in many scenes and / or under various conditions. An exemplary GSR system may analyze this information and / or other information in step 902 to generate a light field model for each scene and / or reconstruct one or more material fields of the scene in step 903. A material field selection function may be employed in step 904 to extract objects of interest from the scene. In one embodiment, such processing may result in a reilluminatable material field (RMF) model that includes both geometry (e.g., shape) and BLIF information in step 905. Such a model may be used as an example in training the machine learning system in step 906. Although not illustrated, in certain embodiments, BLIF parameters may be varied to create multiple training examples from a single RMF model. The result of step 907 is a TMLM.

[0140] An exemplary production application of the TMLM is shown in Figure 10. In step 1001, an image of a new scene is provided. The GSR system processes the image in step 1002 and creates a new light field model to reconstruct a new RMF in step 1003. In one embodiment, a material field selection function can be used in steps 1004 and 1005 to extract portions of the material field to be processed, such as for classification. Finally, in steps 1006 and 1007, the new RMF can be tested using a previously trained TMLM to generate useful output.

[0141] In an embodiment, the system can train and use a TMLM using one or more of the following: one or more material fields, one or more source light fields, one or more window-like light sources, and one or more captured images, which may optionally have associated pose information. Such training can be performed as described herein, including using one or more neural networks, to compute a light field of a scene or subscene. In an embodiment, the TMLM can be trained using one or more models, including models that include neural reconstructions of plenoptic fields, color, intensity, or other radiometric information associated with a scene, or other information. Some embodiments of the TMLM can apply a physically-based modeling approach to perform light field reconstruction, alone or in combination with one or more neural networks. The foregoing embodiments enable the TMLM to create, reconstruct, or otherwise generate one or more images based on inputs to the TMLM, which in some embodiments include a pose associated with the one or more images.

[0142] In some embodiments, the multi-dimensional scene can be stored using a voxel grid or polygon mesh, while other embodiments can avoid their use. In particular, voxels can be expensive to store in terms of data size or required processing, and polygon meshes can often only represent hard surfaces. The system may use one or more fields that define a set of spatial and / or temporal coordinates. In embodiments that use neural networks to model one or more fields, the fields may be referred to as neural fields, or neural graphics primitives when modeling 3D spaces.

[0143] In some embodiments, a light field physics module can be used to model interactions between one or more medial and radials entering and / or exiting one or more medial. In some embodiments of the invention, neural networks can be used to represent light interactions instead of or in combination with parametric functions. In some embodiments, the system can incorporate high frequencies (texture layout) by using a Sine Wave Representation Network (SIREN).

[0144] Although the system described herein may represent light interactions in a scene by using parameterized functions, the system may be configured in other ways. Light interaction functions are complex and the parameter space is often high dimensional. The system may optionally select an ML model that attempts to learn and approximate the light interaction function. One aspect of improving the successful learning of such ML models is to ensure compliance with physical laws related to light interactions. Certain embodiments of the invention may ensure such compliance by using physically informed neural networks. This approach may use prior knowledge of the system to modify the loss function and may model unknown physical components or properties by using neural networks. These neural networks may be configured to use nonlinear activation functions to increase the expressiveness of the model. Using prior information, such as known physical laws of light interactions described elsewhere herein, the system may incorporate additional constraints according to the physical laws.

[0145] There are many approaches to incorporating physics-based modeling into neural networks, including physically-informed neural networks (PINNs), neural network architectures with physical constraints, incorporating physical priors in the loss function, hybrid modeling, and residual modeling.

[0146] 16, in an embodiment using the PINN architecture, the ML model includes two main parts: a fully connected neural network 1602 and a residual layer 1604, both of which can be designed to satisfy the underlying physical equations of the system being modeled. In one embodiment, the inputs to the PINN can be spatial and / or time coordinates 1601, which can be normalized between 0 and 1. The fully connected neural network 1602 can process the inputs and output a predicted solution 1603 for the system being modeled. The PINN can be composed of multiple hidden layers with nonlinear activation functions.

[0147] A residual layer 1604 can be applied to the predicted solution. The residual layer may be optionally designed so that the predicted solution satisfies the governing physics equations of the system. The residual layer can take partial derivatives of the predicted solution from the fully connected neural network with respect to the input coordinates and time, and apply the physics equations governing the predicted solution 1603. The output of the residual layer 1605 is then combined with a loss function, which may include one or both of data constraints (such as known boundary or initial conditions) and physics constraints (such as memory laws or other governing equations). This loss function can be used to train a neural network to minimize the difference between the predicted solution and the observed data while satisfying the underlying physical laws.

[0148] In some embodiments of the present invention, PINNs can be used to enforce physical constraints such as object stiffness, object continuity, and / or object motion invariance during the reconstruction process. These constraints can be incorporated into the neural network to obtain a more accurate and / or robust output model. In some embodiments, the present invention can use PINNs to improve the efficiency of the scene reconstruction process. For example, PINNs can be used instead of or in combination with computationally expensive algorithms, such as algorithms for solving partial differential equations (PDEs). PINNs can approximate the solution of one or more PDEs with a neural network, which can significantly reduce the computation time required for scene reconstruction.

[0149] With reference to FIG. 20, an embodiment of the present invention can combine a physics-based approach (e.g., calculation of BLIF and / or its parameters) with a neural network, which may perform residual modeling. In an exemplary embodiment, the input to the model is incident data (e.g., light rays) 2001. The system can predict the outgoing light intensity 2003 by using a physics-based model 2002 to calculate the BLIF. The physics-based calculation may be configured to be calculated to a certain accuracy, which can reduce processing and / or power demands. The incident light field 2001 and / or the results of the physics-based calculation 2002 are provided to a neural network 2004 to further refine the output to improve prediction performance and / or resolution of the predicted light interactions 2005. In an embodiment, the use of an energy storage constraint (e.g., incorporated into the loss function) can improve the training of the neural network.

[0150] Referring to FIG. 17, several neural network architectures have been developed incorporating physical constraints. For example, the Neural Hamiltonian Network (NHN) architecture is designed to respect Hamiltonian dynamics and conserve the energy of the system. The inputs 1701 to the NHN are the state variables (q, p) of the system, which represent for example the position and momentum of a particle. The neural network can be configured to predict the time derivatives of the state variables based on the input state variables 1702. This neural network may be a fully connected neural network with multiple hidden layers.

[0151] The output of the neural network can be passed through a Hamiltonian layer 1703, which can be configured to calculate the Hamiltonian dynamics of the system based on the predicted derivatives. The Hamiltonian layer can calculate the dot product of the predicted derivatives with a Jacobian matrix that represents the underlying physical laws of the system. Such a configuration ensures that the predictions of the NHN are consistent with the underlying physical laws of the system and that energy is stored over time.

[0152] The output 1704 of the Hamiltonian layer are the predicted state variables at the next time step, which can be used to generate physically plausible trajectories of the system or for other downstream tasks such as control or optimization.

[0153] 18, the system can also be configured to incorporate physical priors into the loss function. The neural network's loss function 1804 can be modified to include physical priors or constraints. For example, in an image processing application where the system is configured to receive an input 1801, process the input using a neural network 1802, and generate a predicted output 1803, the system can incorporate constraints related to the physics of the imaging system, such as the point spread function, into the loss function.

[0154] Referring to FIG. 19, another method for addressing imperfections in physics-based models is residual modeling. When the system is configured to use residual modeling, the ML model can be trained to predict (1905) the errors or residuals made by the physics-based model (1902). An embodiment can provide input data (1901) to the physics model (1902) and the data model (1903). The system can be configured to learn the biases of the physics model (1902) and its output (1904) relative to the observations, and use the predicted biases (1905) to make corrections to the physics model's predictions. Because such an approach models errors instead of physical quantities in a physics-based problem, residual modeling does not have to apply physics-based constraints. Therefore, it may be advantageous to combine residual modeling with another form of modeling to ensure consistency with physical laws.

[0155] In some embodiments, the system can use a combination of physics-based modeling and data-driven modeling, or hybrid modeling, which can provide accurate predictions by taking advantage of the strengths of both approaches.

[0156] In one embodiment, the neural network described above may be designed by directly adding one or more known differential equations to the loss function when training the neural network. In one embodiment, training may be performed by sampling a set of input training locations and passing the samples through the network. The network may compute one or more gradients of the network's output with respect to the input at one or more sampled locations. In one embodiment, the gradients may be computed using automatic differentiation or similar features present in many ML libraries such as PyTorch and TensorFlow. The system may then use the one or more gradients to compute a residual of the underlying differential equation and add this residual as an additional term in the loss function.

[0157] Some embodiments may use a similar approach to predict one or more surface normals in a scene, including based at least in part on the incident light field. The use of a trained ML model may overcome sources of error present in the model. For example, even for volumes that contain air or empty space, it may be difficult to model the errors in the model, and such problems may be further complicated for volumes that contain complex or uneven media. In such situations, the system may be configured to use a neural network to approximate the loss function and add additional constraints to the loss function using known optical physical properties.

[0158] In one embodiment, the invention described herein can utilize Structure from Motion (SfM) techniques. SfM is a technique for 3D scene reconstruction that can estimate the 3D structure of a scene from a set of 2D images. SfM systems can match points in images and use this information to estimate data such as camera pose and / or 3D scene structure. Using SfM, a scene can be reconstructed from a variety of sources including photographs, videos, and LiDAR data.

[0159] Some embodiments of the present invention may utilize Multi-View Stereo (MVS) techniques. MVS is a technique for 3D scene reconstruction that can estimate the depth of a scene from multiple 2D images. MVS systems can find correspondences between points in different images and use this information to estimate the 3D structure of the scene. Using MVS, a scene can be reconstructed from photographs and videos.

[0160] Those skilled in the art will appreciate that SfM and MVS are sometimes referred to as photogrammetry, which some embodiments of the invention use to reconstruct scenes from aerial, satellite, and terrestrial photographs.

[0161] In certain embodiments of the invention, data from a LiDAR system is available. LiDAR is a technology for 3D scene reconstruction that can be used to estimate the position of objects by calculating the time it takes for a laser pulse to be emitted and for the light to bounce back to the light source from objects in the scene. By measuring the time of flight and angle of the laser pulse, LiDAR can generate a point cloud of the scene, which can be used to reconstruct the 3D structure of the scene.

[0162] In some embodiments, the invention can use other data to initialize data structures before making initial assumptions, such image data can be 2D and / or 3D information. In some embodiments, the invention can use 3D datasets, such as datasets from Kinnect RGB-D (RGB and Depth) camera systems, 3D sensing components such as structured light, multi-view stereo, stereo cameras, LiDAR, radar, and / or infrared sensors, photogrammetry software, laser scanners, and other devices that provide 3D image information, Simultaneous Localization and Mapping (SLAM), and other techniques, software, and technologies. Such embodiments can receive input information in 3D, where, in addition to color information, for example, depth information determines the 3D location of each pixel relative to the camera. Such information can be represented as a 3D point cloud, a voxel array, and / or other data storage structures.

[0163] In such an embodiment, the system may use previously captured and / or processed data to make initial assumptions regarding the relative positions of media in a scene and associate a corresponding confidence level with such data. Providing such data may provide advantageous processing results by lowering the initial processing associated with the scene input, possibly providing faster performance. As an example, the system may be operated in combination with photogrammetry or other techniques to provide a sparse mesh model of an object, sub-scene, or scene at relatively low processing cost, followed by the plenoptic processing techniques described herein. Combination of embodiments of the invention with such a system may enable other visualizations of the relative positions of light and / or media in a scene. In an embodiment, ML models may be trained based on light field information, such as incoming and / or outgoing Radial trees, as a means of accelerating the identification of media and / or surfaces in a scene. As a non-limiting example, certain outgoing light fields of Lambertian surfaces may have a disk-shaped artifact. This artifact may represent a cosine falloff around the surface normal of a particular surface, which may accelerate the identification of the relevant surface normal.

[0164] In certain embodiments, the present invention may be used in conjunction with, supplemented by, or implemented in whole or in part with artificial intelligence (AI), machine learning (ML), and neural networks, including neural radiance networks such as Neural Radiance Fields or NeRFs, volumetric scene methods such as PlenOctree or Plenoxel, deep signed distance functions (SDFs), and neural volumes. Such methods can be used in scene reconstruction, novel view synthesis (NVS), and other applications to model radiance, density, or other information as continuous functions in a 3D space, such as a volumetric space, using voxels such as multi-layer perceptrons (MLPs) or voxel arrays. For example, once a location in space with 3D coordinates (x, y, z) and a view direction is determined, the representation returns the color (red, green, blue) and density at that location. The deep SDF system can be configured to learn a signed distance function in a 3D space where the zero level set represents a 2D surface. The neural volume system can be configured as a neural graphics primitive that can be parameterized by a fully connected neural network. The NeFR system can be configured to model the color and density of a scene. Other embodiments operate with alternative input and returned information. In certain embodiments, the returned density may be a differential opacity value that includes in part or in whole an estimate of other information such as radiance and color that may be accumulated by a ray of light in a particular direction through a specified point.

[0165] In some embodiments, such a representation may be initialized with random values. At the start, any specified point and direction may return a meaningless value. The exemplary system may then be trained using calibration images from a variety of known viewpoints (e.g., hundreds of images from random locations on a hemisphere above a real or simulated scene) or other image-related information. In some embodiments, the process begins by selecting one or a set of pixels in the training images. For each pixel, the network may cast a ray from the viewpoint into the scene. The network then performs a query or other processing on a number of points along the ray (e.g., 50, 100, 200, or any other number of points selected for the query). The points may be selected in a variety of ways. In some embodiments, the network, or a portion thereof, may perform an "integral rendering" operation to calculate colors or other values ​​returned along the projected ray and integrate such values ​​in some way to calculate an estimated color or other information for the pixel. In some embodiments, such color values ​​are unrelated to ground truth colors, for example when the network is initialized with random values. In certain embodiments, the network can calculate a characteristic of the difference between the estimated color and the true value (e.g., the sum of the squared differences of the color components). The difference can be used to modify the MLP weights or volumetric information using backpropagation. In some embodiments, the above process can be repeated to further refine the estimated color or other value.

[0166] In some embodiments, the process described above can generate novel viewpoints with high realism after some level of “learning.” For example, this may be via AI that converges on estimated color values ​​in the scene. In certain processes known in the art, generating novel viewpoint images using neural radiance networks or volumetric representations can require significant processing and / or time. A particular query can require perhaps 500,000 to 1,000,000 multiplications and / or other operations for each point on a ray. In some previous systems, a powerful graphics processing unit (“GPU”) such as an Nvidia V100 can take 30 seconds or more to generate a single 800 pixel by 800 pixel image.

[0167] Several methods have been developed to reduce the time, computation, cost, and power required to generate images using these methods. In one embodiment, a partially trained system can be used to generate an octree known as a plenoptic tree, which is a different data structure than the "plenoptic octree." In such an embodiment, the system may generate pixel values ​​by sampling the plenoptic tree at points along the ray rather than using an MLP. Such an embodiment can improve performance by two, five, or more orders of magnitude. The present invention may be implemented to further reduce the computation and hardware required to generate pixel values ​​from an octree data structure while still improving performance.

[0168] In some NeRF architectures, one neural network can be used to predict both the color and density of 3D points in a scene. In some architectures, separate networks can be used to predict different properties. For example, another network can be used to learn material properties such as reflectance, roughness, or transparency. These networks may be configured to predict material properties of different parts of a scene, and may be optionally combined with the color and density predictions to render an image of the scene. In some embodiments, the system may be configured using a two-stage approach with separate networks to predict the shape and material properties of the scene, which may be combined to generate a final rendering.

[0169] However, NeRF-based architectures suffer from certain limitations. For example, such networks have limited scalability and may require large amounts of training data to capture the appearance and lighting variations of real-world scenes; have limited generalization and the method is highly dependent on the quality and diversity of the training data and may not generalize well to scenes that differ significantly from the training data; have limited precision and may introduce errors and artifacts, especially in occluded or poorly lit regions of the scene; and have limited control over the output due to the implicit nature of NeRF and may make it difficult to control certain properties of the output, such as the exact location and orientation of objects in the scene. Using a NeRF-based approach with the systems described herein may address one or more of these limitations.

[0170] In one embodiment, the present invention can generate surface normal vectors by fitting a plane to adjacent surface elements. Such surface normal vectors, optionally together with spatial and color information, can be used in processing operations.

[0171] Certain embodiments of the invention can use ML to reconstruct light fields in a scene, including in some circumstances building a physical model of the interaction of light and matter fields in the scene. In such embodiments, the system can isolate components that may contribute to the light sensed by a camera pixel or another imaging device. This data may be used to determine properties of materials and objects in the scene, including non-Lambertian surfaces (e.g., human skin, fabric, mirror, glass, water). In some embodiments, the specific surface information can be represented in a bidirectional light interaction function (BLIF) for one or more sensed locations on the object, and optionally all sensed locations on the object. The sensed locations may include locations captured by individual pixels of the camera or imaging device. The invention can provide greater refinement using BLIF, and modeling based on BLIF, by extending concepts such as bidirectional reflectance distribution function (BDRF) and / or cosine lobe reflectance models to include light / matter interactions including color, material, roughness, and polarization.

[0172] In some embodiments, this processing can be used in combination with other data regarding the relative positions of materials in the scene. Numerous techniques are known in the art that provide coarse, refined, or highly accurate three-dimensional information, including, for example, photogrammetry (e.g., through software packages such as COLMAP or Metashape), structured light, multi-view stereo, LiDAR, radar, infrared, laser scanners, simultaneous localization and mapping (SLAM), and other technologies and techniques. The ML model can be configured to use a combination of image data and other data to make assumptions about the nature of the media in the scene, such as information that is not readily known from the image data alone or that is better than information that can be obtained from the image data alone. Such assumptions can be used to accelerate the processing of the plenoptic field for the scene. For example, such assumptions provided as assumptions to the processor prior to performing the reconstruction of light in the scene enable the system to predict light interaction characteristics associated with the media in the scene, provide an updated set of assumptions on which the light interactions are based, or render the processing workflow to match the predicted media in the scene.

[0173] In an embodiment, the system can be configured to classify 3D scenes / objects using raw point cloud data, such as may be provided by a LiDAR. For example, the system can use methods such as PointNet (global features) or PointNet++ (local features), which use raw Cartesian point data for classification and segmentation tasks. The system can be configured to use an MLP layer for each point and a symmetric function to handle permutation variances. The system can also be configured to use reilluminable material field (RMF) data, including in combination with a global feature extractor and / or a local feature extractor.

[0174] In some embodiments of the present invention, the latent representations can be used for downstream object recognition, object segmentation, and / or other tasks by implementing NeRF Self-Supervised Object Segmentation (NeRF SOS) or similar processing regimes. In some embodiments, the RMF data can be used as a latent representation of the scene for downstream tasks such as object recognition and / or segmentation.

[0175] Certain embodiments of the present invention may utilize AI and / or ML to perform activities such as object classification. Prior art systems often perform object classification based on images as training input, but the performance of such systems often depends on the quality and scope of the training data. For example, a traffic light detection model trained under normal conditions may not be robust enough to resolve difficult lighting conditions (e.g., the sun is behind or in front of the traffic light) and / or adverse weather conditions (e.g., snow, rain, or fog).

[0176] Certain embodiments of the present invention may have the ability to extract BLIF parameters of the sensed surface, either by direct measurement or by mathematical modeling. Such extracted BLIF parameters can be used to calibrate new light field models to new situations, which in some embodiments allows the system to model light interactions with the reconstructed matter field representing objects or materials in new scenes with different illumination. Thus, in certain embodiments, the system may have the ability to generate realistic renderings of the matter field under various lighting conditions, preferably nearly all lighting conditions, known as "relighting." In such embodiments, the reconstructed matter field becomes a reilluminatable matter field (RMF), which can significantly improve representational robustness and / or support increased levels of realism in applications.

[0177] In some embodiments of the invention, a 3DML system (e.g., Convolutional Neural Network (CNN), NeRF, etc.) may use sensed material "signatures" in the form of BLIFs at observed locations on the surfaces of objects in a scene instead of and / or in combination with sensed color information. In such embodiments, the training model of the ML system may include information about the basic surface materials and properties of the viewed objects, with or without the interaction of scene lighting. In applying such training models, some embodiments of the invention may enable a reduction in the number of lighting and scene situations that must be obtained from the real world for effective training, and / or enable synthetic generation of new training models by varying one or more BLIF parameters in a single object model to account for a variety of properties (e.g., color and / or surface conditions) that may be encountered with the object during productive use. In some embodiments of the invention, such signatures may simplify the training and use of 2DML systems. By acquiring a 3D model of a real-world object that includes BLIF information, a 2D training dataset may be synthetically generated by rendering the model from various viewpoints and / or varying the lighting and BLIF parameters appropriately. In such an embodiment, the system can be used to provide a vast number of training or synthetic data sets to the TMLM.

[0178] In an embodiment, the system can use the relightable material field data for object type classification. In an exemplary configuration, the system can use a CNN-based architecture and / or a transformer-based architecture. As illustrated in FIG. 15, the nodes of the CNN can be treated as a sequence and input to a self-attention based model. In the embodiment illustrated in FIG. 15, a model of the relightable material field 1501 can be used as an input to a model 1502. The model 1502 can include a deep learning based model fθ that can be trained on a parameter θ. The model 1502 can be sequential, convolution-based, or a multi-layer perceptron. The model 1502 can be configured to generate an output 1503 as a classification, shape completion (e.g., via ShapeNet), or another output parameter. For embodiments where there are a variety of sizes, a transformer-based model can be an efficient approach (e.g., by dividing the volume into multiple voxels and treating the voxels as a sequence). For embodiments in which the system is trained to perform shape completion, which may include predicting the volume of one or more objects in a scene, architectures based on variational autoencoders (VAE) and / or generative adversarial networks (GAN) may also be utilized.

[0179] In some embodiments, the system may perform image inpainting and / or outpainting. Image inpainting may predict the condition of damaged, occluded, or missing parts of an image. Outpainting is the generation of new image content that extends beyond the boundaries of the original image. The system may use any number of known methods related to these techniques.

[0180] In one embodiment, the system can use a generative adversarial network (GAN) with an image inpainting task. The GAN can learn the underlying distribution of image data and generate new images. The GAN can use generative and discriminative networks to achieve visually better results. The generator can be trained to fill in missing pixels, and the discriminator can be trained to distinguish between generated and real images.

[0181] Some embodiments of the present invention may use an autoencoder. The autoencoder may include an encoder network and / or a decoder network. The encoder network may be configured to compress an input image into a low-dimensional representation. The decoder network may be configured to reconstruct the image from the compressed representation. In image inpainting, the encoder network may be trained to encode a damaged image and the decoder network may be trained to fill in missing pixels in the encoded representation.

[0182] Certain embodiments of the present invention may use a deep image prior, which may initialize the weights of a deep neural network with random values ​​and optimize the weights to minimize the reconstruction loss between the generated image and the input image data. By optimizing the weights, the network may learn to generate plausible image completions that match the input image data.

[0183] In some embodiments, the system can perform various tasks to reconstruct missing or corrupted parts of a 3D point cloud. For example, a GAN can be trained to learn the distribution of point cloud data in a similar manner to its use with images. One such approach is PCGAN, where a conditional GAN ​​can be used to generate missing points in the input point cloud. In other embodiments, an encoder-decoder architecture such as U-Net or PointCNN can be used. The U-Net or PointCNN architecture may include an encoder that maps the input point cloud to a low-dimensional feature space and a decoder that maps the features back to the reconstructed point cloud. Also, some methods can incorporate an attention mechanism in the encoder-decoder architecture to better capture local and global structures. In some embodiments, a conditional reversible neural network or PI-NeRF can be used (e.g., using a point cloud inpainting network to fill in missing points in the input views and then using NeRF to reconstruct the scene from the completed views).

[0184] In cases where missing parts of a scene are not evident from the input image data, direct inpainting on the generated 3D scene may be more feasible. Some embodiments may be configured to learn the distribution of voxels in a scene and predict the occupancy values ​​of the missing / damaged parts. For example, a 3D-GAN may inpaint missing voxels by training a conditional GAN ​​to generate a complete voxel-based 3D scene when presented with incomplete or corrupted input.

[0185] In one embodiment, the system can predict the material properties as well as the light properties of the missing or corrupted parts. The system can be configured to learn two distribution functions, one for the material field and one for the light field.

[0186] Some embodiments of the present invention may use alternating optimization, i.e., an iterative procedure that jointly optimizes a function over all parameters by alternating between constrained optimizations over individual parameter subsets. For example, as described elsewhere herein, the system may be configured to calculate parameters related to BLIF and / or surface normals. These calculations may be inherently non-convex, meaning that convergence is not guaranteed and there is a risk of minimal effective processing at certain local minima. In situations where joint optimization is ineffective or where doing so would be otherwise advantageous, the system may use alternative optimization to perform the light field and / or material field reconstruction. The alternative optimization may improve optimization time in some cases compared to joint optimization, and may be better at bypassing local optima in some cases.

[0187] One problem with ML models is that such ML models may make processing or decisions based on learned statistics or patterns present in the training data. Because training and inference of ML models are highly data-driven, the output may be adversely affected due to inconsistencies or inaccuracies in the data. In some embodiments, the system may be configured to impose certain limitations, formulations, or other constraints on one or more ML models. For example, the system may impose a formulation of light field physics that consistently follows the laws of nature. Alternatively, the system may configure the ML models to perform parametric modeling using traditional light physics laws in combination with processing via a neural network.

[0188] In some embodiments, the system can incorporate physical constraints into the ML model to improve the model's explainability. For example, if a model is designed to predict the trajectory of a ball incorporating the physical laws of motion, it may help to constrain the model's predictions in ways that help make the system more explainable. A similar approach can be used with the models described herein by including constraints that relate to or characterize the laws of optical physics.

[0189] In certain embodiments of the present invention, a physics-based model can be used to interpret the ML model. For example, a physics-based model can be used to interpret the output of a neural network and / or to explain why the neural network makes a particular prediction. This physics-based model helps to identify the key features or inputs that are driving the model's predictions. In the context of scene or light field reconstruction, a physics-based model can help identify and provide characteristics of specific rays, nodes, or other data that contribute to a particular reconstructed output or result.

[0190] Some embodiments of the present invention may use physics-based simulations to validate ML models. Such embodiments may compare predictions of the ML model to the results of the physics-based simulations. These comparisons may help identify areas where the model may be inaccurate or biased and / or improve the accuracy and / or reliability of the model. For example, the system may be configured to use the output of a non-ML-aided or partially-aided reconstruction technique as a comparison with the output of a machine learning-based approach.

[0191] In some embodiments, the system can combine physics-based models and ML models. This combination can result in one or more hybrid models that are more explainable. For example, the physics-based models can be used to generate initial conditions or constraints for the ML models, and the ML models can be used to improve the predictions of the physics-based models.

[0192] One advantage of the present invention is that it can be adapted to reverse engineer existing models: while legacy techniques such as NeRF can be used to reconstruct a scene, the reconstruction is not necessarily a "decomposable" and / or "reconstructable" technique, in the sense that such techniques cannot be used to reverse engineer an already existing 3D model.

[0193] In some embodiments, the present invention can achieve better resolvability and / or reconstructibility by incorporating one or more non-optical physics-based techniques (e.g., LiDAR, MVS, SfM, photogrammetry, or others) in combination with a physics-based approach. In an example of an ML-based approach, the goal may be to learn the material and / or light fields in the scene. The system may be configured to generate high-resolution images with fine-grained control over various aspects of the image, such as the subject's pose, expression, and appearance. For example, some embodiments of the present invention may be configured to use StyleGAN, which is configured to use "style" vectors. Such vectors can control various properties of the generated images, can be learned during the training process, and can be manipulated to generate new images with different styles, thereby improving generative capabilities for relighting the scene. In a similar manner, the current system may be configured to generate latent variables of light and material properties that help to improve the resolvability and / or reconstructability of the scene.

[0194] The systems described herein can reconstruct scenes in a decomposable and / or reconfigurable manner. For example, the system may be configured to decompose a scene (or sub-scenes, objects, etc.) into parts, which can be done manually or automatically using techniques such as segmentation or clustering. In some embodiments, one or more parts can be reconstructed separately using techniques such as stereo vision, structure from motion, or other techniques. The parts may be combined to form a complete reconstruction. In some embodiments of the system, the robustness of the process can be increased by injecting physical properties. For example, the system can identify separate parts that are the core of an object and attempt to characterize the fundamental properties of the material that comprises such parts. In constructing a model of each part, each part may still retain the decomposable material properties of the object.

[0195] In one embodiment of the present invention, various levels of accuracy can be achieved by using a hierarchical neural network (HNN). The HNN can have a stack of multiple layers of neural networks. Each layer is configured to learn increasingly complex features and patterns from the input data, allowing the HNN to achieve higher levels of accuracy. Early layers can be used to learn low level features. Later layers can use various stacks depending on the desired accuracy.

[0196] Another approach to HNNs is to use a tree-structured network, with each node in the tree corresponding to a different level of abstraction. Such an arrangement allows the network to selectively activate different parts of the network depending on the complexity of the input data. Such a structure can improve efficiency and / or reduce the amount of data required for training. Each node in the network may correspond to a different level of a hierarchical data structure, with lower level nodes representing small voxels and higher level nodes representing larger voxels. The network can learn to selectively activate different parts of the data structure based on the complexity of the input data.

[0197] The system may also be configured to calculate distance functions to evaluate surface models, including, for example, polarimetric surface models. Such distance functions may have multiple uses, including being used to test the compatibility of a model (i.e., a reconstruction of a scene or sub-scene) with other model configurations (e.g., other models known in the industry, such as Unreal Engine 5®). The system may be configured to use statistical comparison methods to determine consistency between the model and other model configurations.

[0198] In some embodiments, the distance function may be used to develop one or more modeled segments or groups of media and / or light interactions as described elsewhere herein. The distance function may be used to compare multiple (e.g., two) different parameter spaces or the same parameter space, and may be done in a Euclidean manner (e.g., in the same parameter space), a non-Euclidean manner (e.g., for different parameter spaces), or a combination of the two. For example, the distance function may be used in platform space (e.g., with Unreal Engine 5®) to map between that space and a different parameter space. For example, if a model is created using a plenoptic parameter space that represents both the light field and the material field in the scene, the distance function may facilitate the conversion to platform space by providing the distance between relative points in each model.

[0199] In an embodiment, the invention described herein is operable to achieve a particular objective, which may be set by the system, specified by a user, determined by the system during processing, in response to conditions, or other factors, including any combination thereof. An exemplary embodiment is where the invention operates to characterize the state of a traffic signal, as described elsewhere herein. In an autonomous vehicle or advanced driver assistance system, such characterization may require both recognizing the traffic signal and understanding the state of the signal, but the challenge is to do all this far enough in advance for the system to make a decision about slowing down, stopping, or otherwise. Those skilled in the art can recognize certain challenges associated with characterizing the state of a traffic signal. For example, there are few standards for traffic signals, so there is a large variation in stacks, orientations (e.g., horizontal, vertical, side-by-side, and other variations), colors (e.g., variations within the colors used), types of bulbs (e.g., incandescent vs. LED), and other features of traffic signals (e.g., red lights are not always located at the top of the traffic signal). Color detection can be even more difficult because red and yellow are close in the color spectrum, while green has a narrow frequency band. Environmental factors can also be an issue (e.g., if the sun is behind the traffic light, the light is shaded and difficult to capture; if the sun is directly behind the observer, the light is difficult to see). The challenge can be more complicated when considering the structure, color of the light, redundant parts, and light sequences that exist in other countries of the world (e.g., the US commonly uses the sequence green, yellow, red, back to green, while the UK commonly uses the sequence green, yellow, red, yellow, back to green).

[0200] Those skilled in the art will recognize certain known approaches, such as narrowing the problem space by first analyzing one of the most common configurations in the United States (e.g., a three-stack traffic light with red, yellow, and green lights from top to bottom). The analysis can then continue to expand the dataset used to build the ML model to include as much variation as possible, such as angles, repair status, range of light colors, bulb types (e.g., incandescent bulbs are white in the model, but may be the appropriate color outside of them), and environmental variations as described above. The dataset can be augmented with synthetically generated images to simulate different sun positions and address imbalances found in capturing the natural world (e.g., less yellow light because yellow is often the shortest signal in time). Performing the aforementioned operations can result in large amounts of data and / or high costs associated with creating a balanced and appropriately varying set.

[0201] In some embodiments, the invention described herein may provide a model of a traffic signal, for example as a voxel field having a boundary around the traffic signal object. Such a model may include a matter field that can be reilluminated such that at least a portion of the incident light field associated with the signal is removed. In some embodiments, the only light associated with the model may be the light emitted by the signal itself. Such a model may be used in any combination with a supervised learning approach with a plenoptic model labeled by the state of the light it emits. The system may be configured to be operable to capture and reconstruct an unlit scene including a traffic signal in any state that may be used for inference against the model.

[0202] In some embodiments, the invention described herein may use image-based machine learning to build machine learning models. Image-based machine learning may include collecting examples of an object and capturing images of as many variations as possible (e.g., for a traffic light, the variations may include each of the illuminated states, types, and environmental conditions such as lighting, weather, quality, location, etc.). An embodiment of the invention may be configured to build a model of an object (e.g., a traffic light) using a plenoptic scene reconstruction of the object itself in a reilluminatable manner to remove external effects such that the only light in the scene may be emission from the object itself (e.g., the illuminated light of the traffic light). For example, the model may be a single sub-scene with a voxel boundary around the traffic light where no light enters the sub-scene. In that case, the only light in the sub-scene may be the light emitted by the traffic light and the downstream response light resulting from that emission.

[0203] By collecting a diverse collection of objects (e.g., traffic lights) in a captured plenoptic sub-scene, the system can enable the ML model to understand the shape and type of object (e.g., that it is a traffic light, its light characteristics, its shape (3-stack, 5-stack, etc.), the type and color of the lens in the traffic light, the illumination state of each bulb, and other features). In an embodiment, it may be advantageous to have a balanced number of objects in each possible state. An embodiment of the present invention may be operable to generate additional plenoptic sub-scene representations, for example by including varying surface qualities, positional aspects, and illumination characteristics.

[0204] In some embodiments of the present invention, the machine learning model can be supervised by providing one or more labels associated with each type of object and its possible states. In the case of a traffic light, the labels can characterize red emitting objects, yellow emitting objects, green emitting objects, etc. in various permutations. The labels can be created manually or at the time of capture or generation. The machine learning model can be built after all sub-scene reconstruction objects are labeled. Alternatively, some embodiments can use a self-supervised training approach where instead of generating labels, the machine learning training approach finds the appropriate category or state of the object. Self-supervised training may be more feasible using a plenoptic sub-scene recapture approach. For example, in the case of a traffic light, the color lens and which traffic light source is emitting light can be calculated more easily than traditional methods from the image source.

[0205] An embodiment may acquire novel objects, process the objects into plenoptic sub-scene objects (e.g., relightable matter fields), and apply the plenoptic scene objects to a machine learning model. Such applications may have many purposes, such as inferring or determining the category or state that an object occupies. In the case of traffic lights, given a machine learning model built from training data of relightable matter fields, a novel relightable matter field characterizing a traffic light may be classified according to which of its lights are on.

[0206] This approach has several potential advantages, including addressing adverse environmental lighting issues (e.g., by computing a re-illuminatable material field that removes the effects of external / incident light), better characterizing the quality of the luminous source (e.g., by providing a model of the signal's luminous quality to more closely match real-world observations), detecting or characterizing the material of the traffic light lens (e.g., by determining the color associated with the lens), detecting bulb luminescence (e.g., luminescence behind a light filter that may help determine color), resolving the luminous source (e.g., determining the type of bulb from incandescent, LED, halogen, etc.), removing weather effects (e.g., water or ice on signals, precipitation that makes imaging difficult, etc.), and avoiding separate model classification categories (e.g., by resolving whether a potential signal detected is an actual signal or an artifact such as a reflection).

[0207] More specifically, a potential advantage of the approach described herein is to account for different light sources present in an object (e.g., types of light bulbs in a traffic light). In the case of a traffic light, the light source may be an incandescent light bulb behind a colored and / or textured lens, an LED light source optionally behind a lens, a halogen light source behind a lens, etc. In some cases, there may be an initial filter that diffuses the light source and one or both of a smooth or textured lens for protection and / or better light emission. Even among the aforementioned types of light bulbs, there may be a wide range of frequencies and qualities of emitted color due to degradation of the quality of the light source, lenses, or other changes. For example, if there is an incandescent light bulb behind a colored lens, the emission in the center tends to be whiter and different in frequency outside of that center. Such changes may make it difficult to distinguish between red and yellow lights. Such changes may require developing larger machine learning models and / or adding categories of light types to handle accurate inference. The embodiments of the invention described herein can directly handle lenses, any additional filtering, and changing light sources, as well as changing color ranges and quality of emission, by addressing these shortcomings with plenoptic sub-scene reconstruction. The embodiments of the invention described herein can use plenoptic matter fields and light field reconstruction to understand and / or process lenses and any filtering. For example, the embodiments described herein can understand how and why the color of the emission varies from the center, for example, by encoding the physical properties of the emission through light field processing.

[0208] More specifically, one potential benefit of the approach described herein is to distinguish between traffic signals and traffic signal reflections, such as reflections in windows of buildings and other vehicles or reflections from shiny surfaces such as the sides of metal truck trailers. In known approaches to this problem, reflections may require adding new categories to the machine learning model of traffic signals in different states reflected by different materials in order to sort the reflections from the actual signals. This approach can be problematic and / or costly. The approach described herein can substantially eliminate these problems by understanding glass or other reflective materials as reflective surfaces. Plenoptic reconstruction allows the system to understand materials in the scene, such as glass on buildings or shiny surfaces on vehicles, eliminating the need to add categories to the machine learning model to handle reflections.

[0209] Additionally, embodiments of the invention described herein that provide a reilluminatable model may be operable to create and / or provide a synthetic dataset of images for training, e.g., as a light-independent model. Such a synthetic dataset may be advantageous in distinguishing between incandescent light bulbs behind red or yellow glass or plastic covers that may have similar light characteristics in some circumstances.

[0210] Some embodiments of the present invention may optionally include a human-computer interface for performing generalized scene reconstruction (GSR) and / or other functions of the system.

[0211] In certain embodiments of the invention, reconstruction of a scene, including using GSR, can be performed via a graphical user interface, a running command line interface, voice input (including speech recognition), or other input to a computing device, including a portable computing device, a personal computer, a server, a mobile phone, an augmented reality or virtual reality device, an unmanned aerial vehicle or other vehicle, or other digital device. In some embodiments, the interface can provide the ability to import or build an approximation of the light and matter fields to be subsequently reconstructed (hereafter referred to as a pre-scene). In such embodiments, this starting point can improve the speed of the reconstruction process and / or avoid errors. Some embodiments of the invention can provide or otherwise make accessible primitive shapes, generic objects, and digitally generated lights to build the pre-scene. When using a mobile device, some embodiments of the invention can provide the user with the option to physically walk through the scene to be reconstructed as part of the pre-scene, placing lights and matter in positions corresponding to the physical location of the device. In some embodiments, the pre-scene can also be constructed from a previous reconstruction, which can be optionally updated with new scene data. For example, the pre-scene may be updated globally, updated only in the light field, updated only in the material field, updated only in specified regions, or any combination of the above.

[0212] In certain embodiments of the invention, the human computer interface can provide control over the sensing devices used to acquire images. For example, a user may be permitted to change device settings and / or view potential image inputs in a preview video feed before initiating capture. In some embodiments, the human computer interface can display an overlay of a selected pre-scene and allow the user to begin capture only after the pre-scene approximately aligns with the preview video feed. In such embodiments, the system may be configured to spatially snap to the preview of the reconstruction. During capture, the human computer interface can also display ongoing video feeds from each input device. If input from the sensing devices is paused, the human computer interface can request and / or assist in alignment between the preview video feed and existing portions of the reconstruction preview before input is resumed.

[0213] In certain embodiments of the invention, the human-computer interface can provide a live preview of the reconstruction during the capture process, updated for each incoming image or video frame. If a pre-scene is used, the reconstruction preview can include the pre-scene. The video feed is displayed in one or more areas of the screen, and the live preview is displayed in one or more other areas of the screen, and the video feed and the reconstruction preview can be compared. The live preview can display the entire light field and material field as illustrated in FIG. 12A, or can display only a portion of the light field and / or material field (such as the BLIF in a small area) as illustrated in FIG. 12B. Analysis information can be overlaid on the camera feed display and / or on the live preview display, including information regarding false coloring related to set parameters (such as resolution) and / or BLIF related to one or more areas on the screen. In certain embodiments of the invention, the live preview of the reconstruction can be optionally navigated by rotating, panning, and / or zooming.

[0214] An embodiment of the invention can arrange the video feed 1101 and the reconstructed preview 1102 in one or more configurations. In an embodiment, the video feed 1101 and the preview 1102 can have clipping masks of dots, checkers, or other shapes that can be arbitrarily sized and arranged in an arbitrarily regular grid, as illustrated in exemplary Figures 11 and 12, to show the reconstructed preview on top of a background layer showing the video feed, both spatially aligned and rendered at the same viewing angle. In an embodiment, the video feed and preview can have clipping masks of arbitrarily sized irregular shapes that can be arbitrarily randomly arranged, to show all or part of the reconstructed preview and / or on top of a background layer showing the video feed, both spatially aligned and rendered at the same viewing angle. In an embodiment, the video feed and preview can have adaptive clipping masks that show specified features of the reconstructed preview on top of a background layer showing the video feed, both masks and layers aligned to the maximum extent possible. In some embodiments, the video feed and preview may have a rectangular window showing the reconstructed preview over a background window showing the video feed, with the preview and background windows aligned as fully as possible.

[0215] In embodiments such as those illustrated in FIGS. 12A and 12B, the reconstruction preview and video feed can be designed to allow a user to navigate the reconstruction process without prior training in the use of 3D software. Certain embodiments of the invention can make the differences between the reconstruction preview (optionally including a pre-scene) and the current camera feed very apparent, such as by using a checkerboard, dot pattern, or other interleaving between the preview and the live capture. In some embodiments, this implementation can make errors or lags in the reconstruction more noticeable. In such embodiments, the user may be able to notice areas of the scene that have changed since the prior reconstruction, which may constitute or include the pre-scene. In some embodiments, the invention can provide information regarding errors or lags, which can provide feedback to the user regarding the speed at which the scene or part of the scene is being captured, view an area of ​​the scene from multiple angles, or obtain additional information about the scene or part of the scene by performing another action. In some embodiments, the system can facilitate the user's understanding of whether the reconstruction is accurate enough before completing the capture process. For example, as illustrated in Figures 12A and 12B, the preview can be designed such that as the fidelity of the reconstruction increases during capture, the differences between the live capture and the preview decrease, including to the point where there are little or no discernible differences between the live capture and the preview.

[0216] In certain embodiments of the invention, a user can provide feedback to a guide 1301 on the process of improving a two-way light interaction function (BLIF) reconstruction by capturing a selected medial from as many angles as possible. An exemplary embodiment of such a BLIF capture guide 1301 is illustrated in FIG. 13. In one embodiment of the invention, the BLIF capture guide 1301 displays a spherical augmented reality overlay over the video feed and / or reconstruction preview. Sections of this spherical overlay can optionally change (e.g., by disappearing, changing color, or undergoing other visible changes) as the user moves around the medial's corresponding location in real space and views it from various angles. In this way, the overlay can assist the user in determining how many angles the user has already viewed.

[0217] In certain embodiments of the invention, the human computer interface may provide a means to create a program that is read during capture, which may optionally guide the operation of the user and / or the automated equipment. For example, the program may include a desired resolution of the light field and / or material field, a desired certainty threshold for reconstruction, and / or goals of the reconstruction process, such as a goal to eliminate gaps in the captured information. The program may also include functionality for responding to triggers encountered during the input capture process. Triggers may include as a change in the specified material field and / or light field structure, the passage of time, and / or the level of uncertainty of the model generated by the input information. Upon encountering these triggers, the software may change its display configuration, add overlays to the display that may be positioned using augmented reality, play audio cues or verbal instructions, change the reconstruction goal, and / or modify the settings of any connected devices. In some embodiments of the invention, the user may link triggers to corresponding functions using a node-based program editor in the graphical user interface. In some embodiments of the invention, the user may also create paths that are used to guide future capture processes by drawing lines in a pre-scene or by moving the capture device along a desired path in physical space. In some embodiments, triggers can be designed to increase the usability or accessibility of the system to a user.

[0218] In certain embodiments of the invention, the human-computer interface can render and display the completed reconstruction. In some embodiments, the display can include analytical visualizations in addition to realistic views. In some embodiments, one or more medials, radiels, voxels, and saels can be rendered as small primitive shapes centered at one or more locations. False coloring can be done corresponding to any property that can be changed by the medial or radiel, which can include Z-depth, ambient occlusion, and / or segmentation. BLIFs are rendered corresponding to the generated outgoing light, or may be replaced by default BLIFs to provide a uniform view of a particular geometry in the scene. In some embodiments, the path of the radiel is traced back a specified number of bounces and can optionally be visualized as an overlay. In some embodiments, the user can move the viewpoint of the rendering camera through the scene in a device-compatible manner, such as scrolling or clicking on a desktop or walking in a VR headset.

[0219] In certain embodiments of the invention, all or a portion of the data captured or reconstructed by the system may be completely hidden from the user and / or automatically deleted. Such a process may include data that may reveal the user's location or other personal or confidential information, the image captured to perform the reconstruction, or the reconstruction data itself. Such data may include all or a portion of either the light field or the matter field or both. For example, in some embodiments, the remaining matter field complete with all light interaction properties may be rendered and displayed in the generic light field and / or a new light field selected by the user. Alternatively, the matter fields discovered during reconstruction may be completely hidden from the user and / or automatically deleted. The remaining light field may be used to illuminate the generic matter field and / or a new material field selected by the user. Alternatively, the light field and the material fields discovered during reconstruction may be completely hidden from the user and / or automatically deleted. The remaining light interaction properties may be applied to the generic matter field and / or the new material field selected by the user, which may be rendered and displayed in the generic light field and / or the new material field selected by the user. In one embodiment, the system may use AI / ML to identify and remove information of concern or identified for deletion or concealment.

[0220] In some embodiments of the invention, the human-computer interface allows the user to edit the light and matter field reconstructions. In some embodiments, the user can transform, distort, or relight all or any part of the reconstruction. In some embodiments, the user can change the light interaction properties of the BLIFs and assign one or more BLIFs to different regions of the matter field. In some embodiments of the invention, the user can manipulate the scene by dragging on anchor points, by entering keyboard shortcuts, or by sculpting and painting on the reconstruction with a brush tool. In some embodiments of the invention, the user can insert new matter and light fields into the reconstruction and / or relight matter fields (in whole or in part) to match a specified light field. In some embodiments, the user can delete all or part of the light and matter fields.

[0221] In certain embodiments of the invention, the human-computer interface allows the user to select medials and radiels, for example by selecting segments, by selecting medials and radiels individually or together, or by using machine learning to make the selection based on semantic descriptors. In some embodiments, the user can group medials and radiels and / or select groups. In some embodiments, groups and segments can be hidden or visible.

[0222] In certain embodiments of the invention, the human computer interface allows the user to perform metric analysis on the reconstruction. For example, the user may take one or more measurements of the matter field structure and optionally use all or a portion of such measurements to calculate geometric properties such as volume. In some embodiments, the measurements and calculations may be saved and exported. In some embodiments, the invention allows the user to spatially search the scene to obtain one or more of a count, selection, or group of light field and / or matter field structures that sufficiently match a query. In certain embodiments of the invention, the query may be provided as a selected region of light and / or matter, presented as descriptive terms that generate a response based on machine learning, or a combination of the two. The characteristics evaluated to determine a match may include the shape of the matter field, the structure of the light field, the radiant intensity, the size, and the BLIF.

[0223] In some embodiments, the system may be designed to use information, such as information related to light fields and / or material fields in a scene, to aid in procedural generation activities. For example, the system may be used to create iterations or extensions of the reconstructed light field and / or material field to vary the size or shape of the reconstruction. Additionally, the system may adapt lighting conditions associated with the generated data based on the original lighting conditions or the computed light field in the scene.

[0224] Further aspects and embodiments of the present invention will become apparent from the following listing of embodiments.

[0225] 1. A method for reconstructing a scene, the method comprising: accessing image data characterizing light in the scene; processing the image data to provide a light field model and a matter field model, where the matter field model includes a representation of a medium in the matter field including a function characterizing an interaction of light with the medium at a location within the matter field; storing the light field model and the matter field model in a data structure, where the data representing the light field model is separately accessible and the data representing the matter field model is separately accessible and configured to be reconstructed using a function that uses light represented in the light field model and properties of light different from the light represented in the light field model; and outputting at least a portion of the matter field model.

[0226] 2. The method of embodiment 1, comprising a step of outputting at least a portion of the light field model.

[0227] 3. The method of embodiment 1, wherein the function is a two-way light interaction function.

[0228] 4. The method of embodiment 1, wherein the functions characterize absorption, transmission, reflection, scattering, refractive index, roughness, polarized diffusion coefficient, unpolarized diffusion coefficient, and extinction coefficient associated with the medium.

[0229] 5. The method of embodiment 1, wherein the matter field model comprises a plenoptic field represented by at least three dimensions that characterize the position, size, and shape of a medium in the scene and at least two dimensions that characterize the interaction of light with the medium.

[0230] 6. The method of embodiment 1, wherein the processing includes using a machine learning model to perform at least a portion of the processing.

[0231] 7. The method of embodiment 6, wherein the machine learning model includes one or more of a physics-informed neural network (PINN), a neural network architecture with physical constraints, incorporating physical prior knowledge in the loss function, hybrid modeling, and residual modeling.

[0232] 8. The method of embodiment 6, wherein the machine learning model provides additional information characterizing the size, shape, position, or interaction with light of the medium.

[0233] 9. The method of embodiment 6, wherein the processing includes determining parameters of the light field model, the matter field model, or both, using neural networks and non-neural networks.

[0234] 10. The method of embodiment 1, further comprising a step of accessing data providing additional information regarding the light field and / or media field in the scene, and the processing further comprises using the additional information to inform the processing.

[0235] 11. The method of embodiment 10, wherein the additional information includes one or more of information characterizing the size, shape, relative position, or light interaction characteristics of the media within the scene.

[0236] 12. The method of embodiment 1, wherein the model is one or more of decomposable, recomposable, and explainable.

[0237] 13. The method of embodiment 1, wherein the matter field model is configured to enable extraction of one or more segments of media within the scene.

[0238] 14. The method of embodiment 13, further comprising the step of placing the extracted segments in different locations in the scene or in different scenes.

[0239] 15. The method of embodiment 1, wherein the processing allows for localized regions of higher resolution or additional computational energy is applied to reconstruct certain scene features with greater accuracy than other regions.

[0240] 16. A method for representing surface light fields in various regions of a scene using a neural network, the scene model including at least one plenoptic field.

[0241] 17. The method of embodiment 16, implemented to support an autostereoscopic screen or intermediate space.

[0242] 18. A method of using a neural network to represent entities in a scene including one or more of a bidirectional light interaction function, a two-dimensional, four-dimensional, or both light field, and a reilluminatable matter field.

[0243] 19. How to construct a reilluminatable material field (RMF) from a light field model.

[0244] 20. A method for simultaneously optimizing parameters of a neural network and classical equations, the classical equations being understood in a field such as physics.

[0245] 21. An optimization method that involves simultaneously determining the parameters of neural and non-neural networks.

[0246] 22. A method for reconstructing a scene, comprising: accessing image data including one or more images of the scene; using a data structure representing a plurality of volumetric elements representing the scene in a memory; using the image data to reconstruct radiometric properties of each of the volumetric elements, where the radiometric properties characterize a light field associated with each of the volumetric elements and an interaction of the one or more light fields with a medium within the volumetric elements, the set of reconstructed radiometric properties being sufficient to reconstruct a viewpoint of the scene from a viewpoint other than the one or more viewpoints; inputting the reconstructed radiometric properties into the data structure; and outputting at least a portion of the data structure representing a viewpoint of the scene from a viewpoint other than the one or more viewpoints.

[0247] 23. The method of embodiment 22, wherein the radiometric characteristics include one or more of: light emitted from the volumetric element, intensity of color in the volumetric element, a level of transparency associated with the volumetric element, a level of transmittance associated with the volumetric element, or a level of opacity associated with the volumetric element.

[0248] 24. The method of embodiment 22, wherein the data structure includes a hierarchical, multi-resolution, spatially sorted data structure, and the step of inputting the data structure includes inputting one or more of an empty data structure, a partially populated data structure, or a data structure populated with previously calculated radiometric characteristics.

[0249] 25. The method of embodiment 22, wherein the step of reconstructing the radiometric characteristics includes using the image data by selecting a viewpoint and calculating radiometric characteristics associated with each volumetric element along one or more paths extending from the viewpoint.

[0250] 26. The method of embodiment 25, wherein the one or more paths are represented by extending a ray outward from the viewpoint and determining one or more volumetric elements through which the ray passes.

[0251] 27. The method of embodiment 22, wherein the light field represents one or more of light flowing into and out of a volumetric element, and interaction with the medium is represented by calculating one or more of transmittance, opacity, transparency associated with the volumetric element, and surfaces present in one or more volumetric elements.

[0252] 28. The method of embodiment 27, further comprising the step of relighting all or part of the scene using media interaction.

[0253] 29. The method of embodiment 22, wherein the step of reconstructing the radiometric characteristics uses one or more of spherical harmonics, interpolation, machine learning, or machine intelligence.

[0254] 30. The method of embodiment 22, wherein the step of reconstructing the radiometric characteristics includes optimizing the reconstructed radiometric characteristics by performing repeated reconstructions until the reconstructed radiometric characteristics exceed a threshold level of accuracy, certainty, reliability, or another factor.

[0255] 31. A system for reconstructing a scene, comprising: a storage medium configured to store image data from one or more viewpoints of the scene and a model of radiometric properties of the scene; accessing at least a portion of the image data; creating a model including radiometric properties associated with a plurality of volumetric elements; reconstructing radiometric properties of each of the volumetric elements of the scene using the image data, where a set of reconstructed radiometric properties is sufficient to enable the scene to be viewed from a viewpoint other than the one or more viewpoints associated with the image data; inputting the reconstructed radiometric properties into the model; and a processor configured to at least temporarily store the model in the storage medium; and an output circuit configured to output the model.

[0256] 32. The system of embodiment 31, wherein the radiometric characteristics include one or more of the following: emitted light from the volumetric element, color intensity in the volumetric element, a level of transparency associated with the volumetric element, a level of transmittance associated with the volumetric element, or a level of opacity associated with the volumetric element.

[0257] 33. The system of embodiment 31, wherein the storage medium includes a hierarchical, multi-resolution, spatially sorted data structure, and the processor is configured to at least temporarily store the model in the data structure.

[0258] 34. The system of embodiment 31, wherein the processor is configured to create a model from one or more of an empty model, a partially populated model, or a model populated with previously calculated radiometric data.

[0259] 35. The system of embodiment 31, wherein the processor is configured to reconstruct radiometric characteristics using the image data by selecting a viewpoint and calculating radiometric characteristics associated with each volumetric element along one or more paths extending from the viewpoint.

[0260] 36. The system of embodiment 35, wherein the one or more paths are formed by extending a ray of light outward from a viewpoint and determining one or more volumetric elements through which the ray of light passes.

[0261] 37. The system of embodiment 31, wherein the reconstructed radiometric characteristics include a reconstruction of a light field in the scene, the light field representing light flowing into and out of one or more volumetric elements.

[0262] 38. The system of embodiment 37, wherein the model further includes a reconstruction of a matter field in the scene, the matter field representing one or more surfaces present in one or more volumetric elements.

[0263] 39. The system of embodiment 38, wherein the matter field is represented by one or more of the refractive index, polarization characteristics, the presence of one or more holes, transmittance, opacity, or transparency associated with the volumetric element.

[0264] 40. The system of embodiment 38, wherein the matter field is a relightable matter field.

[0265] 41. The system of embodiment 31, wherein the processor is configured to reconstruct the radiometric characteristics using one or more of spherical harmonics, machine learning, or machine intelligence.

[0266] 42. The system of embodiment 31, wherein the processor performs the reconstruction iteratively until the processor determines that the model exceeds a threshold level of accuracy.

[0267] 43. A method for training a machine learning model, comprising: providing image data to the machine learning model, where the image data includes information characterizing light in a scene; generating a reilluminatable model of the scene by processing the image data, where such processing includes dividing the scene into a plurality of volumetric elements, analyzing the image data to generate a model of a light field in at least some of the volumetric elements, predicting interactions between light in the scene and media in one or more volumetric elements, and providing information for characterizing the appearance of the media in lighting conditions other than the light in the image data; and outputting the model.

[0268] 44. The method of embodiment 43, wherein the scene includes one or more objects of interest.

[0269] 45. The method of embodiment 44, further comprising the steps of extracting one or more objects of interest from the model and inserting the one or more extracted objects of interest into the second scene.

[0270] 46. ​​The method of embodiment 43, wherein the processing further includes determining shape information associated with a medium in the scene, and the prediction includes calculating a two-way light interaction function associated with the medium.

[0271] 47. The method of embodiment 46, wherein the processing further includes generating a model of the medium in the scene as a material field that characterizes the medium in the scene in at least three dimensions.

[0272] 48. The method of embodiment 43, wherein the image data includes one or more reilluminatable models.

[0273] 49. The method of embodiment 48, further comprising the steps of: varying a two-way light interaction function associated with the output model to create a modified model; inputting the modified model into a machine learning model; and repeating the process to further train the machine learning model.

[0274] 50. The method of embodiment 43, further comprising the steps of receiving a second set of image data, generating a second model by processing the second set of image data using the trained machine learning model, and outputting the second model.

[0275] 51. A system for reconstructing one or more objects in a scene, the system comprising: a processor for processing digital scene data; and an interface for receiving input related to the scene to be captured, the input including the digital scene data in the form of image data representing the scene from an orientation; the processor processes the digital scene data and the input to generate a three-dimensional model of at least a portion of the scene including a material having at least one surface, the processor processes the image data by checking one or more volumetric elements in a material field represented by the image data, and the processor processes the image data by determining whether the material represented in each of the one or more volumetric elements has a surface.

[0276] 52. The system of embodiment 51, wherein the image data is captured by a camera.

[0277] 53. The system of embodiment 51, wherein the orientation is a camera pose.

[0278] 54. The system of embodiment 51, wherein the image data includes data regarding electromagnetic radiation.

[0279] 55. The system of embodiment 54, wherein the data relating to electromagnetic radiation includes one or more radiance values ​​of visible light, infrared light, polarized light, unpolarized light, and / or radar.

[0280] 56. The system of embodiment 51, wherein the digital scene data includes image data representing the scene from at least two directions.

[0281] 57. The system of embodiment 56, wherein the processor sequentially processes image data from at least two directions.

[0282] 58. The system of embodiment 51, wherein the material represented in the volumetric elements is represented by surfels.

[0283] 59. The system of embodiment 58, wherein the data relating to surfels includes one or more of an outgoing light field and an incoming light field.

[0284] 60. The system of embodiment 51, wherein the processor processes the image data by assuming an orientation of the digital scene data.

[0285] 61. The system of embodiment 51, wherein the processor processes the image data by assuming that a surface exists in the volumetric element, assuming one or more of a surface normal, light interaction properties, an outgoing radiance vector, and an incident light field of the surface, calculating a cost for the surface to exist in the volumetric element based on one or more of the assumed surface normal, light interaction properties, an outgoing radiance vector, and an incident light field of the surface, comparing the cost to a cost threshold, and accepting that a surfel exists in the volumetric element if the cost is less than the cost threshold.

[0286] 62. The system of embodiment 61, wherein if the system accepts a surfel as being present in the volumetric element, the surface remains in the scene in subsequent processing of the scene.

[0287] 63. The system of embodiment 61, further comprising updating light field assumptions for one or more other volumetric elements based on the presence of the accepted surfels.

[0288] 64. The system of embodiment 61, wherein the processor performs processing iteratively on one or more volumetric elements.

[0289] 65. The system of embodiment 61, wherein the processor performs processing iteratively on one or more sets of image data.

[0290] 66. The system of embodiment 63, wherein the light field is not delivered to the user.

[0291] 67. The system of embodiment 63, wherein the matter field is not delivered to the user.

[0292] 68. The system of embodiment 63, wherein the light field and matter field are not delivered to the user, and the light interaction properties of the matter field are delivered to the user.

[0293] 69. A method for training a machine learning model, comprising the steps of: providing image data to the machine learning model, where the image data includes one or more objects of interest; processing the image data to generate a model, where such processing includes analyzing the image data to generate one or more light field models of the scene or reconstructions of one or more matter fields in the scene; selecting the objects of interest in the model of the scene; extracting the objects of interest in the model of the scene; and outputting a re-illuminatable matter field model of the objects of interest in the scene.

[0294] 70. The method of embodiment 69, wherein the image data includes reilluminatable material field data.

[0295] 71. The method of embodiment 69, wherein the image data includes one or more of an object of interest in a plurality of scenes and an object of interest under various conditions.

[0296] 72. The method of embodiment 69, wherein the reilluminatable material field is composed of multiple images in two or more dimensions.

[0297] 73. The method of embodiment 69, wherein the reilluminatable material field model includes one or more of shape information and two-way light interaction function (BLIF) information.

[0298] 74. The method of embodiment 69, wherein the light field information is used to calculate light reflectance characteristics of locations within the material field.

[0299] 75. The method of embodiment 69, wherein the light field is not delivered to the user.

[0300] 76. The method of embodiment 69, wherein the matter field is not delivered to the user.

[0301] 77. The method of embodiment 69, wherein the light field and the matter field are not delivered to the user, and the light interaction properties of the matter field are delivered to the user.

[0302] 78. The method of embodiment 69, further comprising the steps of altering BLIF information of the model, inputting the model having the altered BLIF information into a machine learning model, and further training the machine learning model by performing one or more of the aforementioned steps on the model having the altered BLIF information.

[0303] 79. A method for using a machine learning model, comprising: identifying one or more objects of interest in a model of a scene; accessing a relightable matter field of the scene; selecting a portion of the matter field to be processed; extracting at least a portion of the relightable matter field by processing the selected portion of the matter field; and outputting the extracted portion of the relightable matter field.

[0304] 80. The method of embodiment 79, further comprising a step of testing the usefulness of the portion of the reilluminatable material field output by the machine learning model.

[0305] 81. A system for reconstructing one or more objects in a scene, comprising: a processor for processing digital scene data; and an interface for receiving input related to the scene to be captured, wherein the processor generates a three-dimensional model of at least a portion of the scene by processing the digital scene data and the input, the processor directs at least a portion of the processing of the digital scene data by the input, and the processor provides an output including the three-dimensional model of at least a portion of the scene.

[0306] 82. The system of embodiment 81, wherein the input includes at least one of an approximation of at least a portion of a light field in the scene, an approximation of at least a portion of a matter field in the scene, one or more shapes present in the scene, one or more objects in the scene, or information relating to one or more light sources in the scene.

[0307] 83. The system of embodiment 81, wherein the input controls one or more sensing devices that provide digital scene data.

[0308] 84. The system of embodiment 81, wherein the system provides feedback regarding one or more objects being reconstructed within the scene.

[0309] 85. The system of embodiment 84, wherein the feedback includes a preview of one or more objects to be reconstructed within the scene.

[0310] 86. The system of embodiment 85, wherein the system updates the preview as one or more objects are reconstructed with results from such reconstruction.

[0311] 87. The system of embodiment 86, wherein the preview further includes one or more indications regarding one or more parameters of the reconstruction.

[0312] 88. The system of embodiment 85, wherein the preview includes one or more masks representing data related to the generated model and information received from the digital scene data capture device.

[0313] 89. The system of embodiment 84, wherein the feedback includes one or more of information related to a capture rate of the digital scene data, a position at which the digital scene data is captured, a sensor angle at which the digital scene data is captured, an aspect of a light field within the scene, or an aspect of a matter field within the scene.

[0314] 90. The system of embodiment 81, wherein the input is data that enables alignment of the digital scene data with newly received digital scene data.

[0315] 91. The system of embodiment 81, wherein the system further includes a set of instructions for achieving one or more goals for generating a three-dimensional model.

[0316] 92. The system of embodiment 91, wherein the one or more goals include one or more of a desired resolution of the light field, a desired resolution of the matter field, a desired certainty threshold for reconstruction, a threshold for removing gaps in the captured digital scene information, and a trigger for an event encountered during capture of the digital scene information.

[0317] 93. The system of embodiment 92, wherein the trigger includes one or more of a specified matter field structure, a specified light field structure, the passage of time, and a change in the level of model uncertainty.

[0318] 94. The system of embodiment 92, configured to take an action in response to a trigger.

[0319] 95. The system of embodiment 94, wherein the response includes one or more of changing the display configuration, adding an overlay to the display, providing an audio cue, providing a visual cue, changing the reconfiguration target, and changing settings of a device connected to the system.

[0320] 96. The system of embodiment 81, configured to modify one or more characteristics of the model.

[0321] 97. The system of embodiment 96, wherein the modifications include one or more of: editing the light field reconstruction, editing the matter field reconstruction, transforming the model, transforming the model, relighting all or a portion of the model, changing one or more light interaction properties of a BLIF, assigning one or more BLIFs to different regions of a matter field, manipulating the model by dragging on anchor points, entering keyboard shortcuts, or sculpting and painting on the model using a brush tool, inserting a new matter field, inserting a new light field, relighting one or more matter fields (all or a portion), deleting all or a portion of a light field, and deleting all or a portion of a matter field.

[0322] 98. The system of embodiment 81, configured to spatially search models using a search query including one or more parameters.

[0323] 99. The system of embodiment 98, wherein the spatial search includes obtaining one or more of a count, selection, or group of light field structures, or obtaining one or more of a count, selection, or group of material field structures that match one or more parameters of the search query.

[0324] 100. The system of embodiment 98, wherein the search query is provided as a selected region of light.

[0325] 101. The system of embodiment 98, wherein the search query is provided as a region of the selected material.

[0326] 102. The system of embodiment 98, wherein the search query is presented as descriptive terms that generate responses based on machine learning.

[0327] 103. The system of embodiment 98, wherein the one or more parameters include one or more of matter field shape, light field structure, radiometric intensity, size, and BLIF.

[0328] 104. The system of embodiment 81, further comprising a display used to capture digital scene information, wherein during capture, information from multiple sources is presented in three or more adjacent regions of the display as spatially interleaved layers.

[0329] 105. The system of embodiment 104, wherein at least one of the regions is a live reconstruction preview.

[0330] 106. The system of embodiment 104, wherein all layers on the display are aligned to substantially the same viewpoint.

[0331] 107. The system of embodiment 104, wherein all layers on the display contain information about the scene.

[0332] 108. The system of embodiment 104, wherein one of the layers on the display is a pre-scene rendering aligned to substantially the same viewpoint as the other layers.

[0333] 109. The system of embodiment 81, further comprising a display used during capture to show how many angles have already been captured around a particular BLIF.

[0334] 110. The system of embodiment 109, wherein the display is performed by displaying a spherical or hemispherical overlay centered on the selected medial containing the BLIF.

[0335] 111. The system of embodiment 110, wherein at least one section of the spherical overlay changes in response to viewing the medial from various angles relative to a corresponding position of the medial in real space.

[0336] 112. The system of embodiment 111, wherein the change to at least one section of the spherical overlay includes one or more of disappearance, color change, or other visible change.

[0337] 113. A method for reconstructing one or more objects in a scene, the method comprising: accessing digital scene data and an input related to the scene; processing the digital scene data to generate a three-dimensional model of at least a portion of the scene, wherein the processing includes directing at least some aspects of the processing of the digital scene data in response to the input; and outputting the three-dimensional model of at least a portion of the scene.

[0338] 114. The method of embodiment 113, wherein the input includes at least one of an approximation of at least a portion of a light field in the scene, an approximation of at least a portion of a matter field in the scene, one or more shapes present in the scene, one or more objects in the scene, and information relating to one or more light sources in the scene.

[0339] 115. The method of embodiment 113, further comprising a step of controlling one or more sensing devices that provide digital scene data using the input.

[0340] 116. The method of embodiment 113, further comprising the step of providing feedback regarding one or more objects reconstructed within the scene.

[0341] 117. The method of embodiment 116, wherein the step of providing feedback includes providing a preview of one or more objects to be reconstructed within the scene.

[0342] 118. The method of embodiment 117, further comprising updating the preview as one or more objects are reconstructed with results from such reconstruction.

[0343] 119. The method of embodiment 118, further comprising providing one or more instructions to the preview regarding one or more parameters of the reconstruction.

[0344] 120. The method of embodiment 117, further comprising providing the preview with one or more masks representing data related to the generated model and information received from the digital scene data capture device.

[0345] 121. The method of embodiment 116, wherein the step of providing feedback includes providing one or more information related to a capture speed of the digital scene data, a position at which the digital scene data is captured, a sensor angle at which the digital scene data is captured, an aspect of a light field within the scene, or an aspect of a matter field within the scene.

[0346] 122. The method of embodiment 113, further comprising aligning the digital scene data with newly received digital scene data using the input.

[0347] 123. The method of embodiment 113, further comprising the steps of accessing an instruction set and executing the instruction set to achieve one or more goals related to generating the three-dimensional model.

[0348] 124. The method of embodiment 123, wherein the one or more goals include one or more of a desired resolution of the light field, a desired resolution of the matter field, a desired certainty threshold for reconstruction, a threshold for removing gaps in the captured digital scene information, and a trigger for an event encountered during capture of the digital scene information.

[0349] 125. The method of embodiment 124, wherein the trigger includes one or more of a specified matter field structure, a specified light field structure, the passage of time, and a change in the level of model uncertainty.

[0350] 126. The method of embodiment 124, further comprising taking an action in response to the trigger.

[0351] 127. The method of embodiment 126, wherein taking an action includes one or more of changing a display configuration, adding an overlay to the display, providing an audio cue, providing a visual cue, changing a reconfiguration target, and changing settings of a device connected to the system.

[0352] 128. The method of embodiment 113, further comprising the step of modifying one or more features of the model based on the input.

[0353] 129. The method of embodiment 128, wherein the editing includes one or more of: editing the light field reconstruction, editing the material field reconstruction, transforming the model, deforming the model, relighting all or part of the model, changing one or more light interaction properties of a BLIF, assigning one or more BLIFs to different regions of the material field, manipulating the model by dragging on anchor points, entering keyboard shortcuts, or using a brush tool to sculpt and paint the model, inserting a new material field, inserting a new light field, relighting one or more material fields (all or part), deleting all or part of a light field, and deleting all or part of a material field.

[0354] 130. The method of embodiment 113, further comprising a step of spatially searching the model using a search query including one or more parameters.

[0355] 131. The method of embodiment 130, wherein the spatial search includes obtaining one or more of a count, selection, and group of light field structures, or obtaining one or more of a count, selection, and group of material field structures that match one or more parameters of the search query.

[0356] 132. The method of embodiment 130, wherein the search query is provided as a selected region of light.

[0357] 133. The method of embodiment 130, wherein the search query is provided as a region of the selected material.

[0358] 134. The method of embodiment 130, wherein the search query is presented as descriptive terms that generate responses based on machine learning.

[0359] 135. The method of embodiment 130, wherein the one or more parameters include one or more of: matter field shape, light field structure, radiometric intensity, size, and BLIF.

[0360] 136. The method of embodiment 113, further comprising the step of providing a display for capturing digital scene information, wherein during capture, information from multiple sources is presented in three or more adjacent regions of the display as spatially interleaved layers.

[0361] 137. The method of embodiment 136, wherein at least one of the regions is a live reconstruction preview.

[0362] 138. The method of embodiment 136, wherein all layers on the display are aligned to substantially the same viewpoint.

[0363] 139. The method of embodiment 136, wherein all layers on the display contain information about the scene.

[0364] 140. The method of embodiment 136, wherein one of the layers is a pre-scene rendering aligned to substantially the same viewpoint as the other layers.

[0365] 141. The method of embodiment 113, further comprising the steps of providing a display and using the display during capture of digital image data to indicate how many angles have already been captured around a particular BLIF.

[0366] 142. The method of embodiment 141, wherein the display is provided by displaying a spherical or hemispherical overlay centered on the selected medial containing the BLIF.

[0367] 143. The method of embodiment 142, wherein at least one section of the spherical overlay changes in response to viewing the medial from various angles relative to a corresponding position of the medial in real space.

[0368] 144. The method of embodiment 143, wherein the change to at least one section of the spherical overlay includes one or more of disappearance, a change in color, or other visible change.

[0369] 145. The method of any preceding embodiment, wherein the reconstruction of the opaque exterior structure is combined with the reconstruction of the interior structure to form a more complete reconstruction.

[0370] 146. The method of any preceding embodiment, wherein the internal structure does not yet include BLIF information, and the BLIF information is automatically generated based on the external structure.

[0371] 147. A method for operating a machine learning model, the method comprising: creating a training set including a model of an object, where the model has re-illumination properties that enable reconstruction of the model in incident lighting conditions other than lighting conditions associated with the image data from which the model was created, the re-illumination properties including a function that characterizes an interaction of light at a location in the matter field with a medium in the model; accessing the training set with the machine learning model; training the machine learning model with the training set, where training includes configuring the machine learning model to perform one or more of object classification, surface resolution, light field reconstruction, matter field reconstruction, and material signature identification; and characterizing a new object using the trained machine learning model.

[0372] 148. The method of claim 147, wherein the relightable model includes data representing one or more radial solid angle elements in a plurality of volumetric elements.

[0373] 149. The method of claim 148, wherein the machine learning model includes one or more of a physics-informed neural network (PINN), a neural network architecture with physical constraints, incorporating physical prior knowledge into the loss function, hybrid modeling, and residual modeling.

[0374] 150. The method of claim 147, wherein the functions include one or more bidirectional optical interaction functions (BLIFs).

[0375] 151. The method of claim 150, wherein one or more BLIFs are processed using a neural network or a sampled data function.

[0376] 152. The method of claim 150, wherein at least one of the BLIFs is spatially varying.

[0377] 153. The method of claim 150, wherein the BLIF represents one or more light interaction phenomena including absorption, transmission, reflection, and scattering.

[0378] 154. The method of claim 147, wherein the functions represent properties including refractive index, roughness, characteristics of holes in the medium, polarized diffusion coefficient, unpolarized diffusion coefficient, and extinction coefficient.

[0379] 155. The method of claim 147, wherein the classifying includes one of characterizing a state of a traffic signal.

[0380] In the examples described herein, for purposes of explanation and not limitation, specific details of particular nodes, functional entities, techniques, protocols, standards, etc. are shown to facilitate understanding of the described technology. It will be apparent to one skilled in the art that other embodiments may be practiced apart from the specific details described herein. In other examples, detailed descriptions of well-known methods, apparatus, techniques, etc. are omitted to avoid obscuring the description in unnecessary detail. Individual functional blocks are shown. Those skilled in the art will appreciate that the functions of these blocks can be implemented using individual hardware circuits, using software programs and data in combination with a suitably programmed microprocessor or general-purpose computer, using application specific integrated circuits (ASICs), and / or using one or more digital signal processors (DSPs). The instructions and data of the software programs can be stored in a computer-readable storage medium, and the computer or processor performs the functions when the instructions are executed by a computer or other suitable processor control. Although databases are sometimes depicted as tables in this specification, other formats (including relational databases, object-based models, and / or distributed databases) may be used to store and manipulate data.

[0381] Although process steps, algorithms, and the like may be described or claimed in a particular order, such processes may be configured to operate in different orders. That is, the order or sequence of steps that may be explicitly described or claimed does not necessarily indicate a requirement that the steps be performed in that order. Steps of processes described herein may be performed in any order possible. Furthermore, some steps may be performed simultaneously, even though they are described or implied as occurring other than simultaneously (e.g., because one step is described after the other). Furthermore, the illustration of a process by depiction in a drawing does not imply that the illustrated process is exclusive of other variations and modifications, nor does it imply that the illustrated process or any of its steps are necessary to the technology, nor does it imply that the illustrated process is preferred. Furthermore, process steps, algorithms, and the like described as recursive processes may be performed iteratively, and process steps, algorithms, and the like described iteratively may be performed recursively.

[0382] The above-mentioned processor, memory, network interface, I / O interface, and display may be or include hardware devices (e.g., electronic circuits or combinations of circuits) configured to perform various different functions of a computing device.

[0383] In some embodiments, each or any of the processors is or includes, for example, a single-core or multi-core processor, a microprocessor (e.g., sometimes referred to as a central processing unit or CPU), a digital signal processor (DSP), a microprocessor associated with a DSP core, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, or a system on a chip (SOC) (e.g., an integrated circuit that includes a CPU and other hardware components such as memory, networking interfaces, etc.). In some embodiments, each or any of the processors uses an instruction set architecture such as x86 or high performance RISC machine (ARM).

[0384] In some embodiments, each or any of the memory devices is or includes a random access memory (RAM) (such as dynamic RAM (DRAM) or static RAM (SRAM)), flash memory (e.g., based on NAND or NOR technology), a hard disk, magneto-optical media, optical media, cache memory, registers (e.g., holding instructions), or other types of devices that perform volatile or non-volatile storage of data and / or instructions (e.g., software executed on or by a processor). A memory device is an example of a non-volatile computer-readable storage medium.

[0385] In an embodiment, each or any of the network interface devices includes one or more circuits (such as a baseband processor and / or a wired or wireless transceiver) to implement one or more layer 1, layer 2, and / or higher layers of a wired communication technology (such as Ethernet (IEEE 802.3)) and / or a wireless communication technology (such as Bluetooth, WiFi (IEEE 802.11), GSM, CDMA2000, UMTS, LTE, LTE-Advanced (LTE-A), 5G and 5G New Radio (5GNR) (including but not limited to IEEE 1914.1 and 1914.3), enhanced Mobile Broadband (eMBB), ultra-reliable low latency communications (URLLC), massive machine type communications (mMTC), and / or other short-range, medium-range, long-range wireless communication technologies). The transceiver may have circuitry for a transmitter and a receiver. The transmitter and receiver may share a common housing and may share some or all of the circuitry within the housing to transmit and receive. In some embodiments, the transmitter and receiver of a transceiver may not share common circuitry and / or may be in the same housing or in separate housings.

[0386] In some embodiments, each or any of the display interfaces in the I / O interface receives data from the processor, generates corresponding image data based on the received data (e.g., by a discrete GPU, an integrated GPU, a CPU performing graphical processing, etc.), and / or outputs the generated image data to a display device that displays the image data (e.g., via a High Definition Multimedia Interface (HDMI), a DisplayPort interface, a Video Graphics Array (VGA) interface, a Digital Video Interface (DVI), etc.). Alternatively or additionally, in some embodiments, each or any of the display interfaces is or includes, for example, a video card, a video adapter, or a graphics processing unit (GPU).

[0387] In some embodiments, each or any of the user input adaptors of the I / O interface is or includes one or more circuits that receive and process user input data from one or more user input devices included in, attached to, or otherwise in communication with the computing device, and output data based on the received input data to the processor. Alternatively or additionally, in some embodiments, each or any of the user input adaptors is or includes, e.g., a PS / 2 interface, a USB interface, a touch screen controller, etc., and / or the user input adaptors facilitate input from user input devices, e.g., a keyboard, a mouse, a track pad, a touch screen, etc.

[0388] In transmitting data (e.g., sequences of instructions) to a processor, various forms of computer-readable media / transmissions may be involved. For example, data may (i) be delivered from memory to the processor, (ii) be transmitted over any type of transmission medium (e.g., wired, wireless, optical, etc.), (iii) be formatted and / or transmitted according to a number of wired or wireless formats, standards, or protocols, such as Ethernet (or IEEE 802.3), ATP, Bluetooth, TCP / IP, TDMA, CDMA, 3G, etc., and / or (iv) be encrypted to ensure privacy or prevent fraud in any of a variety of ways known in the art.

[0389] As used herein, it is understood that terms such as system, subsystem, service, programmed logic, and the like can be implemented as software, hardware, firmware, and / or any suitable combination thereof. It is also understood that storage locations herein may be any suitable combination of disk drive devices, memory locations, solid state drives, CD-ROMs, DVDs, tape backups, storage area network (SAN) systems, and / or any other suitable tangible computer readable storage media. It is also understood that the techniques described herein can be accomplished by having a processor execute instructions that can be tangibly stored on a computer readable storage medium.

[0390] As used herein, the term "non-transitory computer-readable storage medium" includes registers, cache memory, ROM, semiconductor memory devices (such as D-RAM, S-RAM, and other RAMs), magnetic media such as flash memory, hard disks, magneto-optical media, optical media such as CD-ROMs, DVDs, or Blu-ray disks, or any other type of device for non-transitory electronic data storage. The term "non-transitory computer-readable storage medium" does not include transient, propagating electromagnetic signals.

[0391] In this document, when a certain action is described as "may", "can be performed", or "likely to be performed", when a certain feature or component is described as "may include", "can include", "may include", or "potential to include" or as applicable to a particular situation, when a certain item is described as "may have", "can have", or "likely to have" a certain attribute, or when similar phrases containing the terms "may", "can", or "likely" are used, it should be understood that the certain action, feature, component, attribute, etc. is present in at least one embodiment, but not in all embodiments.

[0392] While the present invention has been described in connection with what are presently considered to be the most practical and preferred embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. 1. A scene reconstruction and machine learning system, comprising: a storage medium configured to store image data, one or more scene models, one or more reilluminatable material fields, information related to the machine learning model, and an output of the machine learning model; an input circuit configured to receive image data characterizing light in a scene, the scene being occupied by materials including objects; A processor; An output circuit; having The processor, reconstructing a scene model representing the scene using the image data, where the scene model represents volumetric regions in the scene occupied by the material that interacts with the light; extracting a reilluminatable matter field from the scene model representing the object, where the reilluminatable matter field characterizes light interactions with the object; storing the scene model and the reilluminatable matter field representing the object in the storage medium; applying the reilluminatable material field as an input to the learning model; Generate an output from the machine learning model after application of the reilluminatable material field as an input It is structured as follows: The output circuitry is configured to output the generated output.

2. 2. The system of claim 1, The reilluminatable matter field characterizes the light interactions using data representing parameters of a neural network.

3. 2. The system of claim 1, The system, wherein the processor is configured to calculate a solid angle component of light in an exiting light field given a solid angle component of light in an entering light field.

4. 2. The system of claim 1, The system, wherein the reilluminatable matter field represents properties including at least one of refractive index, roughness, absorption, transmission, reflection, scattering, characteristics of holes in a medium, polarized diffusion coefficient, unpolarized diffusion coefficient, and extinction coefficient.

5. 5. The system of claim 4, The system, wherein the property is expressed as at least one two-way light interaction function.

6. 6. The system of claim 5, At least one of the two-way light interaction functions is spatially varying.

7. 2. The system of claim 1, The output of the system is one or more of classification, regression, clustering, prediction, pattern recognition, determining a traffic signal status, detecting surface anomalies, characterizing features of an object, and estimating the cost of repairing an object damaged by hail.

8. 1. A method of leveraging a machine learning model and reilluminatable material field data, comprising: accessing image data characterizing light in a scene, the scene being occupied by materials including objects; reconstructing a scene model representing the scene using the image data, the scene model representing volumetric regions in the scene occupied by the material that interacts with the light; extracting a reilluminatable matter field from the scene model representing the object, the reilluminatable matter field characterizing light interactions with the object; storing the scene model and the reilluminatable matter field representing the object in a storage medium; applying the reilluminatable material field as an input to a machine learning model; generating an output from the machine learning model after application of the reilluminatable material field as an input; The method comprising:

9. 9. The method of claim 8, The method of claim 1, wherein the reilluminatable matter field characterizes the light interactions using data representing parameters of a neural network.

10. 9. The method of claim 8, The method further comprising: calculating a solid angle component of light in an exiting light field, given a solid angle component of light in an incoming light field, utilizing data related to the light interactions.

11. 9. The method of claim 8, The method, wherein the reilluminatable matter field represents properties including at least one of refractive index, roughness, absorption, transmission, reflection, scattering, characteristics of holes in a medium, polarized diffusion coefficient, unpolarized diffusion coefficient, and extinction coefficient.

12. 12. The method of claim 11, The method, wherein the property is expressed as at least one two-way light interaction function.

13. 13. The method of claim 12, A method, wherein at least one of the two-way light interaction functions is spatially varying.

14. 9. The method of claim 8, The method further includes using the output for one or more of classification, regression, clustering, prediction, pattern recognition, determining a traffic signal status, detecting surface anomalies, characterizing features of an object, and estimating the cost of repairing an object damaged by hail.

15. 1. A machine learning system for use with reilluminatable matter field data, comprising: a storage medium configured to store reilluminatable material field data, information related to a machine learning model, and an output of the machine learning model; an input circuit for receiving re-illuminatable matter field data representative of one or more objects, at least a portion of the re-illuminatable matter field data characterizing light interactions with the objects; A processor; An output circuit; having the processor is configured to train the machine learning model using the data as a training set, to receive as input a reilluminatable material field of a new object, and to generate an output in response to the input; The output circuitry is configured to output the generated output.

16. 16. The system of claim 15, The system, wherein the reilluminatable material field data characterizes the light interactions with properties of the object including at least one of refractive index, roughness, absorption, transmission, reflection, scattering, characteristics of holes in a medium, polarized diffusion coefficient, unpolarized diffusion coefficient, and extinction coefficient.

17. 17. The system of claim 16, The properties of the light interactions represent parameters of a neural network of the system.

18. 20. The system of claim 17, The system, wherein the light interaction properties are represented as at least one bidirectional light interaction function.

19. 20. The system of claim 17, At least one of the two-way light interaction functions is spatially varying.

20. 16. The system of claim 15, The output of the system is one or more of classification, regression, clustering, prediction, pattern recognition, determining a traffic signal status, detecting surface anomalies, characterizing features of an object, and estimating the cost of repairing an object damaged by hail.

21. 16. The system of claim 15, the storage medium is configured to store one or more scene models; the input circuitry is configured to receive the one or more scene models; the one or more scene models represent volumetric regions in the scene that are occupied by light-interacting materials; the processor is configured to extract reilluminatable material field data from the one or more scene models; the reilluminatable material field data represents an object; At least a portion of the reilluminatable matter field data characterizes a light field emanating from the object given a light field incident on the object.

22. 1. A method for training a machine learning model using reilluminatable material field data, comprising: collecting data for said reilluminatable material field representative of one or more objects; training the machine learning model using the reilluminatable material field data as a training set; having at least a portion of the reilluminatable matter field data characterizes light interactions with the object; The method of claim 1, wherein the trained machine learning model is configured to receive as input a reilluminatable matter field of a new object and to generate an output in response to the input.

23. 23. The method of claim 22, The method, wherein the reilluminatable material field data characterizes the light interactions using properties of the object including at least one of refractive index, roughness, absorption, transmission, reflection, scattering, characteristics of holes in a medium, polarized diffusion coefficient, unpolarized diffusion coefficient, and extinction coefficient.

24. 24. The method of claim 23, The properties of the light interactions represent parameters of a neural network.

25. 24. The method of claim 23, The method, wherein the light interaction properties are represented as at least one two-way light interaction function.

26. 26. The method of claim 25, A method, wherein at least one of the two-way light interaction functions is spatially varying.

27. 23. The method of claim 22, The method, wherein the output is used for one or more of classification, regression, clustering, prediction, pattern recognition, determining a traffic signal status, detecting surface anomalies, characterizing features of an object, and estimating the cost of repairing an object damaged by hail.

28. 23. The method of claim 22, the collecting step includes accessing one or more scene models representing a scene, and extracting a reilluminatable material field from the one or more scene models; the scene model represents volumetric regions in the scene that are occupied by materials that interact with light; The relightable material represents the object.

Images