Codec for processing scenes with nearly infinite details

Abstract

Methods and apparatus are described for a system using a scene codec, where the system is a provider or consumer of multi-way, punctualized, on-demand-only scene data including scenes and scene increments. An exemplary system using a scene codec includes a plenoptic scene database including one or more digitized scene models in which representations and organizations of representations can be distributed across multiple systems such that the multiple systems can collectively represent a scene with nearly infinite details. The system may further include efficient means for processing the representations and organization of the representations to provide punctualization, on-demand only sub-scenes and scene increments required to ensure a maximum degree of continuous user experience with a minimum amount of newly provided scene information, where the efficient means include a spatial processing unit.

Description

[0001] This application is a divisional application of the patent application with application number 201980029663.7, application date May 2, 2019, and invention name “Codec for processing scenes with almost infinite details”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 665,806, filed May 2, 2018, the entire contents of which are incorporated herein by reference. This application is related to PCT Application No. PCT / US2017 / 026994, filed April 11, 2017, the entire contents of which are incorporated herein by reference. Technical Field

[0004] The present disclosure relates to scene representation, processing and acceleration in distributed digital networks. Background Art

[0005] Various codecs are well known in the art and are generally devices or programs that compress data for faster transmission and decompress received data. Typical types of codecs include video (e.g., MPEG, H.264), audio (e.g., MP3, ACC), image (e.g., JPEG, PNG), and data (e.g., PKZIP), where the type of codec encapsulates the type of data and is strongly coupled to the type of data. Although these types of codecs are satisfactory for applications limited to that type of data, inherent in the strong coupling is a limited end-user experience.

[0006] Codecs are inherently "file-based," where a file is a data representation of some real or synthetic pre-captured sensory experience, and where a file (such as a movie, song, or book) necessarily constrains the user's experience to the experience path chosen by the file's creator. Thus, we watch movies, listen to songs, and read books in a largely ordered experience constrained by the creator.

[0007] Technological advances in the market provide more means for expanding data types and experiencing data types. The increase in data types includes data types generally referred to as real-world scene reconstruction, in which sensors (such as cameras and ranging devices) create scene models of real scenes. The inventors proposed significant advances in scene reconstruction in patent application PCT / 2017 / 026994 "Everyday Life Scene Reconstruction Engine" filed on April 11, 2017, the entire contents of which are incorporated herein by reference. Improvements in the means for experiencing data types include higher resolution and better performance 2D and 3D displays, automatic stereoscopic displays, holographic displays, and extended reality devices and methods such as virtual reality (VR) headsets and augmented reality (AR) headsets. Other significant technological advances include the prevalence of automata, in which humans are no longer the only consumers of real-world sensory information, and the prevalence of networks, in which the flow and acquisition of information enables new experience paradigms.

[0008] Some work has been done to develop new scene-based codecs, where the data type is reconstruction of real-world scenes and / or computer generation of synthetic scenes. For evaluation of scene codecs, the reader is invited to read the technical report of the Joint Ad Hoc Group on Digital Representation of Light / Acoustic Fields for Immersive Media Applications, as published by the "Joint Ad Hoc Group on Digital Representation of Light / Acoustic Fields for Immersive Media Applications", which is incorporated herein by reference in its entirety.

[0009] Scene reconstruction and distribution are problematic, where reconstruction presents challenges in representation and organization of representations that adequately describe the complexity of real-world matter and light fields in an efficiently controllable and highly scalable manner, and where distribution presents challenges in managing active (even real-time) scene models across multiple interactive clients (both humans and automatons), each potentially requesting any of a nearly unlimited number of scene perspectives, details, and data types.

[0010] Therefore, there is a need to overcome the shortcomings and deficiencies in the art by providing an efficient and flexible system that addresses the many needs and opportunities in the marketplace. Summary of the invention

[0011] The following overview can provide a basic understanding of some aspects of the system and / or method discussed herein. This overview is not an extensive review of the system and / or method discussed herein. It is not intended to point out all key (key / critical) elements or to delimit the scope of such system and / or method. The only purpose is to present some concepts in a simplified form as a preface to a more detailed description presented later.

[0012] Provided herein are methods and apparatus for supporting a system that utilizes a scene codec, wherein the system is a provider or consumer of multi-channel, just-in-time, on-demand-only scene data, including sub-scenes and sub-scene increments. According to some embodiments, a system that utilizes a scene codec includes: a plenoptic scene database containing one or more digitized scene models, wherein representations and organization of representations are capable of being distributed across multiple systems such that the multiple systems can collectively represent scenes with nearly unlimited detail. The system may further include efficient means for processing these representations and organization of representations, thereby providing the just-in-time, on-demand-only sub-scenes and scene increments required to ensure a maximum continuous user experience with a minimum amount of newly provided scene information, wherein the efficient means includes a spatial processing unit.

[0013] The system according to some embodiments may further include application software that performs operating system functions and user interface functions. The user interface functions include any combination of providing a user interface or communicating with an external user interface. The user interface determines explicit and implicit user indications that are at least partially used to determine a user's request for scene data (and associated other scene data), and provides any of the scene data and other scene data to the user in response to the user's request.

[0014] The system according to some embodiments may further include a scene codec, wherein the codec includes one or both of an encoder and a decoder, thereby allowing the system to be both a scene data provider and consumer. The system may optionally interface with or optionally include any available sensor for sensing real-world, real scene data, wherein any such sensed data may be used to reconstruct a completely new scene or an increment of an existing scene by the system, wherein any system sensing the data may reconstruct the data into scene information, or offload the data to other systems for scene reconstruction, and the other systems performing scene reconstruction will return the reconstructed scene and scene increments to the original sensing system.

[0015] The codec according to some embodiments supports scene models and other types of non-scene data that are integrated with the scene models or stored in association with the scene models. The codec according to some embodiments can support the networking of multiple systems, thereby exchanging control packets including user requests, client status and scene usage data, and scene data packets including requested scene data and non-scene data and optionally request identifiers used by the client when verifying. Support for one-to-one, one-to-many and many-to-many system networking can be provided, wherein any system can sense new scene data again, reconstruct new scene data, provide scene data and consume scene data.

[0016] The system according to some embodiments provides the use of machine learning during the reconstruction and distribution of scene data, where key data records of new types of information provide the basis for machine learning or deterministic algorithms that optimize both individual system performance and networked system performance. For example, the status of all client systems that consume scene data is tracked to ensure that any possible service system has valuable prior knowledge of the client's existing scene data and non-scene data. User requests (including scene types and scene instances) are classified and uniquely identified. Individual systems are identified and classified according to their scene sensing, reconstruction, provision and consumption capabilities. The extent of scene usage is tracked, including usage types and scene consumption paths and durations. The diversity of classified and tracked information provides valuable new data for machine learning, where forward-looking predictions based on cumulative learning intelligently expand user requests for scene data, thereby further ensuring that the maximum degree of continuous user experience is achieved with the minimum amount of newly provided scene information. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] These and other features and advantages will be better and more fully understood through the following detailed description of exemplary non-limiting illustrative embodiments taken in conjunction with the following drawings.

[0018] Figure 1A A block diagram depicts a system using a scene codec in accordance with some embodiments.

[0019] Figure 1B A block diagram depicts a scene codec including an encoder and a decoder according to some embodiments.

[0020] Figure 1C A block diagram depicting a scene codec including an encoder but not a decoder according to some embodiments.

[0021] Figure 1D A block diagram depicting a scene codec including a decoder but not an encoder according to some embodiments.

[0022] Figure 1E A block diagram depicting a network connecting two or more systems using a scene codec according to some embodiments.

[0023] Figure 1F A block diagram depicts a scene codec including an encoder according to some embodiments.

[0024] Figure 1G A block diagram depicts a scene codec including a decoder according to some embodiments.

[0025] Figure 2ABlock diagram depicting the prior art "Light / Acoustic Field Conceptual Workflow" as described in the technical publications ISO / IEC JTC1 / SC29 / WG1N72033, ISO / IEC JTC1 / SC29 / WG11N16352 submitted in June 2016 by the Joint Ad Hoc Group on Digital Representation of Light / Acoustic Fields for Immersive Media Applications in Geneva, Switzerland.

[0026] Figure 2B is captured by a representative real camera and provided to a device such as Figure 1E Combination blocks and schematic diagrams of real-world scenarios of a system such as the system shown in FIG. 1 in a network environment are shown.

[0027] Figure 3 is a diagram of an exemplary network connecting systems using a scene codec according to some embodiments.

[0028] Figure 4A is a schematic diagram of an exemplary real-world scene with infinite or nearly infinite detail, such as an interior house scene with windows viewing an outdoor scene, in accordance with some embodiments.

[0029] Figure 4B is a representation according to some embodiments such as Figure 4A A schematic diagram of a real-world scene depicted, where the representation can be considered an abstract model view of the data included within a plenoptic scene database as well as other objects (such as explained objects and unexplained objects).

[0030] Figure 4C is a block diagram of some data sets of a plenoptic scene database according to some embodiments.

[0031] Figure 5 is a flow diagram of a use case that includes sharing a larger global scene model with a remote client that is consuming any of various types of scene model information, according to some embodiments.

[0032] Figure 6 is similar to Figure 5 , but now we are addressing a variant case where the client first creates a scene model or updates an existing scene model.

[0033] Figure 7 is similar to Figure 5 and Figure 6, but a variation is now being addressed where both the client-side system and the server-side system are capable of reconstructing and distributing the real scene, thereby determining and providing sub-scenes and increments to sub-scenes, the client system first creates a scene model or updates an existing scene model and then captures local scene data for the real scene.

[0034] Figure 8 It is a synthetically generated image of a complex plenoptic scene model (daily life kitchen).

[0035] Fig. 9 is a geometric diagram showing two views of volume elements ("voxels") and solid corner elements ("saels"), according to some embodiments.

[0036] Fig.10 is a geometric diagram of a top-down plan view of a scene model showing an everyday life scene according to some embodiments.

[0037] Fig.11 is a block diagram of a context database according to some embodiments.

[0038] Fig.12 is a class diagram illustrating a hierarchy of primitive types for representing a full light field, according to some embodiments.

[0039] Fig.13 is a synthetically generated image of a complex plenoptic scene model (an everyday kitchen with two points highlighted).

[0040] Fig.14 Contains an image from outside showing an entry according to some embodiments Fig.13 A light cube showing incident light at a point in the open space of a kitchen.

[0041] Fig.15 comprising according to some embodiments Fig.14 Six additional views of the light cube are shown.

[0042] Fig.16 From an internal perspective according to some embodiments Fig.14 An image of the light cube is shown.

[0043] Fig.17 From an internal perspective according to some embodiments Fig.14 An image of the light cube is shown.

[0044] Fig.18 From an internal perspective according to some embodiments Fig.14 An image of the light cube is shown.

[0045] Fig.19According to some embodiments, Fig.13 An image of the exterior of the light cube showing the outgoing light from a point on the surface of a kitchen counter.

[0046] Fig. 20 is an image of a light cube showing the results of applying BLIF to a single incident beam of vertically polarized light, in accordance with some embodiments.

[0047] Fig.21 is a diagram showing a tree structure of an octree, according to some embodiments.

[0048] Fig. 22 According to some embodiments, Fig.21 The nodes in the octree shown represent the geometry of the volume space.

[0049] Fig.23 is a diagram showing a tree structure of a saeltree according to some embodiments.

[0050] Fig.24 According to some embodiments, Fig.23 The nodes in the trie shown represent a geometric diagram of the region of directional space.

[0051] Fig.25 is a geometric diagram showing a cube with the center of an octree node as its origin, according to some embodiments.

[0052] Fig.26 is a geometric diagram showing a space represented by three cubes of a three cube tree in 2D, according to some embodiments.

[0053] Fig. 27 is a geometric diagram showing two outgoing bins of two bin trees and the intersection of the two bins with two volumetric octree (VLO) voxels, according to some embodiments.

[0054] Fig.28 is a geometric diagram showing two incoming voxels attached to a new voxel tree resulting from two outgoing voxels from two voxel trees projected onto a VLO node, according to some embodiments.

[0055] Fig.29 is a geometric diagram showing the generation of outgoing cubes from a new cube tree for a VLO voxel based on the voxel's incoming cube tree and the BLIF associated with the voxel, according to some embodiments.

[0056] Fig.30 is a schematic diagram illustrating the functionality of a spatial processing unit (SPU) according to some embodiments.

[0057] Fig.31is a schematic diagram illustrating sub-functions of the light field manipulation function of the spatial processing unit according to some embodiments.

[0058] Fig.32 is a geometric diagram showing the numbering of the six faces of a surrounding cube of a cube tree, according to some embodiments.

[0059] Fig.33 is a geometric diagram showing a quarter face of a surrounding cube of a vertex tree, wherein a quarter face of one side of the surrounding cube of the vertex tree is highlighted, according to some embodiments.

[0060] Fig.34 is a geometric diagram showing a side view of a quarter face of a surrounding cube of a cube tree, according to some embodiments.

[0061] Fig.35 is a geometric diagram showing a 2D side view of a directional space segment represented by a top cell, according to some embodiments.

[0062] Fig.36 is a geometric diagram showing how the projection of a vertex of a vertex tree onto a projection plane is represented by the intersection of its position on a face of a surrounding cube of the vertex tree, according to some embodiments.

[0063] Fig.37 is a geometric diagram showing the movement of a cube tree in 2D while maintaining cube projection on a projection plane, according to some embodiments.

[0064] Fig.38 is a geometric diagram illustrating movement of a projection plane while maintaining projection of bins of a bin tree according to some embodiments.

[0065] Fig.39 is a geometric diagram illustrating the geometry of a cube's span on a projection plane, according to some embodiments.

[0066] Fig.40 is a geometric diagram showing the relationship between the top and bottom intersection points and the case where the top is below the bottom in the projection plane coordinate system indicating that the projection is invalid (opposite side of the origin of the cube) according to some embodiments.

[0067] Fig.41 is a geometric diagram showing the subdivision of a chunk of a chunk tree into two subtree levels and the spatial regions represented by the nodes in 2D, according to some embodiments.

[0068] Fig.42 is a geometric diagram showing the location where the edge of the new vertex intersects the projection plane, which will be the new top or bottom edge of the sub-vertex, according to some embodiments.

[0069] Fig.43is a geometric diagram showing that an outgoing bin from a bin tree causes an incoming bin to be generated in the bin tree attached to a VLO node that intersects the outgoing bin, according to some embodiments.

[0070] Fig.44 is a geometric diagram showing a front-to-back VLO traversal order in 2D within the directional space shown, according to some embodiments.

[0071] Fig.45 is a geometric diagram showing in 2D the use of a quadtree as a projection mask during projection of a cube into a scene, according to some embodiments.

[0072] Fig.46 is a geometric diagram showing a volumetric space of cubes constructed by the intersection of multiple half-spaces, according to some embodiments.

[0073] Fig.47 is a geometric diagram showing three crossing situations between a cube and three VLO nodes according to some embodiments.

[0074] Fig.48 is a geometric diagram illustrating rotation of a bin as part of a rotation of a bin tree, according to some embodiments.

[0075] Fig.49 is a geometric diagram illustrating the geometric construction of center points and projection planes between blob edges during blob tree rotation, according to some embodiments.

[0076] Fig.50 is a geometric diagram illustrating geometric operations for performing rotation of edges when rotating a bin tree, according to some embodiments.

[0077] Fig.51 is a schematic diagram illustrating the calculation of the geometric relationship between a cube and a projection plane when pushing a cube origin node, a VLO node of a projection plane, or a cube, according to some embodiments.

[0078] Fig.52 is a schematic diagram showing the calculation of the geometric relationship between a cube and a projection plane when a cube origin node, a VLO node of a projection plane, or a cube is pushed (where both the cube origin and the VLO node of the projection plane are pushed simultaneously) according to some embodiments.

[0079] Fig.53 is a table showing a portion of a spreadsheet that tabulates the results of a series of vertex tree origins, VLOs, and vertex pushes, according to some embodiments.

[0080] Fig.54 yes Fig.53 Continuation of .

[0081] Fig.55 It is shown Fig.53 and Fig.54 A table of formulas in a spreadsheet.

[0082] Fig.56 It is shown in Fig.53 and Fig.54 The geometry of the starting relation at the beginning of the push operation sequence listed in .

[0083] Fig.57 According to some embodiments, Fig.53 and Fig.54 A geometric diagram of the geometric relationship between the cube and its projection on the projection plane after iteration #1 (the cube origin SLT node is pushed to child node 3) as shown in the spreadsheet.

[0084] Fig.58 According to some embodiments, Fig.53 and Fig.54 A geometric diagram of the geometric relationship between the cube and its projection on the projection plane after iteration #2 (the cube origin SLT node is pushed to child node 2) as shown in the spreadsheet.

[0085] Fig.59 According to some embodiments, Fig.53 and Fig.54 A geometric diagram of the geometric relationship between the cube and its projection on the projection plane after iteration #3 (the Projection Plane VLO node is pushed to child node 3) as shown in the spreadsheet of FIG.

[0086] Fig.60 According to some embodiments, Fig.53 and Fig.54 A geometric diagram of the geometric relationship between the cube and its projection on the projection plane after iteration #4 (the Projection Plane VLO node is pushed to Child 1) as shown in the spreadsheet.

[0087] Fig.61 According to some embodiments, Fig.53 and Fig.54 A geometric diagram of the geometric relationship between the cube and its projection on the projection plane after iteration #5 (the cube is pushed to child node 1) as shown in the spreadsheet of FIG.

[0088] Fig.62 According to some embodiments, Fig.53 and Fig.54 A geometric diagram of the geometric relationship between the cube and its projection on the projection plane after iteration #6 (the cube is pushed to child node 2) as shown in the spreadsheet of FIG.

[0089] Fig.63According to some embodiments, Fig.53 and Fig.54 A geometric diagram of the geometric relationship between the cube and its projection on the projection plane after iteration #7 (the projection plane VLO node is pushed to child node 0) as shown in the spreadsheet of FIG.

[0090] Fig.64 is a table showing a portion of a spreadsheet that tabulates the results of a series of lnt tree origins, VLOs, and lnt pushes, where the lnt tree origin is not at the center of an octree node, according to some embodiments.

[0091] Fig.65 yes Fig.64 Continuation of .

[0092] Fig.66 is a schematic diagram illustrating application programming interface functionality of a scene codec according to some embodiments.

[0093] Fig.67 is a schematic diagram illustrating the functionality of a query processor function according to some embodiments.

[0094] Fig.68A is a flow chart of a process for implementing a plenoptic projection engine according to some embodiments.

[0095] Fig.68B is a flow chart of a process for extracting sub-scenes from a plenoptic octree for remote transmission, according to some embodiments.

[0096] Fig.69 is a flow diagram of a process for extracting sub-scene models from a scene database for the purpose of generating images from multiple viewpoints, according to some embodiments.

[0097] Fig.70 is a flow chart of a process for accumulating plenoptic primitives that contribute light to a query primitive, according to some embodiments.

[0098] Fig.71 is a flow chart of a process for accumulating medium elements ("mediels") and their contributing light field elements ("radiels") that contribute light to a query cell, according to some embodiments.

[0099] Fig.72 is an image of a kitchen with a small rectangular area highlighting an analytics portal, in accordance with some embodiments.

[0100] Fig.73 is scaled up according to some embodiments Fig.72 An image of a part of a kitchen where Fig.72 The rectangular window in the right corner highlights the analysis entry.

[0101] Fig.74 is scaled up to show the display according to some embodiments Fig.73 The analysis element in the analysis entry Fig.73 An image of the rectangular area shown.

[0102] Fig.75 A schematic diagram is shown relating to evidence of the effectiveness of the embodiment.

[0103] Fig.76 A schematic diagram is shown relating to evidence of the effectiveness of the embodiment.

[0104] Fig.77 A schematic diagram is shown relating to evidence of the effectiveness of the embodiment.

[0105] Fig.78 is a schematic diagram illustrating sub-scene extraction for image generation purposes.

[0106] In the following description, many specific details are set forth, such as examples of specific components, types of use scenarios, etc., to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure can be practiced without these specific details and using alternative implementations, some of which are also described herein. In other cases, well-known components or methods are not described in detail to avoid unnecessarily complicating the present disclosure. Therefore, the specific details set forth are merely exemplary. The specific details may be different from the spirit and scope of the present disclosure and are still considered to be within the spirit and scope of the present disclosure. DETAILED DESCRIPTION

[0107] A comprehensive solution for providing widely varying scene representations such as sub-scenes and increments of sub-scenes, both of which use minimal scene data to satisfy complex user requests while "looking ahead" to anticipate sufficient buffers (extending to the requested scene data) to ensure continuous quality of service. A codec according to an exemplary embodiment addresses ongoing scene reconstruction corresponding to ongoing scene consumption, where multiple entities are either providing scene data or consuming scene data at any moment, where the provided scene data includes reconstructed scene data and newly determined real scene unreconstructed data.

[0108] In certain exemplary embodiments, scene distribution is less "file-based" (i.e., less focused on a one-to-one, unidirectional pipeline of full scene information) and more "file-segment-based" (i.e., more focused on just-in-time, on-demand-only, many-to-many, bidirectional pipelines of sub-scene and sub-scene delta information). In certain exemplary embodiments, this multi-way configuration is self-learning, tracking the provision and consumption of scene data in order to determine the optimal load balance and sharing across a potentially greater number of scene servers and scene clients. Scene processing in exemplary embodiments considers the merging of all types of data, where scene models are indexable, enhanceable, transformable objects with connections to nearly all other types of data, where scenes then provide context for various types of data and can themselves be searched based on all types of data.

[0109] In an exemplary embodiment, a scene can be considered as a region occupied by matter fields and light fields in space and time. An exemplary system according to some embodiments supports scene visualization in free view, free matter, and free light, where free view allows a user to self-navigate a scene, free matter allows a user to materialize, define, quantify, enhance, and otherwise transform a scene, and free light allows a user to reproduce a scene, even taking into account the unique spectral output of various light sources as well as considerations of light intensity and polarization, all of which increase scene model realism. The combination of free matter and free light enables a user to recontextualize a scene into a variety of settings, such as experiencing a Prague city tour on a winter morning or a summer evening.

[0110] While human visualization of scene data will always be important, codecs according to some embodiments provide a range of scene data types and capabilities, including metrics, object recognition, scene, and situational awareness. Scene data can include the entire range of data and metadata that can be determined in the real world, limited only by the range of material field and light field details included in the scene model, where this range of data must then be formatted according to the range of consumers, from humans to AI systems to automatons, such as search and rescue automatons that crawl or fly over a real-time modeled disaster scene, searching for specific objects and people using advanced object recognition. In this way, codecs according to exemplary embodiments are view-free, matter-free, lighting-free, and data-free.

[0111] Codecs according to some embodiments implement new apparatus and methods for efficient sub-scene and scene incrementation, extraction and insertion, wherein such technical improvements in efficiency significantly reduce computer processing requirements, such as computation time and associated power requirements. In view of the expected rise in market demand for multiplexed, just-in-time, on-demand-only scene reconstruction and distribution, new types of scene processing units are needed, including custom computer chips, that embed new kinds of instruction sets optimized for new representations and representation organizations of realistic, complex and highly detailed scenes.

[0112] refer to Figure 1A , a block diagram of key components of a system 1A01 depicting the use of a scene codec according to some exemplary embodiments is shown. System 1A01 provides significant technical improvements in the reconstruction, distribution, and processing of scene models, where real scenes are generally understood as three-dimensional spaces, but a fourth dimension of time may also be included so that spatial aspects of real scenes can change over time. The scene model may be any one or any combination of a real scene reconstruction or a computer-generated scene or scene enhancement. System 1A01 addresses significant challenges of global scene models, where a global scene model is generally understood to represent a larger real-world space that an end user experiences and explores in its spatial increments (referred to herein as sub-scenes). In one instance, the global real scene is a major tourist city such as Prague, where in the real world, exploring Prague would require many days of spatial movement throughout the sub-scenes that include a large amount of spatial detailed information. In particular for larger real scenes, the combination of scene entry points, traversal paths, and viewpoints along the traversal paths creates an almost infinite amount of information, so intelligent scene modeling and processing including compression is needed.

[0113] For the sake of efficient description later, when the present disclosure refers to a scene or sub-scene, it should be understood as a scene model or sub-scene model, and thus as opposed to a real scene or real sub-scene that is understood to exist and from which the model is at least partially derived. However, the present disclosure may from time to time describe the scene as real or real world to discuss the real world without confusion with the modeled world. It should also be understood that the terms "viewer" and "user" can be used interchangeably without distinction.

[0114] The system 1A01 is configured to intelligently provide users with access to virtually unlimited scenes in an efficient real-time or near real-time manner. The global scene can be considered a combination of local scenes, where the local scenes are less extensive but must also be explored in a spatially incremental manner. The local scenes, and therefore the global scene, can have entry points where scene information is first presented to the user. The scene entry points are essentially sub-scenes, where, for example, the scene entry point in the "Prague" global scene model is "The Porch of St. Clement's Cathedral", where again it can be understood that the data provided by the system 1A01 for representing the "Cathedral" sub-scene is typically much less than the entire data of the "Prague" global scene. In some exemplary embodiments, the provided sub-scene, such as "St. Clement's Cathedral", is determined by the system to be a minimum scene representation that is sufficient to meet the requirements of the end use. In some exemplary embodiments, the system provides a number of advantages for the determination of such sufficiency. Typically, the determination of sufficiency includes at least providing sub-scene model information with varying levels of material field and / or light field resolution based on the requested or desired scene viewing direction. For example, higher resolution information can be provided for nearby objects rather than visually distant objects. The term "light field" refers to the flow of light in all directions for all areas in a scene, and the term "matter field" refers to matter occupying areas in a scene. In this disclosure, the term "light" refers to electromagnetic waves at frequencies including the visible, infrared, and ultraviolet bands.

[0115] In addition, according to some exemplary embodiments, the system 1A01 intelligently provides a spatial buffer for sub-scenes for purposes such as, for example, providing a "look-ahead" scene resolution. In the sub-scene example of "St. Clement's Portico", the minimum resolution may be expected for a viewer standing still at the entrance of St. Clement's Cathedral, but then rotating 360 degrees, looking in any direction, such as towards or away from the Cathedral. Although this minimum resolution is sufficient assuming that the viewer is still standing in the portico, if the viewer wishes to approach and enter the Cathedral, this will eventually cause the resolution in the direction of the Cathedral to drop below a Quality of Service (QoS) threshold. The system expects the viewer to request movement, and in response includes an additional non-minimum resolution so that if the viewer moves his free viewpoint, the viewer will not perceive any substantial loss in scene resolution. In this example, this additional non-minimum resolution may include a resolution sufficient to view all of Prague at the QoS threshold, unless this would in turn result in a large amount of excess and most likely unused data processing and transmission, which may adversely affect the uninterrupted real-time viewing experience. Therefore, the concept of the scene buffer is to intelligently determine and provide a certain amount of additional non-minimum resolution based on all known information (including the viewer's possible traversal paths, traversal path viewpoints, and traversal movement rates).

[0116] System 1A01 exhibits a high degree of contextual awareness about scenes and the users' experience and requests to access scenes, wherein in some exemplary embodiments, such contextual awareness is enhanced based on the application of one or both machine learning and the accumulation of scene experience logging performed by system 1A01. For a global scene (such as Prague) experienced by multiple users over time, logging of at least traversal metrics (including selected entry points, traversal paths, traversal path viewpoints, and traversal movement rates) for individual users provides important information to the machine learning component of system 1A01 to help adjust the size of the spatial buffer, thereby ensuring a most (or substantially most) continuous user experience with a minimum (or substantially minimal) amount of provided scene information, wherein in some exemplary embodiments, this most-minimum relationship is the focus of the scene compression technology of system 1A01. Another key aspect of scene compression addressed by system 1A01 is scene processing time, which is highly dependent on the novel arrangement of scene model data representing the real-world scene, wherein this data is generally referred to in this article as a plenoptic scene model and is stored in a plenoptic scene database 1A07.

[0117] Those familiar with the term "holographic" will recognize it as a 5-dimensional (5D) representation of a specific point in a scene from which one can experience 4π steradians of movement, so that any point (x,y,z) in the scene can be considered the center of a sphere from which one can move outward in any direction from the center point. Experience user movement. Those familiar with light field processing will also understand that the full light function can be used to describe at least what is known in the art as a light field. As will be described in detail herein, some exemplary embodiments of the present invention provide a novel representation of the light field and the material field of a real scene, so that the user's effective 5D traversal of the scene model can be efficiently processed in a just-in-time manner to allow a most (or substantially most) continuous user experience to be provided by a minimal (or substantially minimal) amount of newly provided scene information.

[0118] The system 1A01 also includes a spatial processing unit (SPU) 1A09 for substantially processing the plenoptic scene database 1A07 for scene reconstruction and scene distribution. As will be discussed herein, reconstruction is generally the process of adding to or building a database of scenes to augment any of a variety of data representations of the scene, such as, but not limited to: 1) spatiotemporal extension, which is a three-dimensional volume of a real scene, such as from a car hood being inspected for damage to Prague being traversed for tourism; 2) spatial detail, which includes at least a visual representation of the scene relative to the spatial acuity limit perceivable by a user experiencing the scene, where visual spatial acuity is generally understood to be a function of the human visual system and defines a maximum detail resolution per solid angle of approximately 0.5 to 1.0 arc minutes distinguishable by a human user, such that any additional detail is essentially imperceptible to the user unless the user changes their spatial position by moving closer to the scene area to effectively increase the scene area within the solid angle; 3) light field dynamic range, which includes the intensity and color gamut of light representing the perceived scene, where, for example, the dynamic range can be intelligently altered to provide a larger color range for portions of the scene that are considered to be foreground relative to background; and 4) matter field dynamic range, which includes spatial properties (e.g., surface shape) as well as light interaction properties that describe the effects of matter in a scene on the transmission, absorption, and reflection of the scene light field. Sub-scene extraction is then performed by system 1A01 using SPU 1A09 to intelligently and efficiently determine the minimum scene information data set for each dimension of the information representing the scene in the full optical scene database 1A07, where again, most importantly for the user's experience, this minimum data set (sub-scene) provides a basic continuous experience with sufficient scene resolution (e.g., continuity and / or resolution that meets a predetermined QoS threshold).

[0119] In at least some embodiments, system 1A01 may include a scene solver 1A05 for providing machine learning during one or more of a scene reconstruction process and a sub-scene distribution process. In scene solver 1A05, auxiliary scene information, such as, for example, information indicating scene entry points, traversal paths, viewpoints, and effective scene increment paces, may be considered when providing maximum scene compression with minimal or at least acceptable scene loss.

[0120] System 1A01 also includes a request controller 1A13 for receiving requests indicated by a user interface implemented by application software 1A03. The received request is converted into a control packet 1A17 for communicating with another networked system using scene codec 1A11. Therefore, system 1A01 is also capable of receiving requests generated by other networked systems 1A01. The received request is processed by system 1A01 independently by request controller 1A13, or by request controller 1A13 and application software 1A03 in combination. Control packet 1A17 can carry one or both of explicit and implicit user requests, where an explicit request represents a conscious decision of the user, such as selecting a specific available entry point for a scene (e.g., St. Clement's Cathedral as a starting point for a tour of Prague), while an implicit user request can represent a subconscious decision of the user, such as detecting the user's head orientation relative to the current scene (e.g., detected by a camera sensor attached to a holographic display or an inertial sensor set in a virtual reality (VR) headset). Because some user requests are semi-conscious, such as a scene increment pace that might be indicated by movement of a motion controller in a VR system, this distinction between explicit and implicit is intended to be illustrative rather than limiting.

[0121] Scene codec 1A11 is configured to provide, preferably just-in-time, scene data packets in response to user requests that may be contained within control packets 1A17 when and if system 1A01 is acting as a scene provider. Scene codec 1A11 may be further enabled to receive and respond to scene data packets 1A15 when and if system 1A01 is acting as a scene consumer. For example, system 1A01 may be a provider of scene information extracted from plenoptic scene database 1A07 to multiple other systems 1A01 that receive the provided scene information for potential consumption by end users. The scene information included in plenoptic scene database 1A07 may not be limited to strictly visual information, and thus in some exemplary embodiments may also include information that is ultimately received by, for example, a user viewing some form of image output device. It should be understood that in some exemplary embodiments, scene information may also include any amount of meta-information converted at least in part from the material field and light field of the scene, such as scene metering (e.g., the size of a table) or scene identification (e.g., the location of a light source) or related information, such as auxiliary information that is not a material field or a light field but is associated with any combination or portion of the material field and the light field. Exemplary auxiliary information includes, but is not limited to, scene entry points, scene object labels, scene enhancements, and digital scene signage.

[0122] System 1A01 may be configured to output and receive one or both of scene data packets 1A15. Furthermore, the exchange of scene data packets 1A15 between systems such as system 1A01 may not be synchronous or homogeneous, but rather minimally responsive to maximally satisfy a user's request, as expressed primarily in control packet 1A17 or otherwise through a user interface or application interface provided by application software 1A03. In particular with respect to the periodicity of scene data packets 1A15, in contrast to conventional codecs, scene codec 1A11 may operate asynchronously, where, for example, sub-scene data representing a scene increment of a given scene buffer size is provided just-in-time and only on demand or even just-in-time and only on expectation, where "need" is more of a function of explicit user requests and "expected" is more of a function of implicit user requests. In particular with respect to the content construction of scene data packet 1A15, in contrast to conventional codecs, the scene codec 1A11 can be operated to provide heterogeneous scene data packet 1A15, wherein, for example, the punctual packet includes any one or any combination of material field information, light field information, auxiliary information or any transformations thereof.

[0123] It should also be understood that a "user" is not limited to a person, and may include any requestor, such as another autonomous system 1A01 (e.g., see the forthcoming Figure 3 , a land-based robot, UAV, computer, or cloud system depicted in ). As will be well understood by those familiar with autonomous systems, such autonomous systems can have important uses for scene representations that essentially contain visual representation information, for example, the autonomous system 1A01 can use known pictures of the scene in search and find operations to compare with visual information captured by the autonomous system 1A01 in a real-world scene that corresponds to or is similar to the scene included in the plenoptic scene database 1A07. In addition, such autonomous system 1A01 can preferably be used for non-visual information or quasi-visual information, where the non-visual information can include scene and scene object metrics and the quasi-visual information can include scene lighting properties. In some exemplary embodiments, the autonomous or human-operated system 1A01 can also be configured to collect and provide non-visual representations of the scene for possible spatial and even object collocations within the plenoptic scene database 1A07, or at least for further describing the scene as auxiliary information (e.g., see the upcoming illustration of a scene database view). Figure 4C ). For example, non-visual representations include other sensory information such as somatosensory (touch), olfactory (smell), auditory (hearing), or even gustatory (taste). For the purposes of this disclosure, where the focus is on scene representation as visual information, this focus should not be understood as a limitation, but rather as a characteristic, wherein then at least it should be understood that the plenoptic scene database may include any sensory information or transformation of sensory information, especially including audio / visual data often requested by human users.

[0124] Next reference Figure 1B , shows a block diagram of a scene codec 1A11 included in any one of the systems 1A01 according to some exemplary embodiments, wherein the scene codec 1A11 includes an encoder 1B11a and a decoder 1B11b. Figure 1A As described, the primary function of the encoder is to determine and provide scene data packets 1A15, presumably for reception via a network by at least one other system, such as, for example, another system 1A01, enabling the other system to receive and process the scene data packets (by including a decoder, such as, for example, decoder 1B11b). Encoder 1B11a may receive and respond to control packets 1A17. Figure 7 A use case is described for a system 1A01 comprising a scene codec 1A11 including an encoder 1B11a and a decoder 1B11b.

[0125] Next reference Figure 1C , showing a system including only encoder 1B11a (and thus not including Figure 1B The depicted decoder 1B11b) is a block diagram of a scene codec 1A11. Figure 5 and Figure 6 A use case is described for a system 1A01 including scene codecs 1A11 that include only encoders 1B11a.

[0126] Next reference Figure 1D , showing a system including only decoder 1B11b (and thus not including Figure 1B A block diagram of a scene codec 1A11 of the encoder 1B11a) is depicted. Figure 5 and Figure 6 A use case is described for a system 1A01 comprising scene codecs 1A11 which comprise only decoders 1B11b.

[0127] Next reference Figure 1E, a block diagram of a network 1E01 is shown, which includes a transport layer for connecting two or more systems 1A01. Network 1E01 can represent any means or communication infrastructure for transmitting information between any two or more computing systems, where in the exemplary embodiment, the computing system of primary concern is system 1A01, but at least in some embodiments, it is not limited to this. As those familiar with computer networks will well understand, there are currently many network variations, such as personal area networks (PANs), local area networks (LANs), wireless local area networks (WLANs), campus local area networks (CANs), metropolitan area networks (MANs), wide area networks (WANs), storage area networks (SANs), passive optical local area networks (POLANs), enterprise private networks (EPNs), and virtual private networks (VPNs), any and all of which can be implementations of the network 1E01 currently described. As those familiar with computer networks will also well understand, the transport layer is generally understood as a logical division of technology in a protocol layered architecture in a network stack, for example, referred to as layer 4 relative to the open system interconnection (OSI) communication model. For purposes of this disclosure, the transport layer includes the functionality of transmitting information, such as control packets 1A17 and scene data packets 1A15, exchanged over a network 1E01 by any two or more systems 1A01.

[0128] Still reference Figure 1E , computing systems such as system 1A01 that communicate over network 1E01 are often referred to as residing on a server side (such as 1E05) or a client side (such as 1E07). The typical server-client distinction is most often used for Web services (server side) provided to a Web browser (client side). It should be understood that there is no limitation within the exemplary embodiment, for example, the application software 1A03 within the system 1A01 is implemented using a Web browser, rather than another technology such as a desktop application or even an embedded application, and the terms server side and client side are used in the most general sense herein, so that the server is any system 1A01 that determines and provides scene data packets 1A15, and the client is any system 1A01 that receives and processes scene data packets 1A15. Similarly, a server is any system 1A01 that receives and processes control packets 1A17, and a client is any system 1A01 that determines and provides control packets 1A17. System 1A01 can be used as a server or a client, or a single system 1A01 can be used as a server and a client. Therefore, the block diagrams and descriptions provided herein with respect to the network, transport layer, server side, and client side should be considered useful for conveying information rather than as limitations of the exemplary embodiments. Figure 1EIt is also shown that the network 1E01 of systems 1A01 can include one or more systems 1A01 acting as servers at any given time and one or more systems 1A01 acting as clients at any given time, where again it should be understood that a given system 1A01 can alternately or substantially simultaneously act as a server for scene data and a client for scene data.

[0129] exist Figure 1E , one or more optional sensors 1E09 and one or more optional sensor outputs 1E11 are also depicted, which may be included in some exemplary embodiments. It should be understood that neither the one or more sensors 1E09 nor the one or more sensor outputs 1E11 are required for the system 1A01 to perform useful functions, such as receiving scene data packets 1A15 from other systems 1A01 for scene reconstruction, or such as providing scene data packets 1A15 to other systems 1A01 for further scene processing. Alternatively, the system 1A01 may include any one or more sensors 1E09, such as but not limited to: 1) imaging sensors for detecting any of a multi-spectral data range such as ultraviolet, visible, or infrared light filtered for any of light characteristics such as intensity and polarization; 2) distance sensors or communication sensors that can be used at least in part to determine distance, such as lidar, time of flight sensors, ultrasound, ultra-wideband, microwave, and other radio frequency-based systems; and 3) any non-visual sensors that are capable of detecting other sensory information, such as somatosensory (touch), olfactory (smell), audition (hearing), or even taste (taste). It is important to understand that real-world scenes to be represented in the plenoptic scene database 1A07 typically include scenes that are typically understood as visual data, where this data is not necessarily limited to data known to be in the visible spectrum, but real-world scenes also include a large amount of additional information that can be sensed using any of the sensors available today and additional known or unknown data detectable by future sensors. In the spirit of the exemplary embodiments, all such sensors can provide information that can be used to reconstruct, distribute and process scenes as described herein, and are therefore one or more sensors 1E07. Similarly, there are many currently known one or more sensor outputs 1E09, such as but not limited to 2D, 3D, 4D visual presentation devices, where these visual presentation devices often include accompanying 1D, 2D or 3D auditory output devices. One or more sensor outputs 1E09 also include any currently known, future devices that are used to provide any form of sensory information, including vision, hearing, touch, smell and even taste. Any given system 1A01 may include zero or more sensors 1E07 and zero or more sensor outputs 1E09.

[0130] Next reference Figure 1F, shows a block diagram of an encoder 1B11a in a scene codec 1A11 according to some exemplary embodiments, the scene codec including at least the encoder 1B11a. The API interface 1F03 of the scene codec 1A11 receives and responds to application interface (API) calls 1F01 from an API control host, where the host is, for example, application software 1A03. The API 1F03 communicates with various codec components, including a packet manager 1F05, encoder 1B11a, and a non-all-optical data control 1F15. API 1F03 provides: receiving control signals (such as commands) from a host (such as application software 1A03); providing control signals (such as commands) to various codec components including 1F05, 1B11a, and 1F15 based at least in part on any of the host control signals; receiving control signals (such as component status indications) from various codec components including 1F05, 1B11a, and 1F15; and providing control signals (such as codec status indications) to a host (such as application software 1A03) based at least in part on any of the component status indications. The primary purpose of API 1F03 is to provide a single point of interaction for an external host to control scene codec 1A11, where API 1F03 is, for example, a set of software functions executed on a processing element, where in one embodiment, the processing element used to execute API 1F03 is dedicated to scene codec 1A11. In addition, API 1F03 can perform functions for controlling ongoing processes as commanded by host 1F01 such that a single host command generates multiple signals and communications between API 1F03 and various codec components including 1F05, 1B11a, and 1F15. At any time during the internal processing of the scene codec 1A11, API 1F03 determines whether a response, such as a status update, is necessary to provide to host 1F01 based at least in part on an interface contract implemented with respect to API 1F03, all as will be understood by those familiar with software programming, particularly object-oriented programming.

[0131] Still reference Figure 1F, each of the various components 1F05, 1B11a and 1F15 communicate with each other as needed to exchange control signals and data corresponding to any of the internal processing implemented by the scene codec 1A11. During normal operation of the scene codec 1A11, the packet manager 1F05 receives one or more control packets 1A17 for internal processing by the codec 1A11, and provides one or more scene data packets 1A15 based on the internal processing of the codec 1A11. As those familiar with network systems will understand, in one embodiment, the scene codec 1A11 implements a data transmission protocol on a so-called packet-switched network for transmitting data that is divided into units called packets, where each packet includes a header that describes the packet and a payload that is the data being transmitted within the packet. As described with respect to Figure 1E As discussed, exemplary embodiments may be implemented over a variety of network 1E01 types, where, for example, multiple systems using scene codecs 1A01 communicate over the Internet, which is a packet-switched network 1E01. Packet-switched networks 1E01 such as the Internet use transport layer 1E03 protocols such as TCP (Transmission Control Protocol) or UDP (User Datagram Protocol).

[0132] TCP is well known in the art and provides many advantages, such as message acknowledgment, retransmissions and timeouts, and correct ordering of the transmitted data sequence, but is generally limited to what is known in the art as unicast, where a single server system 1A01 provides data to a single client system 1A01 per single TCP stream. Using TCP, a single server system 1A01 can still establish multiple TCP streams with multiple client systems 1A01, and vice versa, as it is understood that the transmitted control packets 1A17 and data packets 1A15 are only exchanged between the two systems forming a single TCP connection. Other data transmission protocols such as UDP (User Datagram Protocol) are known to support what is known in the art as multicast, or to support what is known as broadcast, where, unlike unicast, these protocols allow, for example, multiple client systems 1A01 to receive a stream of the same scene data packets 1A15. UDP is limited in that the transmitted data is not acknowledged by the client as soon as it is received, and the order in which the packets were sent is not maintained. Packet manager 1F05 can be adapted to implement any of the available data transmission protocols based on at least one of the TCP or UDP transport layer protocols used to transmit packets 1A17 and 1A15, wherein it is possible that new protocols will become available in the future or existing protocols will be further modified, such that embodiments should not be unnecessarily limited to any single choice of data transmission protocol or transport layer protocol, but rather should select the protocol selected for implementing a particular configuration of a system using scene codec 1A01 based on the desired implementation of many features of a particular embodiment.

[0133] Still reference Figure 1F , the packet manager 1F05 parses each received control packet 1A17, such as by processing any of the header and payload of the packet, in order to determine various types of packet 1A17 content, including, but not limited to: 1) a user request for plenoptic scene data; 2) a user request for non-plenoptic scene data; 3) scene data usage information; and 4) client state information. The packet manager 1F05 provides any information related to the user request for plenoptic scene data to the encoder 1B11a, where the encoder 1B11a processes the user's request to access the plenoptic scene database 1A07 at least in part using a query processor 1F09. The query processor 1F09 includes, at least in part, a sub-scene extractor 1F11, which is used to efficiently extract the requested plenoptic scene data including sub-scenes or increments of sub-scenes. The extracted requested plenoptic scene data is then provided to the packet manager 1F05 to be inserted as a payload into a scene packet 1A15 for transmission to the requesting (client) system 1A05. In one embodiment, the packet manager 1F05 also preferably inserts information sufficient to identify the original user request in the scene data packet 1A15 including the requested plenoptic scene data, so that the receiving client system 1A05 receives an indication of the original user request and the plenoptic scene data provided to satisfy the original request. During operation, any of the encoder 1B11a, the query processor 1F09, the packet manager 1F05, and in particular the sub-scene extractor 1F11 can call out the codec SPU 1F13 to efficiently process the plenoptic scene database 1A07, wherein the codec SPU 1F13 can be configured to implement the various technical advantages described herein for efficiently processing representations and organization of representations relative to the plenoptic scene database 1A07.

[0134] The representations for representing real-world scenes as plenoptic scene models and the novel organization of these representations in the exemplary embodiment are used for the plenoptic scene database 1A07. The apparatus and method for processing the plenoptic scene database 1A07, and in the exemplary embodiment, in combination with the representations and organization used in the embodiments, provide important technical advantages to then quickly and efficiently extract requested sub-scenes or increments of sub-scenes, such as the ability to efficiently query plenoptic scene databases that may represent very large, complex, and detailed real-world scenes. As will be appreciated by those familiar with computer systems, the scene codec 1A11 may be implemented in many combinations of software and hardware, for example, including higher level programming languages ​​such as C++ running on a general purpose CPU, or embedded programming languages ​​running on an FPGA (field programmable gate array), or including a substantially hard-coded instruction set included within an ASIC (application specific integrated circuit). In addition, any of the scene codec 1A11 components and subcomponents may be implemented in different combinations of software and hardware, with the codec SPU 1F13 in one embodiment being implemented as a substantially hard-coded instruction set such as included within an ASIC. Alternatively, in some embodiments, the implementation of codec SPU 1F13 is a separate hardware chip that communicates with at least the scene codec 1A11 , so that codec SPU 1F13 is actually external to the scene codec 1A11 .

[0135] As will be appreciated by those familiar with computer systems, the scene codec 1A11 may also include a memory or other data storage element for storing at least some or all of the plenoptic scene database 1A07, or a replicated portion of the database 1A07 that is most relevant to the plenoptic scene model, where the replicated portion may be implemented, for example, in a manner known in the art as a cache. It is important to note that although the plenoptic scene database 1A07 is presently depicted as being external to the scene codec 1A11, in alternative embodiments of the scene codec, at least some portions of the plenoptic scene database 1A07 are retained within the scene codec 1A11 or even within the encoder 1B11a. It is therefore important to understand that the presently depicted block diagram of a scene codec having at least one encoder is exemplary and, therefore, should not be viewed as limiting of the exemplary embodiments, as many variations and configurations of the various components and subcomponents of the scene codec 1A11 are possible without departing from the spirit of the described embodiments.

[0136] Still reference Figure 1F, the packet manager 1F05 provides any of the scene data usage information to the encoder 1B11a, wherein the encoder 1B11a inserts the usage information or other calculated information based at least in part on the usage information into the plenoptic scene database 1A07 (see in particular the forthcoming plenoptic database data model view Figure 4C and upcoming use cases Figure 5 , Figure 6 and Figure 7 , for more details about the usage information). As will be discussed further, the usage information is very valuable for optimizing the functionality of the exemplary embodiments, which functionality includes at least determining the degree of informationization of a sub-scene or scene increment that ideally serves a user request. The group manager 1F05 also provides any client state information to the encoder 1B11a, where the encoder 1B11a maintains a client state 1F07 based at least in part on any of the client state information received from the client system 1A01. It is important to understand that the scene codec 1A11 can support multiple client systems 1A01 and maintain a different client state 1F07 for each supported client system 1A01. As will be discussed particularly with respect to upcoming use cases Figure 5 , Figure 6 and Figure 7 As further discussed, the client state 1F07 is at least sufficient to allow the encoder 1B11a to determine the extent of the plenoptic scene database 1A07 information that has been successfully received and is available to the client system 1F07.

[0137] Unlike conventional codecs used to provide some type of other scene data 1F19, such as a movie, a scene codec 1A11 having encoder 1B11a provides plenoptic scene data 1A07 or either of the other scene data 1F19 to a requesting client system 1A01. Moreover, unlike conventional codecs, at least the plenoptic scene data 1A07 provided by the scene codec 1A11 is of the nature that it is not necessarily fully consumed when received and processed by the client system 1A01. For example, in streaming a movie using a conventional codec, the movie comprises a series of image frames, typically encoded in some format, such as MPEG, as the encoded stream of images is decoded by a conventional client system, each next decoded image being presented to the user in substantially real time, after which the decoded images have substantially no further value, or at least no further immediate value, as the next decoded image is then presented to the user, and so on, until the entire stream of images is received, decoded, and presented.

[0138] Instead, the scene codec 1A11 currently provides at least full light scene data 1A07, such as sub-scenes or scene increments, which are both immediately usable by the user of the client system 1A01 and retain additional substantial future value. As will be at least with respect to upcoming use cases Figure 5 , Figure 6 and Figure 7 As further discussed, codec 1A11 with encoder 1B11a transmits, for example, sub-scenes or sub-scene increments representing some requested portion of the server's plenoptic scene database 1A07 within scene data packets 1A15, wherein scene data packets 1A15 are then received and decoded by the requesting client system 1A01 for two substantially simultaneous purposes, including immediate provision of data to the user and insertion into the client plenoptic scene database 1A07. It should then be further understood that by inserting the received plenoptic sub-scenes or sub-scene increments into the client database 1A07, the inserted scene data is then made available for later use directly in responding to potential future user requests from the client plenoptic scene database 1A07 without requiring any additional scene data from the server plenoptic scene database 1A07. After inserting the sub-scene or sub-scene increment into the client plenoptic scene database 1A07, the client system 1A01 then provides client state information as feedback to the scene codec 1A11, wherein the client state information is provided within a control packet 1A17, and wherein the encoder 1B11a then uses the parsed client state information to update and maintain the corresponding client state 1F07.

[0139] By receiving and maintaining the client state 1F07 associated with the stream of scene data packets 1A15 provided to the client system 1A01, the codec 1A11 with encoder 1B11a is then able to determine at least the minimum scope of information of the new server plenoptic scene database 1A07 required to satisfy the next request from the user received from the corresponding client system 1A01. It is also important to understand that in some use cases, the client system 1A01 is receiving plenoptic scene data from two or more server systems 1A01 that include the scene codec 1A11 with encoder 1B11a. In these use cases, the client system 1A01 preferably notifies each server system 1A01 about changes in the client's state information based on the scene data packets 1A15 received from all server systems 1A01. In this arrangement, it is possible to use multiple service systems 1A01 in a load balancing situation to conveniently satisfy user requests issued from a single client system 1A01 by using any of the plenoptic scene databases 1A07 on any of the service systems 1A01, as if all of the service systems 1A07 collectively provide a single virtual plenoptic scene database 1A07.

[0140] Still reference Figure 1F , the group manager 1F05 provides any of the user requests for non-plenioptical scene data to the non-plenioptical data control 1F15, where the non-plenioptical data control 1F15 communicates with one or more non-plenioptical data encoders 1F17. The one or more non-plenioptical data encoders 1F17 include any software or hardware components or systems that provide other scene data from other scene data databases 1F19 to the data control 1F15. It is important to understand that in some embodiments, the codec 1A11 with encoder 1B11a does not need to access other scene data such as included in other scene data databases 1F19, and therefore does not need to access one or more non-plenioptical data encoders 1F17, or even need to implement the non-plenioptical data control 1F15 within the codec 1A11. For embodiments of scene codec 1A11 having encoder 1B11a, which may need or anticipate the need to encode some combination of plenoptic scene data included in server plenoptic scene database 1A07 and other scene data included in other scene data database 1F19, scene codec 1A11 uses, at least in part, other scene data provided by non-plenoptical data encoder 1F15 to determine at least some of the payload of any one or more scene data packets 1A15. Exemplary other scene data 1F19 includes any information that is not plenoptic scene database 1A07 information (see, in particular, forthcoming Figure 4C 1A07), including video, audio, graphics, text, or other digital information, where it is determined that this other data is needed to respond to the client system 1A01 request included in the control packet 1A17. For example, the scene in the plenoptic scene database 1A07 can be a house for sale, where the other scene data stored in the database 1F19 includes any of the related video, audio, graphics, text, or other digital information, such as product videos related to objects in the house, such as furniture.

[0141] It is important to note that the plenoptic scene database 1A07 has a database for storing any of conventional video, audio, graphics, text, or other digital information to be used with the plenoptic scene data (see in particular the forthcoming Figure 4C) and thus the scene codec 1A11 comprising encoder 1B11a is able to provide other data such as video, audio, graphics, text or other digital information received from the server plenoptic scene database 1A07 or other scene data database 1F19. As will also be appreciated by those familiar with computer systems, it is advantageous to store different forms of data in different forms of databases, which may then also reside in different means of data storage and retrieval, where some means are more economical from a data storage cost perspective and other means are more economical from a retrieval time perspective, for example, and therefore it will be apparent to those skilled in the art that, at least in some embodiments, it is preferred to substantially separate the plenoptic scene data from any other scene data.

[0142] The one or more non-all-optical data encoders 1F17 include any processing elements capable of accessing at least another scene data database 1F19 and retrieving at least some other scene data to provide to the data control 1F15. In some embodiments of the present invention, information associating scene data with other non-scene data is maintained within the plenoptic scene database 1A07, so that the one or more non-all-optical data encoders 1F17 are preferably able to access the server plenoptic scene database 1A07 to determine which of any of the other scene data 1F17 should be retrieved from the other scene data databases 1F19 to satisfy the user's request. In one embodiment, the non-all-optical data encoder 1F17 includes any of the processing elements capable of retrieving some other scene data in a first format, converting the first format into a second format, and then providing the converted scene data in the second format to the data control 1F15. In at least one embodiment, the first format is, for example, uncompressed video, audio, graphics, text, or other digital information, and the second format is any of the compressed formats for representing video, audio, graphics, text, or other digital information. In another embodiment, the first format is, for example, any of a first compressed format for representing video, audio, graphics, text, or other digital information, and the second format is any of a second compressed format for representing video, audio, graphics, text, or other digital information. It is also expected that at least in some embodiments, one or more non-all-optical data encoders 1F17 simply extract other scene data from the database 1F19 to provide to the data control 1F15 without format conversion, wherein the extracted other scene data is already in a compressed format or in an uncompressed format.

[0143] Still reference Figure 1F, it is possible that a user requests, for example, a rendered view based on plenoptic scene data for the plenoptic scene data and other scene data. In such a use case, the packet manager 1F05 provides the initial user request to an encoder 1B11a, which extracts the requested plenoptic scene data and then provides the extracted plenoptic scene data to a non-plenioptic data control 1F15. The data control 1F15 then provides the non-plenioptic data to a non-plenioptic data encoder 1F17, which may convert the plenoptic scene data into, for example, a requested scene rendered view, wherein this rendered view as other scene data is then provided to the data control 1F15 to include a payload of one or more scene data packets 1A15. As will be described in particular with respect to the upcoming Figure 1G As will be apparent, instead, the extracted plenoptic scene data may simply be transmitted to the client system 1A01 in one or more scene data packets 1A15, where the codec 1A11 including the decoder 1B11b on the client system 1A01 then renders the requested scene view using the extracted plenoptic scene data received in the one or more scene data packets 1A15. As careful consideration will show, this flexibility provides a communication system network using the scene codec 1A11 with a variety of options to most efficiently satisfy any given user request.

[0144] Still reference Figure 1F There is no limit on the number of concurrent streams of scene data that can be processed by the scene codec 1A11 including the encoder 1B11a, wherein it should be understood that the codec 1A11 with the encoder differs in this respect from a conventional codec with an encoder, which typically provides a single data stream to a single decoder (often referred to as unicast) or to multiple decoders (often referred to as multicast or broadcast). In particular, as described above Figure 1E , Figure 1E As depicted, some exemplary embodiments of the present invention provide a one-to-many relationship between a single service system 1A01 (including a scene codec 1A11 including at least an encoder 1B11a) and multiple client systems 1A01 (including a scene codec 1A11 including at least a decoder 1B11b). Some embodiments of the present invention also provide a many-to-one relationship between a single client system 1A01 and multiple service systems 1A01, and a many-to-many relationship between multiple server systems 1A01 and multiple client systems 1A01.

[0145] Next reference Figure 1G , shows a block diagram of a scene codec 1A11 including at least a decoder 1B11b. Figure 1G Many of the components described in Figure 1F, or are similar to those elements in , and therefore will not be discussed in detail. As discussed previously, the API control host 1F01 is, for example, an application software 1A03 executed on or in communication with a system 1A01 that uses a scene codec, where, for example, the software 1A03 fully or partially implements a user interface (UI) or communicates with the UI. Ultimately, a user such as a person or an automaton uses the UI to provide one or more explicit or implicit indications, where these indications are at least partially used to determine one or more user requests 1G11 for scene data. In a general sense, any system that uses a scene codec 1A01 that determines user requests 1G11 is referred to herein as a client system 1A01. As particularly with respect to the upcoming use cases Figure 5 , Figure 6 and Figure 7 As has been discussed and will be discussed further, it is possible and even desirable for a client system 1A01 to have scene data sufficient to satisfy even a given user request 1G11. However, it should also be understood that the total amount of possible and useful scene data will likely far exceed the capacity of any given client system 1A01, for example, where the computing platform used to implement the client system 1A01 is a mobile computing device or computing element embedded within an autonomous machine such as a drone or robot. Therefore, some exemplary embodiments provide that a given client system 1A01 determining one or more user requests 1G11 has access to any number of other systems using the scene codec 1A01 via a network 1E01, wherein any one or more of these other systems 1A01 may have access to or include scene data sufficient to satisfy the given user request 1G11. As will be further discussed with respect to this figure, the user request 1G11 can then be transmitted via a network to another system 1A01, which includes a scene codec 1A11 including at least an encoder 1B11a, wherein this other system 1A01 is referred to herein as a server system 1A01, and will ultimately provide the client system 1A01 with one or more scene data packets 1A15 that satisfy the user request 1G11.

[0146] There is no limitation that limits any given system using the scene codec 1A01 to the functionality of only the client system 1A01 or only the server system 1A01, and as will be described with respect to Figure 5 , Figure 6 and Figure 7 As discussed, a given system 1A01 can operate as one or both of a client and a server at any time, where the client includes a codec with a decoder 1B11b and the server includes a codec with an encoder 1B11a, such that a system 1A01 including a codec 1A11 with a decoder 1A11b and an encoder 1A11a is able to function as both a client system 1A01 and a server system 1A01.

[0147] Still reference Figure 1G , in a codec 1A11 including a decoder 1B11b, an API host 1F03 communicates with various codec components including a packet manager 1F05, a decoder 1B11b, and a non-plenoptical data control 1F15. The packet manager 1F05 receives user requests 1G11 in any of a number of possible various forms sufficient to communicate user desired scene data to the server system 1A01, where scene data is then broadly understood to include any of the scene data included within a plenoptic scene database 1A07, and / or any of the other scene data included within another scene data database 1G07. The other scene data includes any of video, audio, graphics, text, or other digital information. Other scene data may also be stored within and retrieved from the plenoptic scene database 1A07, particularly as auxiliary information (see with respect to the forthcoming Figure 4C Element 4C21). Throughout this specification, descriptions are provided to describe various data types generally referred to herein as comprising a plenoptic scene database 1A07, which data types include scene models and auxiliary information, the auxiliary information including scene model enhancements, conversions, indexes, and usage history. The scene model may generally include a material field and a light field.

[0148] It should be noted that the various types of scene data and other scene data described in this application can be classified, wherein this classification can be in the form of a GUID (globally unique identifier) ​​or even a UUID (universally unique identifier), for example. In addition, the present structure described herein for reconstructing a real scene into a scene model to be associated with other scene data can be applied to an almost unlimited number of real-world scenes, wherein then providing a classification for the type of possible real-world (or computer-generated) scenes that can be used as scene models is also useful. Therefore, a GUID or UUID can also be assigned to represent various possible scene model types (such as urban landscapes, buildings, cars, houses, etc.). Another GUID or UUID can also be used to uniquely identify a specific instance of a scene type, such as identifying a car type as "2016Mustang xyz". It will also be understood that a given user requesting scene information can be allowed to remain anonymous, or a GUID or UUID can be assigned the same. A GUID or UUID can also be assigned to each system (whether acting as a server and / or client) using the scene codec 1A01. In addition, user requests 1G11 can also be classified into types of user requests (such as "new sub-scene request", "sub-scene increment request", "scene index request", etc.), where the user request type and the actual user request can be assigned a GUID or UUID.

[0149] In some embodiments, one or more identifiers such as a GUID or UUID are included along with a specific user request 1G11 for providing to a packet manager, wherein the packet manager may then include one or more additional identifiers such that a control packet 1A17 issued by a scene codec 1A11 including decoder 1B11b includes significant user request classification data, and wherein any of this classification data may be used at least to: 1) be stored in a database, such as a plenoptic scene database 1A07 maintained by a server system 1A01 servicing user requests, or in an external user request database that is generally available to any one or more systems 1A01 such as a server system 1A01 servicing user requests; and 2) determine any of user request 1G11 / control packet 1A17 routing or scene data providing load balancing, wherein any one or more request traffic processing agents may communicate with any one or more of the client and server systems 1A01 via a network 1E01 to route or reroute control packets 1A17, particularly for the purpose of balancing the load of user requests 1G11 with the availability of the server system 1A01 and the network bandwidth, all as will be understood by those familiar with networking systems and managing network traffic.

[0150] Still reference Figure 1G In one embodiment, the client system 1A01 communicates exclusively with the server system 1A01, wherein the client system 1A01 provides a user request 1G11 included in a control packet 1A17, and the server system in turn provides a scene data packet 1A15 that satisfies the user request 1G11. In another embodiment, the client system 1A01 is served by two or more server systems 1A01. Figure 1F As discussed, server system 1A01 preferably includes identification information within scene data packets 1A15 along with any requested scene data (or other scene data) so that codec 1A11 on client system 1A01 can track the state of received scene data, including answered requests stored within plenoptic scene database 1A07 of client system 1A01 or within other scene data database 1G07 of client system 1A01. In operation, packet manager 1F05 receives and parses a given scene data packet 1A15, wherein any non-plenoptic scene data is then provided to a non-plenoptic data control, any plenoptic scene data is provided to a decoder 1B11b, and any user request identification data is provided to a decoder 1B11b.

[0151] The non-plenioptical data control 1F15 provides any non-plenioptical scene data to any one or more of the one or more non-plenioptical data decoders 1G05 for any of the following: decoding and / or storage in another scene data database 1G07 or the plenoptic scene database 1A07 of the client system 1A01, preferably as auxiliary information (e.g., see Figure 4C ). Again, non-all-optical scene data includes, for example, any of video, audio, graphics, text, or other digital information, wherein decoders for such data are well known in the art and are continually being further developed, and therefore, it should be understood that any of the available or soon-to-be-available non-all-optical scene data decoders may be used by some embodiments as the non-all-optical data decoder 1G05.

[0152] The decoder 1B11b receives the plenoptic scene data and inserts the plenoptic scene data into the plenoptic scene database 1A07 of the client system 1A01 at least in part using a query processor 1G01 having a sub-scene inserter 1G03. Figure 1F As mentioned, the client plenoptic scene database 1A07 may be implemented as any combination of internal or external data memories or storage areas, wherein, for example, the decoder 1B11b includes a high-speed internal memory for storing the substantial portions of the client plenoptic scene database 1A07 that are most expected to be needed and requested by the user, and wherein otherwise, additional portions of the client plenoptic scene database 1A07 are stored external to the decoder 1B11b (but not necessarily external to the system using the scene codec 1A01 including the decoder 1B11b). As with the encoder 1B11a, during operation, any of the decoder 1B11b, the query processor 1G01, and in particular the sub-scene inserter 1G03, may call out the codec SPU 1F13 to efficiently process the plenoptic scene database 1A07, wherein the codec SPU 1F13 is intended to achieve the various technical advantages described herein for efficiently processing representations and organization of representations relative to the plenoptic scene database 1A07.

[0153] Still reference Figure 1G , the decoder 1B11b receives the user request identification data of any of the following: 1) updating the client state 1F05, and 2) notifying the API host 1F01 through the API 1F03 that the user request has been satisfied. The decoder 1B11b may also update the client state 1F05 based on any of the internal operations of the decoder 1B11b, where it is important to see that the purpose of the client state 1F05 includes at least accurately representing the current state of the plenoptic scene database 1A07 and other scene data databases 1G07 available to the client system 1A01. As will be described with respect to the use case Figure 5 , Figure 6 and Figure 7 As discussed in further detail, the client state 1F05 information is useful to at least the client system 1A01, at least in part, for using any of the available client system 1A01 plenoptic scene database 1A07 and other scene data databases 1G07 to efficiently determine whether a given client user request can be satisfied locally on the client system 1A01, or requires additional scene data or other scene data that must be provided by another server system 1A01, in which case the client user request is encapsulated in a control packet 1A17 and transmitted to a particular server system 1A01 or a load balancing component for use in selecting an appropriate server system 1A01 to satisfy the user request. The client state 1F05 information is also useful to any of the load balancing components or ultimately a particular server system 1A01 to efficiently determine at least a minimum amount of scene data or other scene data sufficient to satisfy the user request.

[0154] After receiving an indication through the API 1F03 that a particular user request has been satisfied, the API control host 1F01 (such as application software 1A03) then causes the client system 1A01 to provide the requested data to the user, where again the user can be human or autonomous. It should be understood that there are many possible formats for providing scene data and other scene data, such as free view formats for use with displays for outputting video and audio, the latter such as encoded formats for use with automatons that have requested scene object identification information, which object identification information includes local directions to objects and confirmation of the visual appearance of the objects. The important point to see is that the codec 1A11 including decoder 1B11b is operated to provide user requests 1G11 to one or more server systems 1A01, and then receive and process scene data packets 1A15 so that the end user receives the requested data in a certain format through some user interface means. It is also important to note that codec 1A11 including decoder 1B11b is operative to track current client state 1F05, such that client system 1A01 uses any of the client state 1F05 information to at least partially determine whether a given user request can be satisfied locally on client system 1A01, or requires scene data or other data that must be provided by another server system 1A01. It is further important to note that client system 1A01 using codec 1A11 including decoder 1B11b optionally provides one or more of a variety of possible unique identifiers (e.g., including a classifier) ​​along with any user request 1G11 (particularly encoded in control packet 1A17), where tracking of the various possible unique identifiers by at least either client system 1A01 or service system 1A01 may be used to optimize the overall performance of any one or more clients 1A01 and any one or more servers 1A01 (such as by using machine learning). It is also important to see that, like codec 1A11 including encoder 1B11a, codec 1A11 including decoder 1B11b has access to codec SPU 1F13 to at least significantly increase the speed of execution of various extraction and insertion operations, respectively, all of which will be discussed in more detail herein.

[0155] Still relative to Figure 1G, when codec 1A11 including decoder 1B11b processes scene data packets, any of processing metrics or information may be provided as usage data and changes in client state 1F07, where the usage data differs from a user request at least in that a given user request may be satisfied by providing a sub-scene (such as a scene model of a house for sale), and the client system 1A01 then tracks how the user interacts with this provided scene model, for example where the user is a person and tracking usage refers to the rooms in the house scene model visited by the user, the duration of the visit, the viewing points in each room, etc. It should be understood that just as there are virtually an infinite number of possible scene models representing any combination of real-world and computer-generated scenes, there are also at least a very large number of usage classifications and other information that can be tracked and are valuable at least to the machine learning aspects of the exemplary embodiments, where at least one function of the machine learning described herein is to estimate the optimal degree of informationization of sub-scenes or sub-scene increments in determining how to satisfy a user's request.

[0156] For example, if a user requests to start a tour of a city such as Prague (see in particular Figure 5 ), then the system must decide the degree of information to provide to the user for the initial porch subscene, where providing a greater degree generally allows the user more initial freedom in consuming the scene, but where providing a lesser degree generally allows for faster response times where less scene data is transmitted. As will be discussed, scene freedom includes at least free view, free matter, and free lighting, and where, for example, free view includes spatial movement in the subscene, such as moving from the porch to then entering St. Clement's Cathedral, or moving from the porch to across the street, turning around and capturing a virtual image of the cathedral. As careful consideration will show, each of the possible user choices for consuming the provided subscenes may require an increasingly large range of information, including matter field and light field data. In this regard, exemplary embodiments provide that by tracking scene usage for multiple users and user events, the accumulated usage information can be used by the machine learning components described herein to estimate the range of information, for example, of a material field or light field, that would be necessary to allow "X" amount of scene movement by the user, where X can then be associated with Y amount of time for typically experiencing scene movement, enabling the system to then look ahead and predict when a user might need new scene data based on all known usage and currently tracked user scene movement, where such look ahead can then automatically be used to trigger additional (implicit) user requests 1G11 for more new scene data to be provided by server system 1A01.

[0157] although Figure 1GNot depicted in the figure, any non-all-optical data or other scene data received in scene data packet 1A15 and processed by codec 1A11 including decoder 1B11b may also be provided directly to one or more appropriate sensory output terminals 1E11 (see Figure 1E ) to provide the requested data to the requesting user, where for example the sensory output terminal 1E11 is a conventional display, a holographic display or an extended reality device (such as a VR headset or AR glasses), where directly providing means providing from the codec 1A11 rather than from a process that retrieves equivalent scene data from the plenoptic scene database 1A07 or other scene data database 1G07 after first being stored in the corresponding database 1A07 or 1G07 by the codec 1A11. In addition, it is possible that any of this non-plenoptical data or other scene data provided to the sensory output terminal 1E11 is neither stored as data nor as data in the client plenoptic database 1A07 or other database 1G17, where the storage operation is any of before the provision, substantially simultaneously with the provision, or after the provision, and where the provision to the sensory output terminal 1E11 can be replaced by further processing any of the stored data in the database 1A07 or 1G07 so as to retrieve scene data equivalent to the scene data received in the scene data packet 1A15 for provision to the sensory output terminal 1E11.

[0158] Next reference Figure 2A , shows a block diagram provided on page 13 of a publication entitled "Technical Report of the Joint Ad Hoc Group on Digital Representation of Light / Acoustic Fields for Immersive Media Applications" provided by the "Joint Ad Hoc Group on Digital Representation of Light / Acoustic Fields for Immersive Media Applications", the entire contents of which are incorporated herein by reference. The publication is directed to the processing of "Conceptual Light / Acoustic Fields", where the figure depicts seven steps in the processing flow. The seven steps include: 1) sensor; 2) sensed data transformation; 3) encoder; 4) decoder; 5) renderer; 6) presentation; and 7) interaction commands. The processing flow is intended to provide a real-world or computer-synthesized scene to the user.

[0159] exist Figure 2B, there is shown a combined block diagram and schematic diagram representing exemplary use cases of some exemplary embodiments for providing at least visual information about a real scene 2B01 to a user 2B17 via a sensory output device such as a 2π-4π free-view display 2B19. In this depiction, the real scene 2B01 is sensed using one or more sensors such as real cameras 2B05-1 and 2B05-2, where the real cameras 2B05-1 and 2B05-2 are capable of imaging the real scene 2B01, for example, over a certain field of view that includes the entire spherical 4π steradians (thus 360 x 180 degrees). The real cameras such as 2B05-1 and 2B05-2 and other real-world scene 2B01 sensors provide captured scene data 2B07 to the system 1A01, such as residing on the server side 1E05 of the network 1E01. Although in this depiction, the scene data 2B07 is virtual in nature, as described with respect to Figure 1E As previously discussed, the sensors 1E09 of system 1A01 include, but are not limited to, real cameras such as 2B05-1 and 2B05-2, wherein additional real cameras such as 2B05-1 and 2B05-2 are not limited to 4P spherical cameras (also often referred to as 360 degree cameras), but may be, for example, single sensor narrow field of view cameras. Moreover, as previously discussed, cameras such as 2B05-1 and 2B05-2 may sense a wide range of frequencies, including, for example, ultraviolet, visible, and infrared light. However, in the use case depicted in the present figure where an end user 2B17 is to view visual information, the preferred sensor is a real camera or one capable of sensing real scene depth and color across multiple scene points, all as will be well understood by those familiar with imaging systems.

[0160] Still reference Figure 2B , the sensor data 2B07, which in this example includes a camera image, is provided to the server-side system 1A01. Figure 1A Mentioned and relative to the following upcoming Figure 4C As further described, additional extrinsic and intrinsic information may also be provided to the server-side system 1A01 relative to sensors (such as real cameras 2B05-1 and 2B05-2), wherein such information includes, for example, sensor position and orientation, and may also include sensor resolution, optical and electronic filters, capture frequency, etc. Using either the provided information as well as captured information (such as images including 2B07), the server-side system 1A01 reconstructs the real-world scene 2B01, preferably under the guidance of application software in conjunction with the scene solver 1A05 and SPU 1A09, to form a plenoptic scene model within the plenoptic scene database 1A07. Figure 4A and Figure 4B Provides information such as 2B01 ( Figure 4A) and real-world scenarios ( Figure 4B ) of the abstract model view (or plenoptic scene model). According to some embodiments, the plenoptic scene model describes the matter field 2B09 and the light field 2B11 of the corresponding real-world scene at least with a certain predetermined resolution in various dimensions, including: 1) spatial extension; 2) spatial detail; 3) light field dynamic range; and 4) matter field dynamic range.

[0161] Still reference Figure 2B , a user 2B17 interacting with the client-side system 1A01 requests to view at least a portion of a real-world scene 2B01 represented in a plenoptic scene database 1A07 stored on or accessible to the server-side system 1A01. In a preferred embodiment, application software 1A03 executing on the client-side system 1A01 presents and controls a user interface for determining at least the user request. Figure 5 , Figure 6 and Figure 7 In an exemplary embodiment, the application software 1A03 on the client-side system 1A01 interfaces with a scene codec such as 1B11b including a decoder to transmit the scene codec to the client through the network 1E01 (such as Figure 1E The server-side system 1A01 (shown) transmits the user request within the control packet 1A17 to a scene codec such as 1A11b having, for example, a codec executing on the server-side system 1A01. The server-side system 1A01 application software 1A03 preferably communicates with the server-side encoder 1B11a at least for receiving an explicit user request, such as a user request for receiving scene information starting spatially at a given entry point within the plenoptic scene model (co-located with the spatial entry point in the real-world scene 2B01).

[0162] In processing the request, the server-side system 1A01 preferably determines the relevant sub-scene and extracts the relevant sub-scene from the plenoptic scene database 1A07, as indicated by the requested scene entry point. The extracted sub-scene preferably further includes a sub-scene spatial buffer. Thus, in the present example, the sub-scene includes at least visual data representing a 2π-4π steradian viewpoint located at the entry point, but then includes at most additional portions of the database 1A07 that are sufficient to accommodate any desired path traversal of the scene by a user relative to the entry point and a given minimum time. For example, if the real-world scene is Prague and the entry point is the narthex of St. Clement's Cathedral, the minimum extracted scene will essentially allow the user to perceive a 4π / 2 steradian (half dome) viewpoint located at the narthex of the Cathedral. However, based on a user request or any auxiliary information available within or for use in the server-side system 1A01, such as a typical walking speed and direction of a user based on a given entry point, the application software 1A03 executing on the server-side system 1A01 can determine a scene buffer sufficient to provide additional scene resolution sufficient to support a 30 second walk starting from the front porch in any available direction.

[0163] Still reference Figure 2B , one or more scene data packets 1A15 in the scene stream 2B13 (see Figure 1A ) is provided in the communication of the determined and extracted sub-scenes. Preferably, a minimum number of scene groupings 1A15 are communicated such that the user 2B17 perceives an acceptable application response, with it being understood that transmission of the entire plenoptic scene database 1A07 is generally prohibitive (e.g., due to bandwidth and / or time constraints), particularly as any of the scene dimensions increase, which will naturally be the case at least for global city scenes such as Prague. The techniques for organizing the plenoptic scene database and processing the organized database in the exemplary embodiments provide substantial technical improvements over other known techniques, such that the user experiences real-time or near real-time scene access.

[0164] However, it is possible and affordable that the user experiences some delay when initially entering a scene in favor of then perceiving a continuous experience of the entered scene, where the continuous experience is directly related to both the size of the entry sub-scene buffer and the provision of explicitly or implicitly expressed supplementary scene increments along the scene traversal. Some exemplary embodiments of the present invention provide means for balancing initial entry point resolution and sub-scene buffers, as well as sub-scene increments and resolution based on periodic or non-periodic events. This balance provides a maximum continuous user experience encoded with a minimum amount of scene information, thereby providing a novel scene compression that can meet a predetermined quality of service (QoS) level. Within an asynchronous scene stream 2B13 determined and provided by an exemplary server-side system 1A01, any given transmission of scene data packet 1A15 may include any combination of any form and type of full-optical scene database 1A07 information, wherein, for example, one scene data packet such as 2B13-a or 2B13-d includes at least a combination of matter field 2B09 and light field 2B11 information (e.g., shown in 2B13-a and 2B13-d as having "M" and "L", respectively), while another scene data packet such as 2B13-b does not include at least matter field 2B09 but includes a light field 2B11 (e.g., shown in 2B13-b as having only "L"), and yet another scene data packet such as 2B13-c includes at least a matter field 2B09 but no light field 2B11 (e.g., shown in 2B13-c as having only "M").

[0165] Still reference Figure 2B , the scene stream 2B13 is sent from the server-side system 1A01 through the network 1E01 transport layer to be received and processed by the client-side system 1A01. Preferably, the scene data packets 1A15 are first received on the client-side system 1A01 by the scene codec 1B11b including a decoder, wherein the decoded scene data is then processed under the direction of the application software 1A03, accessing the functions of the SPU 1A09. The decoded and processed scene data is preferably used on the client-side system 1A01 to reconstruct a local plenoptic scene database 2B15 and provide scene information, such as scene entry points in a 2π-4π free view. The provided entry point free view allows the user 2B17 to at least view the angle relative to the viewpoint orientation. and a representation of the spatial viewpoint position (x, y, z) representing the user's current position within the scene, explicitly or implicitly changing the viewpoint. When user 2B17 provides an explicit or implicit request to further explore the scene and thereby move within the scene, the client-side system 1A01 first determines whether the local scene database 2B15 includes sufficient information to provide a sub-scene increment, or whether additional increments should be requested from the server-side system 1A01 to be extracted from the server-side scene database 1A07. A well-functioning scene reconstruction, distribution, and processing system such as that described herein intelligently determines an optimal QoS for user 2B17 that balances multiple considerations and provides an efficient means for storing and retrieving plenoptic scene model information into and from the plenoptic scene database 1A07.

[0166] Next reference Figure 3 , showing a combined block diagram and schematic diagram of a network 1E01 that connects multiple systems using scene codecs 1A01 in various forms representing multiple possible forms, including but not limited to: personal mobile devices (such as mobile phones), display devices (such as holographic TVs), cloud computing devices (such as servers), local computing devices (such as computers), land-based robots, unmanned autonomous vehicles (UAVs) (such as drones), and extended real devices (such as AR glasses). As previously discussed, all systems 1A01 include scene codecs 1A11, wherein the codec 1A11 included in any of the systems 1A01 may also include an encoder 1B11a and a decoder 1B11b, an encoder 1B11a without a decoder 1B11b, or a decoder 1B11b without an encoder 1B11a. Therefore, any of the systems 1A01 represented by the depiction of this figure can be a plenoptic scene data provider and a plenoptic scene data consumer, only a provider, or only a consumer. Although a single system 1A01 (e.g., a computer not connected to the network 1E01) may perform any of the functions described herein as defining a system using a scene codec 1A01, some embodiments may include two or more systems 1A01 that interact via the network 1E01 such that they exchange either captured real scene data 2B07 to be reconstructed into the plenoptic scene database 1A07, or exchange any of the plenoptic scene database 1A07 scene data (see in particular the upcoming use cases Figure 5 , Figure 6 and Figure 7 ).

[0167] Next reference Figure 4A, shows a schematic diagram of an exemplary real-world scene 4A01 having a very high level of detail (e.g., referred to in some instances as infinite or nearly infinite detail), such as an interior house scene with windows viewing an outdoor scene. Scene 4A01 includes, for example, but is not limited to, any one or any combination of the following: opaque objects 4A03, finely structured objects 4A05, distant objects 4A07, emissive objects 4A09, highly reflective objects 4A11, featureless objects 4A13, or partially transmissive objects 4A15. Also depicted is a user of system 1A01 (such as a mobile phone) operating a scene codec that operates alone or in combination with other (not depicted) systems 1A01 to provide the user with, for example, sub-scene images 4A17 and any number of accompanying real-world scene 4A01 transformations, such as object measurements, light field measurements, or scene boundary measurements, such as portions of a scene including small borders versus opaque borders (see, in particular, upcoming figures, e.g. Figure 4B ).

[0168] Still reference Figure 4A , typically, any real-world scene such as 4A01 is converted to a plenoptic scene model 1A07 through a process referred to as "scene reconstruction." As previously described and as will be discussed in greater detail in this disclosure, the SPU 1A09 implements a number of operations to most efficiently perform scene reconstruction as well as other database 1A07 functions, such as, but not limited to, scene enhancement and scene extraction, wherein scene enhancement introduces new real or synthetic scene information into a scene model that would not otherwise necessarily exist or be substantially absent within the corresponding real-world scene 4A01, and wherein scene extraction provides for the determination and processing of representative sub-scenes of the scene model. Synthetic scene enhancement includes, for example, providing a higher resolution of a reconstructed real-world object, such as a tree or a marble floor, such that when a viewer views the real-world object from above a given QoS threshold, the viewer is provided with real-scene reconstruction information represented in the original plenoptic scene model corresponding to the captured real scene. However, as the viewer spatially approaches an object within the rendered sub-scene and eventually crosses a QoS threshold, the system according to some exemplary embodiments intelligently enhances during rendering, or has intelligently included synthetic information within the rendered sub-scene (such as tree bark or marble floor details that were not originally captured (or even rendered) in the real-world scene) as enhancement. As also previously described, the scene solver 1A05 is an optional processing element that typically further applies machine learning techniques to extend the accuracy and precision of any of the scene reconstruction, enhancement (such as QoS-driven synthesis), extraction, or other forms of scene processing.

[0169] As will be well understood by those familiar with computer systems, the combination of any of the system components including application software 1A03, scene solver 1A05, SPU 1A09, scene codec 1A11, and request controller 1A13 provides functional and technical improvements that can be implemented as various arrangements of components without departing from the scope and spirit of the exemplary embodiments. For example, one or more of the various novel functions of the scene solver 1A05 may alternatively be included in the application software 1A03 or SPU 1A09, so that the functional descriptions of the present descriptions describing the various system components should be considered exemplary rather than limiting of the exemplary embodiments, as those skilled in the art of software and computer systems will recognize many possible variations of the system components and component functions without departing from the scope of the exemplary embodiments.

[0170] Still reference Figure 4A , reconstruction of a real-world scene such as 4A01 by system 1A01 includes determining data representations of the matter field 2B09 and light field 2B11 of the real-world scene, wherein these representations and the organization of these representations have important effects on scene reconstruction, but even more importantly on the efficiency of processing speed including sub-scene extraction. Exemplary embodiments provide scene representations and organization of scene representations that essentially make, for example, large global scene models available to users for real-time or near real-time experience. Figure 4A , Figure 4B and Figure 4C These scene representations and the organization of scene representations are collectively directed to be described in some level of detail, and all of them are then further detailed in sections of this disclosure.

[0171] Techniques in accordance with the exemplary embodiments described herein may use hierarchical, multi-resolution, and spatially ordered volumetric data structures to describe the matter field 2B09 and the light field 2B11. This allows the identification of portions of the scene required for remote viewing based on position, resolution, and visibility determined by each user's position and viewing direction or statistically estimated for a group of users. By transmitting only the necessary portions, channel bandwidth requirements are minimized. The use of volumetric models also facilitates advanced functionality in virtual worlds, such as collision detection and physics-based simulations (quality attributes are easy to compute). Thus, based on a novel scene reconstruction process of real-world scenes such as 4A01 into a novel full-optical scene model representation and organization of the representation, as well as a novel process for sub-scene extraction and user-scene interaction monitoring and tracking, the exemplary embodiments provide many use case advantages, some of which will be discussed in the upcoming Figure 5 , Figure 6 and Figure 7, one of the advantages includes providing a free viewpoint viewer experience. In the free viewpoint viewer experience, one or more remote viewers can independently change their viewpoints of the transmitted sub-scene. What is required for the largest free viewpoint experiences (especially for larger global scene models) is that the system 1A01 provides the sub-scenes to the free viewpoint viewers just in time and only as needed or just in time and only as expected.

[0172] Still reference Figure 4A , images and otherwise captured sensor data representing real-world scenes such as 4A01 contain one or more properties of light, such as color, intensity, and polarization. By processing this and other real-scene information, system 1A01 determines shape, surface properties, material properties, and light interaction information about matter field 2B09 for representation in full-light scene database 1A07. A separate determination and characterization of the light field 2B11 of the real scene is used in conjunction with the matter field 2B09 to, among other goals, disambiguate surface properties and material attributes arising from scene lighting (e.g., specular reflections, shadows). The presently described novel processing of real-world scenes into scene models allows for the effective modeling of transparent materials, highly reflective surfaces, and other difficult cases in everyday scenes. This includes, for example, the "discovery" of material and surface properties independent of the actual lighting in the scene, where such discovery and accompanying representation and organization of representations then allow novel sub-scene extraction, including accurate separation of matter field 2B09 that is distinct from light field 2B11 and provided to free-pilot viewers.

[0173] Thus, the free spot viewing experience accomplishes another key goal of free lighting, where, for example, upon accessing a scene model corresponding to a real scene such as 4A01, a viewer is able to request free spot viewing of a scene that may have "morning sunlight" versus "evening sunlight" or even "half moon lighting with available room lights", where a user interface, preferably provided by application software 1A03, allows for insertion of new light sources from a set of template light sources, and where the newly designated or available light sources can then be modified to change, for example, light emission, reflection, or transmission characteristics. Similarly, the viewer can also dynamically change the properties and characteristics of the matter field 2B09, thereby providing free matter as well as free lighting and free viewpoint, where it is particularly important to see that the exemplary embodiments provide a more accurate separation of the matter field 2B09 and the light field 2B11 of the real scene, where the lack of accuracy in the separation in turn limits the end-use experience for accurately changing the properties and characteristics of the matter field 2B09 and / or the light field 2B11. Another advantage of accurate matter fields 2B09 as described herein includes interference and collision detection within matter field objects, where these and other life simulation functions require matter properties such as mass, weight, and center of mass (e.g., for physics-based simulations). As those familiar with object recognition in real-world scenes will also well understand, highly accurate matter fields and light fields provide significant advantages.

[0174] Still reference Figure 4A The matter field 2B09 of a real scene 4A01 comprises elements, which are finite volume representations of material in which light flows or is blocked, and therefore have varying degrees of light transmittance, which can be characterized as degrees of absorption, reflection, transmission, and scattering. Elements are located and oriented in scene space, and have associated properties such as material type, temperature, and a two-way light interaction function (BLIF) that relates the incident light field to the outgoing light field caused by the interaction of light with the elements. Optically, spatially, and temporally homogeneously juxtaposed elements form fragments of an object, which include surfaces with touchable boundaries, where touchable boundaries are generally understood to be boundaries that a person can sense by touch. Using these and other matter field 2B09 properties, objects are distinguished not only spatially but also, and importantly, with respect to their interaction with the light field (2B11). Figure 4A , where these various objects again include: opaque objects 4A03, finely structured objects 4A05, distant objects 4A07, emitting objects 4A09, highly reflective objects 4A11, featureless objects 4A13, or partially transmissive objects 4A15.

[0175] Next reference Figure 4B , showing a representative example Figure 4ASchematic diagram of a real world scene 4A01 depicted in FIG, wherein the representation can be considered an abstract scene model 4B01 view of data within a plenoptic scene database 1A07. The abstract representation of the scene model 4B01 of the real world scene 4A01 includes an external scene boundary 4B03 that contains a plenoptic field 4B07 that includes a matter field 2B09 and a light field 2B11 of the scene. The light field 2B11 is associated with Figure 4A The real world scene 4A01 is captured, for example, by any one or more of real sensors such as a real camera 2B05-1 that captures a real image 4B13, while the scene model 4B01 is converted into a real world data representation using, for example, a virtual camera 2B03 that provides a real world representation image 4A17.

[0176] In addition to Figure 4A In addition to the objects shown in FIG. 4A including opaque objects 4A03, finely structured objects 4A05, distant objects 4A07, emitting objects 4A09, highly reflective objects 4A11, featureless objects 4A13, or partially transmissive objects 4A15, the system according to some embodiments may further allow for explained objects 4B09 and unexplained areas 4B11, where these common objects and areas include, for example, Figure 4A An important aspect is the identification of the matter field 2B09 by a scene reconstruction sufficient to distinguish between multiple types of objects, wherein any single object type uniquely located in the model scene can then be further processed, for example by using machine learning to perform object recognition and classification, changing various characteristics and attributes to cause model rendering effects, such as changes to visualization (usually transformations), object enhancement and labeling (see in particular relative to Figure 4C Model enhancement 4C23 and model index 4C27) and even object removal. Along with object removal, object transformation (see upcoming Figure 4C Model transformations 4C25) in can be specified to perform any number of geometric transformations (such as setting size and rotation) or even object movement based on, for example, object collisions or specified object paths, such as an opaque object 4A03 classified by machine learning, which then rolls along the floor (opaque external scene boundary 4B03) and bounces off a wall (opaque external scene boundary 4B03).

[0177] Still reference Figure 4B, the characteristics and properties of any object can be changed by additionally processing new real scene sensor data, such as but not limited to new camera images, which may be taken at invisible frequencies such as infrared, thereby providing at least new BLIF (Bidirectional Light Field Information). Figure 4A and Figure 4B The object types depicted in should be considered exemplary and not limiting of the embodiments, and it will be clear to those familiar with the software and databases that the data can be updated and the labeling of the associated data forming the matter field objects can be adjusted, including at least the naming of the objects (such as "featureless" versus "finely structured"), or including changes in object classification thresholds, which can be used, for example, to classify objects as "partially transmissive" versus "opaque." Based on this disclosure, other useful variations of object types will be apparent to those skilled in the art of scene processing, as will other variations of matter fields 2B09 and light fields 2B11 in general, so that the most important thing is the representation of real-world scenes such as 4A01 as described herein and the organization of representations and their efficient processing, which combination can be used to provide unique functionality as described herein, where again these unique functionality provides a viewer's free viewpoint, free matter, and free light experience that is visualized at least using a scene model.

[0178] The scene model also includes an outer scene boundary 4B03, which delimits the outermost extent of the represented full light field 4B07. Figure 4AA careful consideration of a real-world scene such as the kitchen depicted in , or an outdoor scene (not depicted) will reveal that some areas of the full light field near the exterior scene boundary 4B03 may be substantially opaque (such as a wall or countertop in a kitchen, or thick fog in an outdoor scene), while other areas near the (imaginary) exterior scene boundary may be substantially window-like, representing an arbitrary light field boundary (such as the sky in an outdoor scene). In a real scene, light may cross back and forth in the space associated with the exterior boundary of the scene model. However, the scene model does not allow for such transmission. Instead, a window-like light field can represent a light field in a real scene (just like a television can display a picture of the moon at night). In the scene model, the opaque areas near the exterior scene boundary 4B03 do not represent substantial transmission of light outside the full light field into the interior of the scene (in a real scene), while the window-like areas near the exterior scene boundary 4B03 do not represent substantial transmission of light outside the full light field into the interior of the scene (in a real scene). In some embodiments, trees and other outdoor matter may be represented as being included in the full light field 4B07, where the exterior space boundary 4B03 is then spatially extended to include at least this matter. However, as objects in the object field become increasingly distant, even reaching a distance known as the no-parallax limit, where features on the objects do not substantially change with changes in viewpoint, it is advantageous to terminate the full light field 4B07 in the exterior scene boundary 4B03. Using one or more window light elements 4B05, the light field incident on the real scene along portions of the exterior scene boundary 4B03 may be represented as if the full light field 4B07 at those portions of the exterior scene boundary 4B03 extended indefinitely.

[0179] For example, refer to Figure 4A , rather than extending the scene boundary 4B03, and therefore the full light field 4B07, to the outdoor area beyond the window to include the tree in the matter field of the scene, it is possible to have the scene boundary 4B03 essentially terminate at a representation of the medium and matter including the workbench, wall, and window, where a plurality of window-like light elements 4B05 included along a portion of the exterior scene boundary 4B03 (representing the window surface spatially) can then be added to effectively inject the window-like light field into the full light field 4B07. Various light field complexities can be achieved using window-like light elements 4B05 including 2D, 3D, and 4D light fields. Note that as used herein, "medium" refers to the contents of a volumetric region including some matter or no matter. A medium can be homogeneous or heterogeneous. Examples of homogeneous media include: empty space, air, and water. Examples of heterogeneous media include the contents of a volumetric region including the surface of a mirror (part air and part broken glass), the surface of a glass pane (part air and part transmissive glass), and the branches of a pine tree (part air and part organic material). Light flows through media through phenomena including absorption, reflection, transmission, and scattering. Examples of partially transmissive media include pine tree branches and glass panes.

[0180] In one exemplary use and advantage of the present system, a Figure 4B and corresponds to the method described in Figure 4A 1A07 of a real scene depicted in the example of FIG. 1B is used to estimate the amount of sunlight based on the time of day that will be transmitted into the scene (e.g., a kitchen), where room temperature or seasonal energy savings opportunities based on the estimated time of day for various types of window coverings can be calculated based at least in part on the estimated amount of transmitted sunlight. Such exemplary calculations are based at least in part on data representing a light field 2B11 that includes a full light field 4B07, whose light field representation and other more basic calculation methods will be addressed in more detail herein. It can be said that the light field 4B07 is treated as a quasi-steady-state light field such that all light propagation is instantaneous relative to the scene, although using the free light principles described herein, a viewer can experience the dynamic state light field by presenting a visual scene representation preferably using application software 1A03.

[0181] Still reference Figure 4B , showing, for example, a real camera 2B05-1 capable of capturing views up to and through 4π steradians, and a virtual camera 2B03 having an exemplary restricted viewpoint 4A17 of less than 4π steradians. Any scene described within the plenoptic scene database 1A07 (such as a real scene 4A01 having a corresponding scene model 4B01) may include any number of real or virtual cameras, such as 2B05-1 and 2B03, respectively. Any of the cameras, such as 2B05-1 and 2B03, may be designated as fixed relative to the scene model or movable relative to the scene model, where the movable camera is associated with a traversal path, for example. It is important to see that the many possible viewpoints of any real or virtual camera, and the images produced therefrom, whether mobile or fixed, whether adjustable in the field of view or fixed in the field of view, may be obtained by processing such as with respect to Figure 4B The scenario model 4B01 is described for estimation.

[0182] Next reference Figure 4C , a block diagram showing the main data sets in one embodiment of a plenoptic scene database 1A07, wherein the database 1A07 stores, for example, data representing Figure 4AThe data model view of the plenoptic scene database 1A07 of the real world scene 4A01 typically includes, for example, any of intrinsic or extrinsic data 4C07 describing a sensor 1E07, such as a real camera 2B05-1 or 2B05-2 or a virtual camera such as 2B03, wherein the intrinsic and extrinsic data are well known in the art based on the type of sensor, and wherein typically the intrinsic attributes are related to the data capture and processing functions of the sensor, and wherein the extrinsic attributes are related to the physical position and orientation of the sensor relative to a typically local scene coordinate system or at least any coordinate system that allows understanding of the spatial position of the sensor relative to the scene including the matter field 2B09 and the light field 2B11.

[0183] The data model view also includes a scene model 4C09, which generally includes a full light field 4C11, objects 4C13, fragments 4C15, BLIF 4C17, and features 4C19. The term "full light field" has a range of meanings in the current art, and in addition, the present disclosure provides a novel representation of the full light field 4C11, wherein this novel representation and the organization of the representation are at least partially the basis for many technical improvements described herein, which include, for example, just-in-time sub-scene extraction, providing a basic continuous visual free view experience with sufficient scene resolution (thus meeting QoS thresholds) achieved with a minimum data set (sub-scene). Therefore, the term and data set full light field 4C11, like other specifically described terms and data sets described herein, should be understood in light of this specification and not only with reference to the current prior art.

[0184] Still reference Figure 4C, the plenoptic field 4C11 includes an organization of representations referred to herein as a plenoptic octree, wherein the plenoptic octree maintains representations of the matter field 2B09 and the light field 2B11. A more detailed discussion of representations and organization of representations relative to the scene model 4C09 in general and relative to the scene model's primary data sets (such as the plenoptic 4C11, objects 4C13, fragments 4C15, BLIF 4C17, and features 4C19) in particular will be presented in the remainder of this specification, wherein the plenoptic octree representation, generally as described herein, includes two types of representations of the matter field 2B09 and one type of representation of the light field 2B11. The matter field 2B09 will be shown as including a (volume) "Medium" type matter representation and a "Surface" type matter representation. The Medium representation describes a homogeneous or heterogeneous material in which light substantially flows (or light is substantially blocked). This includes empty space. Light flows in a medium including a Medium through phenomena including absorption, reflection, transmission, and scattering. The type and degree of modification of light is contained in the property values ​​contained in, or referenced by, the voxels of the plenoptic octree. A surface type representation describes a tangible (touchable) (approximate) planar boundary between matter and empty space (or another medium), where the planar boundary includes medium on both sides, and where the medium on both sides can be the same or different media (where different medium surfaces are referred to as "split surfaces" or "split surfels").

[0185] Surface-type matter comprising juxtaposed media that are homogeneous in both space and time forms a fragment representation 4C15, where the juxtaposed fragments then form a representation of an object 4C13. The effects of the surface-type matter on the light field 2B11 (reflection, refraction, etc.) are modeled by a Bidirectional Light Interaction Function (BLIF) representation 4C17 associated with the surface-type matter, where the granularity of the BLIF representation 4C17 is extended to be associated with at least the fragment 4C15 comprising the object 4C13 but also with a feature representation 4C19, where features are referred to as poses in entities (such as the object 4C13) located in the space described by the scene model 4C09. Examples of features in a scene or image include spots and sparkles from a microscopic perspective or even buildings from a macroscopic perspective. The BLIF representation 4C17 relates the transformation of the light field 2B11 incident on a material (matter) to the light field 2B11 emerging from the material based on the interaction of the light field with the material.

[0186] Still reference Figure 4C, the primary data set of at least one embodiment of some exemplary embodiments includes auxiliary information 4C21, such as any one or any combination of the following: model enhancements 4C23, model transformations 4C25, model indexes 4C27, and model usage history 4C29. Model enhancements 4C23 include any additional metadata to be associated with a portion of the scene model 4C09 that is otherwise not included in the scene model 4C09 and does not otherwise specify changes to the scene model 4C09 (where, for example, model transformations 4C25 describe some mathematical function or the like to attribute to the scene model, whose properties change (alter) an extracted sub-scene or scene model interpretation (such as a metrology)).

[0187] Model augmented representation 4C09 includes, but is not limited to: 1) descriptions of the virtual scene, including text, graphics, URLs, or other digital information, such as may be displayed as increments to a viewed sub-scene (conceptually similar to augmented reality (AR)), examples including room features, object prices, or links to the nearest store to purchase objects, such as relative to the real scene diagram depicted in 4A; 2) sensory information, such as a current temperature reading to be associated with a portion of matter field 2B09 or light field 2B11, where the sensory information is typically not based at least in part on matter 2B09 or light field 2B11, and includes any type of data obtainable from future sensors currently known or not yet known, and particularly relates to data associated with somatic sensations (touch), smell (smell), audition (hearing), or even taste (taste); and 3) metrics related to computations describing either matter field 2B09 or light field 2B11, examples including measurements of quantity (dimensions such as size) or quality (dimensions such as temperature), where preferably the computations are based at least in part on either matter field 2B09 or light field 2B11. Model enhancements 4C23 are associated directly with any context model 4C09, or indirectly with context models 4C09 via either model transformations 4C25 or model indexes 4C27. Model enhancements may be based at least in part on any information included in model usage history 4C29, where, for example, the enhancements are up-to-date statistics on some recorded aspect of context model usage by multiple users.

[0188] Still reference Figure 4C, model transformation 4C25 includes but is not limited to: 1) geometric transformation applied to any of the material field 2B09 or light field 2B11 including the scene model 4C09, wherein the geometric transformation maps the spatial position of any material or light field element to a new position, including spatial movement, rotation, enlargement, reduction, etc.; 2) composite geometric transformations, such as for describing the movement of objects in the scene model 4C09 or otherwise for the trajectory path of any of the material field 2B09 or light field 2B11; 3) virtual scene paths, including waypoint times, which, for example, describe a virtual camera (such as 2B 03) movement and viewpoints within the scene model so that a viewer experiencing a visual representation of the scene model is guided through the scene without the need for free view instructions, or, for example, describing a suggested path through the scene model, such as a set of city tourist destination locations, where a viewer may be transferred from one sub-scene to another sub-scene that may or may not be spatially juxtaposed in the scene model 4C09; and 4) precompilation of any of the data included within the scene model 4C09 and associated auxiliary information 4C21, the auxiliary information including, for example, a jpeg image of St. Clement's Cathedral in Prague. Model transformations 4C25 are associated with any scene model 4C09 directly, or indirectly with the scene model 4C09 through any of model enhancements 4C23 or model indexes 4C27. Model transformations may be referenced by a model usage history 4C29, where, for example, the use of a given transformation is recorded between multiple users.

[0189] The model index 4C27 includes data for presenting to a human or autonomous user any of index elements for selecting a portion of any scene database 1A07 including, in particular, any scene model 4C09 or auxiliary information 4C21, wherein the data includes, but is not limited to: 1) a list of index elements including text, images, video, audio or other digital information, wherein each index element in the list is associated with at least one portion, such as a sub-scene of the scene database 1A07 to be extracted to a requesting user (human or autonomous); or 2) an encoded list of index elements including any of encrypted or unencrypted information that can be used to select a portion of the scene database 1A07 by an executed computer algorithm, wherein, for example, a remote computer of a system using a scene codec 1A01 accesses an encrypted model index of extractable scene information, the extractable scene information including a type of scene information for algorithmic comparison of scene information of a desired type, wherein the algorithm then selects the scene information for extraction based at least in part on the algorithmic comparison. A given model index 4C27 may include associated permission information for allowing or denying any given user (human or autonomous) access to index 4C27 (and therefore access to scenario model 4C09 through index 4C27), where the permission information includes the types of given users allowed, the specific given user associated with access credentials (such as a username and password), and any information related to the payment or sales transaction, including links to interact with remote sales transaction providers (such as PayPal).

[0190] A model index 4C27 is directly associated with any scenario model 4C09. A model index 4C27 may be associated with or trigger the use of a model augmentation 4C23 (e.g., some type of current sensor reading acquired throughout a real-world scenario such as a natural disaster scenario corresponding to a scenario model) or a model transformation 4C25 (e.g., a scene that relights in a morning, daytime, or evening setting, or automatically enters a scene in a particular sub-scene and then automatically moves through the entire scene according to a prescribed path). A model index 4C27 or any of its index elements may be associated with any model usage history 4C29, where association is broadly interpreted to include any formula for a model usage history 4C29, such as a statistical percentage of selection of the index element by multiple users (human or autonomous), and the elapsed time usage of the associated scenario model, where the statistical percentage is then used to rank or represent the index element for a given model index 4C27.

[0191] Still reference Figure 4C, the model usage history 4C29 includes any data known to the system using the scene codec 1A01 that represents a request or instruction from a user, where the user is human or autonomous. Requests and instructions include, but are not limited to, any of the following: 1) model indexing and model index element selection; 2) scene model free view, free matter, or free light adjustment based at least in part on any of explicit or implicit user instructions; 3) any of explicit or implicit user instructions from a user, including body movement, facial expressions, or audible sounds tracked for a human user; 4) summarized scene model propagation information, including the elapsed time spent within a subscene, or at least the elapsed time starting with the provision of a subscene before an increment of a subscene (such as an increase in space to include more total scenes) or before a request to switch to an alternative subscene; or 5) any information recording the use of model enhancements 4C23 (e.g., using a URL to access information external to the scene database 1A07) or model transformations 4C25 (e.g., using a predetermined scene path, such as a tour representing a particular scene, the scene being a city view or a house for sale).

[0192] In addition, relative to Figure 4C As those familiar with computer databases will well appreciate, there are many types of database technologies that are currently available in the market or that will become available in the future, wherein the presently described plenoptic scene database 1A07 may be implemented with any number of these database technologies or even combinations of these technologies, each having trade-offs and each being enhanced by technical improvements in the organization of representation and representation of at least the scene model 4C09 as described herein. As those familiar with computer databases will also appreciate, although one embodiment of database 1A07 has been described as including various data sets 4C07; 4C09 including 4C11, 4C13, 4C15, 4C17, and 4C19; and 4C21 including 4C23, 4C25, 4C27, and 4C29, the data described herein as belonging to a data set may be reorganized into different arrangements of the data sets, or some of the described data sets may be further divided into other data sets or combined into other data sets without departing from the true scope and spirit of the exemplary embodiments. It should also be understood that at least the embodiments described herein are described as including at least the embodiments described herein. Figure 4C , where this other data is also included within plenoptic database 1A07, but may form its own data set that does not include those data sets depicted in the accompanying drawings. A careful reading of this disclosure will also make it clear that not all of the data sets depicted in the accompanying drawings must be present in database 1A07 in order for a system using scene codec 1A01 to perform useful and novel functions, or otherwise provide any of the many technical improvements described herein. Thus, as with respect to Figure 4CThe described plenoptic scene database 1A07 should be considered as exemplary rather than limiting of the exemplary embodiments, as many variations are possible without departing from the scope of the embodiments provided herein.

[0193] Next, common reference Figure 5 , Figure 6 and Figure 7 , depicts a series of three flow charts according to some exemplary embodiments, which describe three variations of a general use case of a system using scene codec 1A01. Each of the three flow charts depicts a series of connected process shapes that are boxes, ovals, or diamonds, where it should be understood that each of these process shapes represents a higher-level function of a specialized computer process that includes a portion of the computational improvements provided in the exemplary embodiments, where then all of the shapes and their interconnections together further describe the technical improvements specified herein. The diamond represents a function that determines an important branch decision of the system relative to processing a user request within the general use case. In general, each shape can be understood to represent a set of executable instructions for processing on a computing device, where these computing devices can be any arrangement of many possible variations, such as a single CPU with multiple cores, each of which performs one or more functions, or multiple CPUs with single or multiple cores, where these multiple CPUs can be distributed on any type of network as previously discussed. In addition, as previously discussed, there are certain computer operations for certain novel technologies described herein that can be further optimized using some form of hardware-specific processing unit, such as an FPGA or ASIC or other well-known hardware that executes what is often referred to as embedded code. In particular, some exemplary embodiments include a spatial processing unit (SPU) 1A09 (see Figure 1A ), the spatial processing unit is preferably a set of operations executed on an embedded system and may even include custom digital circuits optimized for the key plenoptic scene processing functions described in the present disclosure.

[0194] Still a common reference Figure 5 , Figure 6 and Figure 7, exemplary embodiments provide important benefits in the reconstruction, distribution and processing of scene models, wherein it should be understood that, conventionally, most codecs transmit visual information (such as movies), which does not provide many of the benefits described herein, such as free view and on-demand sub-scenes, not, for example, transmission of non-visual scene data (such as scene metering, matter fields or light fields), not, for example, scene data is alternately served to human users and autonomous users, each accessing the same model for different forms and aspects of the scene data. There are some systems for transmitting scene models, including, in particular, virtual reality (VR) systems, but typically, VR systems are based on computer modeling that lacks significant realism, such as Figure 4A . There are other systems for transmitting real-world scenes that have been reconstructed as scene models, however, the present system provides a unique plenoptic octree representation of the real scene, in which the matter field 2B09 and the light field 2B11 are more separated than in existing systems, and in which the organization of the representation, among other aspects, of the matter field 2B09 and the light field 2B11 provides a significant technical improvement over the underlying functionality, enabling real-time or near real-time access and consumption of large real-world reconstructed scenes. Thus, the present system provides a representation of the real scene and the organization of the representation as a plenoptic octree database 1A07, which enables the system to handle large global scenes in a distributed network, where the scene is even undergoing intermittent or continuous reconstruction, and where the basic transmission of information is a stream of punctualized heterogeneous, asynchronous plenoptic data, rather than, for example, just visual data (such as conventional codecs), or even an entire reconstruction model (such as other state-of-the-art scene model codecs).

[0195] Coming Soon Figure 5 , Figure 6 and Figure 7 It will be apparent that there are a variety of ways to process a scene using two or more systems utilizing scene codecs 1A01, where, for example, a first system 1A01 resides on a server and provides scene model information on demand to a human or autonomous consumer (client), where the client then uses a second system 1A01 to receive and process the scene model information (see, for example, Figure 5 ). In other variations, the first system 1A01 is being used by a client who wishes to capture what might be considered a "local" versus a "global" scene, such as their car that was damaged in a recent hail storm, or their house that was damaged in a storm or is just being sold. In this type of use case, the client system 1A01 is also adapted to include raw data for capturing the local scene (see in particular Figure 6) of one or more scene sensors 1E09. It is then possible to transmit this raw scene data over the network to a second system 1A01 running on a server, which assumes the primary responsibility of reconstructing the local client scene and retransmitting the reconstructed sub-scenes and scene increments back to the client system 1A01. In still other variations, a first system 1A01 used by a client in a shared scene (such as a disaster site or industrial warehouse) and a second system 1A01 running on the network share the responsibility of reconstructing the scene data captured by the client into sub-scenes and scene increments, wherein these reconstructed sub-scenes and scene increments are then shared between the first and second systems 1A01 through a codec function, so that the shared scene captured locally by the first system 1A01 is compiled into a larger scene model (through the joint operation of the first and second systems 1A01), which larger scene model can then also be used to share with a third system 1A01, wherein this third system can then be local to the shared scene and also capture raw data, or remote from the first and second systems 1A01, and therefore also remote from the shared scene.

[0196] Common relative to Figure 5 , Figure 6 and Figure 7 , rectangles are drawn to represent the codec 1A11 encoder functions ( Figure 5 505, 515, 517; Figure 6 505, 515, 517; and Figure 7 505 and 517), some indicate codec 1A11 decoder functions ( Figure 5 507, 511; Figure 6 507 and 511; Figure 7 507 and 511), and some indicate application software 1A03 ( Figure 5 501a, 501b, 503, 509, 513a-d; Figure 6 501a, 501b, 503, 509, 513a-d; Figure 7 501a, 501b, 503, 509, 513 in FIG. 5). As will be well understood by those familiar with computer systems and software architecture, it is represented as Figure 5 , Figure 6 and Figure 7 There are many variations in the implementation of the various operations and functions of the shapes in the embodiment, including the use of any one or more processing elements to perform any one or more functions, where for example the processing element is any of the CPU, GPU, or SPU 1A09 defined herein, which communicates with or supports any of the codec 1A11 or application software 1A03 functions as an embedded processor. Therefore, with respect to the upcoming Figure 5 , Figure 6 and Figure 7 The depictions and provisions should be considered as exemplary rather than limiting of the exemplary embodiments, as the functions described may be further combined or further divided, and the functions may be implemented in various combinations and deployed in many variations without departing from the scope and spirit of the exemplary embodiments. It will also be apparent to those skilled in the art of software systems, networks, traditional compression, scene modeling, etc. that some other functions have been omitted for clarity, but these functions may be apparent based on prior knowledge (such as, if a network is to be used and depending on the type of network, transport layer 1E03 will come into play). Figure 5 , Figure 6 and Figure 7 Some connecting lines that are thicker than other connecting lines are also shown, where these thicker lines ( Figure 5 505-507, 517-507; Figure 6 501b-505, 505-517, 517-507; Figure 7 501b-505, 505-517, 517-507) in represents the transmission of any combination of the full optical scene database 1A07 information, such as generally relative to Figure 2B 2B13 described, or scene data 2B07 such as captured by one or more scene sensors 1E09 (such as real cameras 2B05-1 and 2B05-2) for scene reconstruction or annotation purposes (see Figure 2B ) transmission.

[0197] Now only for reference Figure 5 , showing a flowchart of an embodiment, for example, including sharing a larger global scene model with a remote client, which is human or autonomous and is consuming various types of scene model information as described herein, including free view, free matter, free lighting, metering, any of traditional 2D, 3D or 4D visualization, associated (five) sense information, auxiliary information, or any of other related scene model information included in or associated with the plenoptic scene database 1A07 (for example, where the database 1A07 includes URL links embedded within the spatial scene that connect to other Internet accessible content such as current weather conditions, support videos, product information, etc.). In this exemplary embodiment, it can be assumed that the global scene model is not only remote from the consuming client but also too large, thereby precluding the possibility of simply transmitting the entire global scene model to the client. For the purposes of clarity and illustration, the description will be made with respect to only one of many possible specific use cases. Figure 5 Flowchart of, that is, a human user as a remote client requests a city tour relative to a global repository of scenario models city tours available on a server.

[0198] Still reference Figure 5 , human client operations are preferably performed by a client UI (user interface) 501 executed on an application software 1A03 on a client system 1A01 such as a mobile device. Also, zero or more sensors 1E09 (see Figure 1E ) is included within or in communication with the client system 1A01 for sensing at least some data associated with a human user, which data may be used, at least in part, to determine any of the user requests. Exemplary sensors include a mouse, a joystick, a game controller, a web camera, a motion detector, etc., wherein exemplary data preferably includes data that explicitly or implicitly indicates a desired scene movement or view change, the view change including a direction, path, trajectory, etc. relative to a current position / viewpoint tracked within an ongoing scene, which may be used by the system 1A01 to at least partially help determine a viewpoint change and / or a next scene increment for a current subscene, as will be explained in more detail shortly. The client system 1A01 also includes one or more sensory output terminals 1E11 ( Figure 1E ), these sensory outputs are used to provide data to human users, such as 2D displays, VR headsets, or even holographic displays.

[0199] In a first step of this example, a human user accesses a client UI 501 to determine a global scene of interest (SOI) 501a, where, for example, the selection is a number of tourist attractions around the world including major cities around the world, where, for example, the user selects to take a city tour in Prague. Operation 501a communicates with operation 503 of determining and providing a scene index from the global SOI, where, for example, after the user selects to take a virtual tour of the city of Prague, operation 503 provides a plurality of possible tours (and therefore scene entry points with connecting paths, see in particular Figure 4C Model conversion 4C25) and associated auxiliary information, such as images, short videos, consumer ratings, reviewer ratings, text, URL links to websites, hotel and restaurant information and websites, etc., where this auxiliary information and other scene index information can be transmitted using conventional techniques and codecs based on well-known data types and therefore do not need to be included in the full optical flow 2B13 (see Figure 2B ). Although operation 503 may be performed by storing or accessing the full optical scene database 1A07 (see Figure 2B), but operation 501a and other processing of the client UI 501 are preferably performed on the client side 1E07 system 1A01, including determining the initial sub-scene within the global SOI operation 501b. In operation 501b, the user views and selects a sub-scene for entering the scene model from the scene index provided in operation 503, where, for example, the user selects a city cathedral tour starting at the porch of St. Clement's Cathedral.

[0200] Still reference Figure 5 , operation 501b communicates with the extracted initial sub-scene from the global SOI model operation 505, and transmits to operation 505 an indication of a user-selected sub-scene, such as the front porch of St. Clement's. Based at least in part on the user-selected sub-scene, and also based in part on the determined sub-scene buffer size information, operation 505 accesses the plenoptic scene database 1A07 to determine a set of at least initial object field 2B09 and light field 2B11 data for providing to the client system 1A01 as a first independent sub-scene. In one embodiment, operation 505 is primarily a function of the scene codec 1A11, where operation 505 communicates with operation 501b directly or through an intermediate system component such as application software 1A03. In another embodiment, application software 1A03 provides operations that go beyond communication services between operations 501b and 505, where software 1A03 implements, for example, the portion of operation 505 that is primarily responsible for determining the buffer size, and then also causes the scene codec 1A11 to then extract and transmit the initial sub-scene by calling out various application interface (API) calls into the scene codec 1A11. In any of these or other possible embodiments that are possible and will be understood by those familiar with software systems, the processes executed as part of the scene codec 1A11 may then also call out various API calls into the SPU 1A09.

[0201] In yet another embodiment, when determining, for example, a preferred buffer size, a scene solver 1A05 is called, for example, by the application software 1A03 or the scene codec 1A11, wherein the scene solver 1A05 executes a deterministic or non-deterministic (e.g., statistical or probabilistic) algorithm including a machine learning algorithm, in order to preferably at least partially determine the optimal buffer size based on the auxiliary information 4C21 included in the database 1A07 (see Figure 4C) to provide or predict a buffer size, wherein the auxiliary information 4C29 may be particularly useful as a basis for machine learning, based at least in part on data indicating previous buffer sizes and scene movements such as recordings of previous client sessions for other users, accessing the same sub-scene or different sub-scenes. As will be appreciated by those skilled in the art of software systems and architectures, these same deterministic or non-deterministic (e.g., statistical or probabilistic) algorithms including machine learning algorithms may also be a function of the scene codec 1A11, SPU 1A09, application software 1A03, or even some other component (not specifically described herein but apparent based on the description herein), wherein, for example, a system using the scene codec 1A01 includes, for example, another scene usage learning component implemented using any dedicated machine learning hardware that is currently available and known in the market or that will become known.

[0202] At least one technology company known in the market as NVIDIA currently offers technology referred to as "AI chips" that are part of the so-called "Infrastructure 3.0" and are implemented on specialized GPUs that also include what NVIDIA calls "Tensor Cores." The disclosure herein provides novel representations of scene models and organization of representations, and more specifically, plenoptic scene models include auxiliary information 4C21 that is not traditionally considered scene data, but rather data such as model usage history 4C29 that directly relates to how any person or automaton uses the scene model in any or all ways. As those familiar with machine learning will appreciate, while the exemplary embodiments provide some novel methods for implementing scene learning, other methods and implementations may be apparent, particularly with respect to determining buffer sizes, where these implementations may be software executed on general-purpose computing hardware and / or software executed on specialized machine learning hardware, all of which solutions are considered to be within the spirit and scope of the present disclosure.

[0203] Still reference Figure 5, a description is provided with respect to at least effective (just-in-time) subscene extraction, the subscene extraction being based on some determined or provided scene entry point and some determined or provided scene buffer size or other information that foreseeably constrains the subscenes to be extracted from the entire SOI (e.g., global) scene model, such that the subscenes extracted relative to the spatial buffer are substantially guaranteed to provide a maximum continuous user experience with a minimum amount of provided scene information. After the plenoptic scene data is determined from the database 1A07 representing the user-selected subscene, the plenoptic scene data is transmitted as an asynchronous just-in-time stream 2B13 of any combination of matter field 2B09 and light field 2B11 data included in the database 1A07, wherein, for example, a client 1E07 system using a scene codec 1A01 receives the stream 2B13 to process the stream into a sensory output, such as, for example, images and corresponding audio provided to a user 2B17 via a sensory output device 2B19 capable of providing 2π–4π free view manipulation to a human user, wherein the output device 2B19 is a specific instance of any sensory output device 1E11 available through the client UI 501.

[0204] As a first step of receiving the stream 2B13 by the decoder included in the codec 1A01, the function of inserting the next scene data into the client SOI model 507 is performed, resulting in the reconstruction or updating of the client SOI (i.e., the plenoptic scene database 1A07) image, but not equivalent to the global scene model (i.e., the plenoptic scene database 1A07) from which the sub-scene is extracted and provided. It is important to see that it is possible and considered within the scope of exemplary embodiments to transform the provided stream 2B13 including essentially plenoptic scene model data into the requested user data without first being stored in the client ("local") database 1A07, or even not at all, where, for example, the scene transformation is performed by a step of rendering and presenting to a free view or other scene data that satisfies the user request. However, it is preferred and illustrated herein to provide significant benefits that by first or additionally rebuilding the client database 1A07 and not only converting the stream 2B13 into requested scene data such as a user free view visualization, it is also possible to allow ongoing client-side scene data to be provided that is substantially independent of the global scene model or at least semi-independent, wherein it is sometimes necessary to update or further "grow" the local client scene database 1A07 based on a user's request, wherein such growth is referred to as providing a sub-scene increment to be discussed briefly.

[0205] Still reference Figure 5, those familiar with the software systems and architecture will appreciate that, although operation 507 is preferably implemented as part of a decoder within the scene codec 1A11, at least some portion of operation 507 may be performed by the application software 1A03 implementing the client UI 501. For example, the application software 1A03 may include providing indications and changes to the client UI 501 before actually providing the requested scene data in operation 509. In traditional scene processing, if the requested data is, for example, a free view visualization, then operation 509 includes so-called rendering. Relative to current rendering techniques, due to the full light scene database 1A07, the exemplary embodiments provide increased free matter and free lighting options that provide even more realistic free views. As those familiar with the software architecture will appreciate and based on a careful reading of the present disclosure, both the insert operation 507 and the provide requested data operation 509 may call out various application interface (API) calls to the scene processing unit (SPU) 1A09. Unlike conventional codecs, exemplary embodiments provide confirmation that scene data is received 511 into a server of scene data, a feature that is particularly important considering that future provided scene increments are dependent on the initially provided independent sub-scenes and any subsequently provided scene increments.

[0206] Still reference Figure 5 , it is important to see that the client SOI database 1A07 can be sufficient to provide any and all of the SOI data required in operation 509. The extent to which the first independent sub-scene is sufficient to satisfy all future data requests is proportional to the size of the initial sub-scene and inversely proportional to the scope of the scene data to be requested. As the scope of the requested or expected scene data increases, the burden and cost of transmitting the expected initial sub-scene with sufficient scene buffer eventually becomes prohibitive. For example, if the initial sub-scene is the porch of St. Clement's Cathedral, and the user is only expected to enter the cathedral from the porch and stand in the hall, the size of the sub-scene can be limited. However, if the user expects to enter the cathedral or cross the street to enter another building, the size of the sub-scene will inevitably increase. Therefore, the exemplary embodiment provides that the initial sub-scene includes an intelligently determined scene buffer that balances the expected user request up to a certain amount of scene data, with the need to minimize the transmitted scene data and thereby reduce any perceived scene or UI lag, where after the system provides for transmitting other increments of the sub-scene from the global model for satisfying other requests or expected other requests based on either explicit or implicit user indications. Again, preferably, the balance is based at least in part on a history of similar user requests based on machine learning and other deterministic methods so that a maximum continuous user experience is provided with a minimum amount of context information provided initially and incrementally thereafter.

[0207] As will be clear to those familiar with various types of prediction systems, as the "look ahead" (future-facing) time increases, the number of possible scene movement changes grows geometrically or even exponentially, rather than linearly. For example, if a user is provided with an initial sub-scene of the porch of St. Clement's Cathedral, a look ahead time of 1 minute versus 1 hour results in at least a geometric growth in the scene buffer size, such that if a buffer size of X is calculated for 1 minute, then a buffer size of Y for 1 hour may be much greater than 60*X. In this regard, another key technical advantage of certain embodiments is that representations of plenoptic scene models and the organization of these representations will be shown to significantly reduce the processing time necessary to extract any initial sub-scene or scene increment, given any selected buffer size, relative to currently known scene processing techniques. Thus, it will be clear from careful consideration of the trade-offs that the significant reduction in sub-scene or scene increment extraction and processing time supports both larger initial sub-scene buffers for the same system response time and smaller sub-scene increment buffers that facilitate more frequent scene increments, where the smaller and more frequent approach actually reduces the total scene data transmitted when the user requests a reduction in look ahead time.

[0208] Still reference Figure 5 , the remaining process client request operation 513 record consumption operation 515 provides for tracking the user's scenario usage, and intelligently augmenting the client SOI model in the event that it is determined or expected that the initial sub-scenario lacks sufficient scenario data to satisfy the current user request or possible future user requests. As the user interacts with the UI 501, for example to receive updated scenario data, these interactions provide indications of scenario data values ​​and consumption. In addition, the client UI 501 preferably allows the user to express indications that can be interpreted as requests for more scenario data, such as by moving a mouse, joystick, game controller, or VR headset, where the indications are detected using sensor 1E09. Any one or more of these usage or expected usage indications allow the system to track user consumption within the client SOI model (as operation 513a), where the tracked usage is provided as model usage history 4C29 (see). Figure 4C ) is stored in one or both of the server and client plenoptic scene databases 1A07.

[0209] When the user indication is processed by the client UI 501, the processing client request operation 513 includes an operation 513b, which is used to determine whether any of the user indications can be interpreted as a next request for scene data, and then determine whether the next request can be satisfied based only on the scene data already contained in the existing local client SOI model. If the next request can be satisfied based only on the existing client SOI model, the requested scene data is provided to the user through operation 509. If the next request cannot be satisfied based only on the existing client SOI model, operation 513c determines whether the next request is an increment of an existing sub-scene (or sub-scenes) within the client SOI model, or whether the next request is for a completely new (and therefore independent) sub-scene. If the request is for a new sub-scene, operation 513c transfers control or otherwise calls out the client UI 501 to effectively determine what the new sub-scene being requested is (e.g., switching to the Cathedral of St. Lawrence), or even if the user may be requesting a completely new global scene (e.g., switching to a tour of Venice). If the request is not for a new subscene, but to continue exploring an existing subscene in a manner that requires adding increments to the current subscene, then operation 513d determines the next increment vector for the subscene. The next increment vector represents any dimension of the scene model, such as spatiotemporal extension, spatial detail, light field dynamic range, or material field dynamic range, where the vector is indicative of the range of new scene data minimally required to satisfy the user's request. In determining the vector, operation 513d preferably accesses the user history tracked by record consumption operation 515, where the vector determined to minimally satisfy the user's request and the usage history (of the current and all other tracked users) can be combined, at least in part, for use by the system in estimating the next scene increment and the increment buffer size, where again the buffer size expands the scene increment to the minimum satisfying vector scene increment to include the expected "look-ahead" subscene usage.

[0210] Still reference Figure 5, as will be understood by those familiar with the software system, other operation arrangements are possible, while still executing the preferred steps of tracking and recording at least some of the user instructions and scene model consumption. In addition, when still determining whether the user explicitly or implicitly requests additional scene data, and whether this additional scene data already exists in the client scene model database 1A07, other operation arrangements are possible. If the additional scene data does not yet exist, but is an extension of the sub-scene already existing in the client SOI database 1A07, then other operation arrangements for determining the next scene increment and buffer size are possible. In this way, the functions provided for processing client requests 513 and recording consumption 515 in the accompanying drawings should be considered as exemplary, rather than a limitation of the embodiments. In addition, any one of operations 513, 513a, 513b, 513c, 513d and 515 can be implemented as being executed simultaneously on their own processing elements, or can be implemented as being executed sequentially on a single processing element. For example, the tracking user consumption operation 513a and the recording user consumption operation 515 may be performed in parallel with the sequential processing of the request tracking operations 513b and 513c. In another consideration, the recording consumption operation 515 may be run on the client system 1A01 for updating the client SOI database 1A07 usage history 4C29 and on the server system 1A01 for updating the global SOI database 1A07 usage history 4C29. It is also possible to perform some or all of the following on the client system 1A01 or the server system 1A01: determine the next delta vector of the sub-scene 513d (including determining the buffer size).

[0211] It is important to note that user usage of scenario models is tracked and aggregated, and the client system first attempts to satisfy requests for new scenario data based only on the client SOI model currently resident or accessible to the client system 1A01, and if the global SOI model requires additional sub-scenario increments, calculations are performed to determine the minimum amount of sub-scenario increments necessary to provide a maximum continuous user experience relative to an expected look-ahead usage and a determined quality of service (QoS) level, where the determination of the expected look-ahead usage is based at least in part on the history of tracked usage.

[0212] Next reference Figure 6 , showing the Figure 5 , but now addressing a variant case where the client first creates a scene model or updates an existing scene model instead of accessing an existing model. Figure 5 , operations 507, 509, 511, 513 (including 513a, 513b, 513c and 513d), 515 and 517 remain substantially the same Figure 5 described in this Figure 6Minimal additional discussion. Figure 6 An exemplary use case is a user working with a mobile device, such as a cell phone, who uses the scene codec 1A01 to create (or update) a scene model of, for example, their car hood that was damaged in a hail storm. Many other use cases may apply to systems other than the exemplary use case. Figure 5 , Figure 6 and Figure 7 Each of the flowcharts in Figure 6 Modeling car damage. For example, Figure 6 The use case of can also be applied to users capturing context models of any of their assets or properties (including, for example, houses), or perhaps where the user is an agency that needs to capture asset or property models, such as an insurance or real estate agency. Industrial and engineering companies can also use the same use case to capture context models of key assets or properties for sharing with others, where these assets or properties can be of any size and with almost unlimited visual detail.

[0213] Still reference Figure 6 , the client UI 501 includes Figure 5 The same sensor 1E09 and sensor output 1E11, with one difference in the use case being that for Figure 6 As illustrated in FIG. 1 , sensor 1E09 includes one or more sensors for sensing assets, property, or other real-world scenes to be reconstructed into the plenoptic scene model and database 1A07. A typical sensor 1E09 would be such as Figure 2B 5B05-1 and 2B05-2 depicted in the client UI 501, but can be any of a number of sensors and sensor types. The client UI 501 allows the user to instantiate a new client SOI or select an existing client SOI, such as their car or even a car hood. Exemplary embodiments may provide that, for example, there may be a pre-existing plenoptic scene model of the user's assets, property, or other scene, or the user is about to create (instantiate) a new scene model. For example, if the user is a car rental agent and the renter has just returned a rental car to a scanning station, the client UI 501 might allow the agent to scan a barcode from the renter's agreement and then use this information, at least in part, to call up an existing plenoptic scene model of the same vehicle prior to commencing the rental. Thus, the car can be rescanned by a device that is autonomous but still considered a sensor 1E09, where the new scanned image or otherwise sensed data can be used to update the existing scene model of the vehicle.

[0214] As previously described, using this approach, the plenoptic scene exists in all four dimensions, including the three spatial dimensions as well as the time dimension. Therefore, any refinement to an existing plenoptic scene model can permanently change the plenoptic scene, causing the original baseline matter and light field data to be overwritten or lost, or organizing the refinement into another representation, such as being associated with either a specific time (like 10:44 AM Eastern Time on April 25, 2019) or an event name (like Lease Agreement 970445A). Assuming that matter and light fields are organized according to the time dimension of the plenoptic scene database 1A07, it is at least possible to: 1) create any scene data based on before or after the time / event to reconstruct and refine any real scene; 2) measure or otherwise describe the difference between any two time points in the plenoptic scene database 1A07; 3) catalog the history of changes to the plenoptic scene database 1A07, which are filtered out by any of the database 1A07 features, such as some or all of any part of the scene model including the matter field and light field.

[0215] In the instance where a user scans the hood of their own car to, for example, record and measure hail damage, it is also expected that the user can access a remote database of plenoptic scene models of the car, so that the user will first select an appropriate baseline make and model for their own car, and then, based on this, scan their unique data to rebuild and refine the baseline model, rather than instantiating a new model in the absence of any baseline plenoptic scene. It is further expected that in this case, the client UI 501 will also provide the user with intelligent functions that will allow the user to adjust, for example, the material field associated with the baseline model, such as to change the color of the car to a custom paint color added by the user, or any similar type of difference between the baseline and the unique real scene. It is further expected that any part of the material field can be named or labeled, such as "Car Exterior", where this label is auxiliary information 4C21, such as auxiliary information considered to be model enhancement 4C23 (see Figure 4c). As careful consideration will show, by providing a labeled baseline plenoptic scene model, the system provides important leverage for creating and refining new custom scene models.

[0216] Some exemplary embodiments also provide for a variety of labeled plenoptic matter and light field types and instances and a baseline plenoptic scene model, where, for example, a car manufacturer creates various plenoptic matter field types representing various materials used in the construction of any of its car models, where the car models are again represented as baseline plenoptic scenes. In this arrangement, a car salesperson can quickly change the baseline car to select different material (matter field) types to substitute into the baseline, so that these models can be converted to 4C25 (see Figure 4C ) can then be explored as a plenoptic scene model (like Figure 5). As the careful reader will understand, Figure 5 , Figure 6 and Figure 7 The uses of each of the generic use cases presented in this document are nearly endless, not to mention the other use case variations discussed, implied, or otherwise apparent from the description provided herein, which makes it particularly clear that Figure 5 , Figure 6 and Figure 7 The described use cases should be regarded as illustrative rather than limiting.

[0217] Still reference Figure 6 , after establishing the client SOI as a database 1A07, the information is transmitted to, for example, a second server-side system 1A01, where an operation 503 on the server side instantiates / opens a global SOI model corresponding to the client SOI model. As will become clear, the global model and the client model are to be updated to include substantially the same new scene model information captured by the client system 1A01 sensor 1E09, although there is no requirement that the global model and the client model be substantially identical with respect to the scene data. Preferably, operation 503 communicates with the client UI 501 so that after instantiating or substantially instantiating the global SOI model, the client UI 501 indicates to the user and allows the user to begin capturing scene data (see Figure 2B 2B07), such as pictures and videos of a user's car hood with damage. When capturing new scene data, such as images, the captured data is preferably compressed and transmitted using an appropriate conventional codec (such as a video codec for image data). The compressed new scene data is transmitted to the server-side system 1A01, where operation 505 decompresses and then at least partially uses the new scene data to reconstruct or refine the global SOI model, where the reconstruction refers more to completely new scenes and the refinement refers more to existing scenes (like a full-light scene model of a car that already exists but is now being updated). While the reconstruction operation 505 is creating a new portion of the global scene model, the server-side system 1A01 operation 517 provides the next sub-scene increment (of the new portion) from the global SOI model for transmission to the client-side system 1A01. After receiving the new sub-scene increment, the client-side operation 507 inserts the next scene data (sub-scene increment) into the client-side SOI model. As careful consideration will show, the client-side system 1A01 is capturing data while the server-side system 1A01 is doing all of the scene reconstruction, which can be computationally intensive, such that the server-side system 1A01 effectively offloads this computationally intensive task from the client system 1A01.

[0218] Still reference Figure 6, since the client SOI model is constructed based on the received sub-scenario deltas (reconstructed based on the client sensor data) provided from the global SOI model, the client system 1A01 is then able to provide the scenario model information to the user via UI 501, all in accordance with the previous description related to providing the requested SOI data operation 509 and processing the client request 513. In addition, as previously discussed, the client system 1A01 preferably tracks user instructions and usage in operation 513a to record to the global SOI database 1A07 via operation 515.

[0219] Next reference Figure 7 , showing that based on Figure 5 and Figure 6 Flowcharts of some exemplary embodiments constructed by the present invention, but now a variant case is being solved: in the case where both the client-side system 1A01 and the server-side system 1A01 are capable of reconstructing the real scene and providing scene increments, the client first creates a scene model or updates an existing scene model, and then captures the local scene data of the real scene, which is different from Figure 6 Different, in Figure 6 In the example, the real scene is only reconstructed into a scene model on the server-side system 1A01. Figure 5 , operations 507, 509, 511, 513 (including 513a, 513b, 513c and 513d), 515 and 517 remain substantially as in Figure 5 described in this Figure 7 Minimal additional discussion. Figure 6 , operation 501 (including 501a, 501b and sensors 1E09, 1E11) and operation 503 remain substantially as in Figure 6 described in this Figure 7 Minimal additional discussion. Figure 7 An exemplary use case is a user utilizing a mobile device such as an industrial tablet computer that is a system using scene codec 1A01 to create (or update) a scene model of a disaster relief site (where it is conceivable that many users or autonomous vehicles (not depicted) act as clients 1A01 to capture scene sensory data substantially simultaneously). In addition to the exemplary use case, many other use cases are applicable to Figure 5 , Figure 6 and Figure 7 Each flowchart in Figure 7 Modeling disaster scenes. Figure 7 The use cases are equally applicable to users who capture scene models of any shared scene, such as workers in an industrial setting or commuters and pedestrians in an urban setting.

[0220] Still reference Figure 7 ,picture Figure 6 Similarly, the server-side system 1A07 receives compressed raw data captured by or at least partially based on the client-side system 1A07, and in operation 505, reconstructs the compressed raw data into a global SOI model or improves an existing global SOI model. The current use case also includes an operation 517 on the server-side system 1A07, which is used to provide the next sub-scene increment from the global SOI model to the client-side system 1A01 operation 507, where operation 507 then uses the provided sub-scene increment to update the client SOI model so as to provide the requested SOI data to the user through UI 501 in operation 509. Figure 6 Different from the use case of the client-side system 1A01, the client-side system 1A01 also includes an operation 505 for rebuilding the client SOI model, and then also includes an operation 517 for providing the sub-scenario increment to the server-side system 1A01, wherein the server-side system 1A01 then also includes an insertion operation 507 for rebuilding or improving the global SOI model. Both the client and server-side systems include an operation 511 for confirming any received and processed scene data. It is important to note that, preferably, under the guidance of the application software 1A01 running on any of the server-side system 1A01 and the client-side system 1A01, and preferably, in shared communication, at any given time, for any given real scene data captured by or under the command of any one or more client systems 1A01, one or both of the server-side or client-side systems 1A01 can be directed by the corresponding application software 1A01 to rebuild any real scene data, and then also share the reconstructed scene data as the scene increment of any of the other systems 1A01, or not rebuild any real scene data, and then also receive and process any of the scene increments reconstructed by any of the other systems 1A01.

[0221] The value of this operational arrangement becomes more apparent in larger use cases with multiple client-side systems 1A01 and even multiple server-side systems 1A01, where those familiar with computer networks and servers will understand that the application software 1A01 communicating across the multiple systems 1A01 is performing scene reconstruction and distributed load balancing. Some of the clients may be users utilizing mobile devices 1A01, while other clients are autonomous land-based or air-based systems 1A01. Each of these different types of clients 1A01 is expected to have different computational and data transmission capabilities. Each of these individual clients 1A01 is also expected to have a range of potentially different real-scene sensors 1E09 and require full-optical scene data. The load balancing determination of the software 1A01 takes into account, at least in part, any one of the following or any combination thereof: the overall diversity of the sensor 1E09 data collected, the priority of scene reconstruction, the availability of computational power across all server-side and client-side systems 1A01, the availability of computational power across the network 1E01 (see Figure 1E ) Data transmission capabilities between various systems 1A01, and each system's expectations and on-demand requests for scene data. Figure 5 and Figure 6 Same use case as Figure 7 The use case also preferably captures indications and scenario data usage across multiple client-side systems 1A01, and records these indications and usage data in any of the appropriate scenario databases 1A07 across the multiple systems 1A01, where the machine learning (or deterministic) component of the exemplary embodiment is then able to access this recorded scenario usage to optimize load balancing, in addition to the other benefits and uses previously described. It is also desirable to additionally record server-side scenario reconstruction metrics (such as, but not limited to, fluctuations in the type and amount of raw data received and scenario reconstruction processing time) along with the client-side usage, where this additional server-side record is then also used, at least in part, by the machine learning (or deterministic) component to determine or provide load balancing requirements.

[0222] Scene Database

[0223] Figure 8 A kitchen scene is shown with key properties associated with daily living (everyday) scenes: transmissive media (e.g., glass jug and window glass), highly reflective surfaces (e.g., metal cans), finely structured objects (e.g., the potted plant on the right and the outdoor tree), featureless surfaces (e.g., cabinet doors and dishwasher door), and an effective unbounded volume extent (e.g., the outdoor space seen through the window). Figure 8The scene in is an exemplary scene that can be stored in a scene database 1A07 of a system using a scene codec 1A01 for processing in various use cases. A key aspect of such processing is the subdivision of the volumetric and directional (angle) space into addressable containers that are used to hold the full light field elements of the scene.

[0224] Fig. 9 An exemplary representation of volumetric and directional space containers is shown. Voxels 901 are containers that define regions of scene volumetric space. Solid corner elements 903, referred to simply as "vertices," define regions of angular space originating from the vertex of the vertex. (Vertex 903 is shown from two different viewpoints to help convey its 3D shape.) Although vertex 903 is shown as a finite-extent pyramid, a vertex can extend infinitely far outward from its origin. The containers used in one embodiment may or may not have Fig. 9 For example, the use of non-cubic elements or elements that do not have a square cross-section is not excluded. Figure 21 to Figure 65 Additional details are given regarding the efficient hierarchical arrangement of voxels and cubic elements.

[0225] Fig.10 Shown in a different Figure 4B A top plan view of an exemplary scene model 1001 of an everyday life scene in an exemplary embodiment of the described embodiment. Figure 4B In contrast to embodiments more focused on overall codec operation, the embodiments described herein are more narrowly focused on aspects related to sub-scene extraction and insertion. A full light field 1003 is surrounded by an external scene boundary 1005. The full light field 1003 contains plenoptic primitive entities ("plenoptic primitives" or simply "primitives") that represent the matter fields and light fields of the modeled scene. The full light field 1003 is indexed in volume by one or more generally hierarchical arrangements of voxels and in direction by one or more generally hierarchical arrangements of vertical elements. The matter in the full light field is represented as one or more medium elements ("mesocells"), such as 1027, each of which is contained in a voxel. A voxel can also be empty, in which case the voxel is referred to as a "void" or "of type void". Voxels outside the external scene boundary are of type void. Although by definition, these void voxels do not contain plenoptic primitives, they may be associated with (point to) entities other than plenoptic primitives. Light in the full light field is represented as one or more radiation elements ("radiants"), such as 1017, each of which is contained in a cubic cell located at an inter-element (eg, a voxel containing the inter-element).

[0226] The light field at an element (including light fields that represent only negligible light interactions) includes these four component light fields: the incident light field, the response light field, the emission light field, and the window light field. The incident light field represents light transmitted from other elements (including those immediately adjacent to the element in question). The response light field represents light emitted from the element in response to the element's interaction with the incident light. The emission light field represents light emitted from the element due to some physical process other than interaction with the incident light (for example, transformation from another form of energy, such as in a light bulb). The window light field represents light incident into the element due to unspecified processes external to the full light field. An example of this is a window light field representing sunlight, which is incident at the external scene boundary of the full light field when the full light field does not extend far enough to represent the sun itself as an emission source in volume. It is important to note that in some embodiments, a window light field may be composed of multiple window photon fields viewed as "window layers" representing, for example, one layer of light from the sun and another layer of light from the moon. The mesogen interacts with the incident window light field in the same way that it interacts with the incident light field. In the following discussion of BLIF, statements about the incident light field apply equivalently to the window light field. (The response light field is determined by both the incident light field and the window light field.)

[0227] In the full light field 1003, the intermediate element 1027 has an associated BLIF like all intermediate elements. The BLIF represents the relationship between the interesting characteristics of the incident radiation element and the response radiation element in the quasi-steady light field, such characteristics typically including radiation and / or spectrum and / or polarization information. In the context of certain exemplary embodiments, BLIF is useful because it can pragmatically represent the interaction of light with matter without the need for computationally intensive modeling of such interactions at the molecular / atomic level. In a highly general BLIF representation, the response incident ratio of the characteristic of interest can be stored in a sampled / table form at an appropriate fine element granularity. When practiced, an embodiment may use one or more compressed BLIF representations. One such representation is a low-dimensional model that produces a response radiance as an analytical function of the incident irradiance, parameterized on the incident and exit directions, spectral bands, and the polarization states of the incident and response light. Examples of such low-dimensional models include conventional analytical BRDFs, such as the Blinn-Phong and Torrance-Sparrow microfacet reflectivity models. Such compression of BLIF information is well known to those skilled in the art and will be used to compress and decompress BLIF data in some embodiments of the present invention. Embodiments may allow for representation of spatially (volumetrically) varying BLIF, where one or more BLIF parameters vary over the range of a volumetric scene area.

[0228] The external scene boundary 1005 is a closed, piecewise continuous two-dimensional manifold that separates intermediaries in the full light field from void voxels that are outside the full light field. Void voxels are also located inside the internal boundaries 1007 and 1009. The scene model 1001 represents neither the light transmission outside the external scene boundary nor the light transmission inside the internal boundary. Intermediaries placed adjacent to void voxels are called "boundary intermediaries". In addition to the incident light field transmitted from other intermediaries in the full light field, the light field of the boundary intermediaries may also include a window-like light field representing light incident into the full light field due to unspecified phenomena outside the full light field. The window-like light field at one or more boundary voxels in the scene can generally be regarded as a four-dimensional light field that is volumetrically located on a piecewise continuous manifold defined by the boundary.

[0229] An example of an external scene boundary is the sky in an outdoor everyday life scene. In the full light field of the scene model, air elements try to exist at a reasonable distance (e.g., the parallax resolution limit), beyond which there are void voxels. For example, the light of a clear sky or the moon is represented by a window-like light field of air elements at the external scene boundary. Similarly, light due to unspecified phenomena inside the internal scene boundary is represented by a window-like light field of elements bordering the internal scene boundary. An example of an internal scene boundary is the boundary around a volumetric region that has not yet been fully reconstructed. The 4D window-like light field of adjacent boundary elements contains all (currently) available light field information about the bounded void region. This situation will change if a subsequent reconstruction operation successfully discovers a material field model located within the previously void region, which now interprets the previous window-like light field as incident light transmitted from the newly discovered (resolved) element.

[0230] In addition to the full light field 1003, the scene model 1001 also includes other entities. Intermediates 1027 and other nearby non-air intermediates are referenced in various groupings that are useful in display, manipulation, reconstruction, and other potential operations performed by a system using the scene codec 1A01. A grouping is called a feature, in which the full light elements are grouped together by a certain pattern according to their interesting characteristics (which may include spatial posture). 1029 is a shape feature, which means that the constituent intermediates of the feature are grouped according to their spatial arrangement. In one embodiment, a system using the scene codec 1A01 may treat feature 1027 as a protrusion or bump for some purpose. 1021 is a BLIF feature, which means that the constituent intermediates of the feature are grouped based on the pattern of its associated BLIF. A system using the scene codec 1A01 may treat feature 1021 as a contrast boundary, a color boundary, a boundary between materials, etc.

[0231] Plenoptic fragments are a subtype of features defined by similarity in some set of properties (rather than an arbitrary pattern). Fragments 1023 and 1025 are matter field fragments, in which case the fragments are defined by uniformity (within some tolerance) in the BLIF of the elements of each fragment. An object (such as 1019) is a feature subtype of matter field that is defined by one or more humans recognizing it as an "object" in natural language and cognition. Exemplary objects include a kitchen table, a glass window, and a tree.

[0232] Camera path 1011 is a feature subtype that represents a 6-DOF path tracked by a camera observing the full light field 1003. Aspects of potentially useful embodiments of camera paths include kinematic modeling and spherical linear interpolation (slerp). At locations along camera path 1011, focal planes such as 1013 exist at the camera viewpoint that records the light field. The collection of radiation incident on the focal plane is generally referred to as an image. Exemplary embodiments do not limit camera representations to planar arrays with pixels (photosensitive elements). Other arrangements of pixels are also representable. Focal plane 1013 records light exiting object 1019. Features can be defined on a material field, a light field, or a combination of both. Item 1015 is an exemplary feature of a light field, in which case it includes radiation at focal plane 1013. In this case, the pattern of radiation defines the feature. In conventional image processing terms, a system using a scene codec can view 1015 as a feature detected as a 2D pattern in image pixels.

[0233] Fig.11 Shown with reference above Figure 4C A block diagram of a scene database in different embodiments described in the embodiment. Figure 4CIn contrast to embodiments more focused on overall codec operation, the embodiments described herein focus more narrowly on aspects related to subscene extraction and insertion. The scene database 1101 includes one or more scene models, a BLIF library, an activity log, and camera calibration, as well as other entities not shown. The scene model 1103 includes one or more full light fields (such as 1105), and multiple sets of features (such as 1107), which may include features of type fragments (such as 1109), objects (such as 1111), and camera paths (such as 1113). In addition, one or more scene graphs (such as 1115) point to entities in the full light field. The scene graph may also point to analysis entities that are not currently displayed in the full light field. The scene graph is arranged as a hierarchy of nodes, thereby defining spatial and other relationships between the mentioned entities. If a system using a scene codec desires to register multiple full light fields and / or scene graphs into a common spatiotemporal reference frame at some appropriate point in time, the multiple full light fields and / or scene graphs are typically present together in a single scene model. Without this desire, multiple full light fields and / or scene graphs would typically exist in separate scene models.

[0234] The BLIF library 1119 stores BLIF models (representations). As discussed above, the scene database can store BLIF in a variety of forms, ranging from spectral polarization output-incident ratios to effective low-dimensional parameter models. The BLIF library 1119 includes a material sub-library 1125, which represents light interaction properties and other properties of media that can exist in a material field. Examples of entries in the material library 1125 include dielectrics, metals, wood, stone, fog, air, water, and near-vacuum external space. The BLIF library 1119 also includes a roughness sub-library 1127 that represents the roughness properties of the medium. Examples of entries in the roughness library 1127 include various surface micro-facet distributions, sandpaper particle size categories, and impurity distributions in volume scattering media. An intermediary in the full light field can refer to a BLIF library entry, or can have a "locally" defined BLIF that is not included in any BLIF library.

[0235] Activity log 1121 stores sensing (including imaging) activity log 1129, processing activity (including activities related to encoding, decoding and reconstruction) log 1131, and other related activities / events. Camera calibration 1123 stores compensation parameters and other data related to the calibration of cameras used in scene model-based imaging, display or other analysis operations.

[0236] Fig.12A class diagram 1200 showing a hierarchy of types of primitive entities found in a plenoptic field. A root plenoptic primitive 1201 has subtypes medium 1203 and spoke 1205. Medium 1203 represents a medium in a matter field that is resolved as being contained by a particular voxel. A homogeneous medium 1209 is one in which one or more properties of interest of the medium are uniform throughout its voxel, within some tolerance. Examples of homogeneous medium 1211 include reasonably uniform solid glass, air, water, and fog. A homogeneous medium 1211 is one that does not have such uniformity in the property of interest.

[0237] A facet 1225 is a heterogeneous element having two different regions of different media separated by a segmented continuous two-dimensional manifold. In an exemplary embodiment, the manifold has an average spatial orientation represented by a normal vector and has a spatial offset represented by the closest point between the manifold and the volume center of the voxel containing the facet. Subtypes of facet 1225 include simple facets 1227 and split facets 1229. Simple facets 1227 are just like those described for their supertype facet 1225. Examples of simple facets 1227 include the surface of a wall, the surface of a glass sculpture, and a calm water surface. For split facets 1229, on one side of the facet boundary within the element, the element is additionally divided into two sub-regions separated by another segmented continuous two-dimensional manifold. An example of a split facet 1229 is the area where the black squares and white squares of a chessboard surface meet.

[0238] Smoothly varying medium 1211 represents a medium for which one or more properties of interest vary smoothly across the volume of the medium. Spatially varying BLIFs are often employed to represent smooth variations in light interaction properties throughout the volume of smoothly varying medium 1211. Examples of smoothly varying medium 1219 include surfaces painted with smooth color gradients and regions where a layer of fog at ground level gives way to cleaner air above it.

[0239] Radiation 1205 represents light in the light field of a scene that is resolved as being contained by a particular cube. Radiation 1205 has subtypes isotropic radiation 1213 and anisotropic radiation 1215. Isotropic radiation 1213 represents light that is uniform in one or more properties of interest (such as radiance, spectrum, or polarization) within the directional range of the radiation. Anisotropic radiation 1215 represents light that does not have such uniformity in the property of interest. Split radiation 1221 is anisotropic radiation that has two different regions of different light content separated by a segmented continuous one-dimensional manifold (curve). An example of a split radiation 1221 is a radiation that includes the edge of a highly collimated light beam. Smoothly varying radiation 1223 represents light that varies smoothly in one or more properties of interest within the directional range of the radiation. An example of a smoothly varying radiation 1223 is light from a pixel of a laptop screen that exhibits radiance falloff as the exit angle deviates from the vertical.

[0240] Fig.13 The image shown is a rendering of a computerized model of an everyday life scene (a real-world kitchen). Fig.13 Two 3D points in the kitchen scene are indicated in , for use in subsequent figures and discussions. Point 1302 is a typical point in the kitchen open space (a dotted line perpendicular to the floor is shown to make its location clearer). Point 1304 is a point on the marble counter surface.

[0241] The exemplary embodiments described herein can realistically represent such Fig.13 The scene shown. This is so because the technology according to the exemplary embodiments models not only the material fields of the scene, but also the light fields and the interaction between the two. Light entering or leaving a region of volumetric space is represented by one or more radials incident on or exiting from a specified point in the region representing that space. Therefore, this set of radials is called a "point" light field or PLF. The incident and outgoing light of the PLF is represented by one or more radials that intersect a specified region on a "surround" cube centered at the representative point.

[0242] This can be visualized by showing a cube with light passing through the faces of the cube, showing up on the faces on its way to and from the center point. This "light cube" is Fig.14 In the image, it is 1401. Fig.13 The light cube is centered at point 1302. The light cube shows incident light entering point 1302. Therefore, the light intensity shown at a point or area on the surface of the cube is light from items and light sources in the kitchen (or beyond) that pass through that point or area and also intersect the center of the cube (point 1302). Fig.15 Six additional exterior views are shown of light cube 1401. The light cube can also be viewed from inside the cube. Fig.16, Fig.17 and Fig.18 The images in show various views from inside light cube 1401.

[0243] A light cube can also be used to visualize the light emitted from a point (the exit PLF). This light cube is for point 1304 (the point on the marble counter in the kitchen shown in image 13). Fig.19 1902 shown. The surface of the cube shows light emitted from the point that intersects the faces of the cube or areas on the faces of the cube. This is like looking through a straw at the point from all directions located on a sphere surrounding a single point. Note that the surface of the bottom half of the light cube is black. This is because the center of the PLF is on the surface of the opaque material (in this case, the marble). No light leaves the point in the direction of the interior of the counter, so those directions are black in the light cube.

[0244] Light cubes can also be used to visualize other phenomena. Fig. 20 The image in shows a light cube 2001. It shows the effect of the BLIF function in generating an exit PLF based on an incident PLF. In this case, the incident light is a single vertically polarized beam in an incident light element (spoke) 2002. The exit light generated by this single beam is shown on the face of light cube 2001. Based on the details of the BLIF used, a complex exit light pattern appears, as shown in light cube 2001.

[0245] Some exemplary embodiments provide techniques for computing the transport of light in a modeled scene and its interaction with matter. These and other computations involving spatial information are performed in a spatial processing unit or SPU. The computation utilizes a plenoptic octree consisting of two types of data structures. The first data structure is an octree. An example is Fig.21 and Fig. 22 The volumetric octree 2101 shown. An eight-way partitioned hierarchical tree structure is used to represent a cubic space region in a finite cubic universe. At the top of the tree structure is a root node 2103 at level 0, which just represents the universe 2203. The root node has eight child nodes, such as node 2105 at level 1. This node represents a voxel 2205, which is one of eight equal-sized, non-intersecting cubes that just fill the universe. The process will continue to the next level using the same method of subdividing space. For example, node 2107 at level 2 represents a cubic space 2207. The octree portion of the plenoptic octree will be referred to as a "volume octree" or VLO.

[0246] The second data structure used in the plenoptic octree is a cube tree. A cube is a "solid angle element" and is used to represent a region of directional space projected from an "origin" point. A cube is typically used as a container for radiants (light emitted from the origin or incident light falling on the origin from a direction represented by the cube). A cube tree typically represents the directional space of some volumetric region of space (e.g., a voxel) surrounding the origin.

[0247] The space represented by a cube is determined by square regions on the faces of a cube. This cube is the "surrounding cube" and is centered at the origin of the cube tree. The cube can be of any size and does enclose or constrain the cube tree. The cube only specifies the specific geometry of the cubes in the cube tree, with each cube extending from the origin to an infinite distance (but is generally only used within a volume represented by a plenoptic octree). Similar to an octree, a cube tree is a hierarchical tree structure where nodes represent cubes.

[0248] Liyuan Tree 2301 Fig.23 and Fig.24 . The root node 2303 is at level 0 and represents all directions from its origin (the point at the center of the surrounding cube 2403). Although cubes and cube trees encompass an infinite volume of space extending from the origin, they are typically defined and used only within the domain of their plenoptic octree (which is typically the domain of their VLO). As shown in Fig.23 It can be noted that the entity tree root node has six children, while all nodes in the subtrees below have four children (or no children). Node 2305 is one of the six possible children of the root node (only one is shown). This node is at level 1 and represents all of the space that intersects with the faces 2405 of the surrounding cube of the entity tree projected from the origin. Note that when the center of the entity tree is at the center of the universe, its defining faces will be the faces of the universe. When the entity tree is in a different position, its origin will be at another position within the full optical octree, and its surrounding cube will move around the origin. The surrounding cube will no longer be the universe. Since the surrounding cube only determines the orientation of the entity relative to the origin, it can be a cube of any size centered at the origin.

[0249] At the next level of subdivision, node 2307 is one of four level 2 children of node 2305 and represents face square 2407, which is one quarter of the associated face of the universe. At level 3, node 2309 represents the direction space defined by face square 2409, which is one of four equal squares that divide square 2407 (one sixteenth of face 2405). Fig.414101 in 2D for an element tree 4100 with its origin at point 4101. Node 4102 is a non-root node at the nth level in the element tree (the root node will have six children). This node represents a directional space segment 4103. At the next level down, two of the four n+1th level nodes 4104 represent two elements 4105 (the other two represent another two in 3D). At the n+2th level, node 4106 represents four regions 4107 in 2D (16 in 3D). The element tree used in the plenoptic octree will be referred to as an SLT.

[0250] Note that, like octrees, if the attributes in the subtree are adequately represented by the attributes attached to the node, the subdivision of the cube terminates (no subtree). This is also true when a sufficient level of resolution has been reached or for other reasons. Like octrees, cube trees can be represented in a variety of ways. Nodes are usually connected by unidirectional links (parent to child) or bidirectional links (parent to child and child to parent). In some cases, subtrees of an octree or cube tree can be used multiple times in the same tree structure (in which case it technically becomes a graph structure). Therefore, storage can be saved by having multiple parent nodes point to the same subtree.

[0251] Fig.25 A combination of one VLO and three SLTs in a full optical octree 2501 is shown. The overall structure is a VLO structure, where cube voxel 2505 is represented by a level 1 VLO node, and voxel 2502 is represented by a level 2 VLO node. Voxel 2507 is a level 3 voxel, and its origin is at the center (level 0) of the VLO universe. However, voxel 2503 has a different origin. The origin is located at the center of the level 1 VLO node, which serves as its surrounding cube. Since its limiting square is one-fourth of the face of the surrounding cube, it is a level 2 voxel.

[0252] Instead of a single VLO, as mentioned above, a plenoptic octree can be composed of multiple VLOs representing multiple objects or attributes that share the same universe and are typically combined using set operations. They are like layers in an image. In this way, multiple sets of attributes can be defined for the same region of space and displayed and adopted as needed. If the origin is the same point and the nodes have the same alignment, then the elements in multiple element trees can be combined in the same way. For example, this can be used to maintain multiple wavelengths of light that can be combined as needed.

[0253] The SLT and VLO in the plenoptic octree have the same coordinate system and have the same universe, except that the origin of the SLT can be located at different points within the plenoptic octree, not necessarily at the center of a VLO node. Therefore, the surrounding cube of the SLT, while in the same orientation as one or more VLOs in the plenoptic octree, does not necessarily completely coincide with the VLO universe or any other node.

[0254] exist Fig.26 26 shows (in 2D) the use of perspective plenoptic projections (or simply "projections") in a plenoptic octree, as computed by a plenoptic projection engine. Plenoptic octree 2600 includes three SLTs attached to a VLO. SLT A 2601 has its origin at point 2602. From SLT A 2601, one vertical element 2603 is shown projected through the plenoptic octree in the positive x and positive y directions. SLTB 2604 has vertical element 2606 extending into the plenoptic octree, while SLT C 2607 has vertical element 2608 projected in the other direction.

[0255] This is Fig. 27 2604 are output elements. This means that they represent light emitted from the center of their respective origins. Only one element is shown for each SLT. In use, many elements of different resolutions will typically be projected from the origin of each SLT. In this case, the two elements cross two VLO nodes. SLT C 2607 does not have an element that crosses either of the two VLO nodes and therefore is not shown. Fig. 27 Shown in.

[0256] In operation, the intersection of an SLT cube and a VLO node will result in subdivision of the cube and VLO node until some resolution limit (e.g., spatial resolution and angular resolution) is reached. Typically, subdivision will occur until the projection of the cube approaches the size of the VLO node at some resolution level determined by the characteristics of the data and the exigencies of the requesting process.

[0257] exist Fig.28 , some of the outgoing light that falls on voxel 2710 is captured from point 2602 by cube 2603, and from the origin 2604 of the SLT by cube 2606. There are many ways to capture light, depending on the application. This light that hits voxel 2710 is represented by incident SLT D 2810. This light is generated when light falls on a voxel, or if it already exists, it is added to the existing voxel. In this case, the result is two incident cubes 2811 and 2806. Now, this represents the light that falls on the voxel, as represented by the light hitting the center of the node.

[0258] A representative use of SLTs in a plenoptic octree is to use the light entering a voxel (as represented by the incident SLT) to compute the outgoing light emanating from the voxel. Fig.29 This is shown. A known or assumed BLIF function for voxel 2710 is used to generate a second SLT, the exit SLT. This is exit SLT D 2910. Its origin is located at the same point as voxel D 2810. Thus, exit light from multiple locations in the scene has been projected outwards and then falls on the voxel captured in the incident SLT, and is then used to calculate the exit SLT for that voxel.

[0259] Fig.30 The functions of the SPU according to some exemplary embodiments when generating and operating on a full optical octree are shown. The SPU 3001 may include a set of modules, such as a set operation module 3003, a geometry module 3005, a shape transformation module 3007, an image generation module 3009, a spatial filtering module 3011, a surface extraction module 3013, a morphological operation module 3015, a connectivity module 3017, a quality attribute module 3019, a registration module 3021, and a light field operation module 3023. The operations of the SPU modules 3003, 3005, 3007, 3009, 3011, 3013, 3015, 3017, 3019, and 3021 on the octree are well known to those skilled in the art. They know that such modules can be implemented in many ways, including software and hardware.

[0260] In the SPU functionality, several functions have been extended to apply to all-optical octrees and SLTs. Modifying the set operation module 3003 to operate on SLT is a direct extension of the node set operation on the octree. The nodes of multiple SLTs must represent the same cube (directional space region). The nodes are then traversed in the same order to provide the associated attributes contained in the SLT for the operation algorithm. As is well known in the literature, as with the octree, the terminal nodes in one SLT are matched with subtrees in other SLTs by using a "full node push" (FNP) operation.

[0261] Due to the nature of SLTs, the operation of the geometry 3005 process is limited when applied to SLTs. For example, transformations are not applicable because the incoming or outgoing cubes at one point in the plenoptic octree are generally not the same as at another origin. In other words, the light field at one point will generally be different than at another point, and therefore must be recomputed at that point. The cube interpolation and extrapolation light field operations performed in the light field operations module 3023 accomplish this task. The exception where this is not needed is when the same illumination is applied throughout the region (e.g., illumination from a parallax boundary). In such cases, the same SLT can simply be used at any point within the region.

[0262] Geometric scaling within function 3005 is also not applicable to SLT. Individual cubes represent directions that extend infinitely and do not have scalable sizes. Geometric rotations performed by process 3005 can be applied to SLT using the method described below.

[0263] Morphological operations in 3015, such as dilation and contraction, can be applied to cubes in the SLT by extending their limits to, for example, overlapping illumination. This can be achieved by using undersized or oversized rectangles on the faces of the surrounding cube of the SLT. In some cases, the connectivity function 3017 can be extended to merge SLTs by adding attributes to the VLO nodes, indicating that cubes containing attributes such as illumination intersect these nodes. This can then be used with connectivity to identify connected components that have specific relationships to projected attributes (e.g., materials illuminated by a specific light source or materials that are not visible from a specific point in space).

[0264] like Fig.31 As shown, the operation of the light field operation processor 3023 is divided into specific operations. The position invariant light field generation module 3101 is used to generate SLT from light outside the parallax boundary, and can therefore be used anywhere within the area where the parallax boundary is valid. The light can be sampled (e.g., from an image) or synthetically generated based on modeling of the real world (e.g., the sun or moon) or based on computerized models of objects and materials outside the parallax boundary.

[0265] The exit light field generation module 3103 is used to generate point light field information in the form of an SLT located at a specific point in the plenoptic octree scene model. This can come from sampled illumination or synthetic generation. For example, in some cases, the pixel value in the image can be traced back to a certain position on the surface. This illumination is then attached to the surface point as one or more exit cubes attached to (or contributing to) that position in the direction of the camera viewpoint of the image.

[0266] The exit incident light field processing module 3105 is used to generate an incident SLT for a point in the scene called a "query" point (e.g., a point on an object). If an SLT does not already exist, one is generated for that point and its elements are populated with information about the illumination projected into the scene. When the first object in that direction is found, its exit element is accessed to obtain information about the illumination projected back to the starting point. If no element exists in the direction in question, neighboring elements are accessed, possibly with the aid of known or expected BLIF functions, to generate a set of interpolated or extrapolated illumination values. This process continues for other elements contained in the incident SLT at the query point. The incident SLT thus models an estimate of the light landing on the query point from all directions or a subset of directions (e.g., light from inside an opaque object containing a surface query point may not be needed).

[0267] The incident exit light field processing module 3107 may then be used to generate an exit SLT at a point based on the incident SLT at that point, which may have been generated by module 3105. The exit SLT is typically calculated using a BLIF function applied to the incident SLT. The operations of the submodules included in the light field operation module 3123 employ the cube projection and cube rotation methods described below.

[0268] Fig.32 The surrounding cube 3210 of the entity tree is shown. The six square faces of the surrounding cube of the SLT are numbered 1 to 6. The origin of the coordinate system is located at the center of the SLT universe. Face 0 3200 is the SLT face that intersects the -x axis (hidden in the figure). Face 1 3201 is the face that intersects the +x axis, while face 2 3202 intersects the -y axis (hidden). Face 3 3203 intersects the +y axis, face 4 3204 intersects the -z axis (hidden), and face 5 3205 intersects +z.

[0269] Level 0 in SLT includes the owner of the entire region of the sphere (4π steradians) represented in the cube around the origin of the SLT. At level 1 of SLT, six cubes intersect exactly one of the six faces. At level 2, each cube represents one quarter of a face. Fig.33 The numbering of face 5 3205 is shown. Quarter face 0 3300 is along the –x, -y direction, while quarter face #1 3301 is along the +x, -y direction, quarter face 2 3302 is along the –x, +y direction, and quarter face 3 3303 is along the +, +y direction. The following will focus on the quarter face 3 3303 along the +x, +y, +z direction, as shown in Fig.33 Highlighted in. Fig.34 Face 5 3205 is shown looking at the origin from the +z axis. From this viewpoint, quarter face 3401 is seen as a vertical line, i.e., the edge of the quarter face square.

[0270] The cells that intersect the level 2 quarter-faces are called top cells. Since there are six faces, and each face has four quarter-faces, there are 24 top cells in total. In 3D, a top cell is the space enclosed by four planes that intersect the SLT origin, and each plane intersects an edge of a level 2 quarter-face. In 2D, this reduces to two rays that intersect the center and two ends of a quarter-face (such as 3401). An example of a top cell is Fig.35 3502 having origin 3501.

[0271] Cuboids are regions of space that can be used to represent light projections. They are determined by planes that enclose the volume of space. Technically, they are tilted (or incorrect) rectangular pyramids with infinite height. In 2D, planes are represented by rays. For example, ray 3601 in Fig.36 3602. The ray originates from the SLT origin 3602. A particular ray is defined by the origin and its intersection with a projection plane 3603, which is a plane (a line in 2D) parallel to a face of the cube surrounding the cube (in this case perpendicular to the x-axis). The projection plane is typically attached to a node in the VLO and will be used to determine if the cube intersects that node and to perform lighting calculations if appropriate. The intersection point 3604 (t(t x ,t y )) is determined by the origin 3605 of the projection plane 3603 (usually the center of the VLO to which it is attached) and the distance from the projection plane origin 3605 to the intersection point 3604 (which is 3606, i.e., t y ) is determined. The intersection of ray 3601 and cube face 3607 is point "a" 3608. Since the distance from the origin to the face in the x direction is 1 in the illustrated case, the slope of the ray on the xy plane is the y value of point 3608, i.e., a y .

[0272] The SLT is anchored to a specific point in the universe, namely its origin. The anchor point can be located at a customized explicitly defined point with associated projection information calculated for that point. Alternatively, as described here, the SLT origin can start at the center of the universe and move to its anchor point using a VLO push operation while maintaining a geometric relationship with the projection plane (which also moves around in a similar manner). The advantage of this is that when traversing the octree to locate the SLT center, multiple SLTs can be attached to the VLO node and share simplified projection calculations. The VLO octree that locates the SLT center also contains nodes representing material in a unified data set (the plenoptic octree).

[0273] When implementing plenoptic octree projection, the individual 24 top cells can be processed independently in separate processors. To reduce VLO memory access bandwidth, each such processor can have a set of half-space generators. These generators will be used to locally (for each top cell processor) construct the cell pyramid to be interleaved with the VLO. Thus, unnecessary requests to the VLO memory will be eliminated.

[0274] The center of the bottom-level VLO node can be used as the SLT origin. Alternatively, if greater precision is required, an offset relative to the node center can be specified using the projection correction calculated for the geometry calculations.

[0275] In the following, the SLT is positioned in the plenoptic octree by traversing the VLO (with or without offset correction) to locate the origin of the STL. The projection information of the STL relative to the projection plane attached to the center of the sphere is set for the root node of the VLO, and then the projection information is updated at each push down to the SLT origin location. Fig.37 , the center of the SLT (not shown) is at point 3702, which is the center of the VLO root node (only VLO node 3701 at level 1 is shown). To move the SLT, center 3702 moves along with the VLO node as it is pushed in. In the case shown, this is the VLO child node pointing in the +x and +y directions. It is therefore moved in the +x and +y directions to level 1 node center 3703 (for this top vertical element). Therefore, the original ray 3704 (representing either edge of the vertical element) becomes 3705 after the push. The slope of this new ray remains the same as the slope of the original ray 3704, but the intersection with the projection plane 3706 moves. The original intersection point 3708 (t(t x ,t y ))Move to 3709(t'(t' x ,t' y )). Therefore, the value of ty changes to t' y , while the x coordinate of the projection plane remains the same as t x same.

[0276] The step length in y is calculated by taking into account the step to the new origin and the slope of the ray. The edge of the first level VLO node is 1, such as e of 3701 x 3710. Although the magnitude of the edge is the same in all directions along the axis, they are maintained as separate values ​​because the directions will vary during traversal. The y value is e y 3711. When VLO push occurs, the new edge value e' x 3712 and e' y3713 is half of the original value. As shown in the diagram of this push operation:

[0277] e' x =e x / 2 and

[0278] e' y =e y / 2.

[0279] Since the y value of the origin is shifted by e' y 3713, plus the x value of the SLT origin moved 3712e' x , multiplied by the slope of the edge, the new intersection point 3709 moves in the y direction.

[0280] t' y =t y +e' y –slope*e' x

[0281] This calculation can be performed in many ways. For example, instead of performing a product operation each time, the product of the slope and the edge of the VLO domain can be stored in a shift register, and for each VLO push, a right shift operation can be used to divide it by two. This shows that the center of the SLT can be moved by performing a push operation on the VLO while maintaining the projection of the cube on the projection plane.

[0282] The next operation will move the projection plane while maintaining the geometric relationship with the SLT. The projection plane will usually be attached to the center of a different VLO node, which will usually be at a different level of the VLO. When the node to which the projection plane is attached is subdivided, the projection plane and its origin will move in the universe. This is done in Fig.38 . When a push occurs to the level 1 VLO node 3801, the projection plane 3802 is attached to the center of the VLO root node. The projection plane 3802 moves to a new location that becomes projection plane 3803. The projection plane origin moves from the center of the universe 3804 to the center of the child node (point 3805). The original cube edge ray intersection 3806 (t(t x ,t y )) moves to the new intersection point 3807 (t'(t') on the new projection plane 3803 x ,t' y )). As above, push to 3812e' x 3810e x (x edge of node 3801) is divided by two. y edge 3811e y Divide by two, and you get 3813e' y The calculation method is as follows:

[0283] e' x =e x / 2 and

[0284] e' y =e y / 2.

[0285] The y component of the intersection point relative to the new origin becomes:

[0286] t' y =t y –e' y +slope*e' x

[0287] e' y The subtraction is because the origin of the projection plane has been moved in the + direction from 3804 to 3805. Again, the edge multiplied by the slope is placed in the shift register, and this edge is divided by 2 on each pushed right shift. If the two paths in the same tree structure (SLT origin and projection plane) can be pushed and popped separately, the slope values ​​need to be calculated separately, which depends on the details of the actual projection method. For example, the octree structure where the SLT is located can be traversed to the bottom level before starting the VLO traversal, and some registers can be reused.

[0288] A "span" is the line segment in the projection plane between two rays that define the limits of a cube (in one dimension). Fig.39 3902 for a managed entity 3902. It is defined by three points, the origin 3903 of the SLT, the "top" edge 3904, and the "bottom" edge 3905. The edges are defined by where they intersect a projection plane 3906, which has its origin at point 3907. The intersection is point t 3909 for the top edge and point b 3910 for the bottom edge.

[0289] The vertical element emanating from the SLT origin is limited only between the bottom edge and the top edge. It is not limited on the other side of the origin. Fig.40 The first level VLO node 4001 shown contains a cell 4002 with its origin at 4003, which can detect this situation for the cell. The projection plane 4004 moves to the other side of the SLT origin to become 4005 where no cell exists. y The offset value becomes b' y 4006, and t y The offset value becomes 4007t' yAfter the move, the top offset value is lower than the bottom offset value, indicating an undefined cube. It is no longer projected onto the projection plane, and its use in intersection operations with the VLO node needs to be suspended until it returns to the other side of the origin while the geometric relationship is calculated and maintained.

[0290] By calculating the new top and bottom offsets, a cube push operation can be used to subdivide the cube into four sub-cubes. As discussed above, the cube subdivision process is Fig.41 Displayed in. Fig.42 A level 1 VLO node 4201 is shown hosting a cube defined by an origin 4203, a top point t 4204, and a bottom point b 4205. Depending on the child node into which the cube is to be pushed (typically based on geometric calculations performed during the push process), the new sub-cube can be an upper sub-cube or a lower sub-cube. An upper sub-cube is defined by the origin 4203, the top point 4204, and the bottom point 4206, the center between the highest point 4204 and the lowest point 4205. In the case shown, the lower sub-cube is the result of the push, defined by the origin 4203, the original bottom point b 4205, and the new top point t' 4206. The new top point t value (t') is calculated as follows:

[0291] t' y =(t y +b y ) / 2.

[0292] The new bottom edge is identical to the original edge and has the same slope. The top edge defined by t' has a new slope lope_t', which can be calculated as:

[0293] slope_t'=(slope_t+slope_b) / 2.

[0294] Although all cubes at a particular level have the same face area, they do not represent the same solid angle area because the origin is moved relative to the face area. This problem can be corrected by moving the edges of the face rectangles of each cube at a certain level. While this simplifies the lighting calculations, the geometry calculations become more complex. With the preferred approach, an SLT "template" is used. This is a static, pre-computed "shadow" SLT that is traversed simultaneously with the SLT. For light projection, it contains an accurate measurement of the solid area of ​​each cube for use in lighting transfer calculations.

[0295] A cube represents incident or outgoing illumination entering or leaving a point in space (the center of the SLT) (usually the space represented by that point). Although a plenoptic octree can be used for light transport in many ways, a preferred method is to first initialize the geometry variables with the origin of the SLT located at the center of the VLO. The geometric relationships are then maintained as the SLT moves to its position in the global domain. The VLO is then traversed from the root node in a front-to-back (FTB) order starting from the SLT origin so as to proceed in the direction from the cube origin. In this way, some VLO nodes containing matter are encountered, usually in the general order of distance from the cube origin, and these nodes are processed accordingly. Typically, this may need to be performed multiple times to account for multiple groups of cubes (top cubes) in different direction groups.

[0296] As the VLO is traversed in FTB order corresponding to cubes projected from the SLT origin, the first interacting (with light) VLO material node encountered is then checked to determine the next step required. For example, the transfer of illumination from an element to a VLO node can be determined by removing some or all of the illumination from the element and attaching it or a portion of it to the cube to which the VLO node containing the substance is attached. This is typically the incident SLT attached to the VLO node. The transfer can come from properties that may be generated from an image that samples the light field. Or it can come from an outgoing cube attached to an illumination source. The incident illumination can be used with a model of the light interaction properties of the surface to determine, for example, outgoing light to be attached to an existing or newly created cube.

[0297] like Fig.43 As shown, a cube-to-cube transfer is performed from the outgoing cube attached to VLO node 4301 to the incoming cube attached to VLO node 4302. When the projection of the outgoing cube 4303 (whose origin is at 4304) is of the appropriate size relative to VLO node 4302 and generally encompasses its center, the transfer begins. Otherwise, either the outgoing cube or the VLO node containing the incoming SLT (or both) is subdivided and the situation is rechecked for the resulting subtree.

[0298] If a transfer is to be made, then the antipodal cube is accessed or generated at some cube resolution along the origin-to-origin segment 4307 in the incoming cube tree 4305. If the VLO node is too large, it can be subdivided as needed to increase the relative size of the projection. If the incoming cube is too large, it is typically subdivided to reduce the size of the projection.

[0299] A specific traversal order is used to implement FTB ordering of VLO node accesses. Fig.444406). The traversal for VLO node 4401 is shown in . The cell has the origin at 4403 and the edges (top and bottom) intersecting the quarter circle (eighth sphere in 3D) region 4404 (between the bottom edge limit 4405 and the top edge limit 4406). Edge 4407 is a typical edge in this range. The order 0 to 3 4408 will generate the FTB order in VLO node 4401. Other orders are used for other ranges. In 3D, there are equivalent traversal orders for eight child nodes. In the case of VLO, the traversal will be applied recursively. The traversal order ordering is not unique, as multiple orderings can generate an FTB traversal of an area.

[0300] When subdividing cubes, in some algorithms it is necessary to keep track of cubes that contain light that is consumed (e.g. absorbed or reflected) by the encountered VLO nodes containing matter. As with octree image generation, a quadtree will be used to mark "used" cubes. This is done in Fig.45 4502 with origin at 4503 is projected onto VLO node 4501. A quadtree 4504 (edges only shown) is used to track partially or completely inactive or previously used bins in the bin tree.

[0301] Multiple processors can operate on different elements simultaneously. For example, 24 processors can each compute the projection of a different top element and its descendants. This can place a large bandwidth demand on the memory that holds the plenoptic octree (especially the VLO). The SLT center tree can usually be synthesized for each processor, and the top element and its descendants can be divided into separate memory segments, but the VLO memory will be accessed from multiple element processing units.

[0302] As mentioned above, a set of half-space generators can be used for each cell to reduce memory bandwidth requirements. Fig.46 As shown, in 2D, a half-space octree will be generated locally (within each processor) for the two edges (four planes in 3D) that define the sides of a cube. Edge 4602 is the top cube edge. The area below this edge is half-space 4603. Edge 4604 is the bottom edge that defines the upper half-space 4605. In 2D, the volume of a cube is the intersection of the two half-spaces 4606. In 3D, the volume of a cube is the intersection of the four half-spaces that occupy the volume.

[0303] The local cube-shaped octree is then used as a mask to be intersected with the VLO. If a node in the locally generated octree is empty, there is no need to access the VLO octree in memory. Fig.474701. This is illustrated by an upper level VLO node 4701 that contains multiple lower level nodes in its subtree. Node A 4703 is completely disjoint with vertices 4702, so no access is required. vertices 4702 occupy part of the space of node B 4704. Access to VLO memory is required, but any of its child nodes (such as 0, 2, and 3) are disjoint with vertices, so no memory access is required. Node C 4705 is completely surrounded by vertices, so this node and its descendant nodes are needed for processing. These nodes need to be accessed from VLO memory as needed. By interleaving VLO memory with eight fragments corresponding to eight level 1 octree nodes and in other ways, memory access issues can be reduced.

[0304] A "leading edge" is defined here as a surface at a distance from a region in the plenoptic octree such that anything at an equal or greater distance will not exhibit parallax at any point within the region. Thus, light coming from a particular direction does not change regardless of the position within the region of the plenoptic octree. For light coming from outside the leading edge, a single SLT for the entire plenoptic octree may be used. In operation, for a given point, the incident SLT is accumulated for that point to avoid projecting outward. When all such illumination (all illumination from within the leading edge) has been determined, for any element for which such illumination is not found, the elements from the leading edge SLT are used to determine its properties. For example, illumination outside the plenoptic octree but within the leading edge may be represented by, for example, an SLT on a face of the plenoptic octree (not a single SLT).

[0305] In many operations (such as calculations using surface properties such as BLIF), it may be important to rotate the SLT. Fig.48 4801 is an original VLO node containing element 4804. Node 4802 is a rotated VLO node, which contains a rotated version of 4804 generated from it, namely element 4805. The two SLTs share the same origin, namely the VLO center point 4803. The algorithm generates new rotated elements and sub-elements from the original elements and sub-elements. This operation can be performed on all original elements, or, for example, a full light mask or a simple "mask" as used here can be used to prevent the generation of some elements in the new SLT, usually because they are not needed for some reason (e.g., from surface points, directions entering opaque entities, unwanted directions, such as BLIFs of mirrors, where some directions contribute little or no to the outgoing light). Masks can also specify attribute values ​​of interest (e.g., ignore elements with radiation values ​​below a specified threshold). As shown, the faces (edges in 2D) of the new SLT surrounding cube become projection planes (lines in 2D), such as 4806. The spans in the new SLT are projections of the original SLT cubes.

[0306] Fig.49 Show point t(t x ,t y )4901, which is the intersection of the top edge of the new element and the rotated projection plane 4902. Similarly, point b (b x ,b y ) 4903 are the intersection points of the bottom edges. These points correspond to the endpoints of the edge / face spans in the cube of the new SLT. These points will start from the ends or corners of the SLT octree universe. These points will then be subdivided as needed. As shown in the figure, the center point 4904 will now become the new top point t', where:

[0307] t' x =(t x +b x ) / 2 and

[0308] t' y =(t y +b y ) / 2.

[0309] The distance in x between the top and bottom points is d x 4905, and divided by 2 for each push. The change in y is d y 4906 and divided by two in case of each push. The difference for each subdivision will vary with the slope of the edge and will also be divided by two in case of each push. The task will be to keep track of the cubes in the original SLT that are projected onto the new cubes after the subdivision. At the bottom level (highest resolution in directional space), for the nodes that are needed during processing, the value of the new cube will be calculated using the attribute values ​​in the original cube. This can be done by selecting the value from the cube with the largest projection or some weighted average or calculated value.

[0310] Fig.50 5005 and a bottom edge (not shown). The original cell is bounded by a top edge 5005 and a bottom edge (not shown). The original distance in y from point t to t is calculated at the beginning and then maintained as the subdivision continues. This is d in the figure t 5010. There is also an equivalent distance in y from point b to the original point b (not shown). The purpose of the calculation is to calculate the distance of the new point t or point b to the associated original edge. In the figure, this is a new top point t' 5004. An equivalent method can be used to handle the generation of the bottom distance of point b'.

[0311] The calculation involves two slopes, the slope of the edge of the original cube and the slope of the projected edge (a plane in 3D). In either case, for a step size in x (in this case d x / 25014), the distance change of y is the value determined by the slope and divided by two in each push case. These two values ​​can be maintained in a shift register. These values ​​are initialized at the beginning and then shifted as needed during push and pop operations.

[0312] As shown in the figure, in this case, we can first calculate the step size d x / 2 5014 or the value of "a" 5009 determines the movement along the edge of the projection to calculate the new offset distance d t '5004. This offset distance can then be used to determine the distance from the new top point t' to the original vertical intersection with the original top edge. This is the "e" 5011 value in the figure and is equal to a–d t The other part is the distance in y from the original intersection point on the top edge of the original cube to the new intersection point on the top edge. This distance is the slope of the edge in the figure multiplied by d x / 2 or "c" 5007. Therefore, the new distance d t '5006 is the sum e+c.

[0313] When this distance is extended to 3D, an additional value for the step size in the z direction needs to be calculated using the slope information in the new dimension, which is a direct extension of the 2D SLT rotation.

[0314] SLTs are hierarchical in that nodes at higher levels represent a larger volume of space than their descendants. The center of the SLT for a parent node is within this volume, but generally does not coincide with the center of any of its children. If an SLT is generated from, for example, an image, a quadtree can be generated for that image. This can then be projected onto the SLT at the center of the node at multiple levels of resolution.

[0315] In other cases, the upper level is derived from the lower level. SLT reduction is a process for generating values ​​for upper level entities based on the information contained in the lower level entities. This may generate averages, minimums, and maximums. In addition, measures of coverage can be calculated (e.g., the percentage of directional space in sub-elements that have values) and can be accumulated. In some implementations, one or more "eigenvectors" can be used. These eigenvectors are directions in which some attribute of the entity is balanced in some sense in space.

[0316] The SLT is usually assumed to be on or near a locally planar surface. If known, the local surface normal vector of the SLT can be represented as a whole and can be used to improve the value in the reduction process.

[0317] In some cases, especially where the lighting gradient is large, an improved reduction process is to project the lower level cube onto some plane (e.g., a known plane parallel to the surface that passes through the SLT space) or surface, filter the result on the surface (e.g., interpolate the center of the larger parent cube), and then project the new values ​​back onto the SLT. Machine learning (ML) can be used to analyze the lighting based on an earlier training set to improve the reduction process.

[0318] From a light field sample (e.g., an image), an outgoing SLT representing a point in space containing a volumetric region of matter that interacts with light can be gathered. If there is enough information to determine the illumination from multiple directions, it is possible to estimate (or "discover") the BLIF of the represented material. This process can be simplified if the incident SLT can be estimated. ML can be used for BLIF discovery. For example, an "image" (from multiple SLTs) containing entity illumination values ​​for the SLTs in a 2D array (two angles) can be stacked and used to identify the BLIF.

[0319] SLT interpolation is the process of determining the value of an unknown entity based on the value in some other entity set of the SLT. There are many ways to do this. If the BLIF is known, can be estimated, or can be discovered, it can be used to intelligently estimate the unknown entity values ​​of other entities.

[0320] Light sources can often be used to represent real or synthetic lighting. An ideal point light source can often be represented by a single SLT, perhaps with uniform lighting in all directions. Closed point or directional light sources can be represented by using a mask SLT to prevent lighting in occluded directions. Parallel light sources can be represented using a geometric extrusion of "lighting" to generate an octree. Extrusions can, for example, represent orthographic projections of non-uniform lighting (e.g., images).

[0321] Fig.51 A possible plenoptic octree projection processor is shown. This processor implements the projection of the SLT onto the VLO node in the plenoptic octree. Three push operations can occur: push center (push the center of the SLT into the child node), push VLO (push the VLO node into the child node), and push vertical element (push the parent vertical element into the child node). Pop operations are not explicitly included here. It is assumed that all registers are pushed into the stack at the beginning of each operation, and then they are simply popped out. Alternatively, only specific push operations (not values) are stored in the stack, and the pop is performed by reversing the push calculation.

[0322] The processor is used to project the "top" cube towards the face at x = 1. This unit performs the projection calculation in the xy plane. The repeating unit will calculate the calculation results in the yz plane.

[0323] To simplify the operation, all SLT center push operations will be performed first to place the SLT into its location (while maintaining the projection geometry). The two increment registers will be reinitialized, and then the VLO push operation will be performed. Then the SLT push operation will be performed. For example, these operations can be performed simultaneously by duplicating the increment registers.

[0324] The upper register 5101 maintains the y position of the upper plane of the projected element on the projection plane (in this case parallel to face 1). The lower register 5102 maintains the y position of the lower plane. The incremental shift register holds the slope values, namely DDelta_U 5103 for the upper plane and Delta_L 5104 for the lower plane. They have a "lev" (for level) bit to the right, enough to maintain accuracy when performing a pop operation after pushing to the lowest possible level. The incremental register is initialized with the slope of the associated plane in the xy plane. It contains the change in y for a step size in x of 1. For each push (SLT center or VLO), it shifts right by 1. Therefore, for the step size to the child node in the x direction, it becomes the change in y.

[0325] The edge shift register maintains the distance of the edge of the VLO node. They are VLO_Edge 5105 of the node edge during VLO traversal. SLT_Edge 5106 is used for VLO nodes during traversal to locate the vertical element in the full optical octree. The two are usually at different levels in VLO. The edge register also has a "lev" bit to the right to maintain accuracy. Other elements are selectors (5107 and 5108) plus five adders (5110, 5111, 5112, 5113 and 5114). The selectors and adders are controlled by signals A to D according to the following rules. The result is a VLO subdivision signal 5109.

[0326] The operation of the SLT projection unit can be implemented in many ways. For example, if the clock speed in a particular implementation is low enough, the processor instances can be replicated in a serial configuration to form a cascade of push operations that can perform multiple levels of movement in a single clock cycle.

[0327] Alternative designs in Fig.52 , allowing VLO and SLT push operations to be performed simultaneously. Two new increment registers, V_Delta_U 5214 (for VLO increment, upper) and V_Delta_L 5215 (for VLO increment), are added. Now, Fig.51 The increment registers in are only used for SLT push operations. They are now S_Delta_U 5203 and S_Delta_L 5204.

[0328] Fig.56 Show Fig.51 The top vertical element 5602 is located at the origin 5603 (0,0) of the global domain. The projection plane parallel to face 1 intersects the same point and has its origin at the same point. Note the quadrant subnumber 5610 and the sub-vertical element number 5611.

[0329] The registers are initialized as follows:

[0330] Upper = Lower = 0 (Both the upper edge and the lower edge intersect the projection plane at the origin.

[0331] Delta_U = 1 (upper edge slope = 1)

[0332] Delta_L = 0 (lower edge slope = 0)

[0333] VLO_edge=SLT_edge=1 (starting from the edge distance of the first-level node)

[0334] The operation of the projection unit is as follows:

[0335] SLT Center Push

[0336] Shift SLT_edge and Delta_U right by one position

[0337] For SLT sub 0 or 2: A is +; otherwise –

[0338] For SLT sub 0 or 1: C is –; otherwise +

[0339] B is 0

[0340] D is 1

[0341] E is no load (Delta_U and Delta_L registers are not changed)

[0342] VLO Node Push

[0343] Shift VLO_Edge and Delta_U right by one position

[0344] For VLO sub 0 or 2: A is –; otherwise +

[0345] For VLO sub 0 or 1: B is +; otherwise –

[0346] C is 0

[0347] D is 1

[0348] E is no load

[0349] SLT push-in

[0350] The new element is the upper part: D is 1; otherwise 0

[0351] E is the load

[0352] It may be desirable to locate the center of the SLT at some location other than the center of the plenoptic node. For example, this could be used to locate the representative point to a specific point on some infrastructure rather than the center of the local cubic volume represented by the node in space. Or it could be the specific location of a point light source.

[0353] This can be done by incorporating the SLT position into the initialization of the projection processor. This is simplified by the fact that the upper slope starts at 1, and the lower slope starts at 0. Therefore, the initial top projection plane intersection will be at y, which will be the y value of the cube center minus the x value. The bottom value will be the y value of the cube center.

[0354] The projection calculation then proceeds as before. The final push could be used to add the offset value of the offset to the node center of the SLT, but this is generally undesirable, at least not when the SLT center push and VLO push are happening at the same time. The stride value is used to select the next SVO child to visit, so the correct stride is required during the VLO traversal.

[0355] Three types of multiple pushed register values ​​are included in Fig.53 and Fig.54 The two offset values ​​in row 5 are set to 0 to simplify the calculation. The Excel formula used is in Fig.55 (with rows and columns reversed for readability). The offset value is in row 5 (x is F5, y is H5).

[0356] For clarity, the spreadsheet values ​​are in floating point format with geometric diagrams. In a real processor, the registers can use integer-only operations to scale integers. The spreadsheet columns are as follows:

[0357] A. Iteration (the number of sequential push operations.)

[0358] B. SLT push (SLT center child node is being pushed.)

[0359] C. VLO Push (VLO child node is being pushed.)

[0360] D. Liyuan push (Liyuan child node is being pushed.)

[0361] E.SLT to Lev (New SLT position level after pushing.)

[0362] F.VLO to Lev (New VLO node level after push.)

[0363] G. Lev (new level of Lev after push.)

[0364] H.SLT edge (size of the node in the octree used to locate the center of the SLT.

[0365] I. SLT step x (Current step in x pushed to child nodes when locating the SLT center point. Depends on the number of child sites.)

[0366] J. SLT step size y (The current step size in y pushed into the child node when locating the SLT center point. Depends on the number of child sites. The magnitude is the same as SLT step size x, except the sign depends on the number of child sites being pushed.)

[0367] K.SLTx (The x position of the current node center being used to position the SLT.)

[0368] L.SLTy (The y position of the current node center being used to position the SLT.)

[0369] M.VLO edge (length of the current node during push in VLO.)

[0370] N.VLO step x (Current step length in x for moving to a VLO child node. The sign depends on the number of child sites.)

[0371] O. VLO step size y (Current step size in y used to move to a VLO child node. In this implementation the same magnitude as VLO step size x but the sign depends on the number of child nodes.)

[0372] P.VLO x (The x position of the center of the VLO node.)

[0373] Q.VLO y (The y position of the center of the VLO node.)

[0374] R.TOPSlope (The slope of the top (upper) edge of the vertical element. Note: This is the actual slope, not the value of the current x step.)

[0375] S.BOT slope (The slope of the bottom (lower) edge of the vertical element. Note: This is the actual slope, not the value of the current x step.)

[0376] T.t_y (The top (upper) y value of the span's endpoint on the projection plane.)

[0377] U.b_y (The bottom (lower) y value of the span's endpoint on the projection plane.)

[0378] V.comp_t_y (The value of t_y computed independently of the comparison with t_y.)

[0379] W.comp_b_y (The value of b_y computed independently of the comparison with b_y.)

[0380] X. Note (Note about iteration.)

[0381] Initialization values ​​in the first column ("(Start)" in the first column). These values ​​are as listed above (and shown in Fig.56 ). Then 14 iterations of push operations involving the SLT center, VLO, and SLT vertical element follow. The first seven iterations are shown geometrically in Figure 56 to Figure 63 .

[0382] The first two iterations will be SLT pushes, followed by two VLO pushes and then two cube pushes. This is then followed by two VLO pushes (iterations 7 and 8), one SLT push (iteration 9), and a final VLO push.

[0383] The results of iteration #1 are shown in Fig.57 , is an SLT push from the VLO root node to child node 3. In the 1st level VLO node 5701, the cube 5702 is moved from the origin 5706 of the VLO to the center of the 2nd level child node 3, point 5703. The position of the new cube origin is at (0.5, 0.5). The projection plane 5707 remains in the same position and the slope remains unchanged (0 for the bottom edge 5704 and 1 for the top edge 5705).

[0384] Iteration #2 is shown in Fig.58 , is an SLT push to child node 2. This is similar to the last operation, except that it is to child node 2 at level 2, so the step size is half the previous distance, except in a different direction. In node 5801, 5802 moves to point 5803 at (0.25, 0.75). The slope of the bottom edge 5804 remains at 0 and the slope of the top edge 5805 remains at 1. The projection plane 5807 remains in the same position. The intersection of the top edge with the projection plane is lower than the bottom intersection (not shown in the diagram). The vertical element therefore actually crosses the projection plane at the same time and the projection is idle.

[0385] Iteration #3 is a VLO push from the root VLO node to child node 3 (level 1). Fig.59 5901. In VLO node 5901, element 5902 does not move. Projection plane 5907 moves from the center of the VLO root node to point 5906, the center of child node 3 at level 1. Note that the origin of the projection plane now moves to this point, the center of child node 3. The intersection of the edge of 5902 with the projection plane is recalculated due to the movement of the projection plane and the change of its origin.

[0386] Iteration #4 shows Fig.60 This is a VLO push to child node 1. Element 6002 does not move, but projection plane 6007 does and so the intersection of bottom edge 6004 and top edge 6005 must be recalculated. The projection plane moves in +x with its new origin at 6006. The slope of the edge does not change.

[0387] Fig.61 Iteration #5 is shown, a cube push to child node 1. The origin of cube 6102 does not change, but it is split into two sub-cubes, of which the lower sub-cube remains. The bottom edge 6104 remains the same, but the new top edge 6105 moves so that its intersection with the projection plane is halfway between the previous top and bottom intersections or 0.75 distance from the projection plane origin. The bottom distance remains at 0.5. The bottom slope remains at 0, while the top slope is reduced to an average slope of 0.5.

[0388] Iteration #6 shows Fig.62 . This is a cube push to child node 2. Again, cube 6202 is split into two sub-cubes, with the upper sub-cube remaining. Therefore, the top edge remains the same, with the same slope. The bottom edge is moved upward away from the origin of the projection plane and its slope is reset to the average of 0 and 0.5, or 0.25.

[0389] Iteration #7 shows Fig.63 The operation is a VLO push to child node 0. The projection plane moves in the –x direction and its origin moves in the –x and –y directions. Cube 6302 remains in the same position and the edges are unchanged, except that the intersection with the projection plane changes to accommodate the movement.

[0390] The Excel spreadsheet simulation was run again with the SLT center deviation set to a non-zero value (in row 5, the deviation value is 0.125 for x in cell F5 and 0.0625 for y in H5). The results are shown in Fig.64 and Fig.65 in a spreadsheet.

[0391] Volumetric measurement techniques are used to represent matter, including objects, within scenes, VLOs. They are also used to represent light in scenes (light fields) using SLT. As described above, information required for high-quality display and other purposes can be obtained from real-world scenes using a scene reconstruction engine (SRE). This can be combined with synthetically generated objects (generated by the SPU shape conversion module 3007) to form a composite scene. In some exemplary embodiments, the technology uses a hierarchical, multi-resolution and spatially sorted volume data structure for matter and light and their interactions in the SPU 3001. This allows for rapid identification of parts of the scene required for remote use based on positions, resolutions, visibility and other characteristics such as determined by the position and viewing direction of each user or statistically evaluated for several groups of users. In other cases, applications may request subsets of the database based on other considerations. By communicating only the required parts, channel bandwidth requirements are minimized. The use of volumetric models also facilitates advanced functions in the virtual world, such as collision detection (e.g., using the set operation module 3003) and physics-based simulations (e.g., mass properties easily calculated by the mass attribute module 3019).

[0392] Depending on the application, it may be desirable to combine matter and light field models generated by a single SRE or by multiple SREs into a composite scene model for remote visualization and interaction, for example, by one or more users (e.g., musicians or dancers placed on a remote stage). Since the luminescence and material properties are modeled, the lighting from one scene can be applied to displace the lighting in another scene, ensuring that the viewer experiences a consistently lit scene. The light field manipulation module 3023 may be used to calculate the lighting, while the image generation module 3009 generates the image.

[0393] A scene graph or other mechanism is used to represent the spatial relationships between individual scene elements. One or more SREs may generate real-world models that are stored in the plenoptic scene database 1A07. In addition, other real-world or synthetic spatial models represented in other formats (not plenoptic octrees) are stored in the database. This can be almost any representation that can be easily converted to a plenoptic octree representation by the shape conversion module 3007. This includes polygonal models, parametric models, solid models (e.g., CSG (Constructive Solid Geometry) or boundary representations), etc. The function of the SPU 3001 is to perform the conversion once or multiple times as the model changes or the requirement changes (e.g., the viewer moves closer to the object and a higher resolution conversion is required).

[0394] In addition to light fields and material properties, SRE can discover a variety of additional characteristics in the scene. This can be used, for example, to identify visual properties in the scene that can be used to incorporate previously acquired or synthesized models into the scene. For example, if a remote viewer visually moves too close to an object, a higher resolution is required than if the SRE acquired the model from the real world (e.g., a tree). An alternative model (e.g., parametric bark) can be smoothly "switched on" to generate higher resolution visual information for the user.

[0395] Often when the scene graph is modified by an application such as in response to a user request, the SPU modules in 3001 can be used to transform and manipulate the model to accomplish the application requirements. This and other SPU spatial operations can be used to implement advanced functions. This includes interference and collision detection as calculated by the set operation module 3003 plus the need for mass attributes such as mass, weight and center of mass as calculated by the SPU mass attribute module 3019. The modules in the plenoptic scene database are thus modified to reflect real-time scene changes as determined by the user and the application.

[0396] Both types of information matter (VLO) and light (SLT) can be accessed and transmitted for a selected spatial region (directional space in the case of SLT) and at a specified resolution level (angular resolution for SLT). Additionally, characteristic values ​​are typically stored in lower resolution nodes (upper nodes in the tree) in a tree structure representing attributes in a node subtree. This could be, for example, the average or minimum / maximum value of a color in a subtree of an octree node or some representative illumination measurement in a subtree of a tuple node.

[0397] Depending on the needs of the remote process (e.g., one or more users), only the necessary subset of the scene model needs to be transmitted. For viewing, this generally means sending high-resolution information of the part of the scene that is currently being viewed (or expected to be viewed) by module 3009 at a higher resolution than other areas. The resolution information transmitted for adjacent objects is higher than the resolution information of objects far away from the visual distance. Tracking or predicted movement will be used for parts of the scene that are expected to be needed. They will be transmitted with increased priority. The advanced image generation method of the octree model in 3009 can determine blocked areas when rendering images. This indicates that areas of the scene are not needed or can be represented as a lower fidelity level (to explain possible future prospects). This selective transmission capability is an inherent part of the codec. Only parts of the scene at different resolutions are accessed from the memory and transmitted. Control information is transmitted when necessary to maintain synchronization with the remote user.

[0398] When a large number of remote viewers operate simultaneously, their viewing parameters can be aggregated to set transmission priorities. An alternative would be to anticipate viewer preferences based on a probabilistic, possibly empirically based model. Since a version of the model of the entire scene is always available to each viewer at some, possibly limited, resolution level, unanticipated viewing will still result in a view of the scene, but at a lower level of image quality.

[0399] The information required for image generation is maintained in a local database, which is typically a subset of the source scene model database. The composition of the scene is controlled by a local scene graph, which can be a subset of the source's overall scene graph. Thus, especially for large "virtual worlds", the local scene graph can maintain object and light field information as well as other items that are visible or potentially visible to the user or that may be important to the application (e.g., the user's experience).

[0400] The information communicated between the scene server and the client consists of control information and a model in the form of a full-light octree and possibly parts of other models (e.g., shapes in other formats, BLIF functions). The full-light octree contains matter in the form of VLO and light fields in the form of SLT. Each octree is a hierarchical, multi-resolution, and spatially sorted volume tree structure. This allows them to be accessed by a specified area of ​​the modeling space and accessed at a variable resolution that can be specified by a spatial area (the direction space of the octree). The position, viewing direction, and required resolution of each user in the scene space (usually based on the distance from the viewpoint in the viewing direction) plus expected future changes can therefore be used to determine the subset of the scene that needs to be transmitted and the priority of each subset based on various considerations (e.g., how far and how fast the viewpoint can move, the bandwidth characteristics of the communication channel, the relative importance of the image quality of different sections of the scene).

[0401] Depending on the computing power that can be dedicated at the remote site, the functions associated with the server side of the communication channel can be implemented at the remote site. This allows, for example, only the material model (VLO) to be transmitted to the remote site with the light field information (SLT) reconstructed there, rather than having it also transmitted on the channel. Of course, the potential communication efficiency will depend on the details of the situation. Transmitting a simple model of a solid material to the remote site and then calculating the light field and displaying it locally can be more efficient than the transmission of the full light field information. This may be especially true for static objects in the scene. On the other hand, objects that change shape or have complex movement may benefit from transmitting only the light field SLT on demand.

[0402] In a plenoptic octree, the SLT is a 5D hierarchical representation of some location in space within a scene (or in some cases, beyond the scene). The five dimensions are the three position components (x, y, and z) that all voxels in the center conform to plus the two angles that define the voxel. A voxel tree can be positioned at the center of a VLO voxel or at some specified location within a voxel. A VLO node thus contains matter as defined by attributes and optionally also contains a voxel tree. In some embodiments, a voxel in space that contains a substantially opaque (transmissive) medium and is adjacent to a scene boundary (empty voxel) can be referred to as a "window-like" voxel.

[0403] It may be the case that a set of cubes may be similar at multiple points within a scene (at adjacent points on a surface with the same reflective properties). In such cases, a set of cubes with different centers may be represented independently of the center position. If they are the same for multiple center points, they may reference multiple center positions. If the differences are small enough, multiple sets may be represented by individual sets of deviations from one model cube or set of model cubes. Or they may be generated by applying coefficients to a pre-computed set of basis functions (e.g., a cube data set generated from a representative data set obtained using principal component analysis). In addition, other transformations may be used to change a single cube model into a specific set, such as by rotating around the center. Some types of SLTs, such as point light sources, can be repeated by giving additional positions to the model (without interpolation or extrapolation).

[0404] The scene codec operates in a data flow mode with a data source and a data sink. Typically, this takes the form of a request / response pair. The request can be directed to a local codec, where a response is generated (e.g., current status) or transmitted to a remote codec to act on it with a returned response, thereby providing the result of the requested action.

[0405] The request and response are communicated through the scene codec application program interface (API). The core functions of the base codec API 6601 are summarized in Fig.66 The codec is initialized via the operating parameters module 6603. This function can be used to specify or read the operating mode, thereby controlling the parameters and state of the codec. After a link with another scene codec has been established, this function can also be used to control and query the remote codec if specific permissions are given.

[0406] The codec API establish link module 6605, when triggered, attempts to establish a communication link with the specified remote scene codec. This usually initiates a "signal handshake" sequence to establish communication operation parameters (protocol, certificates, expected network bandwidth, etc.). If successful, both codecs report to the call routines that they are ready for communication operations.

[0407] The next step is to establish a scene session. This is set up by API Open Scene Module 6607. This involves establishing a link with the scene database on the remote end to access or update the remote scene database and also establishing a link with the scene database usually on the local end. For example, in order to establish a local sub-scene database from the remote scene database or to update the local scene database simultaneously with the remote scene database.

[0408] Once a connection to one or more scenes is established, two code API modules can be used to access and change the scene database. The remote scene access module 6609 is used to request information about remote scenes and their changes that do not involve sub-scene movement across the communication channel. The local scene access module 6611 is used to perform operations to be performed on the local scene database. The query processor module 6613 is used to perform scene database queries involving sub-scene movement. All actions taken by the codec are recorded through the session recording module 6615.

[0409] The main function of the query processor module 6613 is to transmit a subscene from a remote scene database or request a subscene to be incorporated into it (or removed from it). This may involve, for example, questions about the state of a full-light octree node, requests for calculating quality attributes, and the like. This typically involves subscene extraction and compression of the full-light octree, serialization, and transmission of subtrees and related information that may be encrypted. The subscene to be extracted is typically defined as a collection of geometric shapes, octrees, and other geometric entities specified in the form of some scene graphs, which can produce a region of space, i.e., a region of volume space or directional space or both. In addition, the resolution required in each region of the specified volume or directional space (e.g., reduced from the viewpoint in the case of rendering). The type of information will also be specified so that no additional information is transmitted. In some cases, subscene extraction can be used to perform some forms of spatial queries. For example, a request for subscene extraction of a region but only to the 0th level will return a single node, and if the node is found to be invalid, it indicates that there is no material in the region. This can also be extended to searching for specific features in the full-light scene.

[0410] The sub-functions of the query processor module 6613 are shown in Fig.67 This consists of a state and attribute query module 6703. It is used to obtain information about the plenoptic scene such as the ability to write to it or the attributes that exist in it or where new attributes can be defined, etc. The subscene mask comparison module accepts some form of subscene extraction request and builds a mask plan that completes the request. This is typically a collection of evolving masks that will incrementally send subscenes to the requesting system, as planned by the plan subscene masks module 6705.

[0411] The subscene mask generator 6707 builds a plenoptic octree mask that will be used to select nodes from the scene database for transmission back to the requesting system. It continuously builds the next mask for extraction. The subscene extractor module 6709 performs a traversal of the scene plenoptic octree to select nodes as determined by the mask. They are then serialized and further processed and then input into a packet stream transmitted to the requesting system. The subscene inserter module 6711 is used by the requesting system to modify the local subtree of the scene model using the transmitted plenoptic node request stream.

[0412] The codec may perform subscene extraction or subscene insertion or both. If only one operation is implemented, the modules and functions required for only the other operation may be eliminated. Thus, only the encoder unit will require the subscene extractor 6709, but not the subscene inserter 6711. Only the decoder unit will require the subscene inserter module 6711, but not the subscene extractor module 6709.

[0413] As discussed above, extracting sub-scenes from a plenoptic scene model enables efficient transmission of only portions of a scene database to a client, such as required for immediate or near-term visualization or for other uses. In some embodiments, a plenoptic octree is used for the scene database. The properties of such a data structure facilitate efficient extraction of sub-scenes.

[0414] Various types of information may be contained in the plenoptic octree, as separate VLOs or as attributes contained in or attached to octree or tuple nodes in the plenoptic octree, an auxiliary data structure, a separate database, or in some other manner. The initial subscene extraction request specifies the type of information the client will require. This can be done in a variety of ways that are specific to the application being served.

[0415] The following is an exemplary use where a client is requesting a sub-scene extraction for remote viewing by a display device such as a VR or AR headset. A large plenoptic octree is maintained on the server side. Only a subset is needed on the client to generate an image. The plenoptic octree mask will be used as an example here. Many other methods can be used to complete this example. The mask is in the form of a plenoptic octree, which is used to select nodes in the plenoptic octree using set operations. For example, a sub-portion of the large octree can be selected using a smaller octree mask and a crossover operation. The two octrees share the exact same universe and orientation. Both trees are traversed from the root node at the same time. Any nodes that do not exist as occupied nodes in the large octree are simply skipped and ignored in memory. They simply do not appear in the traversal. They effectively disappear. In this way, a subset of nodes can be selected by traversal and serialized for transmission. The subset is then recreated on the receiving side and applied to the plenoptic octree. This concept is easily extended to the ternary tree.

[0416] In the following, the mask concept is extended using an incremental mask. Therefore, the start mask can be increased or decreased or otherwise modified to select additional nodes to be transmitted to the receiving side. The mask can be modified in many ways for this purpose. The morphological operation of expansion corrosion can be applied using the SPU morphological operation module 3015. Geometry can be added or used to remove parts of the mask bus (mask buy), using the SPU shape conversion module 3007 and the SPU set operation module 3003 to convert these geometries. Usually, the new mask will be subtracted from the old mask to generate an incremental mask. This will be used to traverse the large scene model to locate the new node so that it can be serialized and transmitted to be added or otherwise processed at the receiving end. Depending on the situation, the opposite extraction can be performed, subtracting the new mask from the old mask to determine the node set to be removed. This can be serialized and directly transmitted to be removed on the receiving side (not involving subscene extraction). A similar approach may be used on the receiving side to remove nodes that are no longer needed for some reason (eg, the viewer moves and high resolution information is no longer needed in some areas), thereby notifying the server side of changes to the current mask.

[0417] The purpose of a plenoptic projection engine (PPE) is to efficiently project light from one location to another in a plenoptic scene model, resulting in a light transport. This can be from a light source represented, for example, by an exit point light field (PLF) to an incident PLF attached to an element. Or it can be an incident PLF resulting in the exit light being added to the exit PLF.

[0418] Plenoptic projection utilizes a hierarchical, multi-resolution tree structure that is spatially sorted to efficiently perform the projection process. Three such tree structures are used: (1) VLO or volumetric octree, which holds the elements (although this is considered a single octree, it can be multiple octrees joined together), SOO or trip-tree origin octree (this is the octree in the plenoptic octree that contains the origin of the trip-tree), and (3) SLT, some number of trip-trees in the plenoptic octree (the origin location is at SOO).

[0419] The plenoptic projection engine projects the cells in the SLT onto nodes in the VLO in a front-to-back order starting at the origin of each SLT. When a cell crosses an inter-element node, the size of the projection is compared to the size of the medium voxel. The analysis is based on a variety of factors, such as the spatial or angular resolution currently required, the relative size of the inter-element and the cell projection on it, the presence of higher resolution information at lower levels in the tree structure, and other information. When necessary, the inter-element or cell or both can be subdivided into regions represented by their children. The same analysis is then continued at a higher resolution.

[0420] When the subdivision process is complete, light transmission may occur. A cell in a cell tree may, for example, result in the formation or modification of one or more cells in a cell tree attached to an intermediate cell. In a typical application, the incident radiant cell information may be accumulated in an incident PLF attached to the intermediate cell. When the incident SLT is sufficiently filled, the BLIF of the intermediate cell may be applied, resulting in an exit PLF for the intermediate cell.

[0421] The projection process operates by maintaining a projection plane that projects cubes onto each VLO node visited in the traversal. The projection plane is perpendicular to the axis, according to the top cube to which the cube being projected belongs.

[0422] The process begins by starting the VLO and SOO tree structures at the center of the universe. Thus, the position in the SOO starts at the center of the universe. It will traverse downward to the position of the first SLT to be projected, as determined by any masks applied and any specified traversal order. The projection plane starts as a plane through the origin of the universe, perpendicular to the appropriate axis according to the first vertex. In operation, all three axes can be defined and tracked to account for all top vertex directions.

[0423] The main function of the plenoptic projection engine is to continuously maintain the projection of the oblique pyramid projection, which is the projection of the cube on the projection plane attached to the mesosphere, as the VLO is traversed. This is done by initializing the geometry at the beginning and then continuing to maintain the initialization as the three tree structures are traversed to typically project all cubes in all SLTs into the scene. This can create additional SLTs that can be further traversed during this process or when created later.

[0424] Therefore, the typical flow of the method is to initialize these tree structures, then traverse TOO to place the first SLT at its origin using a series of TOO push operations, thereby maintaining the projection geometry at each step. Next, traverse VLO to close the origin of the first SLT. Next, traverse VLO in a front-to-back order, visiting nodes in the general order of increasing distance from the SLT origin in the top vertical element direction. At each push of VLO, check the projection on the projection plane connected to the node to see if the vertical element continues to intersect with the VLO node. If not, all subtrees are ignored in the traversal.

[0425] If an inter-element VLO node is encountered, the analysis determines the next action to be taken, which, as outlined above, typically involves visiting subtrees of the VLO and / or SLT. When complete, these trees are backed up to a location where the next element can be projected in the same manner. When the last element of the first or subsequent SLT has been processed, the tree structure is presented (POPed) to the point where processing of the next SLT can begin. This process continues until all elements in all SLTs have been processed or optionally processed due to processing of the previous generation of elements.

[0426] The overall procedure is shown in Fig.68A of the projection engine flow chart. This is a sampling of many possible procedures. The process begins by initializing the projection mechanism in operation 68A02. As presented above, the VLO traversal begins at its root node. The target projection plane is therefore attached to the center of the universe (actually three can be tracked). The SSO is also initialized to its root node. The initial SLT point therefore starts at the origin of the universe and will be pushed to the origin of the SLT. The initial cell to be visited is the top cell 0.

[0427] In operation 68A04, the SOO tree structure is traversed to the origin of the next SLT in the plenoptic octree universe using a push operation. In the first use, this will be starting from the origin of the universe. At other times, it will be starting from where the last operation left off. For each operation, a projection of the current top vertex to a projection plane attached to the current VLO projection plane (first attached to the center of the universe) is maintained to reach the next SLT origin. If there are no additional SLTs in the SOO (usually detected by attempting to render from the root node), a decision operation 68A06 terminates the operation and returns control to the request routine.

[0428] Additionally, operation 68A08 traverses the cubes of the current SLT to the first non-empty node (non-empty voxel), which is the cube representing the spoke. Furthermore, the projection geometry between the cube and the projection plane is maintained. If no cube with spokes is retained, control is transferred back to operation 68A04 via decision operation 68A10 to find and traverse the next SLT.

[0429] If a projection cube is needed, operation 68A12 traverses the VLO tree to the node enclosing the current SLT origin. Basically, if the first VLO node with a projection plane is found where the cube projection intersects the VLO node, the VLO node intersects its projection plane. If no such VLO node is found, control is returned to operation 68A08 by decision operation 68A14 to proceed to the next cube to be processed. If the projection of the cube will not intersect the node, control is transferred back to operation 68A12 by decision operation 68A16 to proceed to the next VLO node to be investigated.

[0430] In addition, control is transferred to operation 68A18, where the projection of the current element on the current projection plane is analyzed. If the current rules for this operation are met, control is transferred to operation 68A22 through decision operation 68A20, where radiance transfer or resampling occurs. This generally means that a sufficient resolution level has been reached, which may be based on the change in radiance of the element, and the size of the projection is equivalent to the size of the VLO node in a certain sense. In some cases, some or all of the radiance is transferred to the appropriate element in the SLT attached to the node (created when needed). In other cases, the radiance can be adopted in some other way in the node.

[0431] If the analysis determines that a higher resolution level is required for a cube or VLO node, operation 68A24 determines whether the VLO node needs to be subdivided. If so, control is transferred to operation 68A26 for push. Information about the situation will generally be pushed onto the operation stack for subsequent visits to all sibling nodes. If not, control is transferred to decision 68A28, where the need to subdivide the current cube is processed. If so, control is transferred to operation 68A30, where a cube push is performed. Otherwise, the cube projection onto the current VLO node does not require additional processing and control is transferred back to operation 68A12 to traverse to the next VLO node for inspection and possible radiance transfer.

[0432] The comprehensive pipeline for subscene extraction from the plenoptic octree is shown in Fig.68B The process starts when a subscene request is received. An initial step 68B02 initializes the invalid subscene mask. This is typically a single-node plenoptic octree and related parameters. The request is then analyzed in step 68B04. For the image generation case, this may include the viewer's 3D position in the scene and the viewing direction. Additional information would be the field of view, screen resolution, and other viewing parameters and related parameters.

[0433] For viewing, this will then be used to define the initial view frustum for the first image. This can be represented as a geometric shape and converted to an octree using the SPU shape conversion module 3007. In other cases, a cube tree can be generated using each pixel, thereby generating cubes. The distance to the viewpoint is incorporated as part of the mask data structure or calculated in some other way (e.g., the distance calculated by the neighboring spare during subscene extraction). This will be used during subscene extraction to determine the resolution of the scene model (volume or direction space) to be selected for transmission.

[0434] The secondary analysis performed by module 68B04 builds a plan for a series of subscene masks. The overall strategy is to start with a mask that will generate an initial subscene model at the receiving end, which will produce a usable image for the viewer very quickly. This can, for example, reduce the resolution request for a smaller initial data set for transmission. The following steps usually add gradually higher resolution details. And the information requested from the viewing client can be used to anticipate future changes in viewing needs. This can include, for example, the direction and speed of the viewer's directional and rotational movement. This will be used to expand the mask to account for future expected needs. These extensions will be incorporated into the planning steps for future mask changes.

[0435] The plan then passes to operation 68B06, where the sub-scene mask as defined by the current step in the plan is intersected with the complete plenoptic scene model. The nodes resulting from the particular traversal of the plenoptic octree are then collected into a serial format for transmission to the requestor. Node compression, encryption, and other processing operations may be performed before passing to transmission.

[0436] The next flow operation is performed by decision 68B08 which accepts the next request. If it is a new sub-scene that cannot be accommodated and planned by modifying the current sub-scene mask, the current mask plan is abolished and the new sub-scene mask is initialized in operation 68B02. On the other hand, if the request is a sub-scene that has been anticipated by the current plan, as determined by decision operation 68B10, the next step of the plan is executed in operation 68B12. The sub-scene mask is modified and control is transferred back to operation 68B06 to implement the next sub-scene extraction. If the sub-scene mask end is encountered by decision 68B10, if there is a new sub-scene extraction request, as determined by decision operation 68B14, the next request is used to start the new sub-scene mask in operation 68B02. If no new request is pending, the sub-scene extraction operation 68B00 is placed in a waiting state until the new request is reached.

[0437] Fig.69A flow chart of process 6900 is shown, in one embodiment, extracting a sub-scene (model) from a scene database to generate images from multiple viewpoints in the scene. The first step operation 6901 is to establish a connection with a database containing a complete scene model from which the sub-scene is to be extracted. At operation 6903, the new sub-scene to be output is initialized so that the full light field lacks primitives. In other words, there is no material field or light field in the sub-scene at this time. At operation 6905, a set of "query cubes" is determined based on image generation parameters, including 6-DOF pose, intrinsic parameters and image size of the virtual camera at each viewpoint. Query cubes are cubes defined at some level in the SLT of the full scene, which will be used to spatially query (probe) the scene for plenoptic primitives located within the solid angle volume of the query cube. The query cube set is typically the union of the cube sets for each viewpoint. The cube set for each viewpoint typically covers the FOV (the camera's frustum) so that each image pixel is included in at least one query cube. The query cube can be adap...

Claims

1. A codec for processing plenoptic data, the codec comprising: a scene database configured to store plenoptic scene data; one or both of an encoder or a decoder, the encoder being configured to encode plenoptic scene data and the decoder being configured to decode plenoptic scene data; as well as a communication circuit configured to transmit and / or receive the plenoptic scene data; Wherein the plenoptic scene data comprises light field information and matter field information, and wherein the communication circuit is configured to transmit and / or receive each of the following: matter field and light field information, matter field information but no light field information, or light field information but no matter field information.

2. The codec according to claim 1, wherein: The encoder is configured to calculate a substantial future value of the plenoptic scene data based on one or more of: viewer movement, possible or actual viewer traversal paths, possible or actual viewer traversal path viewpoints, possible or actual traversal movement rates, machine learning algorithms, accumulation of scene experience log records, user's head orientation, sensor input from the user's device, probabilistic factors, or one or more pieces of auxiliary scene information.

3. The codec according to claim 1, wherein: The encoder is configured to encode plenoptic scene data representing a minimal scene representation sufficient to meet end-use requirements.

4. The codec according to claim 1, wherein: The decoder is configured to decode plenoptic scene data representing a minimal scene representation sufficient to meet end-use requirements.

5. The codec according to claim 1, wherein: The plenoptic scene data is associated with one of: a vehicle, a rental car, a building, or a physical facility.

6. The codec according to claim 1, wherein: The encoded plenoptic scene data comprises a windowed light field.

7. A method for encoding and / or decoding scene data, the method comprising: storing plenoptic scene data in a scene database; Do one or both of the following: Encode and transmit plenoptic scene data, or receiving plenoptic scene data and decoding the plenoptic scene data; and The plenoptic scene data includes light field information and matter field information, and the sent and / or received plenoptic scene data includes each of the following items: matter field and light field information, matter field information but no light field information, or light field information but no matter field information.

8. The method of claim 6, further comprising calculating a substantial future value of the plenoptic scene data based on one or more of the following: viewer movement, possible or actual viewer traversal paths, possible or actual viewer traversal path viewpoints, possible or actual traversal movement rates, machine learning algorithms, accumulation of scene experience log records, user's head orientation, sensor input from the user's device, probability factors, or one or more pieces of auxiliary scene information.

9. The method according to claim 6, wherein: One or more of the encoded plenoptic scene data or the decoded plenoptic scene data represents a minimum scene representation sufficient to meet end-use requirements.

10. The method according to claim 6, wherein: The plenoptic scene data is associated with one of the following: a vehicle, a rental car, a building, or a physical facility.

11. The method according to claim 6, wherein: The encoded plenoptic scene data comprises a windowed light field.

Images