Interchangeable representation of scene-based media

Abstract

The present application provides a method for streaming immersive media, which can be executed by a processor, the method comprising: ingesting content of a scene of the immersive media, the content comprising a scene graph having a first scene graph format; converting the scene graph from the first scene graph format to a second scene graph format according to a metadata framework, the metadata framework comprising a first metadata system for representing information shared between the first scene graph format and the second scene graph format; and streaming the content of the immersive media based on a second scene graph, wherein the second scene graph is converted from the first scene graph according to the metadata framework.

Description

Cross-reference to related applications

[0001] This application claims priority to U.S. Provisional Application No. 63 / 664,689, filed June 26, 2024; U.S. Provisional Application No. 63 / 670,114, filed July 11, 2024; and U.S. Application No. 19 / 247,276, filed June 24, 2025, the entire contents of which are hereby expressly incorporated by reference. Technical Field

[0002] The embodiments described in this invention generally relate to architectures, structures, and components for systems and networks that distribute media, including video, audio, geometric (3D) objects, haptic feedback, associated metadata, or other content, to client devices. Specific embodiments relate to systems, structures, and architectures for distributing media content to heterogeneous immersive and interactive client devices. Background Technology

[0003] Video coding and / or decoding techniques employing motion-compensated inter-frame prediction have existed for decades. Uncompressed digital video can consist of a series of images, each with a spatial size of, for example, 1920×1080 luma samples and associated chroma samples. This series of images can have a fixed or variable image rate (also known as the frame rate), such as 60 frames per second or 60 Hz. Uncompressed video has extremely high bitrate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (1920×1080 luma sample resolution at a frame rate of 60 Hz) requires nearly 1.5 Gbit / s of bandwidth. One hour of such video would require more than 600 GB of storage space.

[0004] "Immersive media" generally refers to media that stimulates any or all of the human sensory systems (visual, auditory, somatosensory, olfactory, and possibly gustatory) to create or enhance the user's sense of immersion in the media experience. That is, it extends beyond the scope of timed two-dimensional (2D) video and corresponding audio distributed through existing (e.g., "traditional") commercial networks; such timed media is also known as "traditional media".

[0005] Another definition of "immersive media" is media that attempts to create or mimic the physical world through digital simulations of dynamics and physical laws, thereby stimulating any or all of the human sensory systems to give the user a sense of being immersed in a scene depicting a real or virtual world.

[0006] Presentation devices supporting immersive media can refer to devices equipped with sufficient resources and capabilities to access, resolve, and present immersive media. Regarding media delivered over the network, such devices exhibit heterogeneity in the number and formats of media they can support. Similarly, such media is heterogeneous in the quantity and type of network resources required for large-scale distribution. "Large-scale" can refer to media distribution undertaken by service providers on a scale equivalent to that of traditional video and audio media distributed over the network, such as Netflix, Hulu, Comcast subscription services, and Spectrum subscription services.

[0007] In contrast, traditional presentation devices such as laptop monitors, televisions, and mobile phone displays are homogeneous in terms of capabilities because these devices currently consist of rectangular displays that use frame-based 2D rectangular video or still images as their primary visual media format. Some frame-based visual media formats commonly used in traditional presentation devices include High Efficiency Video Coding / H.265, Advanced Video Coding / H.264, and Universal Video Coding / H.266 for video media.

[0008] The term "frame-based" media refers to visual media that consists of at least one consecutive rectangular image frame. In contrast, "scene-based" media refers to media organized by "scenes," where each scene refers to an independent asset that collectively describes a scene (typically a visual scene). It should be noted that scene-based media can also include other non-visual media, such as tactile and olfactory media.

[0009] Regarding visual media depicting a forest, a contrasting example between frame-based and scene-based visual media is shown. In the frame-based representation, a camera device, such as that on a mobile phone, is used to photograph the forest. The user focuses the camera on the forest, and the frame-based media captured by the phone is the same as what the user sees through the viewfinder of the phone's camera; the frame-based media remains the same regardless of where the user moves the camera. The resulting frame-based representation of the forest is a series of 2D images, typically recorded by the camera at a standard rate of 30 frames per second or 60 frames per second. Each image is a set of pixels, where the information stored in each pixel is consistent across pixels.

[0010] In contrast, a scene-based representation of a forest consists of: individual assets describing each object in the forest, and a generally user-readable scene graph description that presents a wealth of metadata describing that asset or how it is rendered. For example, a scene-based representation might include individual objects called “trees,” where each tree is composed of a set of smaller assets called “trunks,” “branches,” and “leaves.” Each trunk can be further described individually by a mesh and texture, the mesh describing the trunk’s full 3D geometry and the texture applied to the trunk mesh to reflect the trunk’s color and gloss characteristics. Additionally, the trunk may include supplementary information describing the smoothness, roughness, or reflectivity of its surface. The corresponding user-readable scene graph description provides information about the placement of the trunk relative to the viewport of a virtual camera focused on the forest scene. Furthermore, the user-readable description may include information about how many branches to generate from a single branch asset called a “branchone” and where to place these branches in the scene. Similarly, the description may include how many leaves to generate and their positions relative to the branches and trunk. Furthermore, the transformation matrix provides information about how to scale or rotate the leaves so that they don't appear uniform. Overall, the various assets that make up the scene differ in the type and amount of information stored in each asset. Each asset is typically stored in its own file, but these assets are often used to create multiple instances of the objects they were designed to create, such as the branches and leaves of each tree.

[0011] Those skilled in the art will understand that the user-readable portion of a scene graph contains rich metadata that describes not only the logical relationship between assets and their positions within the scene, but also instructions related to how to render objects, such as utilizing various types of light sources, surface properties (to indicate whether an object's surface is metallic or matte), or other materials (porous or smooth textures). Other information typically stored in the user-readable portion of a scene graph includes relationships between certain assets; for example, relationships used to form groups of assets rendered or processed as a single entity, such as a tree trunk with branches and leaves. Other information that may be stored with scene-based media includes instructions on how to access the media based on its physical storage. For example, a media parser might need explicit information on how to locate or access the media, or in some cases, explicit information on how to decrypt the media.

[0012] Examples of scene graphs with user-readable components include glTF 2.0, where the node tree component is provided in JavaScript Object Notation (JSON), a user-readable representation for describing objects. Another example of a scene graph with user-readable components is the Immersive Technologies Media Format (ITMF) (latest version 2.0), where OCS files are generated using XML, another user-readable representation format. ITMF is currently defined by a set of specifications consisting of three parts: the ITMF scene graph specification, the ITMF container specification, and the ITMF data encoding specification.

[0013] The ITMF Scene Graph Specification defines the scene graph description format stored in the "OCS" file, along with its corresponding nodes, pins (e.g., node input and output parameters), and attributes (typically immutable node characteristics). The ITMF Container Specification specifies the binary representation, encryption mechanisms, and organizational design for packaging the scene graph OCS file and the corresponding assets of that scene. The ITMF Data Encoding Specification provides numeric identifiers, or code points, for organizing node types, pin types, attribute types, and other relevant information describing the scene structure.

[0014] Another difference between scene-based and frame-based media is that in frame-based media, the view created for the scene is exactly the same as the view captured by the user through the camera (i.e., the view captured when the media is created). When a client renders frame-based media, the rendered media view is the same as the view captured in that media, for example, the same as the view captured through the camera used to record video. Sometimes, a small number of views of the same scene (e.g., stereoscopic video) can be captured by a small number of corresponding cameras and delivered to a client capable of rendering each of the multiple views (e.g., VR headsets such as the Oculus Quest). However, rendering these multiple views may not allow end-users on the client display to experience an unlimited number of views, a situation that scene-based media may fall into.

[0015] However, for scene-based media, users can use multiple virtual cameras (e.g., thin-lens cameras and panoramic cameras) to view the scene, thus significantly increasing the number of views available. That is, the "virtual camera" can be described or supported by a renderer that allows the user to control which parts of the scene they can experience.

[0016] Distributing any media over a network can employ media distribution systems and architectures that reformat media from an input or network-ingested media format to a distribution media format suitable not only for ingestion by the target client device and its applications but also for streaming over the network. Therefore, the network can perform two processes on the ingested media: 1) converting the media from format A to format B suitable for ingestion by the target client, i.e., based on the client's ability to ingest a specific media format; and 2) preparing the media for streaming.

[0017] In a broad sense, media “streaming” refers to the fragmentation and / or packaging of media so that it can be delivered over a network in consecutive, smaller “chunks” logically organized and ordered according to the media’s temporal and / or spatial structure. Converting media from format A to format B, sometimes called “transcoding,” can be a process typically performed by a network or service provider before the media is distributed to clients. This transcoding can involve converting media from format A to format B based on prior knowledge that format B is, to some extent, the preferred or only format that the target client can ingest, or is more suitable for distribution through limited resources such as commercial networks. In many cases (but not all), both converting the media and preparing it for streaming are necessary before a client can receive and process the media from the network.

[0018] The one or two-step processing steps performed by the network on the ingested media—that is, the processing performed before distributing the media to the client—result in a media format known as a "distribution media format" or simply a "distribution format." Generally, for a given media data object, if the network can obtain information indicating that the client will need to convert and / or stream the media object on multiple occasions (otherwise these occasions would trigger multiple conversions and streaming of the media), then these steps should only be performed once if necessary. That is, data processing and transmission for media conversion and streaming are typically considered sources of latency, consuming potentially substantial network and / or computing resources. Therefore, a network design that cannot obtain information indicating when a client may have stored a particular media data object in its cache or locally on the client will perform worse than a network that can obtain such information.

[0019] For traditional presentation devices, the distribution format can be equivalent to or fully equivalent to the "presentation format" that the client presentation device ultimately uses to create the presentation. That is, a presentation media format is a media format whose attributes (resolution, frame rate, bit depth, color gamut, etc.) are closely matched to the capabilities of the client presentation device. Some examples of distribution and presentation formats include: a high-definition (HD) video signal (1920 pixels in columns × 1080 pixels in rows) distributed over a network to an Ultra-high-definition (UHD) client device with a resolution (3840 pixels in columns × 2160 pixels in rows). In this scenario, the UHD client applies a process called "super-resolution" to the HD distribution format to upscale the video signal from HD to UHD. Therefore, the final signal format presented by the client device is the "presentation format," which in this example is the UHD signal, while the HD signal includes the distribution format. In this example, the HD signal distribution format is very similar to the UHD signal presentation format because both signals use a linear video format, and the process of converting HD to UHD is a relatively straightforward and easy process performed on most traditional client devices.

[0020] Alternatively, the preferred presentation format of the target client device may differ significantly from the ingested format received by the network. Nevertheless, the client may possess sufficient computing, storage, and bandwidth resources to convert the media from the ingested format to the desired presentation format suitable for client display. In this scenario, the network can skip the step of reformatting the ingested media, such as skipping the step of "transcoding" the media from format A to format B, simply because the client has sufficient resources to perform all media conversions without requiring pre-conversion by the network. However, the network can still perform the steps of fragmenting and packaging the ingested media for streaming to the client.

[0021] Another alternative is when the ingested media received by the network differs significantly from the client's preferred presentation format, and the client lacks sufficient computing, storage, and / or bandwidth resources to convert the media to the preferred presentation format. In this scenario, the network can assist the client by performing partial or complete conversion from the ingested format to a format equivalent or nearly equivalent to the client's preferred presentation format. In some architectural designs, this assistance provided by the network on behalf of the client is often referred to as "split rendering" or "adaptation" of the media.

[0022] Over the past decade, numerous devices enabling immersive media have been introduced to the consumer market, including head-mounted displays, augmented reality glasses, handheld controllers, multi-view displays, haptic gloves, and gaming consoles. Similarly, holographic displays and other forms of volumetric 3D displays are expected to enter the consumer market within the next three to five years. Despite the availability or imminent arrival of these devices, a coherent end-to-end ecosystem for distributing immersive media through commercial networks has yet to materialize for various reasons.

[0023] One obstacle to achieving a coherent, end-to-end ecosystem for distributing immersive media over commercial networks lies in the diversity of client devices used as endpoints of distribution networks for immersive displays. Some client devices support specific immersive media formats, while others do not. Some client devices can create immersive experiences based on traditional raster formats, while others cannot. Unlike networks designed solely for traditional media distribution, networks that must support multiple display clients need to acquire a wealth of information, including specific details of each client's capabilities and the format of the media to be distributed, before employing an adaptation process to convert the media into a format suitable for each target display and corresponding application. At a minimum, the network needs to acquire information that directly describes the characteristics of each target display and the media itself to determine media interchangeability. That is, media information may be represented differently depending on how the media is organized according to various media formats; networks supporting heterogeneous clients and immersive media formats need to acquire information to identify when at least one media representation (according to the media format specification) substantially represents the same media information. Therefore, a major challenge in distributing heterogeneous media to heterogeneous client endpoints is achieving media "interchangeability."

[0024] In other words, during the development of the Independent Mapping Space (IMS), it has become clear that the architecture proposed in the CD vote of ISO / IEC 23090 Part 28 needs improvement to better achieve the goal of creating interchangeable representations for various scene-based formats. This proposal suggests updating the IMS architecture in Part 28 and recommends incorporating this update into Part 28, i.e., as an output of MPEG 147, prior to the planned DIS vote.

[0025] The above description is not an admission of prior art, but is provided merely as background information so that those skilled in the art can recognize the technical problems and difficulties associated with the embodiments described in this application and improve upon them. Summary of the Invention

[0026] A method for streaming immersive media is provided, the method being executable by a processor, the method comprising: ingesting content of a scene of the immersive media, the content including a scene graph belonging to the scene and having a first scene graph format; determining whether to convert the scene graph from the first scene graph format to a second scene graph format; converting the scene graph from the first scene graph format to the second scene graph format according to a metadata framework including at least a first metadata system and a second metadata system, the first metadata system including a plurality of first metadata for representing first information shared between the first scene graph format and the second scene graph format, and the second metadata system including a plurality of second metadata for representing second information shared between the first scene graph format and the second scene graph format; and streaming the content of the immersive media based on the second scene graph, wherein the second scene graph is converted from the first scene graph according to the metadata framework.

[0027] An apparatus for streaming immersive media is provided, the apparatus comprising: at least one memory configured to store program code; and at least one processor configured to read the program code and operate according to the instructions of the program code to: ingest content of a scene of the immersive media, the content including a scene graph belonging to the scene and having a first scene graph format; determine whether to convert the scene graph from the first scene graph format to a second scene graph format; convert the scene graph from the first scene graph format to the second scene graph format according to a metadata framework including at least a first metadata system and a second metadata system, the first metadata system including a plurality of first metadata for representing first information shared between the first scene graph format and the second scene graph format, and the second metadata system including a plurality of second metadata for representing second information shared between the first scene graph format and the second scene graph format; and stream the content of the immersive media based on the second scene graph, wherein the second scene graph is converted from the first scene graph according to the metadata framework.

[0028] A non-transitory computer-readable storage medium is provided for storing instructions, the instructions comprising: at least one instruction, which, when executed by at least one processor of a device for streaming immersive media, causes at least one processor to: ingest content of a scene of the immersive media, the content including a scene graph belonging to the scene and having a first scene graph format; determine whether to convert the scene graph from the first scene graph format to a second scene graph format; convert the scene graph from the first scene graph format to the second scene graph format according to a metadata framework including at least a first metadata system and a second metadata system, the first metadata system including a plurality of first metadata for representing first information shared between the first scene graph format and the second scene graph format, and the second metadata system including a plurality of second metadata for representing second information shared between the first scene graph format and the second scene graph format; and stream the content of the immersive media based on the second scene graph, wherein the second scene graph is converted from the first scene graph according to the metadata framework.

[0029] The first system represents the logical organization components of the scene, and each of the multiple first metadata in the first system is identified by a corresponding first tag, each first tag including the first prefix "ims.logical"; the second system represents the physical organization components of the scene, and each of the multiple second metadata in the second system is identified by a corresponding second tag, each second tag including the second prefix "ims.physical".

[0030] The metadata framework includes a third metadata system, which includes multiple third metadata for representing third information shared between the first scene graph format and the second scene graph format. The third system represents the processing direction component of the scene, and each of the multiple third metadata in the third system is identified by a corresponding third tag, each third tag including the third prefix "ims.process".

[0031] The metadata framework also includes metadata for representing geometric information of immersive media.

[0032] The metadata framework also includes metadata for representing the physical storage of immersive media, and the metadata representing the physical storage of immersive media describes any of the following: global attributes of the storage representation of immersive media, local attributes of local aspects of the storage representation of immersive media, a physical stream indicating the existence of a byte stream corresponding to at least one physical element of the scene, a stream block indicating the existence of a byte block corresponding to the physical stream, a stream index indicating a random access point of the physical stream, a directory indicating a list of attributes and other metadata of at least the physical stream, a signature indicating the encrypted information of the scene, and a scene tail indicating information up to the end of the scene.

[0033] The metadata framework also includes processing parameters for representing immersive media, as well as metadata used by either the renderer or the rendering engine. Attached Figure Description

[0034] Other features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, wherein: Figure 1 This is a schematic diagram illustrating the flow of media distributed to a client via a network according to at least one embodiment of this application; Figure 2 This is a schematic diagram of media flowing through a network according to at least one embodiment of the present application, wherein a decision process is employed to determine whether the network should convert the media before distributing it to a client; Figure 3 This is a schematic diagram of an embodiment of a data model for representing and streaming timed immersive media according to at least one embodiment of this application, wherein the timed immersive media includes a list of assets that are reused in a set of N scenes; Figure 4 This is a schematic diagram of an embodiment of a data model for representing and streaming non-timing immersive media according to at least one embodiment of this application, wherein the non-timing immersive media includes a list of assets that are reused in a set of 5 scenes; Figure 5 This is a schematic diagram of the process of capturing a natural scene and converting it into a representation according to at least one embodiment of this application, which can be used as an ingestion format for a network serving heterogeneous client endpoints; Figure 6 This is a schematic diagram illustrating the process of creating a representation of a synthetic scene using 3D modeling tools and formats, which can be used as an ingestion format for networks serving heterogeneous client endpoints. In this example, according to at least one embodiment of this application, IMS metadata may optionally be used to annotate the scene graph; Figure 7 This is a system diagram of a computer system according to at least one embodiment of the present application; Figure 8 This is a schematic diagram of a network serving multiple heterogeneous client endpoints according to at least one embodiment of this application; Figure 9 This is a schematic diagram of a network providing adaptation information for a specific media represented in a media ingestion format. For example, providing the adaptation information occurs before the network adapts the media for use by a specific immersive media client endpoint. In this schematic diagram, according to at least one embodiment of this application, the ingested scene diagram may optionally be labeled; Figure 10This is a system diagram of a media adaptation process, which consists of a media rendering converter that converts source media from its ingested format to a specific format suitable for a particular client endpoint. In this diagram, scene diagrams may optionally be labeled according to at least one embodiment of this application; Figure 11 This is a schematic diagram of a network according to at least one embodiment of the present application formatting adapted source media into a data model suitable for representation and streaming; Figure 12 It is according to at least one embodiment of this application Figure 12 A system diagram of the media streaming process where data models are fragmented into network protocol data packets and their payloads are transmitted. Figure 13 This is a sequence diagram of a network according to at least one embodiment of the present application adapting a specific immersive media in an ingested format to a streaming distribution format suitable for a specific immersive media client endpoint; Figure 14 An example of an architecture for the physical organization of scene-based media according to at least one embodiment of this application is described, the architecture having references to each physical object; Figure 15 An example of a labeled scene diagram according to at least one embodiment of the present application is depicted, wherein such labeling is placed in a user-readable portion of the scene diagram architecture; Figure 16 An example of an annotated scene graph according to at least one embodiment of the present application is depicted, wherein such annotation is placed in the binary (non-user readable) portion of the scene graph architecture; Figure 17 An example of a scene graph architecture that does not distinguish between geometry, physical structure and processing instructions according to at least one embodiment of this application is described; Figure 18 An example architecture according to at least one embodiment of this application is described, which is “extracted” into three main components, including geometric information, physical storage information, and processing information; Figure 19 A hierarchical organization of scene graph information applicable to creating serialization results according to at least one embodiment of this application is described; Figure 20 The use according to at least one embodiment of this application is described. Figure 19 The organization within the framework distinguishes geometric information, physical storage information, and real-time processed information; Figure 21 An embodiment of metadata for physical storage information of a packaging architecture, according to at least one embodiment of this application, is described.

[0035] Figure 22Examples of canonical representations of IMS geometric objects within a scenario according to at least one embodiment of this application are depicted.

[0036] Figure 23 As an application example of the disclosed subject matter, the arrangement of a video encoder and decoder in a streaming media environment according to at least one embodiment of this application is shown. Detailed Implementation

[0037] The features discussed below can be used individually or in any combination in any order. Furthermore, embodiments can be implemented using processing circuitry (e.g., at least one processor or at least one integrated circuit). In one example, at least one processor executes a program stored in a non-transitory computer-readable storage medium.

[0038] The technology provided in this application describes an embodiment of metadata for creating standardized, interchangeable representations of scene-based media, as described in Part 28 of ISO / IEC 23090, to facilitate the interchangeability of 3D scene-based media. This embodiment further defines an architecture for the metadata, wherein the metadata is organized into three categories: 1) a description of the geometry in the scene, 2) a description of the physical characteristics of binary objects within the scene (to facilitate read / write access to the media), and 3) a description of instructions for processing the media into the desired experience (e.g., for a renderer or rendering engine). This architecture and corresponding organization facilitate the interchangeability of scene-based media by providing additional context about how the media should be transformed and / or labeled during interchange. The disclosed technology further describes this metadata for describing the characteristics of the physical organization.

[0039] Figure 1 A network media streaming process 100 is illustrated, which includes a general sequence of steps that can be performed by a network cloud or edge device according to at least one embodiment of this application.

[0040] Figure 2 A media conversion decision process 200 is described, which illustrates a network logic flow of ingested media processed within a network by a manual or automated process according to at least one embodiment of the present application.

[0041] Figure 2 An important aspect of the logic is the decision-making process, which can be performed manually or automatically. This decision-making step must determine whether the media can be streamed in its original ingestion format A, or whether the media must be converted to a different format B so that it can be presented to the client.

[0042] This decision-making process may require information describing aspects or characteristics of the ingested media to help the process make an optimal choice, namely, whether the ingested media needs to be converted before being streamed to the client, or whether the media should be streamed directly to the client in its original ingestion format A.

[0043] Given each of the above scenarios, i.e., where converting media from format A to another format can be done entirely by the network, entirely by the client, or jointly by the network and the client (e.g., for split rendering), it is clearly necessary to have a vocabulary of attributes describing the media format so that both the client and the network have complete information to characterize the media and the work that must be done.

[0044] The media format vocabulary described above can be utilized by the conversion processor. This vocabulary can be further organized into different categories that describe: logical relationships between objects in scene-based media, such as the relationship between branches and trunks; physical storage characteristics of scene-based media, such as how to access or read data in a binary buffer; and instructions on how to render the media, such as how to animate objects in the scene, filters to be applied to objects, or alternative output variables (used to save and store intermediate values ​​during scene serialization).

[0045] Furthermore, a vocabulary describing client capabilities may also be needed, such as those related to available computing resources, available storage resources, and bandwidth access. Additionally, a mechanism is needed to characterize the level of computational, storage, or bandwidth complexity of the ingested format so that the network and client can jointly or independently determine whether or when the network can employ a separate rendering step to distribute media to the client. Moreover, if the client has already completed the conversion and / or streaming of specific media objects required for or to be rendered as part of the work done on the previous scene for that rendering, the network can completely skip the conversion and / or streaming steps of the ingested media, assuming the client still has access to or can obtain the media previously streamed to that client. Finally, if it is determined that the conversion from format A to another format is a necessary step that needs to be performed by or on behalf of the client, a prioritization scheme for ordering the conversion processes of various assets within the scene may benefit an intelligent and efficient network architecture.

[0046] An example of such a descriptive vocabulary used to characterize media is the IMS nomenclature, designed to facilitate the conversion from one scene graph format to another, which may be entirely different. IMS is currently being developed as Part 28 of the ISO / IEC 23090 series of standards; this series is commonly known as "MPEG-I". Depending on the scope of Part 28, IMS consists of metadata and other information describing common aspects of scene-based media formats. For example, scene-based media often provides mechanisms for describing the geometry of a visual scene. One aspect of IMS in ISO / IEC 23090 Part 28 is providing standards-based metadata that can be used to annotate user-readable portions of the scene graph, such annotations guiding conversions from one format to another, i.e., from one description of scene geometry to another. Such annotations can also be attached to the scene graph as separate binary components. The same guided conversion applies to cameras; that is, many scene graph formats provide a means of describing the characteristics of a virtual camera that can be used as part of the rendering process to create the viewport of the scene. IMS in Part 28 also aims to provide metadata for describing common camera types. The purpose of IMS is to provide a naming convention that can be used to describe common aspects of various scene graph formats, thereby enabling conversions from one format to another under the guidance of IMS. This conversion facilitates asset exchange between multiple clients.

[0047] It should be noted that, as of the time of this disclosure, the first version of Part 28 is still under development by ISO / IEC JTC1SC29 / WG7 (MPEG Working Group 7). The latest specification released by WG7 is ISO / IEC JTC1 / SC29 WG7N00870, which was released by WG7 on June 17, 2024. Document N00870 does not provide a complete specification for IMS, particularly in terms of achieving the following objectives: organizing metadata based on descriptors of logical relationships between objects in scene-based media, descriptors of the physical organization of media, and descriptors of media rendering methods.

[0048] Definitions are provided below.

[0049] Scene graph: A general data structure commonly used in vector-based graphics editing applications and modern computer games, which arranges the logical representation of a graphical scene and its usual (but not always) spatial representation; a collection of nodes and vertices in a graph structure.

[0050] Scene: In the context of computer graphics, a scene is a collection of objects (e.g., 3D assets), object attributes, and other metadata, including visual, acoustic, and physically based characteristics describing a particular environment, interactions between objects within that environment, and the spatial or temporal boundaries of that particular environment.

[0051] Node: The basic element of a scene graph, including information related to the logical, spatial, or temporal representation of visual, auditory, tactile, olfactory, gustatory, or related processed information; each node should have at most one output edge, zero or more input edges, and at least one edge (input edge or output edge) connected to that output edge / input edge.

[0052] Base layer: The nominal representation of the asset, which is typically configured to minimize the computational resources or time required to render the asset, or to minimize the time required to transmit the asset over the network.

[0053] Enhancement layer: A set of information that, when applied to the base layer representation of an asset, enhances the base layer to include features or capabilities not supported by the base layer.

[0054] Attributes: Metadata associated with a node, used to describe a specific characteristic or feature of that node in canonical or more complex form (e.g., based on another node).

[0055] Binding LUT: A logical structure that associates metadata from IMS in ISO / IEC 23090 Part 28 with metadata or other mechanisms used to describe the features or functions of a particular scene graph format (e.g., ITMF, glTF, Universal Scene Description, USD).

[0056] Container: A serialization format used to store and exchange information to represent scenes that are entirely natural, entirely synthetic, or a mixture of synthetic and natural, including scene graphs and all media assets required to render the scene.

[0057] Serialization: A process that converts the state of a data structure or object into a format that can be stored (e.g., stored in a file or memory buffer) or transmitted (e.g., transmitted over a network link) and reconstructed later (potentially in a different computing environment). When the resulting bit sequence is reread according to the serialized format, that bit sequence can be used to create a clone that has the exact same semantics as the original object.

[0058] A renderer is a (typically software-based) application or process that, based on a selective fusion of related disciplines such as acoustic physics, optical physics, visual perception, auditory perception, mathematics, and software development, outputs, typically visual and / or audio signals, suitable for rendering on a target device or conforming to the desired characteristics specified by the properties of the rendering target node in the scene graph, given an input scene graph and asset containers. For visual-based media assets, a renderer may output visual signals suitable for the target display or output visual signals for storage as intermediate assets (e.g., repackaging into another container, i.e., for a series of rendering processes in the graphics pipeline); for audio-based media assets, a renderer may output audio signals for rendering in multi-channel speakers and / or binaural headphones or output audio signals for repackaging into another (output) container. Popular examples of renderers include the real-time rendering features of game engines Unity and Unreal Engine.

[0059] Evaluation: Produces results (e.g., similar to evaluating the document object model of a webpage) that prompt the output to be transformed from an abstract form into a concrete result.

[0060] Scripting language: An interpreted programming language that can be executed by the renderer at runtime to handle dynamic inputs and variable state changes to scene graph nodes. These changes affect rendering and the evaluation of spatiotemporal object topology (including physical forces, constraints, inverse kinematics, deformation, and collisions) and energy propagation and transmission (light, sound).

[0061] Shader: A computer program originally used for coloring (generating appropriate levels of lightness and darkness and color within an image), but now performs various specialized functions in all areas of computer graphics effects, or performs video post-processing unrelated to coloring, or even functions completely unrelated to graphics.

[0062] Path tracing: A computer graphics method used to render 3D scenes so that the lighting in the scene is consistent with reality.

[0063] Timing media: Media arranged in chronological order; for example, media with start and end times according to a specific clock.

[0064] Non-timed media: Media organized according to spatial, logical, or temporal relationships; for example, interactive experiences achieved based on actions taken by one or more users.

[0065] Neural network model: A set of parameters and tensors (e.g., matrices) that define the weights (i.e., numerical values) used when performing well-defined mathematical operations on a visual signal to obtain an improved visual output that may include interpolation of a new view of visual signals not explicitly provided in the original signal.

[0066] OCS: The user-readable portion of the ITMF scene graph, which uses a unique identifier represented as "id=nnn", where "nnn" is an integer value.

[0067] IMS: IMS metadata standardized in ISO / IEC 23090 Part 28.

[0068] Pins: Input and output parameters of scene graph nodes.

[0069] Attribute: The property of a given node that cannot be changed by other nodes.

[0070] Over the past decade, numerous devices enabling immersive media have been introduced to the consumer market, including head-mounted displays, augmented reality glasses, handheld controllers, multi-view displays, haptic gloves, and gaming consoles. Similarly, holographic displays and other forms of volumetric 3D displays are expected to enter the consumer market within the next three to five years. Despite the availability or imminent arrival of these devices, a coherent end-to-end ecosystem for distributing immersive media through commercial networks has yet to materialize for various reasons.

[0071] One obstacle to achieving a coherent, end-to-end ecosystem for distributing immersive media over commercial networks lies in the diversity of client devices used as endpoints of distribution networks for immersive displays. Some client devices support specific immersive media formats, while others do not. Some client devices can create immersive experiences based on traditional raster formats, while others cannot. Unlike networks designed solely for traditional media distribution, networks that must support multiple display clients need to acquire a wealth of information, including specific details of each client's capabilities and the format of the media to be distributed, before employing an adaptation process to convert the media into a format suitable for each target display and corresponding application. At a minimum, the network needs to acquire information that directly describes the characteristics of each target display and the media itself to determine media interchangeability. That is, media information may be represented differently depending on how the media is organized according to various media formats; networks supporting heterogeneous clients and immersive media formats need to acquire information to identify when at least one media representation (according to the media format specification) substantially represents the same media information. Therefore, a major challenge in distributing heterogeneous media to heterogeneous client endpoints is achieving media "interchangeability."

[0072] Media interchange can be viewed as the preservation of media properties after the media has been converted (or adapted as described above in the conversion from format A to format B). That is, information represented in format A is either not lost or is represented very closely using format B.

[0073] Immersive media can be organized into “scenes” described by scene graphs, also known as scene descriptions. To date, several mainstream scene-based media formats exist, including FBX, USD, Alembic, and glTF.

[0074] Such scenes refer to scene-based media as described above. A scene graph encompasses the visual, audio, and other forms of immersive assets that constitute a specific environment, which is part of the presented content—for example, the characters and events occurring in a specific location within a building that is part of the presented content (e.g., a film). A list of all scenes that constitute a single piece of presented content can be compiled into a scene inventory.

[0075] The disclosed subject matter addresses the need for an intermediate media representation that facilitates the conversion of scene-based media from one scene format to another. This representation, through its organization, provides context for the conversion process by identifying features common to various scene-based media. For example, such features may include: aspects explicitly identifying aspects defining the scene geometry in the scene-based media; aspects explicitly identifying aspects describing how to read and / or write binary data corresponding to the scene in the scene-based media; and aspects explicitly identifying aspects describing how the media should be rendered, animated, or otherwise processed by a rendering engine. Organizing interchangeable scene representations into these categories provides additional context (to media converters or other network components) compared to representations that do not distinguish between the aforementioned aspects of scene-based media. This application further discloses aspects related to the physical organization of this architecture.

[0076] It should also be noted that the terms "media object" and "media asset" are used interchangeably, both referring to a specific instance of media data in a particular format.

[0077] Figure 1 This is a schematic diagram of the media flow process 100, where media is distributed to clients via a network. Figure 1In this process, the processing of ingested media format A is performed by "cloud" or edge process 104. It should be noted that the same processing can also be performed beforehand manually or by the client. Ingested media 101 is obtained from a content provider. Process 102 performs any necessary transformations or adjustments on the ingested media to create a potential alternative representation of the media, namely distribution format B. Media formats A and B may be representations that follow the same syntax as a specific media format specification, or they may not be that representation, but format B is likely to be adjusted by process 103 to a scheme that facilitates media distribution via network protocols such as TCP or UDP. This "streamable" media is depicted in 105 as media streamed to media storage 106. Client 110 accesses the media from storage 106 via acquisition mechanism 107 (e.g., ISO / IEC 23009 HTTP-based dynamic adaptive streaming). Client 110 has access to some rendering capabilities as depicted in 108. Such rendering capabilities 108 can be basic or complex, depending on the type of target client 110. The rendering process 108 creates a rendering media 109, which may or may not be represented according to a third format specification (e.g., format C).

[0078] Figure 2 This is a schematic diagram of media flow through a network, employing a media conversion decision process 200 to determine whether the network should convert media before distributing it to clients. Figure 2 In this process, ingested media 201, represented in format A, is provided to the network by the content provider. Process 202 acquires attributes describing the processing capabilities of the target client (not depicted). A decision process 203 determines whether the network or the client should perform any format conversion on any media assets contained in the ingested media 201 before streaming the media to the client, such as converting a specific media object from format A to format B. If the network should perform any conversion on any media assets, the network uses process 204 to convert the media object from format A to format B. Converted media 205 is the output of process 204. The converted media is incorporated into preparation process 206 to prepare the media for streaming to the client. Process 207 streams the media to the client or media storage (e.g., ...). Figure 1 (Media storage area 106 in the middle).

[0079] Figure 3 A timing media representation 300 is depicted as an exemplary representation of a streamable format for heterogeneous immersive media used for timing. Figure 4 A non-timing media representation 400 is depicted as an exemplary representation of a streamable format for non-timing heterogeneous immersive media. Both figures illustrate scenarios. Figure 3 Scenario 301 pertains to the use of timed media. Figure 4 Scenario 401 pertains to non-timing media. For both cases, the scenario can be represented by various scenario representations or descriptions.

[0080] For example, in some immersive media designs, scenes can be represented by scene graphs, as multi-plane images (MPI), or as multi-spherical images (MSI). MPI and MSI are examples of techniques that help create display-independent scene representations for natural content—real-world images captured simultaneously by at least one camera. On the other hand, scene graph techniques can be used to represent both natural and computer-generated images in the form of synthetic representations; however, creating such representations is particularly computationally expensive when the content is a natural scene captured by at least one camera. That is, creating scene graph representations of naturally captured content is both time-consuming and computationally resource-intensive, requiring complex analysis of the natural images using photogrammetry and / or deep learning techniques to create a synthetic representation. This synthetic representation can then be used to interpolate and generate a sufficient number of views to fill the view frustum of the target immersive client display. Therefore, using such synthetic representations as candidates for representing natural content is currently impractical because these synthetic representations cannot be created in real-time and do not meet the needs of use cases requiring real-time distribution. Nevertheless, the best candidate representation for computer-generated images is still a scene graph that combines a synthetic model, since computer-generated images are created using 3D modeling processes and tools.

[0081] This fundamental difference in optimal representation between natural and computer-generated content indicates that the optimal ingestion format for naturally captured content differs from that for computer-generated content, and also from that for natural content that is not essential for real-time distribution applications. Therefore, the disclosed subject matter aims to be robust enough to support multiple ingestion formats for visually immersive media, whether these media are captured naturally using physical cameras or created by computers.

[0082] The following are some exemplary techniques that employ scene graphs as a format suitable for representing visually immersive media created using computer-generated techniques, or for representing naturally acquired content for which deep learning or photogrammetry techniques are used to create a corresponding synthetic representation of the natural scene, i.e., the naturally acquired content is not required for real-time distribution applications.

[0083] 1. OTOY's ORBX OTOY's ORBX is one of several scene graph technologies capable of supporting any type of visual media, whether timed or non-timed, including ray-traced, traditional (frame-based), volumetric 3D, and other types of composite or vector-based visual formats. ORBX differs from other scene graphs in that it provides native support for freely available and / or open-source formats of meshes, point clouds, and textures. ORBX is a scene graph designed to facilitate interoperability between technologies from multiple vendors that can manipulate scene graphs. Furthermore, ORBX offers a rich material system, support for open shader languages, a powerful camera system, and support for Lua scripting. ORBX is also the foundation for an immersive technology media format licensed royalty-free by the Immersive Digital Experiences Alliance (IDEA). In the context of real-time media distribution, the ability to create and distribute ORBX representations of natural scenes depends on the availability of computational resources used to perform complex analyses on camera-acquired data and composite that data into a synthetic representation. To date, acquiring sufficient computing power to support real-time distribution is impractical, but not impossible.

[0084] 2. Pixar's USD Pixar's USD is another well-known and established scene graph, widely popular in VFX and professional content production. USD is integrated into NVIDIA's Omniverse platform, a toolset for developers to create and render 3D models using NVIDIA Graphics Processing Units (GPUs). Apple and Pixar released a subset of USD as USDZ. USDZ is supported by Apple's ARKit.

[0085] 3. Khronos' glTF2.0 glTF 2.0 is the latest version of the "Graphics Language Transport Format" specification written by the Khronos 3D Working Group. This format supports a simple scene graph format, typically capable of supporting static (non-timed) objects in the scene, including "png" and "jpeg" image formats. glTF 2.0 supports simple animations, including translation, rotation, and scaling of basic shapes (i.e., geometric objects) described using glTF primitives. glTF 2.0 does not support timed media, and therefore does not support video or audio.

[0086] 4. ISO / IEC 23090 Part 14 Scene Description is an extension of glTF2.0, which adds support for timed media (e.g., video and audio).

[0087] These designs for scene representations of immersive visual media are provided as examples only and do not limit the ability of the disclosed topics to define a process for adapting an input immersive media source to a format suitable for the specific characteristics of the client endpoint device.

[0088] Furthermore, any or all of the exemplary media representations described above currently employ or may employ deep learning techniques to train and create neural network models capable of implementing or facilitating the selection of specific views to fill the view frustum of a particular display based on a specific size of the view frustum. The view selected for the view frustum of a particular display can be obtained by interpolation from existing views explicitly provided in the scene representation (e.g., views from MSI or MPI technologies), or can be directly rendered by the rendering engine based on specific virtual camera positions, filters, or virtual camera descriptions of these rendering engines.

[0089] Therefore, the disclosed subject matter is robust enough to account for the existence of a relatively small but well-known set of immersive media ingestion formats that are sufficient to meet the requirements for real-time or “on-demand” (e.g., non-real-time) distribution of media acquired naturally (e.g., using at least one camera) or created using computer-generated techniques.

[0090] The deployment of advanced network technologies, such as 5G for mobile networks and fiber optic cables for fixed networks, has further facilitated view interpolation from immersive media ingestion formats using neural network models or network-based rendering engines. In other words, these advanced network technologies enhance the capacity and capabilities of commercial networks, as such advanced network infrastructures can support the increasing transmission and distribution of visual information. Network infrastructure management technologies such as Multi-access Edge Computing (MEC), Software Defined Networking (SDN), and Network Functions Virtualization (NFV) enable commercial network service providers to flexibly configure their network infrastructure to adapt to changing needs for specific network resources, such as dynamic increases or decreases in network throughput, network speed, round-trip latency, and computing resources. Furthermore, this inherent ability to adapt to dynamic network needs also helps networks adapt immersive media ingestion formats to suitable distribution formats, thereby supporting a variety of immersive media applications that may employ heterogeneous visual media formats for heterogeneous client endpoints.

[0091] Immersive media applications may have varying network resource requirements, including: gaming applications, which require significantly lower network latency to respond to real-time updates of game status; telepresence applications, which have symmetrical throughput requirements for both uplink and downlink; and passive viewing applications, which may have higher downlink resource requirements depending on the type of display on the client endpoint using the data. Generally, any consumer-facing application can be powered by multiple client endpoints with varying built-in storage, computing, and power consumption capabilities, and also have different requirements for specific media representations.

[0092] Therefore, the disclosed subject matter enables a well-configured network (i.e., a network employing some or all of the characteristics of a modern network) to simultaneously support multiple legacy devices and devices supporting immersive media, according to the characteristics specified in the following: 1. Provides flexibility to leverage media ingestion formats applicable in both real-time and "on-demand" use cases for media distribution.

[0093] 2. Provides flexibility to support both natural and computer-generated content, applicable to both traditional client endpoints and client endpoints that support immersive media.

[0094] 3. Supports both timed and non-timed media.

[0095] 4. Provide a process for dynamically adapting a source media ingestion format to a suitable distribution format based on the characteristics and capabilities of the client endpoint and the requirements of the application.

[0096] 5. Ensure that the distribution format can be streamed over an IP-based network.

[0097] 6. Enables the network to serve multiple heterogeneous client endpoints simultaneously, which may include both legacy devices and applications and devices and applications that support immersive media.

[0098] 7. An exemplary media representation framework is provided that facilitates the organization of distributed media along scene boundaries.

[0099] For example, according to at least one exemplary embodiment, as described below, according to Figures 3 to 16 The processes and components described in the detailed description implement improved end-to-end embodiments brought about by the disclosed subject matter.

[0100] Figure 3 and Figure 4 All employ a single, exemplary inclusive distribution format, adapted from the ingestion source format to match the capabilities of specific client endpoints. As mentioned above, Figure 3The media shown is timed, while Figure 4 The media shown is non-timed. The structure of this particular inclusive format is robust enough to accommodate a variety of media properties, each of which can be layered based on the significant amount of information contributed by each layer to the media presentation. It should be noted that this layering process is a well-known technique at the current technological level, as demonstrated by Progressive JPEG and Scalable Video architectures, such as those specified in ISO / IEC 14496-10 (Scalable Advanced Video Coding).

[0101] 1. Media streamed according to inclusive media formats are not limited to traditional visual and auditory media, but can include any type of media information that can generate signals for machine interaction to stimulate human vision, hearing, taste, touch and smell.

[0102] 2. The media streamed according to the inclusive media format can be timed media, non-timed media, or a combination of both.

[0103] 3. Furthermore, this inclusive media format can also support streaming by employing a base layer and enhancement layer architecture to achieve a layered representation of media objects. In one example, independent base and enhancement layers are computed by applying multi-resolution or multi-segmentation analysis techniques to media objects in each scene. This is similar to the progressive rendering image formats specified in ISO / IEC 10918-1 (JPEG) and ISO / IEC 15444-1 (JPEG2000), but is not limited to raster-based visual formats. In an exemplary embodiment, the progressive representation of a geometric object can be a multi-resolution representation of the object computed using wavelet analysis.

[0104] In another example of the layered representation of this media format, the enhancement layer applies different properties to the base layer, such as refining the surface material properties of the visual objects represented by the base layer. In yet another example, these properties can refine the surface texture of the base layer objects, for example, changing the surface from a smooth texture to a porous texture, or from a matte surface to a glossy surface.

[0105] In yet another example of layered representation, the surface of at least one visual object in the scene can be changed from a Lambertian surface to a ray-traced surface.

[0106] In another example of layered representation, the network distributes the base layer representation to the client, enabling the client to create a nominal representation of the scene while waiting for additional enhancement layers to be transmitted to refine the base representation for resolution or other features.

[0107] 4. The resolution of attributes or refinement information in the enhancement layer is not explicitly coupled to the resolution of objects in the base layer, as is the case in existing MPEG video and JPEG image standards.

[0108] 5. Inclusive media format support enables support for any type of information media that can be rendered or driven by a presentation device or machine, thereby supporting heterogeneous media formats for heterogeneous client endpoints. In one embodiment of a network distributing this media format, the network will first query the client endpoint to determine the client's capabilities, and if the client cannot effectively ingest the media representation, the network will remove an attribute layer that is not supported by the client, or adapt the media from its current format to a format suitable for the client endpoint. In one example of such adaptation, the network utilizes a network-based media processing protocol to convert a volumetric 3D visual media asset into a 2D representation of the same visual asset. In another example of such adaptation, the network may employ neural network processing to reformat the media to an appropriate format, or optionally synthesize the view required by the client endpoint.

[0109] 6. The inventory of complete or partially complete immersive experiences (live events, games, or on-demand asset replays) is organized by scene, which is the minimum amount of information that the rendering and game engines can currently ingest to create a presentation. The inventory includes a list of individual scenes to be rendered for the entire immersive experience requested by the client. Each scene is associated with at least one representation of the geometry within that scene, corresponding to a streamable version of the scene's geometry. One embodiment of a scene representation involves a low-resolution version of the scene's geometry. Another embodiment of the same scene involves enhancement layers for the low-resolution representation of the scene to add additional detail or subdivision to the geometry of the same scene. As described above, each scene may have more than one enhancement layer to progressively increase the detail of the scene's geometry.

[0110] 7. Each layer of a media object referenced within the scene is associated with a token (e.g., a URI) that points to an address on the network from which the resource can be accessed. Such resources are similar to CDNs from which clients can retrieve content.

[0111] 8. The token representing a geometric object can point to a location within the network or within the client. That is, a client can send signaling to the network indicating that its resources are available for network-based media processing.

[0112] Figure 3A timing media representation 300 is depicted, which includes an embodiment of an inclusive media format for timing media, as detailed below. The timing scene list includes a list of scenes 301. Scene 301 references a list of components 302, each describing processing information and the media asset types constituting scene 301. Components 302 reference assets 303, which further reference a base layer 304 and an attribute enhancement layer 305. A list of unique assets not previously used in other scenes is provided in 307.

[0113] Figure 4 A non-timing media representation 400 is depicted, which includes an embodiment of an inclusive media format for non-timing media, as detailed below. The information in scenario 401 is not associated with a clock-based start and end duration. Scenario 401 references a list of components 402, which describe processing information and the types of media assets constituting scenario 401. Component 402 references asset 403, which further references the base layer 404 and attribute enhancement layers 405 and 406. Additionally, scenario 401 references other scenarios 401 for non-timing media. Scenario 401 also references scenario 407 for timing media scenarios. List 406 identifies unique assets associated with a particular scenario that have not previously been used in higher-level (e.g., parent) scenarios.

[0114] Figure 5 An exemplary embodiment of a natural media compositing process 500 for capturing formats from natural content is shown. Camera unit 501 captures a scene of people using a single camera lens. Camera unit 502 captures a scene with five diverging fields of view by mounting five camera lenses around a ring-shaped object. The arrangement at 502 is an exemplary arrangement commonly used for capturing omnidirectional content for VR applications. Camera unit 503 captures a scene with seven converging fields of view by mounting seven camera lenses on the inner diameter portion of a sphere. Arrangement 503 is an exemplary arrangement commonly used for capturing light fields or holographic immersive displays. Natural image content 509 is provided as input to a compositing process 504, which optionally employs a neural network training process 505, which uses a set of training images 506 to generate an optional acquisition neural network model 508. Another process commonly used as an alternative to the training process 505 is photogrammetry. If model 508 is in Figure 5 If the model 508 is created during the process described in 500, then it becomes one of the assets in the natural content ingestion format 510. Optionally, an annotation process 507 is performed to annotate the scene-based media using IMS metadata. Exemplary embodiments of ingestion format 510 include MPI and MSI.

[0115] Figure 6An embodiment of a composite media ingestion creation process 600 for creating an ingestion format for composite media (e.g., computer-generated images) is illustrated. A LiDAR camera 601 acquires a point cloud 602 of the scene. CGI assets 604 are created over a network using CGI tools, 3D modeling tools, or other animation processes for creating composite content on a computer 603. A participant 605 wears a motion capture suit 605A with sensors to acquire digital records of the participant 605's movements, thereby generating animated motion capture data 606. Data 602, 604, and 606 are provided as input to a compositing process 607, which outputs a composite media ingestion format 608. Format 608 can then be input to an optional IMS annotation process 609, the output of which is an IMS-annotated composite media ingestion format 610.

[0116] The aforementioned techniques for representing and streaming heterogeneous immersive media can be implemented in the form of computer software using computer-readable instructions and physically stored in at least one computer-readable medium. For example, Figure 7 A computer system 700 suitable for implementing certain embodiments of the disclosed subject matter is described.

[0117] Computer software can be written using any suitable machine code or computer language. This machine code or computer language may be processed by assembling, compiling, linking, or similar mechanisms to generate code containing instructions. These instructions can be executed directly by the computer's central processing unit (CPU), GPUs, or through interpretation, microcode execution, or other methods.

[0118] These instructions can be executed on various types of computers or their components, including personal computers, tablets, servers, smartphones, gaming devices, Internet of Things devices, and so on.

[0119] Figure 7 The components of the computer system 700 shown are merely exemplary and are not intended to imply any limitation on the scope or functionality of the computer software implementing embodiments of the present invention. The configuration of the components should also not be construed as having any dependency or requirement on any of the components or combinations of components shown in the exemplary embodiments of the computer system 700.

[0120] Computer system 700 may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from at least one human user via, for example, tactile input (e.g., button presses, swipes, data glove movements), audio input (e.g., voice, clapping sounds), visual input (e.g., gestures), or olfactory input (not depicted). Human-machine interface devices may also be used to acquire media that are not necessarily directly related to conscious human input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0121] The human-machine interface input device may include at least one of the following (only one of each is depicted): keyboard 701, mouse 702, touchpad 703, touch screen 710, data glove (not depicted), joystick 705, microphone 706, scanner 707, and camera 708.

[0122] Computer system 700 may also include certain human-machine interface (HMI) output devices. Such HMI output devices can stimulate the senses of at least one human user through, for example, tactile output, sound, light, and smell / taste. These HMI output devices may include tactile output devices (e.g., tactile feedback provided by a touchscreen 710, data gloves (not depicted), or joystick 705, but there may also be tactile feedback devices that do not act as input devices), audio output devices (e.g., speakers 709, headphones (not depicted)), visual output devices (e.g., screens 710, including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touchscreen input functionality, each with or without tactile feedback functionality—some of which are capable of outputting two-dimensional or more than three-dimensional visual output through methods such as stereoscopic output; virtual reality glasses (not depicted), holographic displays, and smoke boxes (not depicted)), and printers (not depicted).

[0123] The computer system 700 may also include user-accessible storage devices and related media, such as optical media or similar media 721 including CD / DVD ROM / RW 720 equipped with CD / DVD, flash memory drive 722, removable hard disk drive or solid state drive (SSD) 723, conventional magnetic media such as magnetic tape and floppy disk (not depicted), and dedicated devices based on ROM / ASIC / PLD such as security dongles (not depicted).

[0124] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not cover transmission media, carrier waves, or other transient signals.

[0125] Computer system 700 may also include an interface connected to at least one communication network. The network may be, for example, a wireless network, a wired network, or an optical network. The network may also be a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), an in-vehicle and industrial network, a real-time network, a latency-tolerant network, etc. Examples of networks include: local area networks such as Ethernet and wireless LANs; cellular networks including GSM, 3G, 4G, 5G, LTE, etc.; TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV; and in-vehicle and industrial networks including CAN buses, etc. Some networks typically require an external network interface adapter (e.g., a USB port of computer system 700) attached to some general-purpose data port or peripheral bus (749); other networks are typically integrated into the core of computer system 700 via a system bus described below (e.g., integrated into a PC computer system via an Ethernet interface, or integrated into a smartphone computer system via a cellular network interface). Using any of these networks, computer system 700 can communicate with other entities. This communication can be one-way (e.g., broadcasting TV), one-way (e.g., CAN bus to certain CAN bus devices), or it can be bidirectional, such as connecting to other computer systems using a local area or wide area digital network. Certain protocols and protocol stacks can be used on each of the networks and network interfaces described above.

[0126] The aforementioned human-machine interface device, user-accessible storage device, and network interface can be attached to the core 740 of the computer system 700.

[0127] Core 740 may include at least one CPU 741, GPU 742, dedicated programmable processing unit in the form of a Field Programmable Gate Array (FPGA) 743, hardware accelerators 744 for certain tasks, etc. These devices, along with read-only memory (ROM) 745, random-access memory (RAM) 746, and internal high-capacity storage devices 747 such as internal non-user-accessible hard drives and solid-state drives, can be connected via system bus 748. In some computer systems, system bus 748 can be accessed via at least one physical connector, allowing for expansion with additional CPUs, GPUs, etc. Peripheral devices can be directly attached to the core's system bus 748 or via peripheral bus 749. Peripheral bus architectures include PCI, USB, etc.

[0128] The CPU 741, GPU 742, FPGA 743, and accelerator 744 can execute certain instructions, which, when combined, constitute the aforementioned computer code. This computer code can be stored in ROM 745 or RAM 746. Temporary data can also be stored in RAM 746, while permanent data can be stored, for example, in an internal mass storage device 747. Fast storage and retrieval of any of the storage devices can be achieved by using a cache memory, which can be closely associated with at least one CPU 741, GPU 742, mass storage device 747, ROM 745, RAM 746, etc.

[0129] The computer-readable medium may contain computer code for performing various computer-implemented operations. The medium and computer code may be designed and constructed specifically for the purposes of this invention, or may belong to a class well-known and available to those skilled in the art of computer software.

[0130] For example, but not as a limitation, a computer system having architecture 700, particularly core 740, can provide functionality generated when one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) execute software embodied in at least one tangible computer-readable medium. Such a computer-readable medium can be a medium associated with certain storage devices (e.g., internal core mass storage 747 or ROM 745) that are user-accessible mass storage devices as described above and with the non-volatile nature of core 740. Software implementing various embodiments of the invention can be stored in such devices and executed by core 740. Depending on specific needs, the computer-readable medium may include at least one storage device or chip. The software can cause core 740, particularly its processors (including CPUs, GPUs, FPGAs, etc.), to execute specific processes or specific portions of specific processes described in this application, including defining data structures stored in RAM 746 and modifying such data structures according to the processes defined by the software. Alternatively or as an alternative, the computer system may provide functionality generated by hard-wired logic or otherwise embodied in circuitry (e.g., accelerator 744), which may replace or operate in conjunction with software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may cover logic, and vice versa. Where appropriate, references to computer-readable media may cover circuitry storing software for execution (e.g., integrated circuits (ICs)), circuitry embodying logic for execution, or both. This invention covers any suitable combination of hardware and software.

[0131] Figure 8An exemplary network media distribution system 800 is shown, supporting various traditional and heterogeneous displays that support immersive media as client endpoints. The content acquisition process 801 uses... Figure 6 or Figure 5 Exemplary embodiments of the system 800 are used to acquire or create media. The ingestion format is generated in content preparation process 802 and subsequently transmitted to a network media distribution system using transmission process 803. Gateway 804 can serve as a customer premises device to provide network access to various client endpoints of the network. Set-top box 805 can also act as a customer premises device to provide access to content aggregated by a network service provider. Radio demodulator 806 can act as a mobile network access point for mobile devices, such as mobile handheld devices and displays 813. In this particular embodiment of system 800, a conventional 2D television 807 is shown as directly connected to gateway 804, set-top box 805, or WiFi router 808. A laptop computer with a conventional 2D display 809 is shown as a client endpoint connected to WiFi router 808. Head-mounted 2D (raster-based) display 810 is also connected to router 808. Lens-mounted light field display 811 is shown connected to gateway 804. Display 811 comprises local computing GPUs 811A, storage device 811B, and a visual rendering unit 811C, which uses light-based lenticular lens optics to create multiple views. Holographic display 812 is shown connected to set-top box 805. Display 812 comprises local computing CPUs 812A, GPUs 812B, storage device 812C, and a Fresnel pattern, wave-based holographic visualization unit 812D. Augmented reality head-mounted device 814 is shown connected to radio demodulator 806. Head-mounted device 814 comprises GPU 814A, storage device 814B, battery 814C, and volumetric 3D visual rendering component 814D. Dense light field display 815 is shown connected to WiFi router 808. The display 815 consists of multiple GPUs 815A, CPUs 815B, storage devices 815C, eye-tracking devices 815D, cameras 815E, and a light field panel 815F based on dense light.

[0132] Figure 9 An embodiment of an immersive media distribution process 900 utilizing a scene analyzer for a default viewport is shown. This process 900 is capable of serving applications such as those previously described. Figure 8 The traditional and heterogeneous displays depicted support immersive media. Content is created or acquired in process 901. Figure 5 and Figure 6 This is further illustrated for both natural content and CGI content. Then, the content 901 is converted into the ingestion format using the web ingestion format creation process 902. Process 902 also... Figure 5 and Figure 6 The details are further illustrated for both natural and CGI content. Optionally, the ingested media is annotated using IMS metadata via a scene analyzer 911 with optional IMS annotations. The ingested media format is transmitted to the network and stored in storage device 903. Optionally, the storage device may be located within the network of the immersive media content producer and remotely accessed by the immersive media network distribution process (unnumbered), as depicted by the dashed line dividing 903 in two. Client- and application-specific information may optionally be available on a remote storage device 904, which may optionally reside remotely in another "cloud" network.

[0133] like Figure 9 As depicted, network orchestration process 905 acts as both the primary source and sink of information to perform the main tasks of distributing the network. In this particular embodiment, process 905 can be implemented using a format consistent with other components of the network. Nevertheless, by Figure 9 The tasks described in process 905 constitute the basic elements of the disclosed subject matter. The orchestration process 905 may further employ a bidirectional messaging protocol with the client to facilitate all media processing and distribution according to the client's characteristics. Furthermore, this bidirectional protocol can be implemented on different transport channels, namely, a control plane channel and a data plane channel.

[0134] Process 905 receives information about the characteristics and attributes of client 908 and further gathers information about the needs of the application currently running on client 908. This information can be obtained from device 904, or, in an alternative embodiment, by directly querying client 908. In the case of directly querying client 908, it is assumed that a bidirectional protocol ( Figure 9 (Not shown in the image) exists and is operational, allowing clients to communicate directly with orchestration process 905.

[0135] The 905 program also initiated the arrangement process. Figure 10 The media adaptation and fragmentation process 910 described herein communicates with the media. While the ingested media is being adapted and fragmented through process 910, the media may optionally be transferred to an intermediate storage device, described as a media storage device 909 to be distributed. When the distributed media is ready and stored in device 909, orchestration process 905 ensures that immersive client 908 receives the distributed media and corresponding description information 906 via its network interface 908B through a "push" request, or client 908 itself may initiate a "pull" request for media 906 from storage device 909. Orchestration process 905 may employ a bidirectional messaging interface (...). Figure 9The client 908 performs a "push" request or initiates a "pull" request (not shown) to perform a "pull" request. The immersive client 908 optionally employs GPUs (or CPUs, not shown) 908C. The media distribution format is stored in the client 908's storage device or storage cache 908D. Finally, the client 908 visually presents the media through its visualization component 908A.

[0136] Throughout the process of streaming immersive media to client 908, orchestration process 905 monitors the client's progress status via client progress and status feedback channel 907. Status monitoring can be achieved through a two-way communication message interface (…). Figure 9 (Not shown in the image) to execute.

[0137] Figure 10 Specific embodiments of a scene analyzer for a media adaptation process 1000 are depicted to properly adapt the ingested source media to meet the needs of client 908. The media adaptation process 1001 comprises multiple components that facilitate the adaptation of the ingested media to the appropriate distribution format required by client 908. These components should be considered exemplary. Figure 10 In the adaptation process 1001, the network state 1005 is received to track the current traffic load on the network; client information 908 is received, including attribute and feature descriptions, application features and descriptions, the current state of the application, and a client neural network model (if available) to assist in mapping the geometry of the client's view frustum to the interpolation capability for ingesting immersive media. This information can be obtained via a bidirectional messaging interface. The adaptation process 1001 ensures that when generating the adaptation output, it is stored in the client-adapted media storage device 1006. A scene analyzer 1007 with an optional IMS annotation process is included. Figure 10 It is described as an optional process that can be executed in advance or as part of a network automation process for media distribution.

[0138] The adaptation process 1001 is controlled by a logic controller 1001F. The adaptation process 1001 also employs a renderer 1001B or a neural network processor 1001C to adapt a specific ingested source media to a format suitable for the client. The neural network processor 1001C uses a neural network model 1001A. Examples of such a neural network processor 1001C include the Deepview neural network model generator, as described in MPI and MSI. If the media is in a 2D format, but the client requires a 3D format, the neural network processor 1001C can invoke a process to derive a volumetric 3D representation of the scene depicted in the video using a highly correlated image from the 2D video signal. An example of a suitable renderer 1001B could be a modified version of the OTOY Octane renderer, modified to interact directly with the adaptation process 1001. The adaptation process 1001 may optionally employ a media compressor 1001D and a media decompressor 1001E, depending on the format of the ingested media and whether the format requested by the client 908 requires these tools.

[0139] Figure 11 The distribution format creation process 1100 is described. The media adaptation packaging process 1103 adapts media from the media adaptation process 1101 (in...). Figure 10 The media described in process 1000 are packaged, and this media currently resides on the client-adapted media storage device 1102. Packaging process 1103 formats the adapted media from process 1101 into a robust distribution format 1104, for example, Figure 3 or Figure 4 The exemplary format shown is illustrated. List information 1104A provides client 908 with a list 1104B of scene data assets it may expect to receive. List 1104B depicts a list of visual assets, auditory assets, and tactile assets, each with its corresponding metadata.

[0140] Figure 12 A packet processing system 1200 is described. The packet processor 1202 divides the adapting media 1201 into individual data packets 1203 suitable for streaming to the client 908.

[0141] Figure 13The components and communications shown in sequence diagram 1300 are described as follows: Client endpoint 1301 initiates a media request 1308 to network distribution interface 1302. Request 1308 includes information identifying the media requested by the client, which is implemented through a URN or other standard naming convention. Network distribution interface (also referred to as client) 1302 responds to request 1308 with a profile request 1309, which requests client 1301 to provide information about its currently available resources (including computing information, storage information, battery percentage information, and other information characterizing the client's current operating state). Profile request 1309 also requests the client to provide at least one neural network model. If such a model is available at the client, the network can use at least one neural network model to perform neural network inference to extract or generate the correct media view through interpolation, thereby matching the characteristics of the client's presentation system. The response 1311 sent by client 1301 to interface 1302 provides a client token, an application token, and at least one neural network model token (if such a neural network model token is available at the client). Then, interface 1302 provides session ID token 1311 to client 1301. Then, interface 1302 sends a request to ingest media server 1303 with ingest media request 1312, which includes the URN or other standard name of the media identified in request 1308. Server 1303 responds to request 1312 with response 1313, which includes the ingest media token. Then, interface 1302 provides the media token from response 1313 in call 1314 to client 1301. Then, interface 1302 initiates an adaptation process for the requested media in 1308 by providing the ingest media token, client token, application token, and neural network model token to adaptation interface 1304. Interface 1304 requests access to the ingested media by providing the ingest media token to server 1303 in call 1316 to request access to the ingested media asset. In response 1317 to interface 1304, server 1303 replies to request 1316 with an ingest media access token. Interface 1304 then requests media adaptation procedure 1305 to adapt the ingested media located at the ingest media access token for the client, application, and neural network inference model corresponding to the session ID token created at 1313. Request 1318 from interface 1304 to procedure 1305 includes the required token and session ID. Procedure 1305 provides the adapted media access token and session ID to interface 1302 in update 1319. Interface 1302 provides the adapted media access token and session ID to packaging procedure 1306 in interface call 1320. Packaging procedure 1306 provides response 1321 to interface 1302, which contains the packaged media access token and session ID.In response 1322, process 1306 provides the packaged assets, URNs, and a packaged media access token for the session ID to the packaged media server 1307. Client 1301 executes request 1323 to initiate streaming of the media assets corresponding to the packaged media access token received in message 1321. Client 1301 executes additional requests and provides a status update to interface 1302 in message 1324.

[0142] Figure 14 An exemplary scene graph architecture 1400 is depicted. A user-readable scene graph description 1401 serves as the spatial, logical, physical, and temporal portion of the scene graph that stores the attached assets. Description 1401 also contains references to binary assets that further constitute the scene. Associated with description 1401 is binary asset 1402. Figure 14 An exemplary diagram is shown with four binary assets, including: binary asset A 1402, binary asset B 1402, binary asset C 1402, and binary asset D 1402. References 1403 from description 1401 are also shown as: reference 1403 to binary asset A, reference 1403 to binary asset B, reference 1403 to binary asset C, and reference 1403 to binary asset D.

[0143] Figure 15 An exemplary labeled scene graph architecture 1500 is provided, in which IMS system metadata 1503? (where ? represents a character in the graph) is directly written into the user-readable portion 1501 of the scene graph architecture 1500. In this example, the IMS system metadata 1503? consists of multiple metadata systems 1503A, 1503B, 1503C, 1503D, 1503E, 1503F, 1503G, and 1503H, each associated with its own unique IMS system identifier code point, which corresponds to the ? depicted for item 1503 in the graph. Mapping 1504? (where ? represents a character in the graph) also provides additional information about unique ITMF code points (derived from the ITMF family of specifications), which fully or partially characterize the information contained in each segment of the user-readable portion 1501. Such mapping 1504? is depicted in the graph. This includes: 1504A, 1504B, 1504C, 1504D, 1504E, 1504F, 1504G, and 1504H. The IMS metadata written to the user-readable portion 1501 includes the information depicted in the mapping 1504? as described above. The scene graph architecture 1500 also includes scene assets 1502.

[0144] Figure 16An exemplary labeled scene graph architecture 1600 is provided, in which IMS system metadata 1606? (where ? represents a character in the graph) is directly written into the binary portion 1603 of the architecture, as a user-readable portion 1601 of the scene graph architecture 1600 for storing such metadata (e.g., Figure 15 The IMS system metadata 1606? is a replacement or supplement to the one depicted in the figure. In this example, the IMS system metadata 1606? consists of multiple metadata systems 1606A, 1606B, 1606C, 1606D, 1606E, 1606F, 1606G, and 1606H, each associated with its own unique IMS system identifier code point, which corresponds to the ? depicted for item 1606 in the figure. Mapping 1604? (where ? represents a character in the figure) also provides additional information about unique ITMF code points (derived from the ITMF family of specifications) that fully or partially characterize the information contained in the user-readable portion 1601. Such mappings 1604? depicted in the figure include: 1604A, 1604B, 1604C, 1604D, 1604E, 1604F, 1604G, and 1604H. The IMS metadata written to the binary portion 1603 includes the information depicted in mapping 1604? as described above. Scene graph architecture 1600 also includes scene assets 1602.

[0145] Figure 17 The organization of an "IMS" according to at least one embodiment of this application is depicted, which may be incorporated into or is currently described in ISO / IEC 23090 Part 28. Items labeled "1701" represent data values ​​(e.g., numerical values, enumerators, geospatial coordinates), items labeled "1702" represent scene features such as lighting, camera, materials, textures, geometry, environment, object layers, surfaces, arbitrary output variables, arbitrary output variable compositors, or transformations, items labeled "1703" represent input or output connectors between other items (e.g., items labeled "1702"), items labeled "1704" represent immutable properties that should not be changed during the serialization process, items labeled "1705" represent groups of other items, items labeled "1706" represent rendering instructions (which may also be considered process-related instructions) or parameters, and items labeled "1707" represent the output of rendering or presentation. In this example, it is clear that the architecture, particularly the item labeled "1702", does not distinguish between items used during the rendering process (such as camera and lighting) and items that describe geometric objects in the scene (such as materials, textures, geometric meshes, etc.).

[0146] Figure 18 Depicting Figure 17The information represented in the image is an "extracted" version, where scene information is organized into 1801 geometric information, 1802 physical organization information (describing how the media is organized, accessed, and written in files, buffers, or other containers), and 1803 processing information, which relates to instructions or parameters used to render or present the media. Information 1801, 1802, and 1803 is provided to a rendering engine 1804, which renders or otherwise presents the scene media. Output 1805 is provided as the output of engine 1804.

[0147] Figure 19 At least one embodiment according to this application is shown as follows Figure 17 The organization shown is different in that the same representation can be used to organize specific serialization results 1901, which are used for geometric information as depicted in 1801, physical information as depicted in 1802, and processing information as depicted in 1803.

[0148] Figure 20 The result 1901 is shown to be usable to represent geometric information as depicted in 1901A, physical information as depicted in 1901B, or processing information as depicted in 1901C. Each set of information (1901A, 1901B, and 1901C) is provided to the rendering engine 1804 to generate output 1805.

[0149] Therefore, as stated above, during the development of IMS, it has become clear that the architecture proposed in the CD vote of Part 28 needs to be improved to better achieve the goal of creating interchangeable representations for various scene-based formats. Furthermore, embodiments of this application provide an update to the IMS architecture in Part 28 and propose incorporating this update into Part 28, i.e., as an output of MPEG 147, prior to the planned DIS vote.

[0150] The text in Section 28 may include a relatively “flat” description of the IMS systems, covering multiple aspects of scene-based media representation. These systems are described in Table 1, which is repeated below for convenience:

[0151]

[0152] There are a total of 21 systems, some of which are interconnected while others are independent. For example, the animation processing, rendering instructions, rendering arbitrary output variables, arbitrary output variable compositor, lighting, camera, kernel, and open color I / O systems each involve specific actions that the rendering engine should perform during scene serialization and / or rendering. Similarly, many systems involve describing the logical relationships and geometric organization between objects within the scene. That is, the materials, scene objects, connection pins, graph types, geometry, and other items in Table 1 involve describing the organization and properties of the geometry in the scene (including relationships between scene objects).

[0153] Further research based on the embodiments of this application and the systems listed in Table 1 shows that the buffer system is the only system that does not belong to any of the categories identified so far, namely, neither the rendering action and instruction category nor the logical geometry organization category. The uniqueness of the buffer system lies in its attempt to solve physical problems related to the storage, access, reading, and writing methods of binary scene media.

[0154] The ITMF technology upon which IMS is based (and is also inspired by) includes a completely independent specification, the ITMF Container Specification, which describes the physical aspects of ITMF media, such as how it is stored, read, and decrypted, as well as many other attributes of the media's physical storage.

[0155] Furthermore, according to embodiments of this application, an architecture similar to ITMF is proposed: IMS should include metadata describing how media should be physically stored, accessed, and otherwise managed by the rendering engine. More specifically, given that the existing 21 systems constituting IMS already have two main categories, a third category should be added and the number of systems expanded to cover all physical aspects of media. With the emergence of these three higher-level categories, it is natural to reconfigure the existing IMS architecture to include: 1. Rendering operations or "instructions" specific to the renderer or rendering engine on how to serialize and render media.

[0156] 2. Information about the geometry of the scene, including relationships between objects and descriptions of individual objects and the scene background, and 3. Information regarding the physical storage methods of media. The following systems in Table 1 are considered to be related to the real-time processing performed by the renderer or rendering engine: animation processing, rendering instructions, rendering arbitrary output variables, arbitrary output variable compositor, lighting, camera, kernel, open color IO, and object layer.

[0157] The following systems in Table 4.1 are considered to be related to the description of the relationships between geometry and scene objects: system values, materials, textures, geometry, surfaces, transformations, graphs, connection pins, data attributes, scene objects, and environment.

[0158] In addition to the buffering systems listed in Table 4.1, it is proposed to add the following new systems to IMS to meet the needs of describing the storage, reading, writing, and access methods of media: a global attribute system for describing the overall physical storage representation of scene media; a local attribute system for describing local aspects of the storage representation of scene media; a physical stream system for signaling the existence of byte streams and the attributes of the byte stream corresponding to at least one physical element of the scene; a stream block system for signaling the existence of byte "blocks" (the unique physical boundary consisting of start and stop indicators) and the attributes of the byte "blocks" corresponding to the physical stream; and a stream index system for signaling random access points of the physical stream. A directory system is used to provide a list of storage attributes and other metadata for at least one stream; a signature system is used to provide encrypted information when the scene is encrypted; The scene end system provides information related to the end of a scene.

[0159] also, Figure 21 It shows in Figure 20 The same diagram depicts physical storage features 2100. Features 2100 include: global attributes, used to signal attributes applicable to the entire physical storage representation of the scene medium; local attributes, which are applicable only to local aspects of the storage representation of the scene medium; physical streams, which indicate the existence of a byte stream corresponding to a physical element of the scene; stream blocks, which indicate the existence of a byte "block" (a unique physical boundary consisting of a start indicator and a stop indicator) corresponding to a physical stream; stream indexes, which can signal random access points of the physical stream; a directory, which indicates a list providing attributes and other metadata for each stream; a signature, which provides encrypted information when the scene is encrypted; and a scene tail, which provides information related to the end of the scene.

[0160] Figure 22Example 2200 illustrates a canonical representation of an IMS geometric object within a scene according to at least one embodiment of this application, wherein element 1 represents a canonical IMS value (e.g., an integer, an enumeration, a floating-point number, etc.), element 2 represents an IMS system for logical organization, physical organization, and processing direction, element 3 represents an input pin or an output pin, element 4 represents an attribute, element 5 represents a node graph, element 6 represents the logical organization of the geometry, element 7 represents the processing direction, element 8 represents the physical organization of the scene graph, element 9 represents a representation engine (not a system in IMS), and element 10 represents a rendering output (not a system in IMS).

[0161] The IMS value (item 1) should represent the input value from the value node system to one of any other IMS systems (item 2), which includes systems that specify the logical organization, physical organization, and processing direction for the scene.

[0162] Input or output relationships between any system nodes should be established via IMS data pins (Element 3). Attributes should represent other IMS system nodes or value nodes that are considered immutable (cannot be changed during the rendering process). A set of system nodes is specified to describe: the logical organization of the geometry (Element 6); the processing direction (Element 7); and the physical organization within the scene (Element 8). The node graph (Element 5) should represent a set of other IMS system nodes, which may also include attributes or rendering-specific metadata (not within the scope of this specification). The rendering engine (Element 9) receives inputs including information about the logical organization, processing direction, and physical organization (not shown in the diagram) to generate the rendering output (Element 10).

[0163] According to an exemplary embodiment, the IMS system used to describe the logical organization components of a scenario is identified by a tag prefixed with "ims.logical". Table 2 specifies the IMS systems that constitute the logical organization components.

[0164]

[0165] The IMS system used to describe the physical organization components of a scene is identified by a label prefixed with "ims.physical". Table 3 specifies the IMS systems that constitute the IMS physical organization components according to the exemplary embodiments.

[0166]

[0167]

[0168] The IMS system used to describe the processing direction component of a scene is identified by a label prefixed with "ims.process". Table 4 specifies the IMS system constituting the IMS processing direction component according to the exemplary embodiments.

[0169]

[0170]

[0171] Figure 23 As an application example of the disclosed subject matter, the arrangement of a video encoder and decoder in a streaming media environment is shown. The disclosed subject matter is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, and storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0172] The streaming media system may include an acquisition subsystem 2303, which may include a video source 2301, such as a digital camera, for creating, for example, an uncompressed video sample stream 2313. The sample stream 2313 may be considered to have a high data volume compared to the encoded video stream and may be processed by an encoder 2302 coupled to the camera 2301. The encoder 2302 may include hardware, software, or a combination of hardware and software to enable or implement aspects of the disclosed subject matter, which are described in more detail below. The encoded video stream 2304 may be considered to have a lower data volume compared to the sample stream and may be stored on a streaming media server 2305 for future use. At least one streaming media client 2312 and 2307 may access the streaming media server 2305 to retrieve copies 2308 and 2306 of the encoded video stream 2304. Client 2312 may include video decoder 2311, which decodes an incoming copy of the encoded video stream 208 and generates an output video sample stream 2310, which can be rendered on display 2309 or other rendering devices. In some streaming media systems, video streams 2304, 2306, and 2308 may be encoded according to certain video encoding / compression standards.

[0173] While several exemplary embodiments have been described in this invention, modifications, substitutions, and various alternative equivalents exist that fall within the scope of this invention. Therefore, it should be understood that those skilled in the art will be able to design numerous systems and methods that, while not explicitly stated or described herein, embody the principles of this invention and are therefore within its spirit and scope.

Claims

1. A method for streaming immersive media, characterized in that, The method, which can be executed by a processor, includes: The content of the scene in the immersive media is captured, and the content includes a scene diagram belonging to the scene and having a first scene diagram format; Determine whether to convert the scene diagram from the first scene diagram format to the second scene diagram format; The scene graph is converted from a first scene graph format to a second scene graph format according to a metadata framework. The metadata framework includes at least a first metadata system and a second metadata system. The first metadata system includes multiple first metadata items, each representing first information shared between the first and second scene graph formats. The second metadata system includes multiple second metadata items, each representing second information shared between the first and second scene graph formats. The content of the immersive media is streamed based on the second scene graph; wherein the second scene graph is converted from the first scene graph according to the metadata framework.

2. The method according to claim 1, characterized in that, The first system represents the logical organization component of the scenario, and each of the plurality of first metadata in the first system is identified by a corresponding first tag, each first tag including the first prefix "ims.logical". The second system represents the physical organization components of the scene, and each of the plurality of second metadata in the second system is identified by a corresponding second tag, each second tag including the second prefix "ims.physical".

3. The method according to claim 2, characterized in that, The metadata framework also includes a third metadata system, which comprises multiple third metadata items representing third information shared between the first scene graph format and the second scene graph format. The third system represents the processing direction component of the scene, and each of the plurality of third metadata in the third system is identified by a corresponding third tag, each third tag including the third prefix "ims.process".

4. The method according to claim 1, characterized in that, The metadata framework includes metadata for representing the geometric information of the immersive media.

5. The method according to claim 1, characterized in that, The metadata framework includes metadata for representing the physical storage of the immersive media.

6. The method according to claim 5, characterized in that, The metadata representing the physical storage of the immersive media describes any of the following: The global attributes of the storage representation of the immersive media. Local attributes of the local aspects of the stored representation of the immersive media. Physical stream, indicating the existence of a byte stream corresponding to at least one physical element of the scene. A stream block indicates the existence of a byte block corresponding to the physical stream. Flow index, indicating the random access point of the physical flow. The directory indicates a list of at least the attributes and other metadata of the physical flow. The signature indicates the encrypted information of the scenario, and The end of the scene indicates information up to the end of the scene.

7. The method according to claim 1, characterized in that, The metadata framework includes processing parameters for representing the immersive media, as well as metadata used by either the renderer or the rendering engine.

8. An apparatus for streaming immersive media, characterized in that, The device includes: At least one memory is configured to store program code; and At least one processor is configured to read the program code and operate according to the instructions of the program code, in order to: The content of the scene in the immersive media is captured, and the content includes a scene diagram belonging to the scene and having a first scene diagram format; Determine whether to convert the scene diagram from the first scene diagram format to the second scene diagram format; The scene graph is converted from a first scene graph format to a second scene graph format according to a metadata framework. The metadata framework includes at least a first metadata system and a second metadata system. The first metadata system includes multiple first metadata items, each representing first information shared between the first and second scene graph formats. The second metadata system includes multiple second metadata items, each representing second information shared between the first and second scene graph formats. The content of the immersive media is streamed based on the second scene graph; wherein the second scene graph is converted from the first scene graph according to the metadata framework.

9. The device according to claim 8, characterized in that, The first system represents the logical organization component of the scenario, and each of the plurality of first metadata in the first system is identified by a corresponding first tag, each first tag including the first prefix "ims.logical". The second system represents the physical organization components of the scene, and each of the plurality of second metadata in the second system is identified by a corresponding second tag, each second tag including the second prefix "ims.physical".

10. The device according to claim 9, characterized in that, The metadata framework also includes a third metadata system, which comprises multiple third metadata items representing third information shared between the first scene graph format and the second scene graph format. The third system represents the processing direction component of the scene, and each of the plurality of third metadata in the third system is identified by a corresponding third tag, each third tag including the third prefix "ims.process".

11. The device according to claim 8, characterized in that, The metadata framework includes metadata for representing the geometric information of the immersive media.

12. The device according to claim 8, characterized in that, The metadata framework includes metadata for representing the physical storage of the immersive media.

13. The device according to claim 12, characterized in that, The metadata representing the physical storage of the immersive media describes any of the following: The global attributes of the storage representation of the immersive media. Local attributes of the local aspects of the stored representation of the immersive media. Physical stream, indicating the existence of a byte stream corresponding to at least one physical element of the scene. A stream block indicates the existence of a byte block corresponding to the physical stream. Flow index, indicating the random access point of the physical flow. The directory indicates a list of at least the attributes and other metadata of the physical flow. The signature indicates the encrypted information of the scenario, and The end of the scene indicates information up to the end of the scene.

14. The device according to claim 8, characterized in that, The metadata framework includes processing parameters for representing the immersive media, as well as metadata used by either the renderer or the rendering engine.

15. A non-transitory computer-readable storage medium for storing instructions, characterized in that, The instructions include: at least one instruction, which, when executed by at least one processor of a device for streaming immersive media, causes the at least one processor to: The content of the scene in the immersive media is captured, and the content includes a scene diagram belonging to the scene and having a first scene diagram format; Determine whether to convert the scene diagram from the first scene diagram format to the second scene diagram format; The scene graph is converted from a first scene graph format to a second scene graph format according to a metadata framework. The metadata framework includes at least a first metadata system and a second metadata system. The first metadata system includes multiple first metadata items, each representing first information shared between the first and second scene graph formats. The second metadata system includes multiple second metadata items, each representing second information shared between the first and second scene graph formats. The content of the immersive media is streamed based on the second scene graph, wherein the second scene graph is derived from the first scene graph according to the metadata framework.

16. The non-transitory computer-readable storage medium according to claim 8, characterized in that, The first system represents the logical organization component of the scenario, and each of the plurality of first metadata in the first system is identified by a corresponding first tag, each first tag including the first prefix "ims.logical". The second system represents the physical organization components of the scene, and each of the plurality of second metadata in the second system is identified by a corresponding second tag, each second tag including the second prefix "ims.physical".

17. The non-transitory computer-readable storage medium according to claim 9, characterized in that, The metadata framework also includes a third metadata system, which comprises multiple third metadata items representing third information shared between the first scene graph format and the second scene graph format. The third system represents the processing direction component of the scene, and each of the plurality of third metadata in the third system is identified by a corresponding third tag, each third tag including the third prefix "ims.process".

18. The non-transitory computer-readable storage medium according to claim 8, characterized in that, The metadata framework includes metadata for representing the geometric information of the immersive media.

19. The non-transitory computer-readable storage medium according to claim 8, characterized in that, The metadata framework includes metadata for representing the physical storage of the immersive media, and the metadata representing the physical storage of the immersive media describes any of the following: The global attributes of the storage representation of the immersive media. Local attributes of the local aspects of the stored representation of the immersive media. Physical stream, indicating the existence of a byte stream corresponding to at least one physical element of the scene. A stream block indicates the existence of a byte block corresponding to the physical stream. Flow index, indicating the random access point of the physical flow. The directory indicates a list of at least the attributes and other metadata of the physical flow. The signature indicates the encrypted information of the scenario, and The end of the scene indicates information up to the end of the scene.

20. The non-transitory computer-readable storage medium according to claim 8, characterized in that, The metadata framework includes processing parameters for representing the immersive media, as well as metadata used by either the renderer or the rendering engine.

Images