Method, apparatus, and computer program for immersive media streaming prioritized by frequency of asset reuse

Abstract

There is provided a method for packaging media to optimize immersive media delivery of a media stream performed by at least one processor, the method comprising: receiving immersive media data for an immersive presentation; obtaining asset information associated with media assets corresponding to a set of scenes included in the immersive media data for the immersive presentation; analyzing characteristics of the media assets based on the asset information, the characteristics including an asset type associated with each media asset and a frequency indicating a number of times each media asset is used in the set of scenes included in the immersive presentation; ordering the media assets into a sequence based on the asset type and frequency associated with each of the media assets; and streaming the immersive media data for the immersive presentation based on the ordered sequence of the media assets.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is based on and claims priority to U.S. Provisional Patent Application No. 63 / 276,536, filed November 5, 2021, and U.S. Patent Application No. 17 / 971,057, filed October 21, 2022, the disclosures of which are incorporated by reference in their entireties into this application.

[0002] This disclosure describes embodiments generally relating to architectures, structures, and components for systems and networks that deliver media, including video, audio, geometric (3D) objects, haptics, associated metadata, or other content for client presentation devices. Some embodiments relate to systems, structures, and architectures for the delivery of media content to heterogeneous immersive and interactive client presentation devices. [Background technology]

[0003] Immersive media generally refers to media that stimulates any or all human sensory systems (e.g., vision, hearing, somatosensation, smell, and possibly taste) to create or enhance a user's perception of being physically present in the media experience, i.e., beyond that distributed over existing (e.g., "legacy") commercial networks for timed two-dimensional (2D) video and corresponding audio; such timed media is also known as "legacy media." Immersive media may also be defined as media that seeks to stimulate any or all human sensory systems to create a perception by a user of being physically present in a scene depicting a real or virtual world by creating or mimicking the physical world through a digital simulation of dynamics and the laws of physics.

[0004] Several immersive media-enabled devices have been introduced (or are poised to emerge) in the consumer market, including head-mounted displays, augmented reality glasses, handheld controllers, multi-view displays, haptic gloves, gaming consoles, holographic displays, and other forms of volumetric displays. Despite the availability of these devices, a coherent end-to-end ecosystem for the delivery of immersive media over commercial networks has not been realized.

[0005] In the related art, one of the obstacles to realizing a coherent end-to-end ecosystem for the delivery of immersive media over commercial networks is the great diversity of all the client devices that serve as the endpoints of such delivery networks for immersive displays. Unlike networks designed solely for the delivery of legacy media, networks that must support a variety of display clients (i.e., heterogeneous clients) require a significant amount of information about the details of each of the client's capabilities and the format of the media to be delivered before such networks can use an adaptation process to convert the media into a format suitable for each target display and corresponding application. Such networks need access to information that describes the characteristics of each target display and the complexity of the ingested media, at a minimum, to ascertain how the network can meaningfully adapt the input media source to a format suitable for the target display and application.

[0006] Therefore, a method is needed for efficiently displaying and streaming heterogeneous immersive media to a variety of clients. Summary of the Invention [Means for solving the problem]

[0007] According to an embodiment, a method is provided for optimizing a system for delivering media assets for a set of at least one scene comprising an immersive media presentation, where the media assets are ordered based on media asset characteristics such as frequency and asset type.

[0008] According to one aspect of the disclosure, a method for packaging media to optimize immersive media delivery is provided, the method comprising: receiving immersive media data for an immersive presentation, obtaining asset information associated with media assets corresponding to a set of scenes included in the immersive media data for the immersive presentation, analyzing characteristics of the media assets based on the asset information, the characteristics including an asset type associated with each media asset and a frequency indicating a number of times each media asset is used in the set of scenes included in the immersive presentation, and ordering the media assets into a sequence based on the asset type and frequency associated with each of the media assets.

[0009] According to another aspect of the present disclosure, a device (or apparatus) for packaging media to optimize immersive media delivery is provided, the device (or apparatus) including at least one memory configured to store computer program code and at least one processor configured to read and operate as instructed by the computer program code. The computer program code includes: a receiving code configured to cause the at least one processor to receive immersive media data for an immersive presentation; an acquiring code configured to cause the at least one processor to acquire asset information associated with media assets corresponding to a set of scenes included in the immersive media data for the immersive presentation; an analyzing code configured to cause the at least one processor to analyze characteristics of the media assets based on the asset information, the analyzing code including an asset type associated with each media asset and a frequency indicating a number of times each media asset is used in the set of scenes included in the immersive presentation; and a sequencing code configured to cause the at least one processor to order the media assets into a sequence based on the asset type and frequency associated with each of the media assets.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable medium storing instructions executed by at least one processor of a device for optimizing immersive media delivery is provided, the instructions causing the at least one processor to receive immersive media data for an immersive presentation, obtain asset information associated with media assets corresponding to a set of scenes included in the immersive media data for the immersive presentation, analyze characteristics of the media assets based on the asset information, the characteristics including an asset type associated with each media asset and a frequency indicating a number of times each media asset is used in the set of scenes included in the immersive presentation, and order the media assets into a sequence based on the asset type and frequency associated with each of the media assets.

[0011] Additional embodiments will be set forth in the description that follows, and in part will be apparent from the description, and / or may be realized by practice of the disclosed embodiments. [Brief description of the drawings]

[0012] [Figure 1A] FIG. 2 is a schematic diagram of media flow through a network for delivery to a client according to an embodiment. [Figure 1B] 1 is a workflow illustrating a reuse logic decision-making process. [Figure 2A] FIG. 2 is a schematic diagram of media flow through a network in which a decision-making process is employed to determine whether the network should convert the media before delivering it to a client, according to an embodiment. [Figure 2B] 1 is a workflow illustrating a media transformation decision-making process using asset reuse logic, according to an embodiment. [Figure 3A] FIG. 2 is a schematic diagram of a data model for displaying and streaming timed immersive media, according to an embodiment. [Figure 3B]FIG. 2 is a schematic diagram of a data model for sequenced presentation and streaming of timed immersive media according to an embodiment. [Figure 4A] FIG. 2 is a schematic diagram of a data model for displaying and streaming non-timed immersive media, according to an embodiment. [Figure 4B] FIG. 2 is a schematic diagram of a data model for sequenced presentation and streaming of non-timed immersive media, according to an embodiment. [Diagram 5] FIG. 1 is a schematic diagram of a natural media synthesis process, according to an embodiment. [Figure 6] 2 is a schematic diagram of an example synthetic media ingest creation process according to an embodiment. [Figure 7] 1 is a schematic diagram of a computer system according to an embodiment. [Figure 8] 1 is a schematic diagram of a network media delivery system according to an embodiment. [Figure 9] FIG. 2 is a schematic diagram of an example workflow of an immersive media delivery process, according to an embodiment. [Figure 10] FIG. 1 is a system diagram of a media adaptation process system according to an embodiment. [Figure 11A] 2 is a schematic diagram of an exemplary delivery format creation process according to an embodiment. [Figure 11B] FIG. 1 is a schematic diagram of an exemplary delivery format creation process with frequency and asset type ordering according to an embodiment. [Figure 12] 4 is a schematic diagram of an exemplary packetization process according to an embodiment. [Figure 13] FIG. 2 is a sequence diagram illustrating an example of a communication flow between components according to an embodiment. [Figure 14A] 1 is a workflow illustrating a method of a media reuse analyzer according to an embodiment. [Figure 14B] 1 is an example of a set of lists of unique assets for a scene in a presentation, according to an embodiment. [Figure 15]FIG. 2 is a block diagram of an example of computer code for a media reuse analyzer according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The following detailed description of the exemplary embodiments refers to the accompanying drawings, in which the same reference numbers in different drawings can identify the same or similar elements.

[0014] The foregoing disclosure provides illustrations and descriptions, but is not intended to be exhaustive or to limit the implementations to the precise forms disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations. Moreover, one or more features or components of one embodiment may be incorporated or combined with another embodiment (or one or more features of another embodiment). Furthermore, in the flow charts and descriptions of operations provided below, it should be understood that one or more operations may be omitted, one or more operations may be added, one or more operations may occur (at least partially) simultaneously, and one or more operations may be reordered.

[0015] It may be apparent that the systems and / or methods described herein may be implemented in different forms of hardware, software, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not intended to limit the implementation. Thus, the operation and behavior of the systems and / or methods are described herein without reference to any specific software code. It is understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0016] Although particular feature combinations are recited in the claims and / or disclosed herein, these combinations are not intended to limit the disclosure of possible implementations. Indeed, many of these features may be combined in ways not specifically recited in the claims and / or disclosed herein. Although each dependent claim listed below may depend directly on only one claim, the disclosure of possible implementations includes each dependent claim in combination with all other claims in the claim set.

[0017] The proposed functions described below may be used separately or combined in any order. Furthermore, the embodiments may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium.

[0018] No element, act, or instruction used herein should be construed as critical or essential unless expressly described as such. Also, as used herein, the articles "a" and "an" are intended to include one or more items and may be used interchangeably with "one or more." When only one item is intended, the term "one" or similar words are used. Also, as used herein, terms such as "has," "have," "having," "include," "including," and the like are intended to be open-ended terms. Furthermore, the phrase "based on" is intended to mean "based at least in part on," unless otherwise noted. Furthermore, expressions such as "at least one of [A] and [B]" or "at least one of [A] or [B]" should be understood as including only A, only B, or both A and B.

[0019] Exemplary embodiments of the present disclosure provide methods and devices for parsing and transforming media assets based on the complexity of the media data for delivery to immersive media-enabled presentation devices. An immersive media-enabled presentation device can refer to being equipped with sufficient resources and capabilities to access, interpret, and present immersive media. Such devices are heterogeneous with respect to the amount and format of media (provided by the network) they can support. Similarly, media is heterogeneous with respect to the amount and type of network resources required to deliver such media on a large scale. "Large scale" can refer to delivery of media by a service provider that achieves delivery equivalent to legacy video and audio media over a network (e.g., Netflix, Hulu, Comcast subscription, Spectrum subscription, etc.). In contrast, legacy presentation devices such as laptop displays, televisions, and mobile handset displays are uniform in their capabilities since all of these devices are comprised of rectangular display screens that consume 2D rectangular video or still images as their primary visual media format. Some of the visual media formats commonly used by legacy presentation devices can include, for example, High Efficiency Video Coding / H.265, Advanced Video Coding / H.264, and Versatile Video Coding / H.266.

[0020] As mentioned above, client devices that act as endpoints for the delivery of immersive media over a network are all very diverse. Some of them support certain immersive media formats, while others do not. Some of them can create immersive experiences from older raster-based formats, while others cannot. To combat this issue, the delivery of any media over a network can employ media delivery systems and architectures that reformat the media from an input or network-ingested media format into a delivery media format that is not only suitable for ingestion by the target client device and its applications, but also lends itself to being streamed over the network. Thus, there may be two processes performed by the network using the ingested media: 1) a process of converting the media from format A to a format B that is suitable for ingestion by the target client device, i.e., based on the client device's ability to ingest a particular media format, and 2) a process of preparing the media to be streamed.

[0021] In embodiments, streaming of media refers broadly to fragmentation and / or packetization of media, whereby media can be delivered over a network in successive small-sized chunks that are logically organized and sequenced according to either or both of the media's temporal or spatial structure. Conversion of media from format A to format B (sometimes referred to as "transcoding") may be a process performed, typically by a network or service provider, prior to delivery of the media to a client device. Such transcoding may consist of converting media from format A to format B based on prior knowledge that format B is the preferred or only format that can be ingested by a target client device, or is more suitable for delivery over constrained resources, such as a commercial network. In many, if not all, steps of converting media and preparing the media to be streamed are necessary before the media can be received from the network and processed by the target client device.

[0022] Converting (or transforming) media and preparing media for streaming are process steps that operate on media captured by a network before delivering the media to a client device. The result of the processing (i.e., converting and preparing for streaming) is a media format called a delivery media format, or simply a delivery format. These steps, when performed for a given media data object, should be performed only once if the network has access to information indicating that a client needs a converted and / or streamed media object for multiple occasions to trigger conversion and streaming of such media multiple times. That is, the processing and transfer of data for converting and streaming media is generally considered a source of latency that requires consuming a potentially significant amount of network and / or computational resources. Thus, a network design that does not have access to information to indicate when a client may already have a particular media data object stored in its cache or stored locally to the client will perform suboptimally for a network that has access to such information.

[0023] An ideal network supporting heterogeneous clients should take advantage of the fact that some of the assets adapted from an input media format to a particular target format can be reused across a set of similar display targets. That is, once converted to a format appropriate for the target displays, some assets can be reused across several such displays having similar adaptation requirements. Thus, according to an embodiment, such an ideal network can use a caching mechanism to store adapted assets in a relative area, similar to the use of content delivery networks (CDNs) in legacy networks, for example.

[0024] Immersive media may be organized into scenes that are described by a scene graph, also known as a scene description. In an embodiment, a scene (in the context of computer graphics) is a collection of objects (e.g., 3D assets), object attributes, and other metadata including visual, acoustic, and physics-based properties that describe a particular setting bounded by either space or time with respect to the interaction of objects within that setting. A scene graph is a generic data structure commonly used by vector-based graphic editing applications and modern computer games to arrange a logical and often (but not necessarily) spatial representation of a graphic scene. A scene graph may be composed of a collection of nodes and vertices in a graph structure. A node may be composed of information regarding the logical, spatial, or temporal representation of visual, audio, tactile, olfactory, gustatory, or related processing information. Each node shall have at most one outgoing edge, zero or more incoming edges, and at least one edge (either incoming or outgoing) connected to it. Attributes or object attributes refer to metadata associated with a node that is used to describe a particular characteristic or feature of that node in either a prescriptive form (e.g., with respect to another node) or in a more complex form. The scope of a scene graph is to describe visual, audio, and other forms of immersive assets that are part of a presentation, e.g., a particular setting that is part of a presentation, e.g., actors and an event that takes place in a particular location within a building that is part of the presentation, such as a movie. A list of all the scenes that comprise a single presentation may be formulated in a scene manifest.

[0025] A further advantage of using a caching mechanism to store adapted assets is that a material statement can be created for pre-prepared content before the content needs to be delivered. The material statement identifies all assets used throughout the presentation and the frequency with which each asset is used among various scenes within the presentation. An ideal network should have knowledge of the existence of cached resources that can be used to meet the asset requirements of a particular presentation. Similarly, a client device presenting a series of scenes may want to have knowledge of the frequency with which any given asset is used across multiple scenes. For example, if a media asset (also known as a media object) is referenced multiple times across multiple scenes that are processed or processed by a client device, the client device should avoid discarding that particular asset from its cache resources until the last scene requiring that particular asset has been presented by the client device. In embodiments of the present disclosure, the terms media "object" and media "asset" can be used interchangeably and both refer to a particular instance of a particular format of media data.

[0026] For legacy media presentation devices, the delivery format may be equivalent or sufficiently equivalent to the "presentation format" that is ultimately used by the client presentation device to create the presentation. That is, the presentation media format is a media format whose properties (e.g., resolution, frame rate, bit depth, color gamut, etc.) are closely aligned to the capabilities of the client presentation device. An example of a delivery versus presentation format includes a high definition (HD) video signal (1920 pixel columns by 1080 pixel rows) delivered by a network to an ultra high definition (UHD) client device with a resolution (3840 pixel columns by 2160 pixel rows). In the example above, the UHD client device will apply super-resolution processing to the HD delivery format to increase the resolution of the video signal from HD to UHD. Thus, the final signal format presented by the client device is the "presentation format," which in this example is a UHD signal, but the HD signal includes the delivery format. In this example, the HD signal distribution format is very similar to the UHD signal presentation format since both signals are linear video formats, and the process of converting the HD format to the UHD format is relatively simple and easy to perform on most legacy media client devices.

[0027] In some embodiments, the preferred presentation format for the client device may be significantly different than the ingest format received by the network. Nevertheless, the client device may have access to sufficient computational, storage, and bandwidth resources to convert the media from the ingest format to the required presentation format suitable for presentation by the client device. In this scenario, the network may bypass the step of reformatting or transcoding the ingested media from format A to format B simply because the client device has access to sufficient resources to perform all media conversion without the network having to do so beforehand. However, the network may still perform the step of fragmenting and packaging the ingest media so that the media may be streamed over the network to the client device.

[0028] In some embodiments, the ingested media may differ significantly from the client's preferred presentation format, and the client device may not have access to sufficient computational, storage, and / or bandwidth resources to convert the media from the ingest format to the preferred presentation format. In such scenarios, the network may assist the client by performing some or all of the conversion on behalf of the client device from the ingest format to a format that is equivalent or nearly equivalent to the client's preferred presentation format. In some architectural designs, such assistance provided by the network on behalf of the client device is commonly referred to as split rendering.

[0029] FIG. 1A is a schematic diagram of a media flow process 100 through a network for delivery to a client, according to some embodiments. FIG. 1A illustrates an example of processing media in format A (hereinafter, “ingest media format A”). The processing (i.e., media flow process 100) may be performed or executed by a network cloud or edge device (hereinafter, “network device 104”) and delivered to a client, e.g., client device 108. In some embodiments, the same processing may be performed in advance by a manual process or a client device. Network device 104 may include an ingest media module 101, a network processing module 102 (hereinafter, “processing module 102”), and a delivery module 103. Client device 108 may include a rendering module 106 and a presentation module 107.

[0030] First, the network device 104 receives ingested media from a content provider or the like. The ingest media module 101 obtains the ingested media stored in an ingest media format A. The network processing module 102 performs any necessary conversion or adjustment of the ingested media to create potential alternative representations of the media. That is, the processing module 102 creates a delivery format for the media objects in the ingest media. As mentioned above, the delivery format is a media format that can be delivered to a client by formatting the media into a delivery format B. The delivery format B is a format that is ready to be streamed to the client device 108. The processing module 102 may include an optimization reuse logic 121 that performs a decision-making process to determine whether a particular media object has already been streamed to the client device 108. The operation of the processing module 102 and the optimization reuse logic 121 will be described in detail with reference to FIG. 1B.

[0031] Media formats A and B may or may not be representations that follow the same syntax of a particular media format specification, but format B is likely to be tailored in a manner that facilitates distribution of the media over a network protocol. The network protocol may be, for example, a connection-oriented protocol (TCP) or a connectionless protocol (UDP). The delivery module 103 streams the streamable media (i.e., media format B) from the network device 104 to the client device 108 over the network connection 105.

[0032] Client device 108 receives the delivered media and prepares the media for presentation as necessary via rendering module 106. Rendering module 106 has access to several rendering capabilities that may be rudimentary or similarly sophisticated depending on the client device 108 being targeted. Rendering module 106 creates presentation media in presentation format C. Presentation format C may or may not be displayed according to a third format specification. Thus, presentation format C may be the same or different from media formats A and / or B. Rendering module 106 outputs presentation format C to presentation module 107, which can present the presentation media within a display (or the like) of client device 108.

[0033] Embodiments of the present disclosure facilitate a decision-making process used by a network to calculate a sequence order for packaging and streaming assets from the network to clients, where all assets utilized across a set of one or more scenes that make up a presentation are analyzed by a media reuse analyzer to determine how frequently each asset is used across all scenes that make up the presentation. Thus, the order in which assets for a particular scene are packaged and streamed to clients can be based on how frequently each asset is used across the set of scenes that make up the presentation.

[0034] Embodiments address the need for a mechanism or process to analyze an immersive media scene to obtain sufficient information that can be used to support a decision-making process that provides instructions as to whether conversion of media objects from format A to format B, when used by a network or client, should be performed entirely by the network, entirely by the client, or via a mix of both (with instructions as to which assets should be converted by the client or network). Such an immersive media data complexity analyzer can be used by either the client or network in an automated context, or manually, for example by a human operating a system or device.

[0035] According to embodiments, the process of adapting an input immersive media source to a particular endpoint client device may be the same as or similar to the process of adapting the same input immersive media source to a particular application running on a particular client endpoint device. Thus, the problem of adapting an input media source to the characteristics of an endpoint device is of the same complexity as the problem of adapting a particular input media source to the characteristics of a particular application.

[0036] 1B is a workflow of the processing module 102, according to an embodiment. More specifically, the workflow of FIG. 1B is a decision-making process of the reuse logic executed by the optimization reuse logic 121 to assist in the decision-making process for determining whether a particular media object has already been streamed to the client device 108.

[0037] In S101, a distribution media creation process is executed. Thus, the reuse logic decision-making process also starts. In S102, a condition logic is executed to determine whether the current media object has already been streamed to the client device 108. The optimization reuse logic 121 can access a list of unique assets for the presentation to determine whether the media object has already been streamed to the client. If the current media object has already been streamed, the process proceeds to S103. In S103, an indicator (later also referred to as a "proxy") is created to identify that the client has already received the current media object and should access a copy of the media object from a local cache or other cache. The process then ends for the current media object (S104).

[0038] If it is determined that the media object has not been previously streamed, processing proceeds to S105, where the processing module 102 continues to prepare the media object for conversion and / or delivery, and a delivery format for the media object is created, after which processing proceeds to S104 and ends.

[0039] Figure 2A is a logical workflow for processing media ingested over a network. The workflow shown in Figure 2A illustrates a media conversion decision-making process 200 according to an embodiment. The media conversion decision-making process 200 is used to determine whether the network should convert the media before delivering it to a client device. The media conversion decision-making process 200 can be handled via a manual or automated process within the network.

[0040] Ingest media, represented in format A, is provided to the network by a content provider. In S201, media is ingested by the network from the content provider. Then, in S202, attributes of the target client are obtained, if not already known. The attributes describe the processing capabilities of the target client.

[0041] In S203, it is determined whether the network (or client) should support conversion of the ingested media. In particular, it is determined whether there is a format conversion (e.g., conversion of one or more media objects from format A to format B) for any of the media assets contained within the ingested media before the media is streamed to the target client. The decision-making process in S203 may be performed manually (i.e., by a device operator, etc.) or may be an automated process. The decision-making process in S203 may be based on a determination of whether the media can be streamed in its original ingested format A, or whether it must be converted to a different format B to facilitate presentation of the media by the client. Such a determination may require access to information describing aspects or characteristics of the ingested media to aid the decision-making process in making the optimal choice (i.e., to determine whether conversion of the ingested media is required before streaming the media to the client, or whether the media needs to be streamed directly to the client in its original ingested format A).

[0042] If it is determined that the network (or client) should support conversion of any media assets (YES at S203), the process 200 proceeds to S204.

[0043] In S204, the captured media is converted from format A to format B to generate converted media 205. The converted media 205 is output and the process proceeds to S206. In S206, the input media undergoes a process of preparing the media for streaming to a client. In this case, the converted media 205 (i.e., the input media) is prepared to be streamed.

[0044] Streaming of immersive media may be in a relatively early stage, especially when such media is "scene-based" rather than "frame-based." For example, streaming of frame-based media may be equivalent to streaming frames of video, where each frame captures a complete picture of the entire scene or object being presented by the client. The sequence of frames is reconstructed by the client from their compressed form and, when presented to a viewer, creates a video sequence that includes the entire immersive presentation or a portion of the presentation. In the case of frame-based streaming, the order in which frames are streamed from the network to the client may conform to a given specification (e.g., ITU-T Recommendation H.264 Advanced Video Coding for General Audio-Visual Services, etc.). However, scene-based streaming of media differs from frame-based streaming because a scene may be composed of individual assets that may themselves be independent of each other. A given scene-based asset may be used multiple times within a particular scene or across a series of scenes. The amount of time a client, or any given renderer, needs to spend to reconstruct a particular asset may depend on many factors, including but not limited to the size of the asset, the availability of computational resources to perform the rendering, and other attributes that describe the overall complexity of the asset. Clients that support scene-based streaming may require some or all of the rendering for each asset in a scene to be completed before any of the presentation of the scene can begin. Thus, the order in which assets are streamed from the network to clients can affect the overall performance of the system.

[0045] The conversion of media from format A to another format (e.g., format B) may be performed entirely by the network, entirely by the client, or jointly between both the network and the client. In split rendering, it becomes clear that a dictionary of attributes describing the media format may be needed so that both the client and the network have complete information characterizing the work that must be done. Furthermore, a dictionary providing attributes of the client's capabilities, e.g., in terms of available computational resources, available storage resources, and access to bandwidth, may be needed as well. Furthermore, a mechanism is needed to characterize the level of computational, storage, or bandwidth complexity of an ingest media format, so that the network and client together, or alone, can decide whether or when the network employs a split rendering process to deliver media to the client.

[0046] If it is determined that the network (or client) should not (or does not need to) support conversion of any media assets (NO at S203), the process 200 proceeds to S206, where the media is prepared for streaming. In this case, the captured data (i.e., the media in its original form) is prepared to be streamed.

[0047] Finally, once the media data is in a streamable format, the media prepared in S206 is streamed to the client (S207). In some embodiments, if conversion and / or streaming of certain media objects that are needed or required by the client to complete the presentation of the media (as described with reference to FIG. 1B) can be avoided, the network can skip converting and / or streaming of the ingested media (i.e., S204-S207), assuming the client still has access to or availability of the media objects that may be needed to complete the presentation of the media for the client. To facilitate the ability of a client to perform at its full potential, it may be desirable for the network to be equipped with sufficient information regarding the order in which scene-based assets are streamed from the network to the client so that the network can determine such an order to improve the performance of the client. For example, such a network that has sufficient information to avoid repetitive conversion and / or streaming steps of assets that are used multiple times in a particular presentation can perform more optimally than a network that is not designed in this way. Similarly, a network that can "intelligently" order the delivery of assets to a client can facilitate the ability of a client to perform at its full potential (i.e., to create a more enjoyable experience for the end user).

[0048] 2B is a media transformation decision-making process according to an embodiment of asset reuse logic 210. Similar to media transformation decision-making process 200, the media transformation decision-making process using asset reuse logic 210 processes ingested media through a network and uses two decision-making processes to determine whether the network should transform the media before delivering the media to a client.

[0049] Ingest media, represented in format A, is provided to the network by a content provider. Then, S221-222 are executed, which are similar to S201-S202 (shown in FIG. 2A). In S221, media is ingested by the network from the content provider. Next, in S202, attributes of the target client are obtained, if not already known. The attributes describe the processing capabilities of the target client.

[0050] If it is determined that the network has already streamed the particular media object or the current media object (YES at S223), the process proceeds to S224, where a proxy is created to replace the already streamed media object to indicate that the client should use a local copy of the already streamed object.

[0051] If it is determined that the network is not already streaming the particular media object (NO in S223), processing proceeds to S225. In S225, it is determined whether the network or the client should perform any format conversion on any of the media assets contained in the ingested media in S221. For example, the conversion may include converting the particular media object from format A to format B before the media is streamed to the client. The processing performed in S225 may be similar to the processing performed in SS203 shown in FIG. 2A.

[0052] If it is determined that the media asset should be converted by the network (YES at S225), proceed to S226. At S226, the media object is converted from format A to format B. The converted media is then prepared to be streamed to the client (S227).

[0053] If it is determined that the media asset should not be transformed by the network (NO in S225), proceed to S227. Next, in S227, the media object is prepared to be streamed to the client.

[0054] Finally, once the media data is in a streamable format, the media prepared in S227 is streamed to the client (S228). The processes performed in S225-S228 of Figure 2B may be similar to the processes performed in S203-S207 shown in Figure 2A.

[0055] The streamable format of media may be timed or non-timed heterogeneous immersive media. Figure 3A shows an example of a timed media presentation 300 of a streamable format of heterogeneous immersive media. Timed immersive media may include a set of N scenes. Timed media is media content ordered by time, e.g., with start and end times according to a particular clock. Figure 4 shows an example of a non-timed media presentation 400 of a streamable format of heterogeneous immersive media. Non-timed media is media content organized by spatial, logical, or temporal relationships (e.g., as in the case of an interactive experience realized according to actions taken by one or more users).

[0056] FIG. 3A illustrates a timed scene for timed media, and FIG. 4 illustrates a non-timed scene for non-timed media. Timed and non-timed scenes can be embodied by various scene representations or scene descriptions. Both FIGS. 3-4 employ a single exemplary enclosing media format adapted from a source ingest media format to match the capabilities of a particular client endpoint. That is, the enclosing media format is a delivery format that can be streamed to a client device. The enclosing media format is robust in its structure to accommodate a wide variety of media attributes, each of which can be layered based on the amount of salient information that each layer contributes to the presentation of the media.

[0057] As shown in FIG. 3A, the timed media presentation 300 includes a timed scene manifest 300A that includes a list of scene information 301. The scene information 301 indicates a list of components 302 that separately describe the types of media assets and processing information that make up the scene information 301, e.g., asset lists and other processing information. The list of components 302 points to proxy assets 308 that correspond to the type of asset (e.g., proxy visual and audio assets, as shown in FIG. 3A). The components 302 point to a list of unique assets that have not already been used in other scenes. For example, a list of unique assets 307 for (timed) scene 1 is shown in FIG. 3A. The components 302 also point to assets 303 that include a base layer 304 and an attribute extension layer 305. The base layer is a nominal representation of the assets that can be formulated to minimize computational resources, the time required to render the assets, and / or the time required to transmit the assets over a network. In this exemplary embodiment, each of the base layers 304 points to a numerical frequency metric that indicates the number of times the asset has been used across the set of scenes that comprise the presentation. An extension layer may be a set of information that, when applied to a base layer representation of an asset, extends the base layer to include features or capabilities that may not be supported in the base layer.

[0058] FIG. 3B shows a timed media display ordered by decreasing frequency 3030. This timed media display is the same display as shown in FIG. 3A, but the assets 3033 in FIG. 3B are ordered in a list by asset type and descending frequency values ​​within each asset type. The timed scene manifest 303A includes a list of scene information 3031. The list of scene information 3031 points to processing information including the list of scene information 3031 and a list of components 3032 that separately describe the type of media asset, e.g., asset lists and other processing information. The components 3032 can reference proxy assets 3038 that correspond to the type of asset (e.g., proxy visual and audio assets, as shown in FIG. 3B). The components 3032 point to assets 3033, which further points to base layers 3034 and attribute extension layers 3035. Each of the base layers 3034 is ordered according to descending values ​​in the corresponding frequency metric. A list of unique assets 3037 that have not already been used in other scenes is also provided.

[0059] As shown in FIG. 4, the untimed media and complexity display 400 includes scene information 401. Scene information 401 is not associated with a start and end time / duration (according to a clock, timer, etc.). An untimed scene manifest (not shown) can reference scene 1.0 where there are no other scenes that can branch into scene 1.0. Scene information 401 points to a list of components 402 that separately describe the types of media assets and processing information that make up scene information 401. Components 402 point to visual assets, audio assets, haptic assets, and timed assets (collectively referred to as assets 403). Assets 403 further point to base layer 404 and attribute extension layers 405 and 406. In this exemplary embodiment, each of base layers 404 points to a numerical frequency value that indicates the number of times an asset has been used across the set of scenes that make up the presentation. Scene information 401 may also point to other non-timed scenes (i.e., referenced in FIG. 4 as non-timed scenes 2.1-2.4) that are for non-timed media sources and / or scene information 407 that is for a timed media scene (i.e., referenced in FIG. 4 as timed scene 3.0). In the example of FIG. 4, the non-timed immersive media includes a set of five scenes (including both synchronized and non-timed). The list of unique assets 408 identifies unique assets associated with a particular scene that have not already been used in a higher-level (e.g., parent) scene. The list of unique assets 408 shown in FIG. 4 includes unique assets for non-timed scene 2.3.

[0060] 4B illustrates non-timed media and an ordered complexity display 4040. The non-timed scene manifest (not shown) references scene 1.0, which has no other scenes that can branch into it. Scene information 4041 does not have associated start and end times according to a clock. Scene information 4041 also points to a list of components 4042 that separately describe the types of processing information and media assets that make up scene information 4041. Components 4042 point to visual assets, audio assets, haptic assets, and timed assets (sometimes collectively referred to as assets 4043). Assets 4043 further point to a base layer 4044 and attribute extension layers 4045 and 4046. In this exemplary embodiment, haptic assets 4043 are organized by increasing frequency values ​​and audio assets 4043 are organized by decreasing frequency values. Scene information 4041 also points to other non-timed scene information 4041 for non-timed media. Scene information 4041 also points to other timed scene information 4047 for timed media. A list of unique assets 4048 identifies unique assets associated with a particular scene that have not already been used in a higher level (e.g., parent) scene.

[0061] Media streamed according to the embracing media format is not limited to legacy visual and audio media. The embracing media format can include any type of media information that can interact with a machine to generate signals that stimulate human vision, sound, taste, touch, and smell. As shown in Figures 3-4, media streamed according to the embracing media format can be timed media or non-timed media, or a mixture of both. The embracing media format is streamable by enabling a hierarchical representation of media objects using a base layer and extension layer architecture.

[0062] In some embodiments, the separate base and enhancement layers are computed by application of multi-resolution or multi-mosaicing analysis techniques to the media objects in each scene, which are not limited to raster-based visual formats.

[0063] In some embodiments, the progressive representation of the geometric object may be a multi-resolution representation of the object computed using wavelet analysis techniques.

[0064] In some embodiments, in a layered display media format, enhancement layers can apply different attributes to a base layer, for example, one or more enhancement layers can enhance the material properties of the surface of a visual object represented by the base layer.

[0065] In some embodiments, in a layered display media format, attributes can refine the texture of the surface of the object represented by the base layer, for example, by changing the surface from a smooth texture to a porous texture, or from a matte surface to a glossy surface.

[0066] In some embodiments, in a layered presentation media format, the surfaces of one or more visual objects in a scene may be changed from Lambertian surfaces to ray-traceable surfaces.

[0067] In some embodiments, in a layered representation media format, the network can deliver a base layer representation to a client so that the client can create a nominal presentation of a scene, and the client awaits the transmission of additional enhancement layers to refine the resolution or other characteristics of the base layer.

[0068] In embodiments, the resolution or refinement information of attributes in the enhancement layer is not explicitly coupled with the resolution of objects in the base layer. Furthermore, the enclosing media format can support any type of information media that can be presented or acted upon by a presentation device or machine, thereby enabling support of heterogeneous media formats to heterogeneous client endpoints. In some embodiments, the network delivering the media format first queries the client endpoint to determine the client's capabilities. If, based on the query, the client is unable to meaningfully capture the media representation, the network can remove layers of attributes that are not supported by the client. In some embodiments, if the client is unable to meaningfully capture the media representation, the network can adapt the media from its current format to a format appropriate for the client endpoint. For example, the network can adapt the media by converting a stereoscopic visual media asset into a 2D representation of the same viewing angle asset using a network-based media processing protocol. In some embodiments, the network can adapt the media by employing a neural network (NN) process to reformat the media into an appropriate format, or optionally by synthesizing a view required by the client endpoint.

[0069] A scene manifest for a complete (or partially complete) immersive experience (live streaming event, game, or on-demand asset playback) is organized by scenes that contain the minimum amount of information required to render and ingest to create a presentation. The scene manifest contains a list of individual scenes to be rendered for the entirety of the immersive experience requested by the client. Associated with each scene are one or more representations of the geometric objects in the scene that correspond to a streamable version of the scene geometry. An embodiment of a scene can point to a low-resolution version of the geometric objects of the scene. Another embodiment of the same scene can reference an enhancement layer for a low-resolution representation of the scene to add further detail of the geometric objects of the same scene or to increase mosaicking. As previously mentioned, each scene can have one or more enhancement layers to progressively increase the detail of the geometric objects of the scene. Each layer of a media object referenced in a scene can be associated with a token (e.g., a Uniform Resource Identifier (URI)) that points to an address of where the resource can be accessed in the network. Such resources are similar to a content delivery network (CDN) where content can be fetched by a client. The tokens for representations of geometric objects may point to locations within the network or within the client, i.e., the client may signal to the network that its resources are available to the network for network-based media processing.

[0070] According to an embodiment, a scene (timed or untimed) may be embodied by a scene graph as a multi-planar image (MPI) or a multi-spherical image (MSI). Both MPI and MSI techniques are examples of techniques that aid in the creation of display-independent scene representations for natural content (i.e., real-world images captured simultaneously from one or more cameras). Scene graph techniques, on the other hand, may be used to represent both natural and computer-generated images in the form of a synthetic representation. However, such representations are particularly computationally intensive to create when the content is captured as a natural scene by one or more cameras. Scene graph representations of naturally captured content are both time and computationally intensive to create, requiring complex analysis of the natural images using photogrammetry or deep learning or both techniques to create a synthetic representation that can later be used to interpolate a sufficient and appropriate number of views to fill the viewing frustum of the target immersive client display. As a result, such synthetic representations are impractical to consider as candidates for displaying natural content, since they cannot actually be created in real time to account for use cases that require real-time delivery. Therefore, the best representation of computer-generated images is to use a scene graph in the synthetic model. This is because computer-generated imagery is created using 3D modeling processes and tools, and using a scene graph in the synthetic model provides the best representation of the computer-generated imagery.

[0071] 5 illustrates an example of a natural media synthesis process 500, according to an embodiment. The natural media synthesis process 500 converts an ingest format from a natural scene into a representation that can be used as an ingest format for a network that serves heterogeneous client endpoints. To the left of the dashed line 510 is the content capture portion of the natural media synthesis process 500. To the right of the dashed line 510 is the ingest format synthesis (for natural images) of the natural media synthesis process 500.

[0072] As shown in FIG. 5, the first camera 501 captures a scene, for example of a person (i.e., the actor shown in FIG. 5), using a single camera lens. The second camera 502 captures a scene with five diverging fields of view by mounting five camera lenses around a ring-shaped object. The arrangement of the second camera 502 shown in FIG. 5 is an exemplary arrangement commonly used to capture omnidirectional content for VR applications. The third camera 503 captures a scene with seven converging fields of view by mounting seven camera lenses on the inner diameter of a sphere. The arrangement of the third camera 503 is an exemplary arrangement commonly used to capture a light field or a light field for a holographic immersive display. The embodiment is not limited to the configuration shown in FIG. 5. The second camera 502 and the third camera 503 may include multiple camera lenses.

[0073] Natural image content 509 is output from a first camera 501, a second camera 502, and a third camera 503 and is input to a synthesizer 504. The synthesizer 504 can use NN training 505, which uses a set of training images 506, to generate a capture NN model 508. The training images 506 can be predefined or stored from a previous synthesis process. A NN model (e.g., capture NN model 508) is a collection of parameters and tensors (e.g., matrices) that define weights (i.e., numerical values) used in well-defined mathematical operations that are applied to the visual signal to arrive at an improved visual output that can include the interpolation of new views of the visual signal that were not explicitly provided by the original signal.

[0074] In some embodiments, a photogrammetry process can be performed in place of NN training 505. If a capture NN model 508 is created during the natural media composition process 500, the capture NN model 508 becomes one of the assets in an ingest format 507 of the natural media content. The ingest format 507 may be, for example, MPI or MSI. The ingest format 507 may also include media assets.

[0075] 6 illustrates an example of a synthetic media ingest creation process 600 according to an embodiment. The synthetic media ingest creation process 600 creates an ingest media format for synthetic media, such as computer-generated images.

[0076] As shown in FIG. 6, a camera 601 can capture a point cloud 602 of a scene. The camera 601 can be, for example, a LIDAR camera. A computer 603 creates synthetic content (i.e., a representation of the synthetic scene that can be used as an ingest format for a network serving heterogeneous client endpoints) using, for example, a common gateway interface (CGI) tool, a 3D modeling tool, or another animation process. The computer 603 can create CGI assets 604 over the network. Additionally, a sensor 605A can be attached to an actor 605 in the scene. The sensor 605A can be, for example, a motion capture suit with sensors attached. The sensor 605A captures a digital recording of the actor 605's movements to generate animated motion data 606 (or MoCap data). Data from the point cloud 602, the CGI assets 604, and the motion data 606 are provided as inputs to a compositor 607 that creates a synthetic media ingest format 608. In some embodiments, the synthesizer 607 can use the NN and training data to create a NN model and generate the synthetic media ingest format 608.

[0077] Both natural and computer-generated (i.e., synthetic) content can be stored in a container. The container can include a serialization format for storing and exchanging information representing all-natural, all-synthetic, or a mix of synthetic and natural scenes, including a scene graph and all media resources required to render the scene. The content serialization process involves converting data structures or object states into a format that can be stored (e.g., in a file or memory buffer) or transmitted (e.g., over a network connection link) and later reconstructed in the same or a different computing environment. When the resulting string of bits is reread according to the serialized format, it can be used to create a semantically identical clone of the original object.

[0078] The dichotomy of optimal display of both natural and computer-generated (i.e., synthetic) content suggests that the optimal ingest format for naturally captured content will be different from the optimal ingest format for computer-generated content or natural content that is not mandatory for real-time delivery applications. Thus, according to embodiments, the network aims to be robust enough to support multiple ingest formats for visually immersive media, whether created naturally using, for example, physical cameras, or created by a computer.

[0079] Technologies such as ORBX by OTOY, Universal Scene Description by Pixar, and the Graphics Language Transmission Format 2.0 (glTF2.0) specification written by the Khronos 3D Group embody the scene graph as a format suitable for representing visually immersive media created using computer-generated techniques, or naturally captured content where deep learning or photogrammetry techniques are used to create corresponding synthetic representations of natural landscapes (i.e., not required for real-time delivery applications).

[0080] ORBX by OTOY is one of several scene graph technologies that can support any type of visual media, timed or untimed, including ray-traceable, legacy (frame-based), stereoscopic, and other types of synthetic or vector-based visual formats. ORBX is different from other scene graphs because it natively supports freely available and / or open source formats for meshes, point clouds, and textures. ORBX is a scene graph purposefully designed to facilitate exchange across multiple vendor technologies that operate on the scene graph. In addition, ORBX provides a rich material system, support for an open shader language, a robust camera system, and support for Lua scripting. ORBX is also the basis of an immersive technology media format released for license on royalty-free terms by the Immersive Digital Experience Alliance (IDEA). In the context of real-time delivery of media, the ability to create and deliver ORBX representations of natural scenes is a function of the availability of computational resources to perform complex analysis of the data captured by the cameras and compositing of that same data into a synthetic representation.

[0081] Pixar's USD is a scene graph widely used in visual effects and professional content creation. USD is incorporated into Nvidia's Omniverse platform, a developer toolset for 3D modeling and rendering with Nvidia's graphics processing units (GPUs). A subset of USD released by Apple and Pixar is called USDZ, which is supported by Apple's ARKit.

[0082] glTF2.0 is a version of the Graphics Language Transmission Format specification written by the Khronos 3D Group. The format supports simple scene graph formats, including PNG and JPEG image formats, that are generally capable of supporting static (non-timed) objects in a scene. glTF2.0 supports simple animation, including support for translation, rotation, and scaling of basic shapes, i.e., geometric objects, described using glTF primitives. glTF2.0 does not support timed media, and therefore does not support video or audio media input.

[0083] These designs for immersive visual media scene displays are provided by way of example only and do not limit the disclosed subject matter in its ability to specify a process for adapting an input immersive media source into a format suitable for the particular characteristics of a client endpoint device. Additionally, any or all of the above exemplary media displays may use or may employ deep learning techniques to train and create NN models that enable or facilitate the selection of specific views to fill a particular display's view frustum based on the particular dimensions of the frustum. The views selected for a particular display's view frustum may be interpolated from existing views explicitly provided in the scene display, for example from MSI or MPI techniques. Views may also be rendered directly from a rendering engine based on a particular virtual camera position, filter, or description of the virtual camera of these rendering engines.

[0084] The methods and devices of the present disclosure are robust enough to take into account that there is a relatively small but well-known set of immersive media ingest formats that can adequately meet the requirements for real-time or on-demand (e.g., non-real-time) delivery of media that is either captured naturally (e.g., using one or more cameras) or created using computer-generated techniques.

[0085] Interpolation of views from immersive media ingest formats through the use of either NN models or network-based rendering engines will become even easier as advanced network technologies (e.g., 5G for mobile networks) and fiber optic cables are deployed in fixed networks. These advanced network technologies will increase the capacity and capabilities of commercial networks, as such advanced network infrastructure can support the transmission and delivery of increasingly large amounts of visual information. Network infrastructure management technologies, such as multi-access edge computing (MEC), software-defined networking (SDN), and network function virtualization (NFV), will enable commercial network service providers to flexibly configure their network infrastructure to adapt to changes in demand for specific network resources, e.g., to respond to dynamic increases and decreases in demand for network throughput, network speed, round trip delay, and computational resources. Furthermore, this inherent ability to adapt to dynamic network requirements will likewise facilitate the network's ability to adapt immersive media ingest formats into appropriate delivery formats to support a variety of immersive media applications with potentially heterogeneous visual media formats for heterogeneous client endpoints.

[0086] Immersive media applications themselves may also have different requirements for network resources, including gaming applications that require extremely low network latency to respond to real-time updates on the state of the game, telepresence applications that have symmetric throughput requirements for both the uplink and downlink portions of the network, and passive viewing applications that may have increasing demands on downlink resources depending on the type of display of the client endpoint that is consuming the data. In general, any consumer application may be supported by a variety of client endpoints with different on-board client capabilities for storage, computation, and power, as well as different requirements for the particular media display.

[0087] Thus, the embodiments of the present disclosure enable a well-equipped network, i.e., a network that uses some or all of the characteristics of a modern network, to simultaneously support multiple legacy and immersive media-enabled devices according to the characteristics specified within the devices. Thus, the immersive media distribution method and process described herein provides the flexibility to leverage practical media ingest formats for both real-time and on-demand use cases for the distribution of media, the flexibility to support both natural and computer-generated content for both legacy and immersive media-enabled client endpoints, and the flexibility to support both timed and non-timed media. The method and process also dynamically adapts the source media ingest format to an appropriate distribution format based on the capabilities and capabilities of the client endpoints, and based on the requirements of the application. This ensures that the distribution format is streamable over an IP-based network, allowing the network to simultaneously serve multiple heterogeneous client endpoints, which may include both legacy and immersive media-enabled devices. Furthermore, the embodiments provide an exemplary media presentation framework that facilitates the organization of the distributed media along scene boundaries.

[0088] An end-to-end implementation of heterogeneous immersive media delivery according to an embodiment of the present disclosure that provides the aforementioned improvements is achieved in accordance with the processes and components described in the detailed description of Figures 3-16, which are described in further detail below.

[0089] The techniques for displaying and streaming heterogeneous immersive media described above may be implemented as computer software using computer readable instructions at both the source and destination, and may be physically stored on one or more non-transitory computer readable media, or by one or more specifically configured hardware processors. Figure 7 illustrates a computer system (700) suitable for implementing certain embodiments of the disclosed subject matter.

[0090] The computer software may be coded using any suitable machine code or computer language that may be subjected to mechanisms such as assembly, compilation, linking, etc. to produce code containing instructions that may be executed by a computer central processing unit (CPU), graphics processing unit (GPU), etc., directly, or via interpretation, microcode execution, etc.

[0091] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, and Internet of Things devices.

[0092] 7 for computer system 700 are exemplary in nature and are not intended to suggest any limitations as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The component configurations should not be construed as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system 700.

[0093] The computer system 700 may include certain human interface input devices that may respond to input by one or more human users, for example, via tactile input (such as keystrokes, swipes, data glove movements), audio input (such as voice, clapping), visual input (such as gestures), or olfactory input. Human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as sound (such as speech, music, ambient sounds, etc.), images (such as scanned images, photographic images obtained from a still image camera, etc.), and video (such as two-dimensional video, three-dimensional video including stereoscopic video, etc.).

[0094] The input human interface devices may include one or more of a keyboard 701, a trackpad 702, a mouse 703, a screen 709 (which may be, for example, a touch screen, a data glove, a joystick 704, a microphone 705, a camera 706, and a scanner 707) (only one of each is shown).

[0095] The computer system 700 may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include tactile output devices (e.g., haptic feedback via a screen 709, data gloves, or joystick 704, although there may also be haptic feedback devices that do not function as input devices), audio output devices (e.g., speakers 708, headphones), visual output devices (such as screens 709, including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability, some of which may output two-dimensional visual output or three-dimensional or more output via means such as stereo output, virtual reality glasses, holographic displays, and smoke tanks), and printers.

[0096] The computer system 700 may also include human accessible storage devices and their associated media such as optical media including CD / DVDROM / RW 711 having media 710 such as CDs / DVDs, thumb drives 712, removable hard drives or solid state drives 713, legacy magnetic media such as tapes and floppy disks, dedicated ROM / ASIC / PLD based devices such as security dongles.

[0097] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitory signals.

[0098] The computer system 700 may also include an interface 715 to one or more communication networks 714. The network 714 may be, for example, wireless, wired, optical. The network 714 may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of the network 714 include local area networks such as Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicular and industrial including CANBus, etc. Certain networks 714 typically require an external network interface adapter (e.g., graphics adapter 725) attached to a particular general-purpose data port (e.g., a USB port of the computer system 700) or peripheral bus 716, while other networks are typically built into the core of the computer system 700 by attachment to a system bus described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks 714, computer system 700 may communicate with other entities. Such communications may be unidirectional, receive only (e.g., broadcast TV), transmit only unidirectional (e.g., CANbus to a particular CANbus device), or bidirectional, for example, to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks may be used with each of these networks and network interfaces, as described above.

[0099] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to core 717 of computer system 700 .

[0100] The core 717 may include one or more central processing units (CPUs) 718, graphics processing units (GPUs) 719, dedicated programmable processing units in the form of field programmable gate areas (FPGAs) 720, hardware accelerators 721 for specific tasks, etc. These devices may be connected via a system bus 726 along with read only memory (ROM) 723, random access memory (RAM) 724, internal mass storage 722 such as an internal non-user accessible hard drive, SSD, etc. In some computer systems, the system bus 726 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus 726 or via the peripheral bus 716. Architectures for peripheral buses include PCI, USB, etc.

[0101] The CPU 718, GPU 719, FPGA 720, and accelerator 721 may execute certain instructions that, in combination, may constitute the aforementioned machine code (or computer code). That computer code may be stored in ROM 723 or RAM 724. Transient data may also be stored in RAM 724, while permanent data may be stored, for example, in internal mass storage 722. Rapid storage and retrieval in any of the memory devices may be enabled by the use of cache memories, which may be closely associated with one or more of the CPU 718, GPU 719, mass storage 722, ROM 723, RAM 724, etc.

[0102] The computer-readable medium may bear computer code for performing various computer-implemented operations. The medium and computer code may be specially designed and constructed for the purposes of the present disclosure, or they may be of a type well known and available to those skilled in the computer software arts.

[0103] By way of example and not limitation, a computer system having the architecture of computer system 700, and in particular core 717, may provide functionality as a result of a processor (including CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be user-accessible mass storage devices as previously described, as well as media associated with a particular storage device of core 717 that is of a non-transitory nature, such as core internal mass storage 722 or ROM 723. Software implementing various embodiments of the present disclosure may be stored in such devices and executed by core 717. Computer-readable media may include one or more memory devices or chips, depending on particular needs. The software may cause core 717, and in particular the processors therein (including CPU, GPU, FPGA, etc.) to perform certain processes, or certain portions of certain processes, described herein, including defining data structures stored in RAM 724 and modifying such data structures according to processes defined by the software. Additionally, or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator 721), which may operate in place of or in conjunction with software to perform a particular process, or a particular portion of a particular process, described herein. References to software may encompass logic, and vice versa. References to computer-readable media may encompass circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry embodying logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software.

[0104] The number and arrangement of components shown in Figure 7 are provided as an example. In practice, an input human interface device may include additional components, fewer components, different components, or components arranged differently than those shown in Figure 7. Additionally or alternatively, a set of components (e.g., one or more components) of an input human interface device may perform one or more functions that are described as being performed by another set of components of the input human interface device.

[0105] In an embodiment, any one of the operations or processes of FIGS. 1-6 and 8-15 may be performed by or using any one of the elements shown in FIG.

[0106] 8 illustrates an example network media distribution system 800 serving multiple heterogeneous client endpoints. That is, the system 800 supports a variety of legacy and heterogeneous immersive media-enabled displays as client endpoints. The system 800 can include a content acquisition module 801, a content preparation module 802, and a transmission module 803.

[0107] The content acquisition module 801 captures or creates source media using, for example, the embodiments described in FIG. 6 and / or FIG. 5. The content preparation module 802 creates ingest formats that are sent to the network media distribution system using the transmission module 803. The gateway 804 can service customer premises equipment to provide network access to various client endpoints for the network. The set-top box 805 can also function as customer premises equipment to provide access to aggregated content by the network service provider. The wireless demodulator 806 can function as a mobile network access point for mobile devices, for example, as shown by the mobile handset display 813. In this particular embodiment of the system 800, a legacy 2D television 807 is shown as directly connected to one of the gateway 804, the set-top box 805, or the WiFi (router) 808. A laptop 2D display 809 (i.e., a computer or laptop with a legacy 2D display) is shown as a client endpoint connected to the WiFi (router) 808. A head mounted 2D (raster based) display 810 is also connected to the WiFi (router) 808. A lenticular light field display 811 is shown connected to one of the gateways 804. The lenticular light field display 811 can include one or more GPUs 811A, a storage device 811B, and a visual presentation component 811C that uses ray based lenticular optics technology to create multiple views. A holographic display 812 is shown connected to a set top box 805. The holographic display 812 can include one or more CPUs 812A, a GPU 812B, a storage device 812C, and a visualization component 812D. The visualization component 812D can be a Fresnel pattern, wave based holographic device / display. An augmented reality (AR) headset 814 is shown connected to the wireless demodulator 806.The AR headset 814 can include a GPU 814A, a storage device 814B, a battery 814C, and a volumetric visual presentation component 814D. A high density light field display 815 is shown connected to a WiFi (router) 808. The high density light field display 815 can include one or more GPUs 815A, a CPU 815B, a storage device 815C, an eye tracking device 815D, a camera 815E, and a high density beam-based light field panel 815F.

[0108] The number and arrangement of components shown in Figure 8 are provided as an example. In practice, system 800 may include additional components, fewer components, different components, or components in a different arrangement than that shown in Figure 8. Additionally or alternatively, a set of components (e.g., one or more components) of system 800 may perform one or more functions that are described as being performed by another set of components of the device or respective display.

[0109] Figure 9 illustrates an example workflow of an immersive media delivery process 900 that can serve legacy and heterogeneous immersive media enabled displays as previously described in Figure 8. The immersive media delivery process 900 performed by the network can provide adaptation information about particular media represented in a media ingest format, for example, prior to the network's process of adapting the media for consumption by a particular immersive media client endpoint (as described with reference to Figure 10).

[0110] The immersive media delivery process 900 can be divided into two parts: immersive media production to the left of dashed line 912, and immersive media network delivery to the right of dashed line 912. The immersive media production and immersive media network delivery may be performed by a network or a client device.

[0111] First, media content 901 is created or acquired from a network (or client device) or content source, respectively. Methods for creating or acquiring data are embodied, for example, in Figures 5 and 6 for natural and synthetic content, respectively. The created content 901 is then converted to an ingest format using a network ingest format creation process 902, which is also embodied in Figures 5 and 6 for natural and synthetic content, respectively. The ingest format may also be updated to store information about assets potentially reused across multiple scenes, for example from a media reuse analyzer 911 (discussed in more detail below with reference to Figures 10 and 14A). The ingest format is sent to the network and stored in an ingest media storage 903 (i.e., a storage device). In some embodiments, the storage device may be within the immersive media content creator's network or may be accessed remotely for immersive media network distribution 920. Client and application specific information is optionally available in a remote storage device, client specific information 904. In some embodiments, the client specific information 904 may reside remotely in an alternative cloud network and may be transmitted to the network.

[0112] Then, the network orchestrator 905 is executed. The network orchestrator acts as a central source and sink of information to perform the primary tasks of the network. The network orchestrator 905 may be implemented in a unified manner with other components of the network. The network orchestrator 905 may be a process that further uses a two-way message protocol with the client devices to facilitate all processing and delivery of the media according to the characteristics of the client devices. Furthermore, the two-way protocol may be implemented across different delivery channels (e.g., control plane channels and / or data plane channels).

[0113] 9, the network orchestrator 905 receives information regarding the characteristics and attributes of a client device 908. The network orchestrator 905 gathers requirements regarding applications currently running on the client device 908. This information may be obtained from client specific information 904. In some embodiments, the information may be obtained by querying the client device 908 directly. When a client device is directly queried, it is assumed that a two-way protocol exists and is operational so that the client device 908 can communicate directly with the network orchestrator 905.

[0114] The network orchestrator 905 may also initiate and communicate with the media adaptation and fragmentation module 910 (which is described in FIG. 10). Once the ingested media has been adapted and fragmented by the media adaptation and fragmentation module 910, the media may be transferred to an inter-media storage device such as media prepared for distribution 909. Once the distributed media is prepared and stored in a storage device that is media prepared for distribution 909, the network orchestrator 905 ensures that the client device 908 can receive the distributed media and description information 906 via a "push" request or the client device 908 can initiate a "pull" request for the distributed media and description information 906 from the stored media prepared for distribution 909. Information may be "pushed" or "pulled" through the network interface 908B of the client device 908. The "pushed" or "pulled" distributed media and description information 906 may be description information corresponding to the distributed media.

[0115] In some embodiments, the network orchestrator 905 uses a two-way message interface to perform "push" requests or initiate "pull" requests with the client devices 908. The client devices 908 can optionally use a GPU 908C (or a CPU).

[0116] The distributed media format is then stored in a storage device or storage cache 908D located on the client device 908. Finally, the client device 908 visually presents the media via a visualization component 908A.

[0117] Throughout the process of streaming immersive media to client devices 908, the network orchestrator 905 monitors the client's progress via a client progress and status feedback channel 907. In some embodiments, the status monitoring may be performed via a two-way communication message interface.

[0118] 10 illustrates an example of a media adaptation process 1000, which may be performed, for example, by the media adaptation and fragmentation module 910. By performing the media adaptation process 1000, the captured source media may be appropriately adapted to match the requirements of a client (e.g., client device 908).

[0119] As shown in Figure 10, media adaptation process 1000 includes multiple components that facilitate adaptation of ingested media to an appropriate delivery format for client device 908. The components shown in Figure 10 should be considered as exemplary. In practice, media adaptation process 1000 may include additional components, fewer components, different components, or components in a different arrangement than those shown in Figure 10. Additionally or alternatively, a set of components (e.g., one or more components) of media adaptation process 1000 may perform one or more functions described as being performed by another set of components.

[0120] In Fig. 10, the adaptation module 1001 receives an input of network state 1005 to track the current traffic load on the network. As mentioned above, the adaptation module 1001 also receives information from the network orchestrator 905. The information may include a description of the attributes and capabilities of the client device 908, the capabilities and descriptions of the application, the current state of the application, and the client NN model (if available) to help map the geometry of the client frustum to the interpolation capabilities of the ingested immersive media. Such information may be obtained by a two-way message interface. The adaptation module 1001 ensures that the adapted output, once generated, is stored in a storage device for storing the client adapted media 1006.

[0121] The media reuse analyzer 911 may be an optional process that may be run in advance or as part of a network automated process for the distribution of media. The media reuse analyzer 911 may store the ingest media formats and assets in a storage device (1002). The ingest media formats and assets may then be transmitted from the storage device (1002) to the adaptation module 1001.

[0122] The adaptation module 1001 may be controlled by a logic controller 1001F. The adaptation module 1001 may also use a renderer 1001B or a processor 1001C to adapt the particular ingest source media to a format suitable for the client. The processor 1001C may be a NN-based processor. The processor 1001C uses the NN model 1001A. Such a processor 1001C may include, for example, the Deepview NN model generator as described in MPI and MSI. If the media is in a 2D format but the client must have a 3D format, the processor 1001C may invoke a process that uses highly correlated images from the 2D video signal to derive a stereoscopic representation of the scene depicted in the media.

[0123] The renderer 1001B may be a software-based (or hardware-based) application or process based on a selective mix of fields related to acoustic physics, optical physics, visual perception, audio perception, mathematics, and software development, which, given an input scene graph and asset containers, emits (usually) visual and / or audio signals suitable for presentation on a target device or conforming to desired characteristics specified by attributes of the render target nodes in the scene graph. In the case of visual-based media assets, the renderer may emit visual signals suitable for target display or suitable for storage as intermediate assets (e.g., repackaged in another container and used in the sequence of rendering processes in the graphics pipeline). In the case of audio-based media assets, the renderer may emit audio signals for presentation on multi-channel loudspeakers and / or binauralized headphones or for repackaging in another (output) container. Renderers include, for example, real-time rendering capabilities of source and cross-platform game engines. The renderer may include a scripting language (i.e., an interpreted programming language) that can be executed by the renderer at run-time to process dynamic inputs and variable state changes made to the scene graph nodes. Dynamic inputs and variable state changes can affect the rendering and evaluation of spatial and temporal object topology (including physical forces, constraints, inverse kinematics, deformations, collisions), as well as energy propagation and transport (light, sound). The evaluation of spatial and temporal object topology produces results that move the output from the abstract to the concrete (e.g., similar to the evaluation of a document object model for a web page).

[0124] Renderer 1001B may be, for example, a modified version of the OTOY Octane renderer that is modified to interact directly with adaptation module 1001. In some embodiments, renderer 1001B implements computer graphics methods (e.g., path tracing) that render three-dimensional scenes such that the lighting of the scene is realistic. In some embodiments, renderer 1001B may use shaders (i.e., a type of computer program that was originally used for shading (producing appropriate levels of light, darkness, and color in an image) but now performs various specialized functions in various areas of computer graphics special effects, video post-processing unrelated to shading, and other functions unrelated to graphics).

[0125] The adaptation module 1001 may perform compression and decompression of the media content using a media compressor 1001D and a media decompressor 1001E, respectively, depending on the compression and decompression needs based on the format of the ingested media and the format required by the client device 908. The media compressor 1001D may be a media encoder, and the media decompressor 1001E may be a media decoder. After performing compression and decompression (if necessary), the adaptation module 1001 outputs client adapted media 1006 suitable for streaming or delivery to the client device 908. The client adapted media 1006 may be stored in a storage device for storing adapted media.

[0126] FIG. 11 illustrates an example of a delivery format creation process 1100. As shown in FIG. 11, the delivery format creation process 1100 includes an adaptation media packaging module 1103 that packages the media output from the media adaptation process 1000 and stored as client adaptation media 1006. The media packaging module 1103 formats the adaptation media from the client adaptation media 1006 into a robust delivery format 1104. The delivery format may be, for example, the format shown in FIG. 3 or FIG. 4. The information manifest 1104A may provide the client device 908 with a list of scene data assets 1104B. The list of scene data assets 1104B may also include metadata describing how frequently each asset is used across the set of scenes that make up the presentation. The list of scene data assets 1104B illustrates a list of visual assets, audio assets, and haptic assets, each with corresponding metadata. In this exemplary embodiment, each asset in the list of scene data assets 1104B references metadata that includes a numerical frequency value indicating the number of times a particular asset is used across all the scenes that make up the presentation.

[0127] FIG. 11B illustrates a delivery format creation using an ordered frequency process 1110. The delivery format creation using an ordered frequency process 1110 includes similar components as those of FIG. 11A. Accordingly, it is described again herein with reference to FIG. 11B. The media packaging module 11043 is similar to the media packaging module 1103. However, the media packaging module 11043 orders the assets in the list of scene data assets 11044B based on frequency. For example, the visual assets in the list of scene data assets 11044B may be ordered by increasing frequency, and the audio and haptic assets in the list of scene data assets 1104B may be ordered by decreasing frequency. As shown in FIG. 11B, the assets in the delivery format 11044 are ordered first by asset type, and then by how frequently the assets are used throughout the presentation, for example, in either ascending or descending frequency values ​​based on asset type (i.e., visual, haptic, audio, etc.). The information manifest 11044A provides the client device 908 with a list of scene data assets 11044B that it can expect to receive, along with optional metadata indicating how frequently all of the assets are used across the set of scenes that make up the entire presentation. The list of scene data assets 1104B includes a list of visual assets, audio assets, and haptic assets, each with corresponding metadata.

[0128] The media may be further packetized before streaming. Figure 12 illustrates an exemplary packetization process 1200. The packetization system 1200 includes a packetizer 1202. The packetizer 1202 may receive a list of scene data assets 1104B (or 11044B) (as shown in Figure 12) as input media 1201. In some embodiments, the client adapted media 1006 or the delivery format 1104 is input to the packetizer 1202. The packetizer 1202 separates the input media 1201 into individual packets 1203 suitable for presentation or streaming to a client device 908 over a network.

[0129] 13 is a sequence diagram illustrating an example of data and communication flow between components according to an embodiment. The sequence diagram of FIG. 13 is of a network adapting a particular immersive media in an ingest format into a streamable and appropriate delivery format for a particular immersive media client endpoint. The data and communication flow is as follows:

[0130] The client device 908 initiates a media request 1308 to the network orchestrator 905. In some embodiments, the request may be made to a network delivery interface of the client device. The media request 1308 includes information to identify the media requested by the client device 908. The media request may be identified, for example, by a uniform resource name (URN) or another standard nomenclature. The network orchestrator 905 then responds to the media request 1308 with a profile request 1309. The profile requests the client 1309 to provide information about currently available resources, including computation, storage, battery charge, and other information to characterize the client's current operating state. The profile request 1309 also requests the client to provide one or more NN models that may be used by the network for NN inference to extract or interpolate the correct media view to match the characteristics of the client's presentation system, if such NN models are available at the client endpoint.

[0131] The client device 908 then sends a response 1310 from the client device 908 to the network orchestrator 905 provided as a client token, an application token, and one or more NN model tokens (if such NN model tokens are available at the client endpoint). The network orchestrator 905 then provides a session ID token 1311 to the client device. The network orchestrator 905 then requests ingest media 1312 from the ingest media server 1303. The ingest media server 1303 may include, for example, the ingest media storage 903 or the ingest media format and assets of the storage device 1002. The request for ingest media 1312 may also include a URN or other standard name for the media identified in the request 1308. The ingest media server 1303 responds to the ingest media 1312 request with a response 1313 that includes the ingest media tokens. The network orchestrator 905 then provides the media token from the response 1313 to the client device 908 in a call 1314. The network orchestrator 905 then initiates the adaptation process for the requested media by providing the ingest media token, the client token, the application token, and the NN model token to the adaptation and fragmentation module 910 in a request 1315. The adaptation and fragmentation module 910 requests access to the ingest media by providing the ingest media token to the ingest media server 1303 in a request 1316 requesting access to the ingest media asset.

[0132] The ingest media server 1303 responds to the request 1316 with the ingest media access token in a response 1317 to the adaptation and fragmentation module 910. The adaptation and fragmentation module 910 then requests that the media adaptation process 1000 adapt the ingest media located in the ingest media access token for the client, application, and NN inference model corresponding to the session ID token created and sent in the response 1313. A request 1318 is made from the adaptation and fragmentation module 910 to the media adaptation process 1000. The request 1318 includes the necessary token and session ID. The media adaptation process 1000 provides the adapted media access token and session ID to the network orchestrator 905 in an update response 1319. The network orchestrator 905 then provides the adapted media access token and session ID to the media packaging module 11043 in an interface call 1320. The media packaging module 11043 provides a response 1321 to the network orchestrator 905 with the packaged media access token and the session ID in response 1321. The media packaging module 11043 then provides the packaged assets for the session ID, the URN, and the packaged media access token in response 1322 to the packaged media server 1307 where they are stored. The client device 908 then performs a request 1323 to the packaged media server 1307 to begin streaming the media assets corresponding to the packaged media access token received in response 1321. Finally, the client device 908 performs other requests and provides a status update in message 1324 to the network orchestrator 905.

[0133] Figure 14A illustrates a workflow for the media reuse analyzer 911 shown in Figure 9. The media reuse analyzer 911 analyzes metadata regarding the uniqueness of objects in scenes contained in the media data.

[0134] In S1401, media data is obtained, for example, from a content provider or a content source. In S1402, initialization is performed. Specifically, an iterator "i" is initialized to 0. The iterator may be, for example, a counter. A set of lists of unique assets 1420 (shown in FIG. 14B) for each scene is also initialized, which identify the unique assets that occur in all the scenes that comprise the presentation (as shown in FIG. 3 and / or FIG. 4).

[0135] In S1403, a decision process is performed to determine whether the value of iterator "i" is less than the total number N of scenes that contain the presentation. If the value of iterator "i" is equal to (or greater than) the number N of scenes that contain the presentation (NO in S1403), processing proceeds to S1404, where the reuse analysis ends (i.e., the process ends). If the value of iterator "i" is less than the number N of scenes that contain the presentation (YES in S1403), processing proceeds to S1405. In S1405, the value of iterator "j" is set to 0.

[0136] Then, in S1406, a decision process is performed to determine whether the value of iterator "j" is less than the total number X of media assets (also called media objects) in the current scene. If the value of iterator "j" is equal to (or greater than) the total number X of media assets in scene s (NO in S1406), processing proceeds to S1407, where iterator "i" is incremented by 1, and then processing returns to S1403. If the value of iterator "j" is less than the total number X of media assets in scene s (YES in S1406), processing proceeds to S1408.

[0137] At S1408, the characteristics of the media asset are compared to assets already analyzed from a scene prior to the current scene (ie, scene s) to determine if the current media asset has already been used.

[0138] If the current media asset is identified as a unique asset (NO in S1408), i.e., the current media asset has not already been parsed in a scene associated with a smaller iterator "i" value, processing proceeds to S1409. At S1409, a unique asset entry is created in the set of lists of unique assets 1420 that correspond to the current scene (i.e., scene s). The unique asset entry is also assigned a unique identifier, and the number of times (e.g., frequency) the asset has been used across scenes 0 to N-1 is set to 1. Processing then proceeds to S1411.

[0139] If the current media asset is identified as an asset that has been used in one or more scenes prior to scene s (YES in S1408), processing proceeds to S1409. In S1410, the number of times the current media asset has been used across scenes 0 to N-1 is incremented by 1 in the set 1420 of lists of unique assets corresponding to the current scene (i.e., scene s). Processing then proceeds to S1411.

[0140] In S1411, the value of the iterator "j" is incremented by 1. After that, the process returns to S1406.

[0141] In some embodiments, the media reuse analyzer 911 can further signal to a client, e.g., a client device 108, that the client should use a copy of the asset for each instance (after the first time the asset is delivered to the client) that the asset is used in a set of scenes.

[0142] It should be noted that the sequence diagrams and workflow steps described with reference to Figures 13-14 are not intended to limit the configuration of data and communication flows in embodiments. For example, one or more of the steps may be performed simultaneously, data may be stored, and / or flow in a direction not explicitly shown in the flows of Figures 13-14A, etc.

[0143] 14B is an example of a list of unique assets 1420 initialized in S1402 (and potentially updated in S1409-S1410) for all scenes upon completion of a presentation, according to an embodiment. The list of unique assets in the set of lists of unique assets 1420 may be pre-identified or pre-defined by the network or client device. The set of lists of unique assets 1420 shows a sample list of entries of information describing assets that are unique to the entire presentation, including an indicator of the type of media (e.g., mesh, audio, or volume) that the asset comprises, a unique identifier for the asset, and the number of times the asset is used across the set of scenes that comprise the entire presentation. As an example, for scene N-1, no assets are included in the list because all assets required for scene N-1 have been identified as assets that are also used in scene 1 and scene 2.

[0144] 15 is a block diagram of an example of computer code 1500 for packaging media to optimize immersive media delivery, according to an embodiment. In an embodiment, the computer code may be, for example, program code or computer program code. According to an embodiment of the present disclosure, an apparatus / device may be provided that includes at least one processor having a memory that stores the computer program code. The computer program code, when executed by the at least one processor, may be configured to perform any number of aspects of the present disclosure.

[0145] As shown in FIG. 15, computer code 1500 can include receiving code 1510, acquiring code 1520, parsing code 1530, sequencing code 1540, and streaming code 1550.

[0146] The receiving code 1510 is configured to cause the at least one processor to receive immersive media data for an immersive presentation.

[0147] The obtaining code 1520 is configured to cause at least one of the processors to obtain asset information associated with a media asset corresponding to a set of scenes included in the immersive media data for the immersive presentation.

[0148] The analysis code 1530 is configured to cause the at least one processor to analyze characteristics of the media assets based on the asset information, the characteristics including an asset type associated with each media asset and a frequency indicating the number of times each media asset is used within a set of scenes included in the immersive presentation.

[0149] The sequencing code 1540 is configured to cause the at least one processor to order the media assets into a sequence based on an asset type and a frequency associated with each media asset.

[0150] The streaming code 1550 is configured to cause the at least one processor to stream the immersive media data for an immersive presentation based on the ordered sequence of the media assets.

[0151] Although Figure 15 shows example blocks of code, in some implementations an apparatus / device may include additional, fewer, different, or differently arranged blocks than those shown in Figure 15. Additionally or alternatively, two or more of the blocks of an apparatus / device may be combined. In other words, although Figure 15 shows separate blocks of code, the various code instructions need not be separate and may be intermixed.

[0152] While this disclosure describes several exemplary embodiments, there are modifications, permutations, and various substitute equivalents that fall within the scope of this disclosure. Thus, it can be appreciated that those skilled in the art can devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope. [Explanation of symbols]

[0153] 100 Media Flow Process 101 Ingest Media Module 102 Processing Module 103 Distribution Module 104 Network Devices 105 Network Connection 106 Rendering Module 107 Presentation Module 108 client devices 121 Optimization Reuse Logic 200 Media conversion decision-making process 205 Media 210 Asset Reuse Logic 300 Timed Media Display 300A Timed Scene Manifest 301 Scene Information 302 Components 303 Assets 303A Timed Scene Manifest 304 Base Layer 305 Attribute Extension Layer 307 Unique Assets 308 Proxy Assets 400 Non-Timed Media and Complexity Indicators 401 Scene Information 402 Components 403 Assets 404 Base Layer 405 Attribute Extension Layer 407 Scene Information 408 Unique Assets 500 Natural Media Synthesis Process 501 First Camera 502 Second Camera 503 Third Camera 504 Synthesizer 505 Neural Network (NN) Training 506 Training Images 507 Ingest Format 508 Capture NN model 509 Natural Image Content 510 dashed line 600 Synthetic Media Ingest Creation Process 601 Camera 602 point cloud 603 Computer 604 Common Gateway Interface (CGI) Assets 605 Actor 605A Sensor 606 Movement Data 607 Synthesizer 608 Synthetic Media Formats 648 System Bus 700 Computer Systems 701 Keyboard 702 Trackpad 703 Mouse 704 Joystick 705 Microphone 706 Camera 707 Scanner 708 Speaker 709 Screen 710 CD / DVD and other media 712 Thumb Drive 713 Removable Hard Drive or Solid State Drive 714 Network 715 Network Interface 716 Surrounding Bus 717 Cores 718 Computer Central Processing Unit (CPU) 719 Graphics Processing Unit (GPU) 720 Field Programmable Gate Area (FPGA) 721 Hardware Accelerator 722 Internal Mass Storage 723 Read-Only Memory (ROM) 724 Random Access Memory (RAM) 725 Graphics Adapter 726 System Bus 800 Network Media Distribution System 801 Content Acquisition Module 802 Content Preparation Module 802 Transmitting Module 804 Gateway 805 Set-Top Box 806 Radio Demodulator 807 Legacy Two-Dimensional (2D) Television 808 WiFi (router) 809 Laptop 2D Display 810 Head-mounted 2D display 811 Lenticular Light Field Display 811A GPU 811B Storage Device 811C Visual Presentation Components 812 Holographic Display 812A CPU 812B GPU 812C Storage Device 812D Visualization Components 813 Mobile Handset Display 814 Augmented Reality (AR) Headset 814A GPU 814B Storage Device 814C Battery 814D Volumetric Visual Presentation Components 815 High-density light field display 815A GPU 815B CPU 815C Storage Device 815D Eye Tracking Device 815E Camera 815F Light Field Panel 900 Immersive Media Delivery Process 901 Content 902 Network Ingest Creation Process 903 Ingest Media Storage 904 Client Specific Information 905 Network Orchestrator 906 Distribution media and description information 907 Client Progress and Status Feedback Channel 908 Client Device 908A Visualization Components 908B Network Interface 908C GPU 908D Storage devices or storage caches 909 Media Prepared for Distribution 910 Media Adaptation and Fragmentation Module 911 Media Reuse Analyzer 912 dashed line 1000 Media Adaptation Process 1001 Adaptation Module 1001A NN model 1001B Renderer 1001C Processor 1001D Media Compressor 1001E Media Decompressor 1001F Logic Controller 1002 Storage Device 1005 Network Status 1006 Client Adaptation Media 1100 Delivery format creation process 1103 Media Packaging Module 1104 Delivery Format 1104A Information Manifesto 1104B Scene Data Asset 1110 Frequency Process 1200 Packetization process, packetization system 1201 Input Media 1202 Packetizer 1203 packets 1303 Ingest Media Server 1307 Media Server 1308 Media Request 1309 Profile Request 1310 Response 1311 Session ID Token 1312 Ingest Media 1313 Responses 1314 Call 1315 request 1316 request 1317 Responses 1318 request 1319 Update Response 1320 Interface Call 1321 Responses 1322 Responses 1323 request 1324 Messages 1420 Unique Assets 1500 Computer Code 1510 receiving code 1520 Acquisition Code 1530 Analysis Code 1540 Sequencing Code 1550 Streaming Code 3030 Frequency 3031 Scene Information 3032 Components 3033 Assets 3034 Base layer 3035 Attribute Extension Layer 3037 Unique Assets 3038 Proxy Assets 4040 Ordered Complexity Display 4041 Scene Information 4042 Components 4043 Assets 4044 Base layer 4045 Attribute Extension Layer 4047 Timed Scene Information 4048 Unique Assets 11043 Media Packaging Module 11044 Delivery Format 11044A Information Manifesto 11044B Scene Data Asset

Claims

1. 1. A method for packaging media to optimize immersive media delivery, executed by at least one processor, the method comprising: receiving immersive media data for an immersive presentation; obtaining asset information associated with media assets corresponding to a set of scenes included in the immersive media data for the immersive presentation; analyzing characteristics of the media assets based on the asset information, the characteristics including an asset type associated with each media asset and a frequency indicating the number of times the each media asset is used within the set of scenes included in the immersive presentation; ordering the media assets into a sequence based on the asset type and the frequency associated with each of the media assets; A method comprising:

2. The method of claim 1 , wherein the sequence of media assets is ordered first by asset type and then by increasing or decreasing frequency.

3. 2. The method of claim 1 , wherein the immersive media data includes one or more scenes, the one or more scenes being timed scenes, non-timed scenes, or a combination of timed and non-timed scenes, the timed scenes being scenes ordered by time, and the non-timed scenes being scenes organized by spatial, logical, or temporal relationships.

4. the asset information includes a base representation of each of the media assets and a set of asset extension layers, the set of asset extension layers including attribute information corresponding to the characteristics of the media assets; 2. The method of claim 1, wherein when the set of asset extension layers is applied to the base representation of the media asset, the base representation of the respective media asset is extended to include features that are not supported in a base layer that includes the base representation of the media asset.

5. separating the sequence of media assets into individual packets for display and streaming over a network; streaming the immersive media data for the immersive presentation based on the ordered sequence of the media assets; The method of claim 1, further comprising:

6. determining, based on scene complexity, whether a format of the immersive media data corresponding to the scenes in the set of scenes should be converted from a first format to a second format prior to immersive media delivery; determining whether a source of the immersive media data or a client of the immersive media data should perform a conversion from the first format to the second format based on a determination that the immersive media data corresponding to the scene should be converted; The method of claim 1, further comprising:

7. determining whether the respective media asset has already been streamed; if it is determined that the respective media asset has already been streamed, creating a proxy to replace the respective media asset within the sequence of media assets; The method of claim 1, further comprising:

8. A device for packaging media to optimize immersive media delivery, the device comprising processing circuitry configured to perform the method of any one of claims 1 to 7.

9. A computer program configured, when executed by at least one processor, to cause said at least one processor to perform the method of any one of claims 1 to 7.

Images