Immersive media analyzer for reusing scene assets

Abstract

A method for optimizing media delivery of a media stream executed by at least one processor is provided, comprising the steps of receiving immersive media data for an immersive presentation from a content source, obtaining asset information corresponding to media assets in a set of scenes included in the immersive media data, analyzing characteristics of media assets used in the set of scenes from the asset information to determine whether each media asset is unique among the media assets, and generating metadata information that uniquely identifies each media asset in the set of scenes for reusing each media asset based on determining that each media asset is not unique among the media assets in the set of scenes included in the immersive media data for the immersive presentation.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is based on and claims priority to U.S. Provisional Patent Application No. 63 / 276,540, filed November 5, 2021, and U.S. Patent Application No. 17 / 970,712, filed October 21, 2022, the disclosures of which are incorporated by reference in their entireties herein.

[0002] This disclosure describes embodiments relating to architectures, structures, and components for systems and networks that deliver media, including video, audio, geometric (3D) objects, haptics, associated metadata, or other content for client presentation devices. Some embodiments relate to systems, structures, and architectures for the delivery of media content to heterogeneous immersive and interactive client presentation devices. [Background technology]

[0003] Immersive media generally refers to media that stimulates any or all human sensory systems (e.g., vision, hearing, somatosensation, smell, and possibly taste) to create or enhance the user's perception of being physically present in the media experience, such as beyond that delivered over existing (e.g., "legacy") commercial networks for timed two-dimensional (2D) video and corresponding audio; such timed media is also known as "legacy media." Immersive media may also be defined as media that seeks to create or mimic the physical world through digital simulation of dynamics and the laws of physics, thereby stimulating any or all human sensory systems to create the perception by the user of being physically present in a scene depicting the real or virtual world.

[0004] Several immersive media-enabled devices have been introduced (or are poised to emerge) in the consumer market, including head-mounted displays, augmented reality glasses, handheld controllers, multi-view displays, haptic gloves, gaming consoles, holographic displays, and other forms of volumetric displays. Despite the availability of these devices, a coherent end-to-end ecosystem for the delivery of immersive media over commercial networks has not been realized.

[0005] In the prior art, one of the obstacles to realizing a coherent end-to-end ecosystem for the delivery of immersive media over commercial networks is the great diversity of all the client devices that serve as the endpoints of such delivery networks for immersive displays. Unlike networks designed solely for the delivery of legacy media, networks that must support a variety of display clients (i.e., heterogeneous clients) require a significant amount of information about the details of each of the client's capabilities, and the format of the media being delivered, before such networks can employ an adaptation process to convert the media into a format appropriate for each target display and corresponding application. Such networks need access to information that describes the characteristics of each target display and the complexity of the ingested media, at a minimum, to ascertain how the network can meaningfully adapt the input media source to a format appropriate for the target display and application.

[0006] Therefore, there is a need for a method for efficiently representing and streaming heterogeneous immersive media to a variety of clients. Summary of the Invention [Means for solving the problem]

[0007] According to an embodiment, a method is provided for optimizing a system for delivering media assets to at least one set of scenes comprising an immersive media presentation, where each media asset that is used more than once across the set of at least one scene is delivered only once to a client that has or has access to sufficient storage resources to store a copy of the media asset that the client receives.

[0008] According to one aspect of the present disclosure, there is provided a method for optimizing media delivery, executed by at least one processor, comprising the steps of receiving immersive media data for an immersive presentation from a content source, obtaining asset information corresponding to media assets in a set of scenes included in the immersive media data, For each scene included in the immersive media data, Each media asset whether the unique media asset is not used in a scene prior to the scene in the immersive media data; From the asset information judgement and each of the media assets The unique media asset Based on what was judged ,before In order to reuse the above media assets, indicating the unique media assets associated with each scene in the immersive media data; and generating metadata information.

[0009] According to another aspect of the present disclosure, there is provided a device (or apparatus) for optimizing media delivery, the device (or apparatus) including at least one memory configured to store computer program code and at least one processor configured to read and operate as instructed by the computer program code, the computer program code comprising: a receiving code configured to cause the at least one processor to receive immersive media data for an immersive presentation from a content source; an acquiring code configured to cause the at least one processor to acquire asset information corresponding to media assets in a set of scenes included in the immersive media data; and an acquiring code configured to cause the at least one processor to: For each scene included in the immersive media data, Each media asset whether the unique media asset is not used in a scene prior to the scene in the immersive media data; From the asset information judgement and a code for processing at least one of the media assets, The unique media asset Based on what was judged ,before In order to reuse the above media assets, indicating the unique media assets associated with each scene in the immersive media data; and generating code configured to generate the metadata information.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable medium storing instructions for execution by at least one processor of a device for optimizing media delivery is provided, the instructions causing the at least one processor to receive immersive media data for an immersive presentation from a content source, obtain asset information corresponding to media assets in a set of scenes included in the immersive media data, For each scene included in the immersive media data, Each media asset Whether the unique media asset is not used in a scene prior to the scene in the immersive media data, From the asset information judgement Each of the media assets The unique media asset Based on what was judged ,before In order to reuse the above media assets, indicating the unique media assets associated with each scene in the immersive media data; Generate metadata information.

[0011] Additional embodiments will be set forth in the description that follows, and in part will be apparent from the description, and / or may be realized by practice of the presented embodiments of the present disclosure. [Brief description of the drawings]

[0012] [Figure 1A] FIG. 2 is a schematic diagram of media flow through a network for delivery to a client, according to an embodiment. [Figure 1B] 1 is a workflow illustrating a reuse logic decision-making process. [Figure 2A] 1 is a schematic diagram of media flow through a network in which a decision-making process is employed to determine whether the network should convert the media before delivering it to a client, according to an embodiment. [Figure 2B]1 is a workflow illustrating a media transformation decision-making process using asset reuse logic, according to an embodiment. [Diagram 3] FIG. 2 is a schematic diagram of a data model for representing and streaming timed immersive media, according to an embodiment. [Figure 4] FIG. 2 is a schematic diagram of a data model for representing and streaming non-timed immersive media, according to an embodiment. [Diagram 5] FIG. 1 is a schematic diagram of a natural media synthesis process, according to an embodiment. [Figure 6] FIG. 2 is a schematic diagram of an example synthetic media ingest creation process, according to an embodiment. [Figure 7] FIG. 1 is a schematic diagram of a computer system, according to an embodiment. [Figure 8] 1 is a schematic diagram of a network media distribution system, according to an embodiment. [Figure 9] FIG. 1 is a schematic diagram of an exemplary workflow of an immersive media delivery process, according to an embodiment. [Figure 10] FIG. 1 is a system diagram of a media adaptation process system, according to an embodiment. [Figure 11] 2 is a schematic diagram of an exemplary delivery format creation process, according to an embodiment. [Figure 12] FIG. 2 is a schematic diagram of an exemplary packetization process, according to an embodiment. [Figure 13] FIG. 2 is a sequence diagram illustrating an example of communication flow between components according to an embodiment. [Figure 14A] 1 is a workflow illustrating a method of a media reuse analyzer for analyzing uniqueness of objects / assets in a scene, according to an embodiment. [Figure 14B] FIG. 2 illustrates an example set of a list of unique assets for a scene in a presentation, according to an embodiment. [Figure 15] FIG. 13 is a block diagram of an example of computer code for a media reuse analyzer for analyzing uniqueness of objects / assets in a scene, according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The following detailed description of the exemplary embodiments refers to the accompanying drawings, in which the same reference numbers in different drawings may identify the same or similar elements.

[0014] The foregoing disclosure provides illustrations and descriptions, but is not intended to be exhaustive or to limit the implementations to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations. Moreover, one or more features or components of one embodiment may be incorporated or combined with another embodiment (or one or more features of another embodiment). Furthermore, in the flowcharts and operational descriptions provided below, it is understood that one or more operations may be omitted, one or more operations may be added, one or more operations may be performed simultaneously (at least partially), and the order of one or more operations may be switched.

[0015] It will be apparent that the systems and / or methods described herein may be implemented in different forms of hardware, software, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not intended to limit the implementation form. Thus, the operation and behavior of the systems and / or methods are described herein without reference to any specific software code. It is understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0016] Although certain combinations of features are recited in the claims and / or disclosed herein, these combinations are not intended to limit the disclosure of possible implementations. Indeed, many of these features may be combined in ways not specifically recited in the claims and / or disclosed herein. Although each dependent claim listed below may depend directly on only one claim, the disclosure of possible implementations includes each dependent claim in combination with all other claims in the claim set.

[0017] The proposed functions described below may be used separately or combined in any order. Furthermore, the embodiments may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored in a non-transitory computer-readable medium.

[0018] No element, act, or instruction used herein should be construed as critical or essential unless expressly described as such. Also, as used herein, the articles "a" and "an" are intended to include one or more items and may be used interchangeably with "one or more." When only one item is intended, the term "one" or similar words are used. Also, as used herein, terms such as "has," "have," "having," "include," "including," and the like are intended to be open-ended terms. Furthermore, the phrase "based on" is intended to mean "based at least in part on," unless otherwise noted. Furthermore, expressions such as "at least one of [A] and [B]" or "at least one of [A] or [B]" should be understood to include only A, only B, or both A and B.

[0019] Exemplary embodiments of the present disclosure provide methods and devices for analyzing and transforming media assets for delivery to immersive media-enabled presentation devices. An immersive media-enabled presentation device can refer to being equipped with sufficient resources and capabilities to access, interpret, and present immersive media. Such devices are heterogeneous with respect to the amount and format of media (provided by the network) they can support. Similarly, media is heterogeneous with respect to the amount and type of network resources required to deliver such media at scale. "At scale" can refer to the delivery of media by a service provider that achieves delivery equivalent to legacy video and audio media over a network (e.g., Netflix, Hulu, Comcast subscription, Spectrum subscription, etc.). In contrast, legacy presentation devices such as laptop displays, televisions, and mobile handset displays are uniform in their capabilities, as all of those devices are comprised of rectangular display screens that consume 2D rectangular video or still images as their primary visual media format. Some of the visual media formats commonly used by legacy presentation devices may include, for example, High Efficiency Video Coding / H.265, Advanced Video Coding / H.264, and Versatile Video Coding / H.266.

[0020] As mentioned above, the client devices that act as endpoints for the delivery of immersive media over a network are all very diverse. Some of them support certain immersive media formats while others do not. Some of them can create immersive experiences from legacy raster-based formats while others cannot. To combat this issue, the delivery of any media over a network may employ media delivery systems and architectures that reformat the media from an input or network ingest media format to a delivery media format that is not only suitable for being ingested by the target client device and its applications but also lends itself to being streamed over the network. Thus, there may be two processes performed by the network with the ingested media, namely, 1) a process of converting the media from format A to a format B that is suitable for being ingested by the target client device, i.e., based on the client device's ability to ingest a particular media format, and 2) a process of preparing the media to be streamed.

[0021] In embodiments, streaming media refers broadly to the fragmentation and / or packetization of media so that it can be delivered over a network in successive, small-sized chunks that are logically organized and sequenced according to either or both of the media's temporal or spatial structure. The conversion of media from format A to format B (sometimes referred to as "transcoding") may be a process performed, typically by a network or service provider, prior to delivery of the media to a client device. Such transcoding may consist of converting media from format A to format B based on prior knowledge that format B is the preferred or only format that can be ingested by the target client device, or is more suitable for delivery over constrained resources, such as a commercial network. In many, if not all, steps of converting media and preparing the media to be streamed are necessary before the media can be received from the network and processed by the target client device.

[0022] Converting (or transforming) media and preparing media for streaming are process steps that operate on media captured by a network before delivering the media to a client device. The result of the process (i.e., converting and preparing for streaming) is a media format called a delivery media format, or simply a delivery format. These steps, when performed for a given media data object, should be performed only once if the network has access to information indicating that a client needs the converted and / or streamed media object for multiple opportunities to trigger the conversion and streaming of such media multiple times. That is, the processing and transfer of data for converting and streaming media is generally considered a source of latency that requires the consumption of a potentially significant amount of network and / or computational resources. Thus, a network design that does not have access to information to indicate when a client may already have a particular media data object stored in its cache or stored locally to the client will perform suboptimally for a network that has access to such information.

[0023] An ideal network supporting heterogeneous clients should exploit the fact that some of the assets adapted from an input media format to a particular target format may be reused across a set of similar display targets. That is, once converted to a format appropriate for the target displays, some assets may be reused across several such displays having similar adaptation requirements. Thus, according to an embodiment, such an ideal network may employ a caching mechanism to store adapted assets in a relative area, similar to, for example, the use of content delivery networks (CDNs) in legacy networks.

[0024] Immersive media may be organized into scenes that are described by a scene graph, also known as a scene description. In an embodiment, a scene (in the context of computer graphics) is a collection of objects (e.g., 3D assets), object attributes, and other metadata including visual, acoustic, and physics-based properties that describe a particular setting bounded by either space or time with respect to the interaction of objects within that setting. A scene graph is a common data structure typically used by vector-based graphics editing applications and modern computer games to arrange a logical and often (though not necessarily) spatial representation of a graphic scene. A scene graph may be composed of a collection of nodes and vertices in a graph structure. A node may be composed of information about the logical, spatial, or temporal representation of visual, audio, tactile, olfactory, gustatory, or related processing information. Each node shall have at most one outgoing edge, zero or more incoming edges, and at least one edge (either incoming or outgoing) connected to it. Attributes or object attributes refer to metadata associated with a node that is used to describe a particular characteristic or feature of that node, either in a canonical form or in a more complex form (e.g., with respect to other nodes). The scope of a scene graph is to describe the visual, audio, and other forms of immersive assets that are part of a presentation, including the particular setting, actors, and events that take place in a particular location within a building that are part of the presentation (e.g., a movie). A list of all the scenes that comprise a single presentation may be formulated into a scenes manifest.

[0025] An additional benefit of employing a caching mechanism to store adapted assets is that a bill of materials can be created for pre-prepared content before the content needs to be delivered. The bill of materials identifies all assets used throughout a presentation and the frequency with which each asset is used across various scenes within the presentation. An ideal network should have knowledge of the existence of cached resources that can be used to meet the asset requirements of a particular presentation. Similarly, a client device presenting a series of scenes may want to have knowledge of the frequency with which any given asset is used across multiple scenes. For example, if a media asset (also known as a media object) is referenced multiple times across multiple scenes that have been or will be processed by a client device, the client device should avoid discarding that particular asset from its cache resources until the last scene requiring that particular asset is presented by the client device. In embodiments of the present disclosure, the terms media "object" and media "asset" may be used interchangeably, both referring to a particular instance of a particular format of media data.

[0026] For legacy media presentation devices, the delivery format may be equivalent or sufficiently equivalent to the "presentation format" that is ultimately used by the client presentation device to create the presentation. That is, the presentation media format is a media format whose properties (e.g., resolution, frame rate, bit depth, color gamut, etc.) are closely aligned to the capabilities of the client presentation device. An example of a delivery versus presentation format includes a high definition (HD) video signal (1920 pixel columns by 1080 pixel rows) delivered by a network to an ultra high definition (UHD) client device with a resolution (3840 pixel columns by 2160 pixel rows). In the example above, the UHD client device would apply super-resolution processing to the HD delivery format to increase the resolution of the video signal from HD to UHD. Thus, the final signal format presented by the client device is the "presentation format," which in this example is a UHD signal, but the HD signal includes the delivery format. In this example, the HD signal distribution format is very similar to the UHD signal presentation format since both signals are linear video formats, and the process of converting the HD format to the UHD format is relatively simple and easy to perform on most legacy media client devices.

[0027] In some embodiments, the preferred presentation format for the client device may be significantly different than the ingest format received by the network. Nevertheless, the client device may have access to sufficient computational, storage, and bandwidth resources to convert the media from the ingest format to the required presentation format suitable for presentation by the client device. In this scenario, the network may bypass the step of reformatting or transcoding the ingested media from format A to format B, simply because the client device has access to sufficient resources to perform all media conversion without the network having to do so beforehand. However, the network may still perform the steps of fragmenting and packaging the ingest media so that the media may be streamed over the network to the client device.

[0028] In some embodiments, the ingested media may differ significantly from the client's preferred presentation format, and the client device may not have access to sufficient computational, storage, and / or bandwidth resources to convert the media from the ingest format to the preferred presentation format. In such scenarios, the network may assist the client by performing some or all of the conversion from the ingest format to a format equivalent or nearly equivalent to the client's preferred presentation format on behalf of the client device. In some architectural designs, such assistance provided by the network on behalf of the client device is commonly referred to as split rendering.

[0029] FIG. 1A is a schematic diagram of a media flow process 100 through a network for delivery to a client, according to some embodiments. FIG. 1A illustrates an example of processing media in format A (hereinafter, "ingest media format A"). The processing (i.e., media flow process 100) may be performed or implemented by a network cloud or edge device (hereinafter, "network device 104") and delivered to a client, e.g., client device 108. In some embodiments, the same processing may be performed in advance by a manual process or a client device. Network device 104 may include an ingest media module 101, a network processing module 102 (hereinafter, "processing module 102"), and a delivery module 103. Client device 108 may include a rendering module 106 and a presentation module 107.

[0030] First, the network device 104 receives ingested media from a content provider or the like. The ingest media module 101 obtains the ingested media stored in an ingest media format A. The network processing module 102 performs any necessary conversion or adjustment of the ingested media to create potential alternative representations of the media. That is, the processing module 102 creates a delivery format for the media objects in the ingest media. As described above, the delivery format is a media format that can be delivered to a client by formatting the media into a delivery format B. The delivery format B is a format that is ready to be streamed to the client device 108. The processing module 102 may include an optimization reuse logic 121 that performs a decision-making process to determine whether a particular media object has already been streamed to the client device 108. The operation of the processing module 102 and the optimization reuse logic 121 will be described in detail with reference to FIG. 1B.

[0031] Media formats A and B may or may not be representations that follow the same syntax of a particular media format specification, but format B is likely to be tailored in a manner that facilitates distribution of the media over a network protocol. The network protocol may be, for example, a connection-oriented protocol (TCP) or a connectionless protocol (UDP). The delivery module 103 streams the streamable media (i.e., media format B) from the network device 104 to the client device 108 over the network connection 105.

[0032] Client device 108 receives the delivery media and, optionally, prepares the media for presentation via rendering module 106. Rendering module 106 has access to several rendering capabilities that may be rudimentary or similarly sophisticated, depending on the targeted client device 108. Rendering module 106 creates presentation media in presentation format C. Presentation format C may or may not be expressed according to a third format specification. Thus, presentation format C may be the same as or different from media format A and / or B. Rendering module 106 outputs presentation format C to presentation module 107, which may present the presentation media within a display (or the like) of client device 108.

[0033] The embodiments of the present disclosure facilitate a decision-making process employed by the network and / or client to determine whether the network should convert some or all of the ingested media from format A to format B to further facilitate the client's ability to generate a media presentation in a potentially third format C. To aid in such a decision-making process, the embodiments describe an immersive media optimizer for scene asset reuse. The immersive media optimizer is introduced herein as a mechanism that analyzes one or more media objects comprising either a part or an entire immersive media scene across a set of scenes that make up the presentation. The immersive media optimizer creates information metadata about the uniqueness of each object in each scene that makes up the presentation. The metadata may include information that uniquely identifies the object according to its properties. The immersive media optimizer further tracks how many times a unique object has been used across at least one scene. Such an immersive media optimizer may be implemented separately or together with an immersive media analyzer that determines the complexity of each object in each scene. Thus, once all such metadata is available for some or all of an immersive media scene, the decision-making process is better equipped with information indicating how frequently particular objects are used across scenes for the complete presentation and may therefore decide to skip the step of converting and / or streaming the original media if the client has previously received a copy of such media from the network.

[0034] Embodiments address the need for a mechanism or process to obtain sufficient information that can be used to support a decision-making process that analyzes an immersive media scene to provide an indication as to whether conversion of media objects from format A to format B, when employed by a network or a client, should be performed entirely by the network, entirely by the client, or via a mix of both (with indication of which assets should be converted by the client or network). Such an immersive media data complexity analyzer may be used by either the client or network in an automated context, or manually, for example by a human operating a system or device.

[0035] According to embodiments, the process of adapting an input immersive media source to a particular endpoint client device may be the same as or similar to the process of adapting the same input immersive media source to a particular application running on a particular client endpoint device. Thus, the problem of adapting an input media source to the characteristics of an endpoint device is of the same complexity as the problem of adapting a particular input media source to the characteristics of a particular application.

[0036] 1B is a workflow of the processing module 102, according to an embodiment. More specifically, the workflow of FIG. 1B is a decision-making process of the reuse logic executed by the optimization reuse logic 121 to assist in the decision-making process for determining whether a particular media object has already been streamed to the client device 108.

[0037] In S101, the delivery media creation process is started. Thus, the reuse logic decision-making process is also started. In S102, the condition logic is executed to determine whether the current media object has been previously streamed to the client device 108. The optimization reuse logic 121 may access a list of unique assets for the presentation to determine whether the media object has been previously streamed to the client. If the current media object has been previously streamed, the process proceeds to S103. In S103, an indicator (later also referred to as a "proxy") is created to identify that the client has already received the current media object and should use its local copy of the media object. Processing of the current media object then ends (S104).

[0038] If it is determined that the media object has not been previously streamed, the process proceeds to S105, where the processing module 102 continues to prepare the media object for conversion and / or delivery, and a delivery format for the media object is created, after which the process proceeds to S104, where processing of the current media object ends.

[0039] 2A is a logic workflow for processing media ingested over a network. The workflow shown in FIG. 2A illustrates a media conversion decision-making process 200 according to an embodiment. The media conversion decision-making process 200 is employed to determine whether the network should convert the media before delivering it to a client device. The media conversion decision-making process 200 may be handled via a manual or automated process within the network.

[0040] Ingest media, represented in format A, is provided to the network by a content provider. In S201, media is ingested by the network from the content provider. Next, in S202, attributes of the target client are obtained, if not already known. The attributes describe the processing capabilities of the target client.

[0041] In S203, it is determined whether the network (or client) should assist in conversion of the ingested media. In particular, it is determined whether there is a format conversion for any of the media assets contained within the ingested media (e.g., conversion of one or more media objects from format A to format B) before the media is streamed to the target client. The decision-making process in S203 may be performed manually (i.e., by a device operator, etc.) or may be an automated process. The decision-making process in S203 may be based on a determination of whether the media can be streamed in its original ingest format A or whether it must be converted to a different format B to facilitate presentation of the media by the client. Such a decision may require access to information describing aspects or characteristics of the ingested media to assist the decision-making process in making an optimal choice (i.e., to determine whether conversion of the ingested media is required before streaming the media to the client or whether the media needs to be streamed directly to the client in its original ingest format A).

[0042] If it is determined that the network (or client) should assist in the conversion of any of the media assets (YES in S203), the process 200 proceeds to S204.

[0043] At S204, the captured media is transformed to convert the media from format A to format B, generating transformed media 205. The transformed media 205 is output and the process proceeds to S206. At S206, the input media undergoes a process to prepare the media for streaming to a client. In this case, the transformed media 205 (i.e., the input media) is prepared to be streamed.

[0044] The conversion of media from format A to another format (e.g., format B) may be performed entirely by the network, entirely by the client, or between both the network and the client together. In split rendering, it becomes apparent that a dictionary of attributes describing the media format may be required so that both the client and the network have complete information characterizing the work that must be done. Furthermore, a dictionary providing attributes of the client's capabilities, e.g., in terms of available computational resources, available storage resources, and access to bandwidth, may be required as well. Furthermore, a mechanism is required to characterize the level of computational, storage, or bandwidth complexity of an ingest media format, so that the network and client together, or alone, may decide whether or when the network employs a split rendering process to deliver media to the client.

[0045] If it is determined that the network (or client) should not (or does not need to) support conversion of any media assets ("NO" at S203), the process 200 proceeds to S206, where the media is prepared for streaming. In this case, the ingested data (i.e., the media in its original form) is prepared to be streamed.

[0046] Finally, once the media data is in a streamable format, the media prepared in S206 is streamed to the client (S207). In some embodiments, if the conversion and / or streaming of certain media objects that are or will be required by the client to complete the presentation (as described with reference to FIG. 1B) have already been done as part of the work to process previous scenes of the presentation, the network may skip converting and / or streaming the ingest media (i.e., S204-S207), assuming the client still has access to or availability of the media that was previously streamed to the client.

[0047] 2B is a media transformation decision-making process by asset reuse logic 210, according to an embodiment. Similar to media transformation decision-making process 200, the media transformation decision-making process using asset reuse logic 210 processes ingested media over a network and employs two decision-making processes to determine whether the network should transform the media before delivering the media to a client.

[0048] Ingest media, represented in format A, is provided to the network by a content provider. Then, S221-S222 are executed, similar to S201-S202 (shown in FIG. 2A). In S221, media is ingested by the network from the content provider. Next, in S202, attributes of the target client are obtained, if not already known. The attributes describe the processing capabilities of the target client.

[0049] If it is determined that the network has previously streamed the particular media object or the current media object (YES in S223), the process proceeds to S224, where a proxy is created to replace the previously streamed media object to indicate that the client should use a local copy of the previously streamed object.

[0050] If it is determined that the network has not previously streamed the particular media object ("NO" at S223), the process proceeds to S225. At S225, it is determined whether the network or the client should perform any format conversion on any of the media assets contained within the ingested media at S221. For example, the conversion may include converting the particular media object from format A to format B before the media is streamed to the client. The processing performed at S225 may be similar to the processing performed at S203 shown in FIG. 2A.

[0051] If it is determined that the media asset should be converted over the network (YES in S225), the process proceeds to S226, where the media object is converted from format A to format B. The converted media is then prepared to be streamed to the client (S227).

[0052] If it is determined that the media asset should not be converted over the network (NO in S225), the process proceeds to S227. Next, in S227, the media object is prepared to be streamed to the client.

[0053] Finally, once the media data is in a streamable format, the media prepared in S227 is streamed to the client (S228). The processes performed in S225 to S228 of Figure 2B may be similar to the processes performed in S203 to S207 shown in Figure 2A.

[0054] The streamable format of media may be timed or non-timed heterogeneous immersive media. Figure 3 shows an example of a timed media representation 300 of a streamable format of heterogeneous immersive media. Timed immersive media may include a set of N scenes. Timed media is media content ordered by time, e.g., with start and end times according to a particular clock. Figure 4 shows an example of a non-timed media representation 400 of a streamable format of heterogeneous immersive media. Non-timed media is media content organized by spatial, logical, or temporal relationships (e.g., as in the case of an interactive experience realized according to actions taken by one or more users).

[0055] FIG. 3 refers to timed scenes for timed media, and FIG. 4 refers to non-timed scenes for non-timed media. Timed and non-timed scenes may be embodied by various scene representations or scene descriptions. Both FIG. 3 and FIG. 4 employ a single exemplary generic media format adapted from a source ingest media format to match the capabilities of a particular client endpoint. That is, the generic media format is a delivery format that can be streamed to a client device. The generic media format is robust in its structure to accommodate a wide variety of media attributes, each of which may be layered based on the amount of salient information each layer contributes to the presentation of the media.

[0056] As shown in FIG. 3, a timed media representation 300 includes a list of timed scenes 301. The timed scenes 301 point to processing information and a list of components 302 that separately describe the types of media assets that make up the timed scene 301, e.g., asset lists and other processing information. The list of components 302 may reference proxy assets 308 that correspond to the types of assets (e.g., proxy visual and auditory assets, as shown in FIG. 3). The visual asset list points to proxy assets 310 that correspond to the types of assets (e.g., proxy visual and auditory assets, as shown in FIG. 3). The components 302 point to a list of unique assets that have not been used before in other scenes. For example, a list of unique assets 307 for (timed) scene 1 is shown in FIG. 3. The components 302 also point to assets 303, which further point to a base layer 304 and an attribute enrichment layer 305. The base layer is a nominal representation of the assets that may be formulated to minimize computational resources, the time required to render the assets, and / or the time required to transmit the assets over a network. An enhancement layer may be a set of information that, when applied to a base layer representation of an asset, extends the base layer to include features or capabilities that may not be supported in the base layer.

[0057] As shown in FIG. 4, the non-timed media representation 400 includes information about a scene 401. The scene 401 is not associated with a start and end time / duration (by a clock, timer, etc.). The scene 401 points to a list of components 402 that separately describe the types of processing information and media assets that make up the scene 401. The components 402 point to visual assets, auditory assets, haptic assets, and timed assets (collectively referred to as assets 403). The assets 403 further point to a base layer 404 and attribute enhancement layers 405 and 406. The scene 401 can also point to other non-timed scenes that are for non-timed media sources (i.e., referenced in FIG. 4 as non-timed scenes 2.1-2.4) and / or a scene 407 that is for a timed media scene (i.e., referenced in FIG. 4 as timed scene 3.0). In the example of FIG. 4, the non-timed immersive media includes a set of five scenes (including both timed and non-timed). The list of unique assets 408 identifies unique assets associated with a particular scene that have not been previously used in a higher level (e.g., parent) scene. The list of unique assets 408 shown in Figure 4 includes unique assets for non-timed scene 2.3.

[0058] Media streamed according to the generic media format is not limited to legacy visual and audio media. The generic media format may include any type of media information that can interact with a machine to generate signals that stimulate a human's senses of sight, hearing, taste, touch, or smell. As shown in Figures 3-4, media streamed according to the generic media format may be timed media or non-timed media, or a combination of both. The generic media format is streamable by enabling layered representation of media objects using a base layer and enhancement layer architecture.

[0059] In some embodiments, the separate base and enhancement layers are computed by application of multi-resolution or multi-mosaicing analysis techniques to the media objects in each scene, which computation techniques are not limited to raster-based visual formats.

[0060] In some embodiments, the progressive representation of the geometric object may be a multi-resolution representation of the object computed using wavelet analysis techniques.

[0061] In some embodiments, in a layered representational media format, an enhancement layer may apply different attributes to a base layer, for example, one or more enhancement layers may improve the material properties of the surface of a visual object represented by the base layer.

[0062] In some embodiments, in a layered representational media format, an attribute may refine the texture of the surface of an object represented by a base layer, for example, by changing the surface from a smooth texture to a porous texture, or from a matte surface to a glossy surface.

[0063] In some embodiments, in a layered representation media format, the surfaces of one or more visual objects in a scene may be changed from Lambertian surfaces to ray-traceable surfaces.

[0064] In some embodiments, in a layered representation media format, the network may deliver a base layer representation to a client so that the client can create a nominal presentation of the scene, and the client awaits the transmission of additional enhancement layers to refine the resolution or other characteristics of the base layer.

[0065] In embodiments, the resolution or refinement information of attributes in the enhancement layer is not explicitly coupled with the resolution of objects in the base layer. Furthermore, the generic media format may support any kind of information media that can be presented or acted upon by a presentation device or machine, thereby enabling support of heterogeneous media formats to heterogeneous client endpoints. In some embodiments, the network delivering the media format first queries the client endpoint to determine the client's capabilities. If, based on the query, the client is unable to meaningfully capture the media representation, the network may remove layers of attributes that are not supported by the client. In some embodiments, if the client is unable to meaningfully capture the media representation, the network may adapt the media from its current format to a format appropriate for the client endpoint. For example, the network may adapt the media by converting a stereoscopic visual media asset into a 2D representation of the same visual asset using a network-based media processing protocol. In some embodiments, the network may adapt the media by employing a neural network (NN) process to reformat the media into an appropriate format, or optionally by synthesizing a view required by the client endpoint.

[0066] A scene manifest for a complete (or partially complete) immersive experience (live streaming event, game, or on-demand asset playback) is organized by scenes that contain the minimum amount of information required to render and ingest to create a presentation. The scene manifest contains a list of individual scenes for which the entire immersive experience requested by the client is rendered. Associated with each scene are one or more representations of the geometric objects in the scene that correspond to a streamable version of the scene geometry. An embodiment of a scene may reference a low-resolution version of the scene's geometric objects. Another embodiment of the same scene may reference enhancement layers for the low-resolution representation of the scene to add further detail to the geometric objects of the same scene or to increase mosaicking. As described above, each scene may have one or more enhancement layers to progressively increase the detail of the scene's geometric objects. Each layer of a media object referenced in a scene may be associated with a token (e.g., a uniform resource identifier (URI)) that points to an address where the resource can be accessed in the network. Such resources are similar to a content delivery network (CDN) where content can be fetched by a client. Tokens for representations of geometric objects may point to locations within the network or within the client, i.e., the client may signal to the network that its resources are available to the network for network-based media processing.

[0067] According to an embodiment, a scene (timed or untimed) may be embodied by a scene graph as a multi-planar image (MPI) or a multi-spherical image (MSI). Both MPI and MSI techniques are examples of techniques that aid in the creation of display-independent scene representations for natural content (i.e., real-world images captured simultaneously from one or more cameras). Scene graph techniques, on the other hand, may be employed to represent both natural and computer-generated images in the form of synthetic representations. However, such representations are particularly computationally intensive to create when the content is captured as a natural scene by one or more cameras. Scene graph representations of naturally captured content are both time and computationally intensive to create, requiring complex analysis of the natural imagery using photogrammetry or deep learning or both techniques to create a synthetic representation that can later be used to interpolate a sufficient and appropriate number of views to fill the viewing frustum of the target immersive client display. As a result, such synthetic representations are not practical to consider as candidates for representing natural content, since they cannot be practically created in real time to account for use cases that require real-time delivery. Therefore, the best representation of a computer-generated image is to use a scene graph in a synthetic model, because computer-generated imagery is created using 3D modeling processes and tools, and using a scene graph in a synthetic model results in the best representation of the computer-generated imagery.

[0068] 5 illustrates an example of a natural media synthesis process 500, according to an embodiment. The natural media synthesis process 500 converts an ingest format from a natural scene into a representation that can be used as an ingest format for a network that serves heterogeneous client endpoints. To the left of the dashed line 510 is the content capture portion of the natural media synthesis process 500. To the right of the dashed line 510 is the ingest format synthesis (for natural images) of the natural media synthesis process 500.

[0069] As shown in FIG. 5, the first camera 501 uses a single camera lens to capture a scene, for example, of a person (i.e., an actor shown in FIG. 5). The second camera 502 captures a scene of five diverging fields of view by mounting five camera lenses around a ring-shaped object. The arrangement of the second camera 502 shown in FIG. 5 is an exemplary arrangement commonly used to capture omnidirectional content for VR applications. The third camera 503 captures a scene of seven converging fields of view by mounting seven camera lenses on the inner diameter part of a sphere. The arrangement of the third camera 503 is an exemplary arrangement commonly used to capture a light field or a light field for a holographic immersive display. The embodiment is not limited to the configuration shown in FIG. 5. The second camera 502 and the third camera 503 may include multiple camera lenses.

[0070] Natural image content 509 is output from a first camera 501, a second camera 502, and a third camera 503 and serves as input to a synthesizer 504. The synthesizer 504 may employ NN training 505, which uses a set of training images 506 to generate a capture NN model 508. The training images 506 may be predefined or may be stored from a previous synthesis process. A NN model (e.g., capture NN model 508) is a collection of parameters and tensors (e.g., matrices) that define weights (i.e., numerical values) used in well-defined mathematical operations that are applied to a visual signal to arrive at an improved visual output, which may include interpolation of new views of the visual signal that were not explicitly provided by the original signal.

[0071] In some embodiments, a photogrammetry process may be performed in place of NN training 505. If a capture NN model 508 is created during the natural media composition process 500, the capture NN model 508 becomes one of the assets in an ingest format 507 of the natural media content. The ingest format 507 may be, for example, MPI or MSI. The ingest format 507 may also include media assets.

[0072] 6 illustrates an example of a synthetic media ingest creation process 600, according to an embodiment. The synthetic media ingest creation process 600 creates an ingest media format for synthetic media, such as computer-generated images.

[0073] As shown in FIG. 6, a camera 601 may capture a point cloud 602 of a scene. The camera 601 may be, for example, a LIDAR camera. A computer 603 employs, for example, a common gateway interface (CGI) tool, a 3D modeling tool, or another animation process to create synthetic content (i.e., a representation of a synthetic scene that can be used as an ingest format for a network serving heterogeneous client endpoints). The computer 603 may create CGI assets 604 over the network. Additionally, a sensor 605A may be attached to an actor 605 in the scene. The sensor 605A may be, for example, a motion capture suit with sensors attached to it. The sensor 605A captures a digital recording of the movement of the actor 605 to generate animated motion data 606 (or MoCap data). Data from the point cloud 602, the CGI assets 604, and the motion data 606 are provided as inputs to a synthesizer 607 that creates a synthetic media ingest format 608. In some embodiments, the synthesizer 607 may use the NN and training data to create a NN model and generate the synthetic media ingest format 608.

[0074] Both natural and computer-generated (i.e., synthetic) content may be stored in a container. The container may contain a serialization format for storing and exchanging information for representing all natural scenes, all synthetic scenes, or a combination of synthetic and natural scenes, including a scene graph and all media resources required to render the scene. The content serialization process involves converting data structures or object states into a format that can be stored (e.g., in a file or memory buffer) or transmitted (e.g., over a network connection link) and later reconstructed in the same or a different computing environment. When the resulting string of bits is reread according to the serialized format, it can be used to create a semantically identical clone of the original object.

[0075] The dichotomy in optimal representation of both natural and computer-generated content suggests that the optimal ingest format for naturally captured content is different from the optimal ingest format for computer-generated (i.e., synthetic) content or natural content that is not essential for real-time delivery applications. Thus, according to embodiments, the network aims to be robust enough to support multiple ingest formats for visually immersive media, whether created naturally using, for example, physical cameras or created by a computer.

[0076] Technologies such as OTOY's ORBX, Pixar's Universal Scene Description, and the Graphics Language Transmission Format 2.0 (glTF2.0) specification written by the Khronos 3D Group embody the scene graph as a format suitable for representing visual immersive media that is created using computer-generated techniques, or naturally captured content where deep learning or photogrammetry techniques are used to create corresponding synthetic representations of natural scenes (i.e., not required for real-time delivery applications).

[0077] ORBX, from OTOY, is one of several scene graph technologies that can support any type of visual media, timed or non-timed, including ray-traceable, legacy (frame-based), stereoscopic and other types, synthetic or vector-based visual formats. ORBX differs from other scene graphs because it natively supports freely available and / or open source formats for meshes, point clouds and textures. ORBX is a scene graph purposefully designed with the goal of facilitating exchange across multiple vendor technologies that operate on the scene graph. In addition, ORBX offers a rich material system, support for an open shader language, a robust camera system, and support for Lua scripting. ORBX is also the basis of the Immersive Technologies Media Format, released for license on royalty-free terms by the Immersive Digital Experience Alliance (IDEA). In the context of real-time delivery of media, the ability to create and deliver ORBX representations of natural scenes is a function of the availability of computational resources to perform complex analysis of the data captured by the camera and compositing of that same data into a synthetic representation.

[0078] Pixar's USD is a scene graph widely used in visual effects and professional content creation. USD is integrated into Nvidia's Omniverse platform, a set of developer tools for 3D modeling and rendering with Nvidia's graphics processing units (GPUs). A subset of USD released by Apple and Pixar is called USDZ, which is supported by Apple's ARKit.

[0079] glTF2.0 is a version of the Graphics Language Transmission Format specification written by the Khronos 3D Group. The format supports simple scene graph formats, including PNG and JPEG image formats, that are generally capable of supporting static (non-timed) objects in a scene. glTF2.0 supports simple animation, which includes support for translation, rotation, and scaling of basic shapes (i.e., geometric objects) described using glTF primitives. glTF2.0 does not support timed media, and therefore does not support video or audio media input.

[0080] These designs for immersive visual media scene representations are provided only as examples and do not limit the disclosed subject matter in its ability to specify a process for adapting an input immersive media source to a format suitable for the particular characteristics of a client endpoint device. Additionally, any or all of the above exemplary media representations may employ or employ deep learning techniques to train and create NN models that enable or facilitate the selection of a particular view to fill a particular display's view frustum based on the particular dimensions of the frustum. The views selected for a particular display's view frustum may be interpolated from existing views explicitly provided in the scene representation, for example from MSI or MPI techniques. Views may also be rendered directly from a rendering engine based on a particular virtual camera position, filter, or description of the virtual camera of these rendering engines.

[0081] The methods and devices of the present disclosure are robust enough to take into account that there is a relatively small but well-known set of immersive media ingest formats that can adequately meet the requirements of both real-time or on-demand (e.g., non-real-time) delivery of media that is captured naturally (e.g., with one or more cameras) or created using computer-generated techniques.

[0082] Interpolation of views from immersive media ingest formats through the use of either NN models or network-based rendering engines will become even easier as advanced network technologies (e.g., 5G for mobile networks) and fiber optic cables are deployed in fixed networks. These advanced network technologies will increase the capacity and capabilities of commercial networks as such advanced network infrastructure can support the transmission and delivery of increasingly large amounts of visual information. Network infrastructure management technologies such as Multi-access Edge Computing (MEC), Software Defined Networks (SDN), and Network Functions Virtualization (NFV) will enable commercial network service providers to flexibly configure their network infrastructure to adapt to changes in demand for specific network resources, e.g., to respond to dynamic increases and decreases in demand for network throughput, network speed, round-trip latency, and computational resources. Furthermore, this inherent ability to adapt to dynamic network requirements similarly facilitates the network's ability to adapt immersive media ingest formats into appropriate delivery formats to support a variety of immersive media applications with potentially heterogeneous visual media formats for heterogeneous client endpoints.

[0083] Immersive media applications themselves may also have different requirements for network resources, including gaming applications that require significantly lower network latency to respond to real-time updates on the state of the game, telepresence applications that have symmetric throughput requirements for both the uplink and downlink portions of the network, and passive viewing applications that may have increasing demands on downlink resources depending on the display type of the client endpoint that is consuming the data. In general, any consumer application may be supported by a variety of client endpoints that include different on-board client capabilities for storage, computation and power, as well as equally different requirements for the particular media presentation.

[0084] Thus, the embodiments of the present disclosure enable a well-equipped network, i.e., a network that uses some or all of the characteristics of a modern network, to simultaneously support multiple legacy and immersive media-enabled devices according to the characteristics specified within the devices. Thus, the immersive media delivery method and process described herein provides the flexibility to leverage practical media ingest formats for both real-time and on-demand use cases for the delivery of media, flexibility to support natural and computer-generated content for both legacy and immersive media-enabled client endpoints, and support for both timed and non-timed media. The method and process also dynamically adapts the source media ingest format to an appropriate delivery format based on the capabilities and capabilities of the client endpoints, as well as based on the requirements of the application. This ensures that the delivery format is streamable over an IP-based network, enabling the network to simultaneously serve multiple heterogeneous client endpoints, which may include both legacy and immersive media-enabled devices. Furthermore, the embodiments provide an exemplary media representation framework that facilitates the organization of the delivered media along scene boundaries.

[0085] An end-to-end implementation of heterogeneous immersive media delivery according to an embodiment of the present disclosure that provides the aforementioned improvements is achieved in accordance with the processes and components described in the detailed description of Figures 3-16, which are described in further detail below.

[0086] The techniques for representing and streaming heterogeneous immersive media described above may be implemented as computer software using computer readable instructions at both the source and destination and physically stored on one or more non-transitory computer readable media or by one or more specifically configured hardware processors. Figure 7 illustrates a computer system 700 suitable for implementing certain embodiments of the disclosed subject matter.

[0087] The computer software may be coded using any suitable machine code or computer language that may be subjected to mechanisms such as assembly, compilation, linking, etc. to produce code containing instructions that may be executed by a computer central processing unit (CPU), graphics processing unit (GPU), etc., directly, or via interpretation, microcode execution, etc.

[0088] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[0089] 7 for computer system 700 are exemplary in nature and are not intended to suggest any limitations as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system 700.

[0090] The computer system 700 may include certain human interface input devices. Such human interface input devices may respond to input by one or more human users, for example, via tactile input (keystrokes, swipes, data glove movements, etc.), audio input (voice, clapping, etc.), visual input (gestures, etc.), olfactory input, etc. The human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (speech, music, ambient sounds, etc.), images (scanned images, photographic images obtained from a still image camera, etc.), video (two-dimensional video, three-dimensional video including stereoscopic video, etc.).

[0091] The input human interface devices may include one or more of a keyboard 701, a trackpad 702, a mouse 703, a screen 709, which may be, for example, a touchscreen, a joystick 704, a microphone 705, a camera 706, and a scanner 707 (only one of each is shown).

[0092] The computer system 700 may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a screen 709, data gloves, or joystick 704, although there may also be haptic feedback devices that do not function as input devices), audio output devices (e.g., speakers 708, headphones, etc.), visual output devices (e.g., screens 709, including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without haptic feedback capabilities, some of which may be capable of outputting two-dimensional visual output or three-dimensional hypervisual output via means such as stereographic output, virtual reality glasses, holographic displays, and smoke tanks), and printers.

[0093] The computer system 700 may also include human accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW 711 with CD / DVD or similar media 710, thumb drives 712, removable hard drives or solid state drives 713, legacy magnetic media such as tapes and floppy disks, specialized ROM / ASIC / PLD based devices such as security dongles, etc.

[0094] Additionally, those skilled in the art will appreciate that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitory signals.

[0095] The computer system 700 may also include an interface 715 to one or more communication networks 714. The network 714 may be, for example, wireless, wired, optical. The network 714 may further be local, wide area, city, vehicular and industrial, real-time, delay tolerant, etc. Examples of the network 714 include local area networks such as Ethernet, WLAN, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicular and industrial including CANBus, etc. A particular network 714 typically requires an external network interface adapter (e.g., graphics adapter 725) attached to a particular general-purpose data port (e.g., a USB port of the computer system 700) or peripheral bus 716, while other networks are typically integrated into the core of the computer system 700 by attaching to a system bus described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks 714, computer system 700 may communicate with other entities. Such communications may be one-way receive only (e.g., broadcast TV), one-way transmit only (e.g., CANbus to a particular CANbus device), or two-way, such as to other computer systems using local area or wide area digital networks. Particular protocols and protocol stacks may be used with each of these networks and network interfaces, as described above.

[0096] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to the core 717 of the computer system 700.

[0097] The core 717 may include one or more central processing units (CPUs) 718, graphics processing units (GPUs) 719, specialized programmable processing units in the form of field programmable gate areas (FPGAs) 720, hardware accelerators 721 for specific tasks, etc. These devices may be connected via a system bus 726 along with read only memory (ROM) 723, random access memory (RAM) 724, internal mass storage devices 722 such as internal non-user accessible hard drives, SSDs, etc. In some computer systems, the system bus 726 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus 726 or via the peripheral bus 716. Architectures for peripheral buses include PCI, USB, etc.

[0098] The CPU 718, GPU 719, FPGA 720, and accelerator 721 may execute certain instructions that may combine to constitute the machine code (or computer code) described above. That computer code may be stored in ROM 723 or RAM 724. Also, temporary data may be stored in RAM 724, while persistent data may be stored, for example, in internal mass storage device 722. Rapid storage and retrieval from any memory device may be enabled by the use of cache memory, which may be closely associated with one or more of the CPU 718, GPU 719, mass storage device 722, ROM 723, RAM 724, etc.

[0099] The computer-readable medium may bear computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the available kind well known to those skilled in the computer software arts.

[0100] By way of example and not limitation, a computer system having the architecture of computer system 700, and in particular core 717, may provide functionality as a result of a processor (including CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be user-accessible mass storage devices as described above, as well as media associated with specific storage devices of the core 717 that are non-transitory in nature, such as core internal mass storage device 722 or ROM 723. Software implementing various embodiments of the present disclosure may be stored in such devices and executed by the core 717. The computer-readable media may include one or more memory devices or chips, depending on particular needs. The software may cause the core 717, and in particular the processors therein (including CPU, GPU, FPGA, etc.) to perform certain processes, or certain portions of certain processes, described herein, including defining data structures stored in RAM 724 and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator 721), which may operate in place of or in conjunction with software to perform a particular process, or a particular portion of a particular process, described herein. References to software may encompass logic, and vice versa. References to computer-readable media may encompass circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software.

[0101] The number and arrangement of components shown in Figure 7 are provided as an example. In practice, an input human interface device may include additional components, fewer components, different components, or components in a different arrangement than those shown in Figure 7. Additionally or alternatively, a set of components of an input human interface device (e.g., one or more components) may perform one or more functions that are described as being performed by another set of components of the input human interface device.

[0102] In an embodiment, any one of the operations or processes of FIGS. 1-6 and 8-15 may be performed by or using any one of the elements shown in FIG.

[0103] 8 illustrates an exemplary network media distribution system 800 serving multiple heterogeneous client endpoints. That is, the system 800 supports a variety of legacy and heterogeneous immersive media-enabled displays as client endpoints. The system 800 may include a content acquisition module 801, a content preparation module 802, and a transmission module 803.

[0104] The content acquisition module 801 captures or creates source media using, for example, the embodiments described in FIG. 6 and / or FIG. 5. The content preparation module 802 creates ingest formats that are sent to the network media distribution system using the transmission module 803. The gateway 804 may service customer premises equipment to provide network access to various client endpoints for the network. The set-top box 805 may also function as customer premises equipment to provide access to aggregated content by the network service provider. The wireless demodulator 806 may function as a mobile network access point for mobile devices, for example, as shown by the mobile handset display 813. In this particular embodiment of the system 800, a legacy 2D television 807 is shown as directly connected to one of the gateway 804, the set-top box 805, or the WiFi (router) 808. A laptop 2D display 809 (i.e., a computer or laptop with a legacy 2D display) is shown as a client endpoint connected to the WiFi (router) 808. A head mounted 2D (raster based) display 810 is also connected to the WiFi (router) 808. A lenticular light field display 811 is shown connected to one of the gateways 804. The lenticular light field display 811 may include one or more GPUs 811A, a storage device 811B, and a visual presentation component 811C that uses ray based lenticular optics technology to create multiple views. A holographic display 812 is shown connected to a set top box 805. The holographic display 812 may include one or more CPUs 812A, a GPU 812B, a storage device 812C, and a visualization component 812D. The visualization component 812D may be a Fresnel pattern, wave based holographic device / display. An augmented reality (AR) headset 814 is shown connected to the wireless demodulator 806.The AR headset 814 may include a GPU 814A, a storage device 814B, a battery 814C, and a stereoscopic visual presentation component 814D. A high density light field display 815 is shown connected to a WiFi (router) 808. The high density light field display 815 may include one or more GPUs 815A, a CPU 815B, a storage device 815C, an eye tracking device 815D, a camera 815E, and a high density beam-based light field panel 815F.

[0105] The number and arrangement of components shown in Figure 8 are provided as an example. In practice, system 800 may include additional, fewer, different, or differently arranged components than those shown in Figure 8. Additionally or alternatively, a set of components (e.g., one or more components) of system 800 may perform one or more functions that are described as being performed by another set of components in a device or respective display.

[0106] Figure 9 illustrates an example workflow of an immersive media delivery process 900 that can serve legacy and heterogeneous immersive media enabled displays as previously described in Figure 8. The immersive media delivery process 900 performed by the network may, for example, provide adaptation information regarding particular media represented in a media ingest format prior to the network's process of adapting the media for consumption by a particular immersive media client endpoint (as described with reference to Figure 10).

[0107] The immersive media delivery process 900 may be divided into two parts: immersive media production to the left of dashed line 912 and immersive media network delivery to the right of dashed line 912. The immersive media production and immersive media network delivery may be performed by a network or a client device.

[0108] First, media content 901 is created or acquired from a network (or client device) or content source, respectively. Methods for creating or acquiring data are embodied, for example, in Figures 5 and 6 for natural and synthetic content, respectively. The created content 901 is then converted to an ingest format using a network ingest format creation process 902. The network ingest creation process 902 is also embodied in Figures 5 and 6 for natural and synthetic content, respectively. The ingest format may also be updated to store information about assets potentially reused across multiple scenes, for example from a media reuse analyzer 911 (discussed in more detail below with reference to Figures 10 and 14A). The ingest format is sent to the network and stored in an ingest media storage 903 (i.e., a storage device). In some embodiments, the storage device may be within the immersive media content creator's network or may be accessed remotely for immersive media network distribution 920. Client and application specific information is optionally available in a remote storage device, client specific information 904. In some embodiments, the client-specific information 904 may reside remotely in an alternative cloud network or may be transmitted to the network.

[0109] Then, the network orchestrator 905 is executed. The network orchestrator serves as the main source and sink of information to perform the main tasks of the network. The network orchestrator 905 may be implemented in a unified format with other components of the network. The network orchestrator 905 may be a process that further employs a two-way message protocol with the client device to facilitate all processing and delivery of the media according to the characteristics of the client device. Furthermore, the two-way protocol may be implemented across different delivery channels (e.g., control plane channels and / or data plane channels).

[0110] As shown in Figure 9, the network orchestrator 905 receives information regarding the characteristics and attributes of a client device 908. The network orchestrator 905 gathers requirements regarding applications currently running on the client device 908. This information may be obtained from client specific information 904. In some embodiments, the information may be obtained by directly querying the client device 908. When a client device is directly queried, it is assumed that a bidirectional protocol exists and is operational such that the client device 908 may communicate directly with the network orchestrator 905.

[0111] The network orchestrator 905 may also initiate and communicate with the media adaptation and fragmentation module 910 (which is depicted in FIG. 10). Once the ingest media has been adapted and fragmented by the media adaptation and fragmentation module 910, the media may be transferred to an inter-media storage device, such as media prepared for delivery 909. Once the distributed media is prepared and stored in the media prepared for delivery 909 storage device, the network orchestrator 905 ensures that the client device 908 receives the distributed media and description information 906 via a "push" request, or the client device 908 may initiate a "pull" request for the distributed media and description information 906 from the stored media prepared for delivery 909. Information may be "pushed" or "pulled" through the network interface 908B of the client device 908. The "pushed" or "pulled" distributed media and description information 906 may be description information corresponding to the distributed media.

[0112] In some embodiments, the network orchestrator 905 uses a two-way message interface to perform "push" requests or initiate "pull" requests with the client devices 908. The client devices 908 may optionally employ a GPU 908C (or CPU).

[0113] The distributed media format is then stored in a storage device or storage cache 908D included in the client device 908. Finally, the client device 908 visually presents the media via a visualization component 908A.

[0114] Throughout the process of streaming immersive media to client devices 908, the network orchestrator 905 monitors the status of the client's progress via a client progress and status feedback channel 907. In some embodiments, the status monitoring may be performed via a two-way communication message interface.

[0115] 10 illustrates an example of a media adaptation process 1000, for example, performed by the media adaptation and fragmentation module 910. By performing the media adaptation process 1000, the captured source media can be appropriately adapted to match the requirements of a client (e.g., client device 908).

[0116] As shown in Figure 10, media adaptation process 1000 includes multiple components that facilitate adaptation of ingested media to an appropriate delivery format for client device 908. The components shown in Figure 10 should be considered as exemplary. In practice, media adaptation process 1000 may include additional components, fewer components, different components, or a different arrangement of components compared to the components shown in Figure 10. Additionally or alternatively, a set of components (e.g., one or more components) of media adaptation process 1000 may perform one or more functions described as performed by another set of components.

[0117] In FIG. 10, the adaptation module 1001 receives an input network status 1005 to track the current traffic load on the network. As described above, the adaptation module 1001 also receives information from the network orchestrator 905. The information may include a description of the attributes and capabilities of the client device 908, the capabilities and descriptions of the application, the current status of the application, and the client NN model (if available) to help map the geometry of the client's frustum to the interpolation capabilities of the immersive media. Such information may be obtained by a two-way message interface. The adaptation module 1001 ensures that the adapted output, once generated, is stored in a storage device for storing the client adapted media 1006.

[0118] The media reuse analyzer 911 may be an optional process that may be run in advance or as part of a network automated process for the distribution of media. The media reuse analyzer 911 may store the ingest media formats and assets in a storage device (1002). The ingest media formats and assets may then be transmitted from the storage device (1002) to the adaptation module 1001.

[0119] The adaptation module 1001 may be controlled by the logic controller 1001F. The adaptation module 1001 may also employ a renderer 1001B or a processor 1001C to adapt the particular ingest source media to a format suitable for the client. The processor 1001C may be a NN-based processor. The processor 1001C uses the NN model 1001A. Such a processor 1001C may include, for example, the Deepview NN model generator as described in MPI and MSI. If the media is in a 2D format but the client must have a 3D format, the processor 1001C may invoke a process that uses highly correlated images from the 2D video signal to derive a stereoscopic representation of the scene depicted in the media.

[0120] The renderer 1001B may be a software-based (or hardware-based) application or process based on a selective mix of fields related to acoustic physics, optical physics, visual perception, audio perception, mathematics, and software development, which, given an input scene graph and asset containers, emits (usually) visual and / or audio signals suitable for presentation on a target device or conforming to desired characteristics specified by attributes of the nodes to be rendered in the scene graph. In the case of visual-based media assets, the renderer may emit visual signals suitable for a target display or suitable for storage as an intermediate asset (e.g., repackaged in another container and used in the sequence of rendering processes in a graphics pipeline). In the case of audio-based media assets, the renderer may emit audio signals for presentation on multi-channel loudspeakers and / or binauralized headphones, or for repackaging in another (output) container. The renderer may include, for example, the real-time rendering capabilities of the source and cross-platform game engines. The renderer may include a scripting language (i.e., an interpreted programming language) that may be executed by the renderer at runtime to process dynamic input and variable state changes made to scene graph nodes. The dynamic input and variable state changes may affect the rendering and evaluation of spatial and temporal object topology (including physical forces, constraints, inverse kinematics, deformations, collisions), and energy propagation and transport (light, sound). The evaluation of spatial and temporal object topology produces results that move the output from abstract to concrete (e.g., similar to the evaluation of a document object model for a web page).

[0121] Renderer 1001B may be, for example, a modified version of the OTOY Octane renderer that is modified to interact directly with adaptation module 1001. In some embodiments, renderer 1001B implements computer graphics methods (e.g., path tracing) that render three-dimensional scenes such that the lighting of the scene is true to life. In some embodiments, renderer 1001B may employ shaders (i.e., a type of computer program originally used for shading (the generation of appropriate levels of light, darkness, and color in an image) but which now performs various specialized functions in various areas of computer graphics special effects, video post-processing unrelated to shading, and other functions unrelated to graphics).

[0122] The adaptation module 1001 may perform compression and decompression of the media content using a media compressor 1001D and a media decompressor 1001E, respectively, depending on the compression and decompression needs based on the format of the ingested media and the format required by the client device 908. The media compressor 1001D may be a media encoder, and the media decompressor 1001E may be a media decoder. After performing compression and decompression (if necessary), the adaptation module 1001 outputs client adaptation media 1006 suitable for streaming or delivery to the client device 908. The client adaptation media 1006 may be stored in a storage device for storing adaptation media.

[0123] FIG. 11 illustrates an example of a delivery format creation process 1100. As shown in FIG. 11, the delivery format creation process 1100 includes an adaptation media packaging module 1103 that packages the media output from the media adaptation process 1000 and stored as client adaptation media 1006. The media packaging module 1103 formats the adaptation media from the client adaptation media 1006 into a robust delivery format 1104. The delivery format may be, for example, the exemplary format shown in FIG. 3 or FIG. 4. The information manifest 1104A may provide the client device 908 with a list of scene data assets 1104B. The list of scene data assets 1104B may also include complexity metadata that describes the complexity of all assets in the list of scene data assets 1104B. The list of scene data assets 1104B illustrates a list of visual assets, auditory assets, and haptic assets, each with corresponding metadata.

[0124] The media may be further packetized prior to steaming. Figure 12 illustrates an exemplary packetization process 1200. The packetization system 1200 includes a packetizer 1202. The packetizer 1202 may receive a list of scene data assets 1104B as input media 1201 (shown in Figure 12). In some embodiments, the client adapted media 1006 or the delivery format 1104 is input to the packetizer 1202. The packetizer 1202 separates the input media 1201 into individual packets 1203 suitable for presentation or streaming to a client device 908 over a network.

[0125] 13 is a sequence diagram illustrating an example of data and communication flows between components according to an embodiment. The sequence diagram of FIG. 13 is of a network adapting a particular immersive media in an ingest format into a streamable and appropriate delivery format for a particular immersive media client endpoint. The data and communication flows are as follows:

[0126] The client device 908 initiates a media request 1308 to the network orchestrator 905. In some embodiments, the request may be made to a network delivery interface of the client device. The media request 1308 includes information to identify the media requested by the client device 908. The media request may be identified, for example, by a uniform resource name (URN) or another standard nomenclature. The network orchestrator 905 then responds to the media request 1308 with a profile request 1309. The profile request 1309 requests the client to provide information about currently available resources (including computation, storage, battery charge, and other information to characterize the client's current operating state). The profile request 1309 also requests the client to provide one or more NN models that may be used by the network for NN inference to extract or interpolate the correct media view to match the characteristics of the client's presentation system, if such NN models are available at the client endpoint.

[0127] The client device 908 then continues with a response 1310 from the client device 908 to the network orchestrator 905 provided as a client token, an application token, and one or more NN model tokens (if such NN model tokens are available at the client endpoint). The network orchestrator 905 then provides a session ID token 1311 to the client device. The network orchestrator 905 then requests ingest media 1312 from the ingest media server 1303. The ingest media server 1303 may include, for example, the ingest media formats and assets of the ingest media storage 903 or the storage device 1002. The request for ingest media 1312 may also include the URN or other standard name of the media identified in the request 1308. The ingest media server 1303 responds to the request for ingest media 1312 with a response 1313 including the ingest media tokens. The network orchestrator 905 then provides the media token from the response 1313 to the client device 908 in a call 1314. The network orchestrator 905 then initiates the adaptation process for the requested media by providing the ingest media token, the client token, the application token, and the NN model token to the adaptation and fragmentation module 910 in a request 1315. The adaptation and fragmentation module 910 requests access to the ingest media by providing the ingest media token to the ingest media server 1303 in a request 1316 requesting access to the ingest media asset.

[0128] The ingest media server 1303 responds to the request 1316 with the ingest media access token in a response 1317 to the adaptation and fragmentation module 910. The adaptation and fragmentation module 910 then requests that the media adaptation process 1000 adapt the ingest media located in the ingest media access token for the client, application, and NN inference model corresponding to the session ID token created and sent in the response 1313. A request 1318 is made from the adaptation and fragmentation module 910 to the media adaptation process 1000. The request 1318 includes the necessary token and session ID. The media adaptation process 1000 provides the adapted media access token and session ID to the network orchestrator 905 in an update response 1319. The network orchestrator 905 then provides the adapted media access token and session ID to the media packaging module 1103 in an interface call 1320. The media packaging module 1103 provides a response 1321 to the network orchestrator 905 with the packaged media access token and the session ID in response 1321. The media packaging module 1103 then provides the packaged assets for the session ID, the URN, and the packaged media access token in response 1322 to the packaged media server 1307 where they are stored. The client device 908 then performs a request 1323 to the packaged media server 1307 to begin streaming the media asset corresponding to the packaged media access token received in response 1321. Finally, the client device 908 performs another request and provides a status update in message 1324 to the network orchestrator 905.

[0129] Figure 14A shows a workflow for the media reuse analyzer 911 shown in Figure 9. The media reuse analyzer 911 analyzes metadata regarding the uniqueness of objects in scenes contained in media data.

[0130] In S1401, media data is obtained, for example, from a content provider or content source. In S1402, initialization is performed. Specifically, an iterator "i" is initialized to 0. The iterator may be, for example, a counter. A set of lists of unique assets for each scene 1420 (shown in FIG. 14B) is also initialized, which identifies the unique assets that occurred across all scenes that comprise the presentation (shown in FIG. 3 and / or FIG. 4).

[0131] At S1403, a decision process is performed to determine whether the value of iterator "i" is less than the total number N of scenes that contain the presentation. If the value of iterator "i" is equal to (or greater than) the number N of scenes that contain the presentation ("No" at S1403), processing proceeds to S1404, where the reuse analysis ends (i.e., the process ends). If the value of iterator "i" is less than the number N of scenes that contain the presentation ("Yes" at S1403), processing proceeds to S1405. At S1405, the value of iterator "j" is set to 0.

[0132] Then, in S1406, a decision process is performed to determine whether the value of iterator "j" is less than the total number X of media assets (also called media objects) in the current scene. If the value of iterator "j" is equal to (or greater than) the total number X of media assets in scene s ("No" in S1406), processing proceeds to S1407, where iterator "i" is incremented by 1, and then processing returns to S1403. If the value of iterator "j" is less than the total number X of media assets in scene s ("Yes" in S1406), processing proceeds to S1408.

[0133] At S1408, the characteristics of the media asset are compared to previously analyzed assets from scenes prior to the current scene (ie, scene s) to determine if the current media asset has been used previously.

[0134] If the current media asset is identified as a unique asset ("No" at S1408), i.e., the current media asset has not been previously analyzed in a scene associated with a smaller value of the iterator "i", processing proceeds to S1409. At S1409, a unique asset entry is created in the set 1420 of lists of unique assets corresponding to the current scene (i.e., scene s). The unique asset entry is also assigned a unique identifier, and the number of times the asset has been used across scenes 0 to N-1 is set to 1. Processing then proceeds to S1411.

[0135] If the current media asset is identified as an asset that was used in one or more scenes prior to scene s ("Yes" at S1408), processing proceeds to S1409. At S1410, the number of times the current media asset was used across scenes 0 to N-1 is incremented by 1 in the set 1420 of lists of unique assets that correspond to the current scene (i.e., scene s). Processing then proceeds to S1411.

[0136] In S1411, the value of the iterator "j" is incremented by 1. Then, the process returns to S1406.

[0137] In some embodiments, the media reuse analyzer 911 may further signal to a client, e.g., a client device 108, that the client should use a copy of the asset for each instance the asset is used in a set of scenes (after the asset is delivered to the client for the first time).

[0138] It should be noted that the sequence diagrams and workflow steps described with reference to Figures 13-14 are not intended to limit the configuration of data and communication flows in embodiments. For example, one or more of the steps may be performed simultaneously, data may be stored, and / or flow in a direction not explicitly shown in the flows of Figures 13-14A.

[0139] FIG. 14B is an example of a set 1420 of lists of unique assets initialized in S1402 (and potentially updated in S1409-S1410) for all scenes upon completion of a presentation, according to an embodiment. The list of unique assets in the set 1420 of lists of unique assets may be pre-identified or pre-defined by the network or client device. The set 1420 of lists of unique assets shows a sample list of entries of information describing assets that are unique to the entire presentation, including an indicator of the type of media (e.g., mesh, audio, or volume) that makes up the asset, a unique identifier for the asset, and the number of times the asset is used across the set of scenes that make up the entire presentation. As an example, for scene N-1, no assets are included in its list because all assets required for scene N-1 have been identified as assets that are also used in scene 1 and scene 2.

[0140] 15 is a block diagram of an example of computer code 1500 for optimizing media delivery, according to an embodiment. In an embodiment, the computer code may be, for example, program code or computer program code. According to an embodiment of the present disclosure, an apparatus / device may be provided that includes at least one processor having a memory that stores the computer program code. The computer program code, when executed by the at least one processor, may be configured to perform any number of aspects of the present disclosure.

[0141] As shown in FIG. 15, computer code 1500 may include receiving code 1510, obtaining code 1520, analyzing code 1530, and generating code 1540.

[0142] The receiving code 1510 is configured to cause the at least one processor to receive immersive media data for an immersive presentation from a content source.

[0143] The acquisition code 1520 is configured to cause the at least one processor to acquire code configured to cause the at least one processor to acquire asset information corresponding to media assets in a set of scenes included in the immersive media data.

[0144] The analysis code 1530 is configured to cause the at least one processor to analyze characteristics of the media assets used in the set of scenes from the asset information to determine whether each media asset is unique among the media assets.

[0145] The generation code 1540 is configured to cause the at least one processor to generate metadata information that uniquely identifies each media asset for reuse of each media asset in the set of scenes based on determining that each media asset is not unique among the media assets in the set of scenes included in the immersive media data for the immersive presentation.

[0146] Although Figure 15 illustrates example blocks of code, in some implementations an apparatus / device may include additional, fewer, different, or differently arranged blocks relative to the blocks illustrated in Figure 15. Additionally or alternatively, two or more of the blocks of an apparatus / device may be combined. In other words, although Figure 15 illustrates separate blocks of code, the various code instructions need not be separate and may be intermixed.

[0147] While this disclosure has described several exemplary embodiments, there are alterations, permutations, and various substitute equivalents that fall within the scope of this disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope. [Explanation of symbols]

[0148] 100 Media Flow Process 101 Ingest Media Module 102 Network Processing Module 103 Distribution Module 104 Network Devices 105 Network Connection 106 Rendering Module 107 Presentation Module 108 client devices 121 Optimization reuse logic 200 Media conversion decision-making process 205 Converted Media 210 Asset Reuse Logic 300 Time-limited media expression 301 Timed Scene 302 Components 303 Assets 304 Base Layer 305 Attribute enhancement layer 307 Unique Asset List 308 Proxy Assets 310 Proxy Assets 400 Non-time-limited media expressions 402 Components 403 Assets 404 Base Layer 405 Attribute enhancement layer 406 Attribute enhancement layer 408 Unique Asset List 500 Natural Media Synthesis Process 501 First Camera 502 Second Camera 503 Third Camera 504 Synthesizer 505 NN Training 506 Training Images 507 Ingest Format 508 Capture NN model 509 Natural Image Content 510 dashed line 600 Synthetic Media Ingest Creation Process 601 Camera 602 point cloud 603 Computer 604 CGI assets 605 Actors 605A Sensor 606 Motion Data 607 Synthesizer 608 Synthetic Media Ingest Format 700 Computer Systems 701 Keyboard 702 Trackpad 703 Mouse 704 Joystick 705 Microphone 706 Camera 707 Scanner 708 Speaker 709 Screen 710 Medium 711 CD / DVD ROM / RW 712 Thumb Drive 713 Removable Hard Drive or Solid State Drive 714 Communication Network 715 Interface 716 Surrounding Bus 717 Cores 718 Central Processing Unit (CPU) 719 Graphics Processing Unit (GPU) 720 Field Programmable Gate Area (FPGA) 721 Hardware Accelerator 722 core internal mass storage 723 Read-Only Memory (ROM) 724 Random Access Memory (RAM) 725 Graphics Adapter 726 System Bus 800 Network Media Distribution System 801 Content Acquisition Module 802 Content Preparation Module 803 Transmitting Module 804 Gateway 805 Set-Top Box 806 Radio Demodulator 807 Legacy 2D TV 808 WiFi (router) 809 Laptop 2D Display 810 Head-mounted 2D (raster-based) display 811 Lenticular Light Field Display 811A GPU 811B Storage Device 811C Visual Presentation Components 812 Holographic Display 812A CPU 812B GPU 812C Storage Device 812D Visualization Components 813 Mobile Handset Display 814 Augmented Reality (AR) Headset 814A GPU 814B Storage Device 814C Battery 814D Stereoscopic Visual Presentation Components 815 High-density light field display 815A GPU 815B CPU 815C Storage Device 815D Eye Tracking Device 815E Camera 815F Light Field Panel 900 Immersive Media Delivery Process 901 Media Content 902 Network Ingest Format Creation Process 903 Ingest Media Storage 904 Client-specific information 905 Network Orchestrator 906 Delivery media and description information 907 Status Feedback Channel 908 Client Device 908A Visualization Components 908B Network Interface 909 Media Prepared for Distribution 910 Media Adaptation and Fragmentation Module 911 Media Reuse Analyzer 912 dashed line 920 Immersive Media Network Distribution 1000 Media Adaptation Process 1001 Adaptation Module 1001A NN model 1001B Renderer 1001C Processor 1001D Media Compressor 1001E Media Decompressor 1001F Logic Controller 1002 Storage Device 1005 Input Network Status 1006 Client Adaptation Media 1100 Delivery format creation process 1103 Adaptive Media Packaging Module 1104 Delivery Format 1104A Information Manifesto 1104B Scene Data Asset 1200 Packetization system, packetization process 1201 Input Media 1202 Packetizer 1203 packets 1303 Ingest Media Server 1307 Media Server 1308 Media Request 1309 Profile Request 1310 Response 1311 Session ID Token 1312 Ingest Media 1313 Responses 1314 Call 1315 request 1316 request 1317 Responses 1318 request 1319 Update Response 1320 Interface Call 1321 Responses 1322 Responses 1323 request 1324 Messages 1420 Set a list of unique assets 1500 Computer Code 1510 receiving code 1520 Acquisition Code 1530 Analysis Code 1540 Generated Code

Claims

1. 1. A method for optimizing media delivery, executed by at least one processor, the method comprising: receiving immersive media data for an immersive presentation from a content source; obtaining asset information corresponding to media assets in a set of scenes included in the immersive media data; For each scene included in the immersive media data, determining from the asset information whether each media asset included in the scene is a unique media asset that has not been used in a scene prior to the scene in the immersive media data; and generating metadata information indicating the unique media assets associated with each scene in the immersive media data based on determining that each of the media assets is a unique media asset, in order to reuse each of the media assets.

2. creating a list of the unique media assets associated with scenes in the set of scenes when each of the media assets is identified as a unique media asset; assigning a unique identifier to the unique media asset and creating a counter to track the number of times the unique media asset is used across the set of scenes for the immersive presentation; signaling to a client device when each of the media assets is identified as a unique media asset that has not been used in a previous scene, the signaling step instructing the client device to use a copy of each of the media assets for each instance after a first instance in which the each of the media assets is used in the set of scenes; The method of claim 1 further comprising:

3. initializing the counter and the list of unique media assets associated with each scene in the set of scenes upon receiving the immersive media data for the immersive presentation; incrementing the counter for each instance in which the unique media asset is identified in the immersive presentation; The method of claim 2 , further comprising:

4. the asset information includes a base representation of each of the media assets and a set of asset enhancement layers, the set of asset enhancement layers including attribute information corresponding to characteristics of the media assets; 2. The method of claim 1 , wherein when the set of asset enhancement layers is applied to the base representation of each media asset, the base representation of each media asset is extended to include features that are not supported in a base layer that includes the base representation of each media asset.

5. The method of claim 4 , further comprising identifying two or more scenes that share a same media asset, and reusing the same media asset in at least one of the two or more scenes.

6. 2. The method of claim 1, further comprising: determining whether to convert a format of the immersive media data corresponding to the scenes in the set of scenes from a first format to a second format prior to delivery to a client device based on a complexity of the scene.

7. 1. A device for optimizing media delivery, comprising: at least one memory configured to store computer program code; and at least one processor configured to read the computer program code and to operate as instructed by the computer program code, the computer program code including code that causes the at least one processor to perform the method of any one of claims 1 to 6.

8. A computer program product which, when executed by at least one processor of a device for optimizing media delivery, causes said at least one processor to carry out the method according to any one of claims 1 to 6.

Images