Bidirectional presentation data stream using control and data plane channels

Abstract

Aspects of the present disclosure provide methods and apparatus for media processing. In some embodiments, the apparatus includes a processing circuit. The processing circuit is capable of exchanging a plurality of control messages with a server device over a control plane channel using a first transport protocol. The plurality of control messages belong to a control plane of a two-way protocol for immersive media delivery. The processing circuit receives a first plurality of data messages from the server device over a first data plane channel using a second transport protocol. The first plurality of data messages belong to the data plane of the two-way protocol and carry immersive media content. The processing circuit presents the immersive media content carried by the first plurality of data messages.

Description

[Technical field]

[0001] Incorporation by Reference This application claims the benefit of priority to U.S. Patent Application No. 17 / 851,838 entitled "BIDIRECTIONAL PRESENTATION DATASTREAM USING CONTROL AND DATA PLANE CHANNELS," filed June 28, 2022, which claims the benefit of priority to U.S. Provisional Application No. 63 / 217,049 entitled "INTELLIGENT BIDIRECTIONAL PRESENTATION DATASTREAM USING CONTROL AND DATA PLANE CHANNELS," filed June 30, 2021. The disclosure of the prior application is incorporated herein by reference in its entirety.

[0002] Technical Field This disclosure describes embodiments relating generally to media processing and distribution. [Background technology]

[0003] The background discussion provided herein is intended to generally present the context of the present disclosure. Work of the presently cited inventors is not expressly or impliedly admitted as prior art to the present disclosure to the extent that such work is described in this background section, or in aspects of the description that may not otherwise qualify as prior art at the time of filing.

[0004] What is immersive media? in general , Either Or all human senses system Stimulate the senses (vision, hearing, somatosensation, smell, and possibly taste) to create or enhance the user's perception of being physically present in the media experience. This refers to media that is known as "legacy media." It goes beyond what is delivered over existing commercial networks for time-limited two-dimensional (2D) video and corresponding audio. Immersive and legacy media both Timed Media or Untimed Media It can be characterized as either:

[0005] Timed Medi a Timed media refers to media that is structured and presented according to time. Examples include movie features, news reports, and episodic content, all of which are organized according to time. R Legacy video and audio are generally considered timed media.

[0006] Untimed Medi a Untimed media is media that is not structured by time, but is structured by logical, spatial, and / or temporal relationships. Examples include video games in which the user has control of the experience created by the gaming device. Another example of untimed media is a still image photograph taken by a camera. Untimed media may incorporate timed media, for example, in a continuously looped audio or video segment of a video game scene. Conversely, timed media may incorporate untimed media, for example, a video with a fixed still image background.

[0007] An immersive media-enabled device may refer to a device with the capability to access, interpret, and present immersive media. Such media and devices are heterogeneous with respect to the amount and format of media and the number and type of network resources required to distribute such media on a large scale, i.e., to achieve distribution equivalent to legacy video and audio media over a network. In contrast, legacy devices such as laptop displays, televisions, and mobile handset displays are homogenous with respect to their capabilities, since all of these devices consist of rectangular display screens and consume 2D rectangular video or still images as their primary media format. Summary of the Invention

[0008] Aspects of the present disclosure provide methods and apparatus for media processing. In some embodiments, the apparatus includes a processing circuit. The processing circuit is capable of exchanging a plurality of control messages with a server device over a control plane channel using a first transport protocol. The plurality of control messages belong to a control plane of a two-way protocol for immersive media delivery. The processing circuit receives a first plurality of data messages from the server device over a first data plane channel using a second transport protocol. The first plurality of data messages belong to the data plane of the two-way protocol and carry immersive media content. The processing circuit presents the immersive media content carried by the first plurality of data messages.

[0009] In some embodiments, the first transport protocol is the Transmission Control Protocol (TCP) and the second transport protocol is the User Datagram Protocol (UDP).

[0010] In some embodiments, the first transport protocol is a connection-based transport protocol and the second transport protocol is a connectionless transport protocol.

[0011] In some embodiments, the processing circuitry is configured to: Therefore , server device Between The first data plane channel is configured.

[0012] In some embodiments, the processing circuitry provides one or more specific characteristics of the device to the server device over the control plane channel, the one or more specific characteristics being: computational resources of the device; storage resources of the device; service level agreements of a network service provider with the device; immersive application requirements; type a model of the device; a neural network model of the device.

[0013] In some embodiments, the processing circuitry transmits a second plurality of data messages to the server device over a second data plane channel using a second transport protocol, the second plurality of data messages carrying at least one of layer information of a neural network model in the device and media content rendered by the device.

[0014] In some embodiments, the apparatus is a first client device, and the control messages enable a server device to share immersive media content with a second client device. In an embodiment, the processing circuitry transmits, in response to a request from the server device, over a control plane channel, uniform resource identifiers (URIs) of assets cached in immutable storage and URIs of assets available for sharing. type In another embodiment, the processing circuitry responds to a request from the server device by providing a list of the status of each asset accessible by the first client device over the control plane channel. update In another embodiment, the processing circuitry provides, in response to a request from the server device, over a control plane channel, a current status of a particular asset type and one of a particular server-assigned identifier and a particular asset uniform resource identifier (URI).

[0015] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform a method for media processing. [Brief description of the drawings]

[0016] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Figure 1]FIG. 1 illustrates an end-to-end process for timed media delivery in one embodiment. [Diagram 2] FIG. 2 is a diagram illustrating a standard media format used for streaming timed media in one embodiment. [Diagram 3] FIG. 3 illustrates a data model for representing and streaming timed immersive media in some embodiments. [Figure 4] FIG. 4 is a diagram of a data model for untimed immersive representation and streaming in some embodiments. [Diagram 5] FIG. 5 illustrates the process of capturing a natural scene and converting it into a representation that can be used as a capture format for network serving heterogeneous client endpoints in some embodiments. [Figure 6] FIG. 6 illustrates a process of using three-dimensional (3D) modeling tools and formats to create a representation of a synthetic scene that can be used as a capture format for a network that provides endpoints for heterogeneous clients in some embodiments. [Figure 7] FIG. 7 illustrates a network that provides multiple heterogeneous client endpoints in accordance with some embodiments. [Figure 8] FIG. 8 illustrates a network that, in some embodiments, provides adaptation information about particular media represented in a media capture format prior to the network's process of adapting the media for consumption by a particular immersive media client endpoint. [Figure 9] FIG. 9 illustrates a system diagram of a media adaptation process that includes a media rendering transformation to convert source media from a capture format to a specific format suitable for a particular client endpoint in some embodiments. [Figure 10]FIG. 10 illustrates a network that formats adapted source media into a data model suitable for presentation and streaming in some embodiments. [Figure 11] FIG. 11 illustrates a system of media streaming processes that breaks down the data model of FIG. 11 into network protocol packet payloads in some embodiments. [Figure 12] FIG. 12 illustrates a network sequence that adapts a particular immersive media in a capture format to a suitable streamable delivery format for a particular immersive media client endpoint in some implementations. [Figure 13] FIG. 13 is a diagram of a capture media format according to some embodiments. [Figure 14] FIG. 14 is a diagram illustrating the carriage of neural network model information along with an encoded video stream in some embodiments. [Figure 15] FIG. 15 illustrates the carriage of neural network model information along with input immersive media and assets in some embodiments. [Figure 16A] FIG. 16A shows a diagram of messages sent from a server side to a client side using a two-way protocol for immersive media delivery in some implementations. [Figure 16B] FIG. 16B shows a diagram of messages sent from a server side to a client side using a two-way protocol for immersive media delivery in some implementations. [Figure 16C] FIG. 16C shows a diagram of messages sent from a client side to a server side using a bidirectional protocol for immersive media delivery in some embodiments. [Figure 17A] FIG. 17A is a diagram illustrating a definition of a message sent from the server side to the client side in some embodiments. [Figure 17B]FIG. 17B is a diagram illustrating a definition of a message sent from the server side to the client side in some embodiments. [Figure 17C] FIG. 17C illustrates a definition of a message sent from the server side to the client side in some embodiments. [Figure 17D] FIG. 17D illustrates a definition of a message sent from the server side to the client side in some embodiments. [Figure 17E] FIG. 17E illustrates a definition of a message sent from the server side to the client side in some embodiments. [Figure 17F] FIG. 17F illustrates a definition of a message sent from the server side to the client side in some embodiments. [Figure 17G] FIG. 17G illustrates a definition of a message sent from the server side to the client side in some embodiments. [Figure 18] FIG. 18 is a diagram illustrating the definition of a message sent from the client side to the server side in some embodiments. [Figure 19A] FIG. 19A illustrates a bidirectional presentation data stream on separate control plane and data plane channels. [Figure 19B] FIG. 19B is a diagram illustrating a bidirectional presentation data stream on separate control plane and data plane channels. [Figure 19C] FIG. 19C illustrates a bidirectional presentation data stream on separate control and data plane channels. [Figure 19D] FIG. 19D is a diagram illustrating a bidirectional presentation data stream on separate control plane and data plane channels. [Figure 19E] FIG. 19E is a diagram illustrating a bidirectional presentation data stream on separate control plane and data plane channels. [Figure 20]FIG. 20 illustrates a media system that uses a bidirectional protocol over separate control and data plane channels for the delivery of immersive media in some embodiments. [Figure 21] FIG. 21 is a flowchart outlining a process according to some embodiments of the present disclosure. [Figure 22] FIG. 22 is a flowchart outlining another process according to some embodiments of the present disclosure. [Figure 23] FIG. 23 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Aspects of the present disclosure provide techniques for bidirectional presentation data streams over separate data plane and control plane channels. This disclosure generally describes embodiments relating to architectures, structures, and components for systems and networks that distribute media including video, audio, geometric (3D) objects, haptics, associated metadata, or other content for client presentation devices. Some embodiments are directed to systems, structures, and architectures for distributing media content to heterogeneous immersive and interactive client presentation devices.

[0018] Immersive media generally refers to media that stimulates any or all of the human sensory systems (sight, hearing, somatosensation, smell, and possibly taste) to create or enhance the user's perception of being physically present in the media's field of experience; i.e., it is a type of media that is not part of the traditional visual experience known as "legacy media." Limited Beyond what is distributed over existing commercial networks for two-dimensional (2D) video and corresponding audio. Both immersive and legacy media are Timed Media or UntimedIt can be characterized as a medium.

[0019] Timed media is time-based. Therefore Refers to media that is structured and presented over time. Examples include video features, news reports, and episodic content. Therefore Legacy video and audio are generally considered timed media.

[0020] Untimed Medi a Untimed media refers to media that is not structured by time, but is structured by logical, spatial and / or temporal relationships. Examples include video games where the user can control the experience created by the gaming device. Another example of untimed media is a still image photograph taken by a camera. Untimed media may incorporate timed media, for example, in a continuously looped audio or video segment of a scene for a video game. Conversely, timed media may incorporate untimed media, for example, a video with a fixed still image background.

[0021] An immersive media-enabled device may refer to a device with the capability to access, interpret, and present immersive media. Such media and devices are heterogeneous with respect to the amount and format of media and the number and type of network resources required to distribute such media on a large scale, i.e., to achieve distribution equivalent to legacy video and audio media over a network. In contrast, legacy devices such as laptop displays, televisions, and mobile handset displays are homogenous in their capabilities, since all of these devices consist of rectangular display screens and consume 2D rectangular video or still images as their primary media format. Similarly, the number of audio formats supported by legacy devices is limited to a relatively small set.

[0022] The delivery of any media over a network may employ media delivery systems and architectures that reformat the media from an input or network-ingested format into a final delivery format that is suitable for the target client device and its applications as well as being suitable to be streamed over a network. Streaming media broadly refers to the fragmentation and packetization of source media so that the media can be delivered over a network in successive, small-sized "chunks" that are logically organized and sequenced according to either or both of the media's temporal and spatial structures. In such delivery architectures and systems, the media may undergo a compression or layering process so that only the most salient media information is delivered to the client initially. In some cases, a client may receive all of the salient media information for a portion of the media before the client can present any of the same media portions to an end user.

[0023] The process of reformatting input media to match the capabilities of the target client endpoint. MediumA neural network process may employ a network model that may encapsulate prior knowledge of a particular media. For example, a particular model may be tuned to recognize an outdoor park scene (with trees, plants, grass, and other objects common to park scenes), while another particular model may be tuned to recognize an indoor dinner scene (with a dinner table, serving utensils, people sitting at the table, etc.). In some embodiments, the network model may be tuned to recognize objects from a particular context, e.g., objects in a park scene, and a neural network process with a network model tuned to match the content of a particular scene may produce better visual results than a network model that is not so tuned. Thus, it is advantageous to provide scene-specific network models to a neural network process tasked with reformatting input media to fit the capabilities of a target client endpoint.

[0024] A mechanism for associating a neural network model with a particular scene of two-dimensional (2D) media can be achieved, for example, by compressing the network model and inserting the compressed network model directly into the 2D coded bitstream of the visual scene, for example, using the Supplemental Enhancement Information (SEI) structured field commonly used to attach metadata to video streams coded in H.264, H.265, and H.266 video compression formats. The presence of an SEI message containing a particular neural network model within the context of a portion of a coded video bitstream can be used to indicate that the network model should be used to interpret and adapt the video content within the portion of the bitstream in which the model is embedded. In some embodiments, the SEI message can be used to signal which neural network model is to be used, by way of an identifier for the network model, in the absence of the actual model itself.

[0025] The mechanism for associating an appropriate neural network with an immersive media may be achieved by the immersive media itself referencing the appropriate neural network model to be used. This referencing may be achieved by embedding the network model and its parameters directly into the object, on an object-by-object or per-scene basis, or some combination thereof. R In some embodiments, rather than embedding one or more neural network models within the media, a media object or scene may reference a particular neural network model by an identifier. 。

[0026] In some other examples, the mechanism for referencing an appropriate neural network to adapt media for streaming to a client endpoint is for the particular client endpoint itself to provide at least one neural network model and corresponding parameters to the adaptation process to be used. Such a mechanism may be implemented, for example, by the client communicating with the adaptation process to provide the neural network model(s) when the client connects itself to a network.

[0027] Following adaptation of the video to the target client endpoint, the adaptation process in the network may choose to apply a compression algorithm to the result, which in some examples may separate the adapted video signal into layers that correspond to the most salient to the least salient portions of the visual signal.

[0028] In some embodiments, a compression and layering process, such as the progressive format of the JPEG standard (ISO / IEC 10918 Part 1), can separate the image into layers that initially are out of focus, i.e., from the lower DCT coefficients for the entire image scan, causing the entire image to be displayed with only basic contrast and color, and then into layers of additional detail, i.e., from the lower DCT coefficients for the entire image scan, that bring the image in focus.

[0029] The process of breaking down media into smaller pieces, organizing it into payload portions of successive network protocol packets, and distributing these protocol packets is referred to as streaming the media, while the process of converting the media into a format suitable for presentation at one of a variety of heterogeneous client endpoints running one of a variety of heterogeneous applications is referred to as adapting the media.

[0030] Definitions of some terms used in this disclosure are provided in the following paragraphs.

[0031] Scene Graph: A general-purpose data structure commonly used by vector-based graphics editing applications and modern computer games to arrange a logical, and often (though not necessarily) spatial, representation of a graphical scene; a collection of nodes and vertices in a graph structure.

[0032] Node: A basic element of a scene graph consisting of information related to logical or spatial or temporal representation of visual, audio, tactile, olfactory, gustatory, or related processing information; each node shall have at most one outgoing edge, zero or more incoming edges, and at least one edge (either incoming or outgoing) connected to it.

[0033] Base Layer: A nominal representation of an asset, typically formulated to minimize the computational resources or time required to render the asset or transmit the asset over a network.

[0034] Enhancement Layer: A set of information that, when applied to a base layer representation of an asset, augments the base layer to include features or capabilities not supported in the base layer.

[0035] Attribute: Metadata associated with a node that is used to describe a particular characteristic or feature of that node in a standard or more complex format (e.g., with respect to other nodes).

[0036] Container: A serialized format for storing and exchanging information for representing an all-natural, all-composite, or mixed synthetic and natural scene, including the scene graph and all the media resources required to render the scene.

[0037] Serialization: The process of converting a data structure or the state of an object into a format that can be stored (e.g., in a file or memory buffer) or transmitted (e.g., over a network connection link) and later reconstructed (perhaps in another computing environment). The resulting series of bits, when reread according to the serialized format, can be used to create a semantically identical clone of the original object.

[0038] Renderer: A (usually software-based) application or process based on a selective mix of acoustic physics, optical physics, visual recognition, speech recognition, mathematics, and related fields of software development that, given an input scene graph and an asset container, typically emits visual and / or audio signals suitable for presentation on a target device or conforming to desired properties specified by attributes of the render target node in the scene graph. For visual-based media assets, the renderer may emit visual signals suitable for a target display or suitable for storage as an intermediate asset (e.g., to be repackaged into another container, i.e., used in the sequence of rendering processes in a graphics pipeline); for audio-based media assets, the renderer may emit audio signals for presentation on multi-channel loudspeakers and / or binauralized headphones, or for repackaging into another (output) container. Examples of popular renderers include Unity, Unreal, etc.

[0039] Evaluate: Generate results that move the output from abstract to concrete results (e.g., similar to evaluating a Document Object Model for a web page).

[0040] Scripting Language: An interpreted programming language that is executed by the renderer at runtime and can process dynamic input and variable state changes made to scene graph nodes, which affects the rendering and evaluation of spatial and temporal object topology (including physical forces, constraints, IK, transformations, collisions), and energy propagation and transmission (light, sound).

[0041] Shader: A type of computer program originally used for shading (producing the correct levels of light, darkness, and color in an image), but now performing a variety of specialized functions in various areas of computer graphics special effects, or for video post-processing unrelated to shading, or even functions that are not related to graphics at all.

[0042] Path Tracing: A computer graphics method of rendering three-dimensional scenes so that the lighting in the scene is realistic.

[0043] Timed Media: Media that is ordered by time; e.g., has a start and end time according to a particular clock.

[0044] Untimed Media: Media that is organized according to spatial, logical, or temporal relationships; for example, as an interactive experience that is realized according to actions taken by a user(s).

[0045] Neural Network Model: A collection of parameters and tensors (e.g., matrices) that define the weightings (i.e., numerical values) used in well-defined mathematical operations applied to visual signals to arrive at an improved visual output, which may include the interpolation of new views for the visual signal that were not explicitly provided by the original signal.

[0046] According to some aspects of the present disclosure, immersive media may be considered as one or more types of media that, when presented to a human by an immersive media-enabled device, stimulates any of the five senses: sight, hearing, taste, touch, and sound in a manner that is more realistic and consistent with human understanding of experiences in the natural world; i.e.; greater than that achieved with legacy media presented by a legacy device. In this context, the term legacy media refers to two-dimensional (2D) visual media, image frames, either still or moving, and / or corresponding audio where user interaction capabilities are limited to pausing, playing, fast forwarding, or rewinding. Legacy devices may refer to televisions, laptops, displays, and mobile devices whose capabilities are limited to displaying only legacy media.

[0047] In some consumer application scenarios, a presentation device for immersive media (i.e., an immersive media-enabled device) is a consumer hardware device that is specifically equipped with the capability to exploit the specific information embodied by the immersive media, such that the device can create a presentation that more closely resembles human understanding and interaction with the physical world, i.e., beyond the capabilities of legacy devices to do so. Whereas legacy devices are performance constrained to only display legacy media, immersive media devices are not similarly constrained.

[0048] Over the past decade, a multitude of immersive media-enabled devices have been introduced into the consumer market, including head-mounted displays, augmented reality glasses, handheld controllers, haptic gloves, game consoles, etc. Similarly, holographic displays and other formats of volumetric displays are poised to emerge within the next decade. Despite the immediate or imminent availability of these devices, a consistent end-to-end ecosystem for the delivery of immersive media over commercial networks has not materialized for several reasons.

[0049] One of these reasons is the lack of a single standard representation for immersive media that can address the two main use cases associated with the current delivery of media at scale over commercial networks: 1) real-time delivery for live action events, i.e., content is created and delivered to client endpoints in real-time or near real-time; 2) Non-real-time delivery, where the content does not need to be delivered in real time, i.e., the content is physically captured or created. These two use cases can potentially be compared to currently existing broadcast and on-demand formats of distribution, respectively.

[0050] For real-time delivery, content may be captured by one or more cameras or created using computer-generated techniques. In some embodiments, content captured by a (one or more) cameras is referred to herein as natural content, and content created using computer-generated techniques is referred to herein as synthetic content. Media formats representing synthetic content may be formats used in the 3D modeling, visual effects, and CAD / CAM industries and may include object formats and tools such as meshes, textures, point clouds, structured volumes, amorphous volumes (e.g., for fire, smoke, and fog), shaders, procedurally generated geometry, materials, lighting, virtual camera definitions, animations, and the like. Although synthetic content is generated by a computer, synthetic media formats can be used for both natural and synthetic content. However, the process of converting natural content into synthetic media formats (e.g., into a synthetic representation) may be a time-consuming and computationally intensive process and may not be practical for real-time applications and use cases.

[0051] For real-time delivery of natural content, the content captured by the camera can be delivered in a raster format, which is suitable for legacy display devices, as many of these devices are designed to display raster formats as well, i.e., delivery of raster formats is best suited for displays that can only display raster formats, given that legacy displays are homogenously designed to display raster formats.

[0052] However, immersive media-capable displays do not need to be constrained to display raster-based formats. Furthermore, some immersive media-capable displays cannot display media that is only available in raster-based formats. The availability of displays optimized to create immersive experiences based on formats other than raster-based formats is another important reason why there is not yet a consistent end-to-end ecosystem for the delivery of immersive media. be .

[0053] Yet another problem with creating a consistent delivery system for multiple different immersive media devices is that current and emerging immersive media-enabled devices themselves can differ significantly. For example, some immersive media devices are explicitly designed to be used by only one user at a time, such as head-mounted displays. Some other immersive media devices are designed to be used by multiple users simultaneously. For example, the "Looking Glass Factory 8K Display" (hereafter referred to as the "Lenticular Light Field Display") can display content that can be viewed by up to 12 users simultaneously, with each user experiencing their own unique view of the displayed content.

[0054] Further complicating the development of a consistent delivery system is that the number of unique views each display can generate can vary widely. In most cases, legacy displays can only create one view of the content. Lenticular light field displays, on the other hand, can support multiple users, with each user experiencing unique views of the same visual scene. To achieve this creation of multiple views of the same scene, a specific volumetric viewing frustum is created in lenticular light field displays, where 45 unique views of the same scene are required as input to the display. This means that 45 slightly different unique star representations of the same scene must be captured and delivered to the display in a format specific to this one specific display, or view frustum. In contrast, the view frustum of legacy displays is limited to a single two-dimensional plane, and therefore there is no way to display multiple views of the content through the viewing platform of the display, regardless of the number of simultaneous viewers experiencing that display.

[0055] In general, immersive media displays can differ significantly depending on the following characteristics of all displays: the dimensions and volume of the view frustum, the number of viewers supported simultaneously, the optical technology used to fill the view frustum, which can be point-based, ray-based, or wave-based technology, the density of optical units (either points, rays, or waves) occupying the view frustum, the availability of computing power and type of computing (CPU or GPU), the availability of power and power (battery or hardwired), the amount of local storage or cache, access to auxiliary resources such as cloud-based computing and storage. These characteristics contribute to the heterogeneity of immersive media displays, in contrast to the homogeneity of legacy displays, complicating the development of a single delivery system that can support all displays, including both legacy and immersive displays.

[0056] The disclosed subject matter is directed to the development of a network-based media delivery system capable of supporting both legacy and immersive media displays as client endpoints within the context of a single network. Specifically, a mechanism is presented herein for adapting an input immersive media source to a format suitable for the particular characteristics of a client endpoint device, including the application currently running on that client endpoint device. Such an input immersive media source adapting mechanism includes reconciling the characteristics of the input immersive media with the characteristics of a target endpoint client device, including the application running on the client device, and then adapting the input immersive media to a format suitable for the target endpoint and its application. Additionally, the adaptation process may include interpolating additional views, e.g., new views, from the input media to create additional views required by the client endpoint. Such interpolation may be performed with the aid of a neural network process.

[0057] It should be noted that the remainder of the disclosed subject matter assumes, without loss of generality, that the process of adapting an input immersive media source to a particular endpoint client device is the same as or similar to the process of adapting the same input immersive media source to a particular application running on a particular client endpoint device, i.e., the problem of adapting an input media source to the characteristics of an endpoint device is of the same complexity as the problem of adapting a particular input media source to the characteristics of a particular application.

[0058] Legacy devices supported by legacy media are widely prevalent among consumers. They are similarly supported by an ecosystem of legacy media content providers that generate standards-based representations of legacy media, and commercial network service providers that provide the network infrastructure to connect legacy devices to sources of standard legacy content. Beyond their role in distributing legacy media over their networks, commercial network service providers can also facilitate the pairing of legacy client devices to access legacy content on content distribution networks (CDNs). Once paired with access to content in the appropriate format, the legacy client device can request or pull the legacy content from the content server to the device for display to the end user. Nevertheless, an architecture in which a network server pushes the appropriate media to the appropriate client is equally relevant without introducing additional complexity into the overall architecture and solution design.

[0059] BooksAccording to some aspects of the disclosure, a media delivery network supporting heterogeneous clients can exploit the fact that some of the assets adapted from an input media format to a particular target format can be reused across a set of similar display targets (client devices). For example, some assets, once converted to a format suitable for a target display, can be reused across a number of such displays having similar adaptation requirements. In some embodiments, the media delivery network can employ a caching mechanism to store the adapted assets in relatively immutable storage.

[0060] Books According to an aspect of the disclosure, the immersive media comprises: S Also called line description Scene Graph Described by 、 The presentation may be organized into "scenes". The scope of a scene graph is to describe the visual, audio, and other formats of immersive assets that make up a particular setting that is part of the presentation, for example describing an event (such as a movie) and actors that take place in a particular location within a building that is part of the presentation. The list of all scenes that make up a presentation can be formulated in a scene manifest.

[0061] In some embodiments, for content prepared in advance of the need to deliver such content, a "bill of materials" can be created that identifies all assets that will be used throughout the presentation and the frequency with which each asset is used across various scenes within the presentation. The media delivery network can be implemented to recognize the existence of cached resources that can be used to satisfy asset requirements for a particular presentation.

[0062] BooksSome aspects of the disclosure can provide a two-way protocol that can be used between a media distribution network (e.g., a server device in a media distribution network that interfaces the media distribution network with client devices) and client devices. In some embodiments, the two-way protocol can be used in a media distribution network that delivers immersive media. The two-way protocol can support a variety of different client devices that require asset types in different formats. In some embodiments, the two-way protocol enables reuse of assets that were previously adapted for use by a particular client device.

[0063] It is noted that in this disclosure, without loss of generality, the process of adapting an input immersive media source to a particular endpoint client device is the same as or similar to the process of adapting the same input immersive media source to a particular application running on a particular endpoint client device. The techniques for adapting an input media source to the characteristics of an endpoint device may be similar to the techniques for adapting an input media source to the characteristics of a particular application.

[0064] BooksAccording to aspects of the disclosure, legacy devices supported by legacy media have achieved widespread consumer adoption because they are similarly supported by an ecosystem of legacy media content providers that generate standards-based representations of the legacy media, and commercial network service providers that provide the network infrastructure to connect legacy devices to sources of standard legacy content. Beyond their role in distributing legacy media over their networks, commercial network service providers can also facilitate the pairing of legacy client devices to access legacy content on content distribution networks (CDNs). Once paired with access to the content in the appropriate format, the legacy client device can request or pull the legacy content from the content server to the device for display to the end user. Nevertheless, media distribution networks may utilize an architecture in which a network server "pushes" the appropriate media to the appropriate client.

[0065] BooksSome aspects of the disclosure provide a media distribution network that includes an interface device (also referred to as a server device) for interfacing the media distribution network with client devices. The server device can employ a two-way communication protocol (also referred to as a two-way protocol) for communicating with the client devices, and can facilitate adapting media to suit the uniqueness of the client device or requirements arising from an application running on the client device. The server device can also use the two-way protocol to stream entirely newly adapted assets or previously adapted and cached assets to a particular client device. In some embodiments, the server device can use the two-way protocol to support the functionality of a client device requesting specific assistance from the server device, for example, to assist the client device in rendering the asset in preparation for presenting the asset.

[0066] Aspects are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer-readable media according to various embodiments. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0067] This disclosure describes embodiments relating generally to architectures, structures, and components for systems and networks that distribute media including video, audio, geometric (3D) objects, haptics, associated metadata, or other content for client devices. Particular embodiments are directed systems, structures, and architectures for delivering media content to heterogeneous immersive and interactive client devices.

[0068] Figure 1 shows a diagram of an end-to-end process (100) of timed legacy media distribution. In Figure 1, timed audiovisual content is captured by a camera or microphone (101A) or generated by a computer (101B) to create a sequence of 2D images and associated audio (102) that is input to a preparation module (103). The output of the preparation module (103) is the edited content (e.g. for post-production including language translation, subtitling and other editing functions) and is referred to as a standard contribution format, e.g. for live events, or a master format, e.g. for on-demand media, ready to be converted by a converter module (104) into a standard Mezzanine format. In one embodiment, media is ingested by a commercial network service provider and an adaptation module (105) packages the media into various bit rates, temporal resolutions (frame rates) or spatial resolutions (frame sizes) packaged into a standard distribution format. The resulting adaptations are stored on a content delivery network (CDN) 106, from which various clients 108A-108C make pull requests 107A-107C to fetch the media and present it to the end user. It is important to note that the master format may consist of a hybrid of both 101A or 101B media, and the 101A format may be obtained in real time, e.g., from a live sporting event. Additionally, the clients 108A-108C are responsible for selecting the particular adaptation that best suits their configuration and / or current network conditions, although it is equally possible that a network server (not shown in FIG. 1) determines the appropriate content and then pushes it to the clients 108A-108C.

[0069] FIG. 2 shows a diagram of a standard media format (200) used in some embodiments for the delivery of legacy timed media, e.g., video, audio, and supporting metadata (including timed text such as used for subtitling). As shown in FIG. 1 with CDN (106), the media is stored in a standards-based delivery format on a CDN, such as CDNs (201A)-(201C) in FIG. 2. The standards-based format is shown as an MPD (202), which is made up of multiple sections including periods, such as Period (203A) and Period (203B), with start and end times corresponding to a clock. In an embodiment, each period (e.g., (203A)(203B)) refers to one or more adaptation sets (204A)-(204F). Each adaptation set (204A)-(204F) is generally used for a single type of media, such as video, audio, or timed text. In some embodiments, for any given period (e.g., 203A), multiple adaptation sets (e.g., 204A-204C) may be provided; for example, one for video and multiple for audio, such as used for translation into various languages. Each adaptation set (204A-204F) is associated with one or more representations (205) that provide information about the frame resolution (for video), frame rate, and bit rate of the media. The multiple representations (205) may be used to provide access to respective representations (205) of, for example, ultra-high definition, high definition, or standard definition video. Each representation (205) is associated with one or more segment files (206) where the media is actually stored for fetching by a client (such as those shown in FIG. 1 as 108A-108C) or for distribution by a network media server (not shown in FIG. 1) (a push-based architecture).

[0070] Figure 3 shows, in an embodiment, a representation of a streamable format (300) for timed or timed heterogeneous immersive media. Figure 4 shows, in an embodiment, a representation of a streamable format (400) for untimed or timed heterogeneous immersive media. In the case of Figure 3, Figure 3 shows a timed media scene (301). In the case of Figure 4, Figure 4 shows an untimed media scene (401). In both cases, the scene may be implemented by a variety of scene representations, or scene descriptions.

[0071] For example, in some immersive media designs, a scene may be implemented as a scene graph, or a multi-plane image (MPI), or a multi-sphere image (MSI). Both MPI and MSI technologies are examples of technologies that support the creation of display-agnostic scene representations for natural content, i.e., real-world images captured simultaneously from one or more cameras. Scene graph technologies, on the other hand, may be used to represent both natural and computer-generated images in the form of synthetic representations, but such representations are particularly computationally intensive to create for cases where the content is captured as a natural scene by one or more cameras. That is, scene graph representations of naturally captured content are time-consuming and computationally intensive to create, and require complex analysis of the natural imagery using photogrammetry and / or deep learning techniques to create a synthetic representation that can then be used to interpolate a sufficient and appropriate number of views to fill the view frustum of the target immersive client display. As a result, such synthetic representations cannot be practically created in real time to account for use cases that require real-time delivery, and therefore are currently not practical to consider as candidates for representing natural content. In some embodiments, since computer-generated images are created using 3D modeling processes and tools, the best candidate representation for computer-generated images is to employ the use of a scene graph with a synthetic model.

[0072] This dichotomy in optimal representations of both natural and computer-generated content suggests that the optimal capture format for naturally captured content may be different from the optimal capture format for computer-generated content or natural content that is not critical for real-time delivery applications. Thus, the disclosed subject matter aims to be robust enough to support multiple capture formats of visually immersive media, whether naturally created or computer-generated.

[0073] The following are examples of techniques that embody a scene graph as a format suitable for representing visually immersive media created using computer-generated techniques, or naturally captured content that uses deep learning or photogrammetry techniques to create corresponding synthetic representations of natural scenes, i.e., not required for real-time delivery applications.

[0074] 1. ORBX® by OTOY

[0075] ORBX by OTOY is one of several scene graph technologies that can support any kind of visual media, timed or untimed, including ray-traceable, legacy (frame-based), volumetric, and other kinds of synthetic or vector-based visual formats. According to one aspect, ORBX is unique from other scene graphs because it provides native support for freely available and / or open source formats for meshes, point clouds, and textures. ORBX is a scene graph purposefully designed to facilitate exchange across multiple vendor technologies that operate on the scene graph. In addition, ORBX provides a rich material system, support for open shader languages, a robust camera system, and support for Lua scripting. ORBX is also the basis of an immersive technology media format licensed under royalty-free terms by the immersive digital experiences alliance (IDEA). In the context of real-time delivery of media, the ability to create and deliver ORBX representations of natural scenes is a function of the availability of computational resources to perform complex analysis of camera-captured data and composite that same data into a synthetic representation. Currently, it is not practical to have sufficient computing available for real-time delivery, but it is not impossible.

[0076] 2. Universal Scene Description by Pixar

[0077] Universal Scene Description (USD) by Pixar is an alternative scene graph that can be used by the visual effects (VFX) and professional content creation communities. USD is integrated into NVIDIA's Omniverse platform, a developer toolset for creating and rendering 3D models with NVIDIA's GPUs. A subset of USD has been published by Apple and Pixar as USDZ. USDZ is supported by Apple's ARKit.

[0078] 3. glTF 2.0 by Khronos

[0079] glTF 2.0 is the latest version of the graphics language transmission format specification written by the Khronos 3D Group. The format supports a simple scene graph format that can typically support static (untimed) objects in a scene (such as the "png" or "jpeg" image formats). glTF 2.0 supports simple animation, including support for moving, rotating, and scaling basic shapes, or geometric objects, described using glTF primitives. glTF 2.0 does not support timed media, and therefore does not support video or audio.

[0080] It should be noted that the above scene representations of immersive visual media are provided by way of example only and do not limit the disclosed subject matter in its ability to specify a process for adapting an input immersive media source into a format suitable for the particular characteristics of a client's endpoint device.

[0081] Additionally, any or all of the above example media representations currently employ or may employ deep learning techniques to train and create neural network models that enable or facilitate the selection of specific views to fill a particular display's view frustum based on the particular dimensions of the frustum. The views selected for a particular display's view frustum may be interpolated from existing views that are explicitly provided in the scene representation (e.g., from MSI or MPI techniques) or may be rendered directly from a render engine based on, for example, the particular virtual camera positions, filters, or virtual camera descriptions of those render engines.

[0082] The disclosed subject matter is therefore robust enough to consider that there is a relatively small but well-known set of immersive media capture formats sufficient to meet the requirements for both real-time or "on-demand" (e.g., non-real-time) delivery of media that is naturally captured (e.g., by one or more cameras) or created using computer-generated techniques.

[0083] Interpolation of views from immersive media ingest formats, using either neural network models or network-based rendering engines, will become even easier as advanced network technologies such as 5G for mobile networks and fiber optic cable for fixed networks are deployed. In other words, these advanced network technologies increase the capacity and capabilities of commercial networks, as such advanced network infrastructure can support the transmission and delivery of increasingly large amounts of visual information. Network infrastructure management technologies such as multi-access edge computing (MEC), software-defined networking (SDN), and network function virtualization (NFV) allow commercial network service providers to flexibly configure their network infrastructure to adapt to changes in demand for specific network resources, for example, in response to dynamic increases and decreases in network throughput, network speed, round-trip latency, and demand for computing resources. Furthermore, this inherent ability to adapt to dynamic network requirements facilitates the network's ability to adapt immersive media ingest formats to appropriate delivery formats to support a variety of immersive media applications with potentially heterogeneous visual media formats for heterogeneous client endpoints.

[0084] Immersive media applications themselves may have different requirements for network resources, including gaming applications that require significantly lower network latency to respond to real-time updates on the state of the game, telepresence applications that have symmetric throughput requirements for both the uplink and downlink portions of the network, and passive viewing applications that may have increasing demands on downlink resources depending on the type of client endpoint display that is consuming the data. In general, consumer applications may be supported by a variety of client endpoints with different on-board client capabilities for storage, computing, and power, and may likewise be supported by different requirements for the particular media presentation.

[0085] The disclosed subject matter thus enables a fully equipped network, i.e., a network that adopts some or all of the characteristics of modern networks, to simultaneously support multiple legacy and immersive media capable devices in accordance with the characteristics specified therein.

[0086] 1. It provides the flexibility to leverage practical media ingest formats for both real-time and on-demand media delivery use cases.

[0087] 2. It provides the flexibility to support both natural and computer-generated content across both legacy and immersive media-enabled client endpoints.

[0088] 3. Supports both timed and untimed media.

[0089] 4. Provide a process that dynamically adapts the ingest format of the source media to an appropriate delivery format based on the features and capabilities of the client endpoint and based on the requirements of the application.

[0090] 5. Ensure that your delivery format can be streamed over IP-based networks.

[0091] 6. Allows the network to simultaneously serve multiple heterogeneous client endpoints, which may include both legacy and immersive media capable devices.

[0092] 7. We provide an exemplary media representation framework that facilitates the organization of distributed media along scene boundaries.

[0093] An end-to-end embodiment of the improvements enabled by the disclosed subject matter is accomplished according to the processes and components described in the detailed description of FIGS.

[0094] Each of Figures 3 and 4 employs an exemplary generic delivery format that can be adapted from an ingestion source format to fit the capabilities of a particular client endpoint. As can be seen, the media shown in Figure 3 is timed and the media shown in Figure 4 is untimed. The particular generic format is sufficiently robust in its structure to accommodate a variety of media attributes that can each be hierarchically layered based on the amount of salient information each layer contributes to the media presentation. It is noted that the hierarchical layering process can be applied, for example, to progressive JPEG and scalable video architectures (e.g., as specified in ISO / IEC 14496-10 Scalable Advanced Video Coding).

[0095] According to one aspect, media streamed according to the generic media format is not limited to legacy visual and audio media, but may include any type of media information capable of generating signals that interact with a machine to stimulate human senses for sight, sound, taste, touch, and smell.

[0096] According to another aspect, media streamed according to the generic media format can be both timed media or untimed media, or a mixture of both.

[0097] According to another aspect, the generic media format is further streamable by enabling layered representations of media objects using a base layer and enhancement layer architecture. In one embodiment, separate base and enhancement layers are computed by application of multi-resolution or multi-tessellation analysis techniques to the media objects of each scene. This is similar to progressively rendered image formats specified in ISO / IEC 10918-1 (JPEG) and ISO / IEC 15444-1 (JPEG 2000), but is not limited to raster-based visual formats. In an embodiment, the progressive representation of the geometric object may be a multi-resolution representation of the object computed using wavelet analysis.

[0098] In another embodiment of a layered representation of a media format, an enhancement layer applies different attributes to a base layer, such as refining the material properties of the surface of a visual object represented by the base layer. In yet another embodiment, the attributes may refine the texture of the surface of a base layer object, such as changing the surface from a smooth texture to a porous texture, or from a matte surface to a glossy surface.

[0099] In yet another example of a layered representation, the surfaces of one or more visual objects in a scene may be modified from Lambertian to ray-traceable.

[0100] In yet another example of a layered representation, the network delivers a base layer representation to a client, allowing the client to create a nominal presentation of the scene while the client waits for the transmission of additional enhancement layers to improve the resolution or other characteristics of the base representation.

[0101] According to another aspect, the resolution of the attributes or refinement information of the enhancement layer is not explicitly tied to the resolution of the objects in the base layer, as is currently done in existing MPEG video and JPEG image standards.

[0102] According to another aspect, the generic media format supports any kind of information media that can be presented or acted upon by a presentation device or machine, thereby enabling support of heterogeneous media formats for heterogeneous client endpoints. In one embodiment of a network for delivering a media format, the network first queries the client endpoint to determine the client's capabilities, and if the client is unable to meaningfully capture the media representation, the network removes layers of attributes not supported by the client or adapts the media from its current format to a format appropriate for the client endpoint. In one example of such adaptation, the network converts volumetric visual media assets into 2D representations of the same visual assets by using a network-based media processing protocol. In another example of such adaptation, the network employs neural network processes to reformat the media into an appropriate format or optionally synthesize a view required by the client endpoint.

[0103] According to another aspect, a manifest for a fully or partially fully immersive experience (live streaming event, game, or on-demand asset playback) is organized by scenes, which are the minimum amount of information that rendering and game engines can currently ingest to create a presentation. The manifest includes a list of individual scenes to be rendered for the entire immersive experience requested by the client. Associated with each scene are one or more representations of the geometric objects in the scene that correspond to a streamable version of the scene geometry. One embodiment of a scene representation relates to a low-resolution version of the geometric objects for the scene. Another embodiment of the same scene relates to enhancement layers of the low-resolution representation of the scene, adding additional details to the geometric objects of the same scene and increasing the tessellation. As mentioned above, each scene can have multiple enhancement layers to progressively increase the details of the geometric objects of the scene.

[0104] According to another aspect, each layer of a media object referenced in a scene is associated with a token (e.g., a URI) that points to an address where the resource can be accessed in the network, similar to a CDN where a client can obtain the content.

[0105] According to another aspect, a token for a representation of a geometric object may point to a location in the network or a location in the client, i.e., the client can signal to the network that its resources are available on the network for network-based media processing.

[0106] 3 illustrates an embodiment of a generic media format for timed media in some embodiments. In some embodiments, a timed scene manifest includes a scene (also referred to as scene information) (301). The scene (301) includes processing information and a media asset. type The components (302) are associated with a list of assets (303) which further reference a base layer (304) and an attribute enrichment layer (305).

[0107] FIG. 4 illustrates an embodiment of a generic media format for untimed media in some implementations. An untimed scene manifest includes a scene (401). A scene (also referred to as scene information) (401) is not associated with a start and end duration according to a clock. A scene (401) is associated with a list of components (402) that individually describe the processing information and type of media asset that comprises the scene (401). The components (402) are associated with assets (403) (e.g., visual, audio, haptic assets) that further reference a base layer (404) and an attribute enrichment layer (405). Additionally, the scene (401) is associated with other scenes (411) for untimed media. A scene (401) may be associated with a timed media scene (407) in some implementations.

[0108] 5 shows a diagram of a process 500 for synthesizing a capture format from natural content. The process 500 includes a first sub-process for content capture and a second sub-process for capture format synthesis for natural images.

[0109] In the embodiment of FIG. 5, in the first sub-process, the camera units can be used to capture natural image content (509). For example, the camera unit (501) can capture a scene of a person using a single camera lens. The camera unit (502) can capture a scene with five diverging fields of view by mounting five camera lenses around a ring-shaped object. The device of (502) is an exemplary device for capturing omnidirectional content for VR applications. The camera unit (503) captures a scene with seven converging fields of view by mounting seven camera lenses on the inner diameter part of a sphere. The device of (503) is an exemplary device for capturing a light field or a light field for a holographic immersive display.

[0110] In the example of FIG. 5, in the second sub-process, natural image content (509) is synthesized. For example, the natural image content (509) is provided as input to a synthesis module (504), which may employ a neural network training module (505) using a collection of training images (506) to generate a capture neural network model (508). Another process commonly used instead of a training process is photogrammetry. When a model (508) is created during the process (500) shown in FIG. 5, the model (508) becomes one of the assets of a natural content capture format (507). Exemplary embodiments of the capture format (507) include MPI and MSI.

[0111] FIG. 6 shows a diagram of a process (600) for creating synthetic media (608), such as a capture format for computer-generated imagery. In the embodiment of FIG. 6, a LIDAR camera (601) captures a point cloud (602) of a scene. A computer-generated imagery (CGI) tool, a 3D modeling tool, or another animation process for creating synthetic content is used by a computer (603) to create a CGI asset over a network (604). A sensored motion capture suit (605A) is worn by an actor (605) to capture a digital recording of the actor's (605) movements and generate animated motion capture (MoCap) data (606). Data (602), (604), and (606) are provided as inputs to a synthesis module (607), which may also use a neural network and training data to create a neural network model (not shown in FIG. 6).

[0112] FIG. 7 illustrates a network media distribution system (700) that supports a variety of legacy media and heterogeneous immersive media-enabled displays as client endpoints in some embodiments. In the embodiment of FIG. 7, a content acquisition module (701) captures or creates media using the exemplary embodiment of FIG. 6 or FIG. 5. Ingestion formats are created in a content preparation module (702) and then transmitted to one or more client endpoints in the network media distribution system using a transmission module (703). A gateway (704) may provide services to customer premise equipment to provide network access to various client endpoints to the network. A set-top box (705) may also function as customer premise equipment to provide access to content aggregated by a network service provider. A wireless demodulator (706) may function as a mobile network access point for mobile devices (such as, for example, a mobile phone or a display (713)). In one or more embodiments, a legacy 2D television (707) may connect directly to the gateway (704), a set-top box (705), or a WiFi router (708). A computer laptop with a legacy 2D display (709) may be a client endpoint connected to the WiFi router (708). A head-mounted 2D (raster-based) display (710) may also connect to the router (708). A lenticular light field display (711) may connect to the gateway (704). The display (711) may consist of a local computing GPU (711A), a storage device (711B), and a visual presentation unit (711C) that creates multiple views using ray-based lenticular optics technology.The holographic display (712) may be connected to the set-top box (705) and may include a local computing CPU (712A), a GPU (712B), a storage device (712C), and a Fresnal pattern wave-based holographic visualization unit (712D). The augmented reality headset (714) may be connected to the wireless demodulator (706) and may include a GPU (714A), a storage device (714B), a battery (714C), and a volumetric visual presentation component (714D). Possible The high density light field display (715) may connect to the WiFi router (708) and may include multiple GPUs (715A), a CPU (715B), and a storage device (715C); an eye tracking device (715D); a camera (715E); and a high density beam-based light field panel (715F).

[0113] FIG. 8 shows a diagram of an immersive media delivery module (800) capable of accommodating legacy media and heterogeneous immersive media-enabled displays as previously shown in FIG. 7. Content is created or acquired in module (801), which is depicted in FIGS. 5 and 6 for natural content and CGI content, respectively. The content is then converted to an ingest format using a network ingest format creation module (802). Several embodiments of module (802) are depicted in FIGS. 5 and 6 for natural content and CGI content, respectively. The ingested media format is sent to the network and stored in a storage device (803). In some other embodiments, the storage device may reside in the immersive media content producer's network and be accessed remotely by the immersive media network delivery module (800), as indicated by the bisecting dotted line. Client and application specific information may be available on a remote storage device (804) in some embodiments, and may optionally reside remotely in an alternative cloud network in embodiments.

[0114] As shown in FIG. 8, the client interface module (805) (e.g., referred to as a server device in some examples) serves as the primary source and sink of information for performing the primary tasks of the distribution network. In this particular embodiment, the client interface module (805) may be implemented in a unified format with other components of the network. Nevertheless, the tasks illustrated by the client interface module (805) of FIG. 8 form elements of the disclosed subject matter in some examples. The client interface module (805) may further employ a bidirectional protocol for communication with the client device to facilitate all processing and distribution of media according to the characteristics of the client device. Furthermore, the bidirectional protocol may be implemented across different distribution channels, i.e., control plane channels and data plane channels.

[0115] Books According to some aspects of the disclosure, the client interface module (805) may further employ a bidirectional protocol for communicating with client devices to facilitate processing and delivery of media (e.g., immersive media) in accordance with the characteristics of the client devices.

[0116] The client interface module (805) receives information about the characteristics and attributes of client devices, such as the client (808) of FIG. 8 (also referred to as the client device (808)), and also gathers requirements for applications currently running on the client (808). This information can be obtained from the device (804) or, in another embodiment, by directly querying the client (808). In some examples, a bidirectional protocol is used to enable direct communication between the client interface module (805) and the client (808). For example, the client interface module (805) can send a query directly to the client (808). Examples of messages for the bidirectional protocol are provided with reference to FIGS. 16A-C, 17A-G, and 18.

[0117] The client interface module (805) also initiates and communicates with a media adaptation and fragmentation module (810) as described in FIG. 9. As the ingested media is adapted and fragmented by the module (810), the media is transferred to an intermedia storage device, shown in some embodiments as prepared media for distribution storage device (809). Once the distribution media is prepared and stored on the device (809), the client interface module (805) transmits the media to an immersive client (808) via its network interface (808B) which receives the distribution media and corresponding description information (806) through a push request, or the client (808) itself may initiate a pull request for the media (806) from the storage device (809). The immersive client (808) may employ a GPU (or CPU, not shown) (808C) in some embodiments. The distribution format of the media is stored in the client's (808) storage device or storage cache (808D). Finally, the immersive client (808) visually presents the media via its visualization component (808A).

[0118] Throughout the process of streaming immersive media to an immersive client (808), the client interface module (805) can monitor the status of the client's progress via a client progress and status feedback channel (807).

[0119] FIG. 9 shows a diagram of a media adaptation process in some embodiments, so that the captured source media is appropriately adapted to fit the requirements of the immersive client (808). The media adaptation module (901) comprises several components that facilitate the adaptation of the captured media to an appropriate delivery format for the immersive client (808). In FIG. 9, the media adaptation module (901) receives an input network status (905) to track the current traffic load on the network. The immersive client (808) information can include attributes and feature descriptions, application characteristics and descriptions, and the current status of the application, as well as a client neural network model that helps map the geometry of the client's frustum to the interpolation functions of the captured immersive media. The media adaptation module (901) ensures that the adapted output, as created, is stored in the client adaptive media storage device (906).

[0120] In some embodiments, the media adaptation module (901) employs a renderer (901B) or a neural network processor (901C) to adapt the particular source media to a format suitable for the client. In one embodiment, the media adaptation module (901) receives client information (904) from a client interface module (903), such as a server device. The client information (904) can include a client description and current status, an application description and current status, and a client neural network model. The neural network processor (901C) uses the neural network model (901A). An example of such a neural network processor (901C) includes the DeepView neural network model generator, as described in MPI and MSI. In some embodiments, the media is in a 2D format, but the client requires a 3D format, and then the neural network processor (901C) can invoke a process that uses highly correlated images from the 2D video signal to derive a volumetric representation of the scene depicted in the video. An example of such a process could be the neural radiance fields (NeRF) from one or more image processes developed at the University of California, Berkeley. An example of a suitable renderer (901B) could be a modified version of the OTOY Octane renderer (not shown) that is modified to interact directly with the media adaptation module (901). The media adaptation module (901) may, in some examples, employ a media compressor (901D) and a media decompressor (901E) as required by these tools with respect to the format of the captured media and the format required by the immersive client (808).

[0121] Figure 10 illustrates an adaptive media packaging process that, in some implementations, ultimately converts the adapted media from a media adaptation module (1001) (e.g., corresponding to the media adaptation module (901) of Figure 9) currently residing on a client adaptation media storage device (1002). For example, a media packing module (1003) formats the adapted media from the media adaptation module (1001) into a robust delivery format (1004), e.g., the exemplary formats shown in Figures 3 or 4. The manifest information (1004A) provides the immersive client (808) with a list of scene data that the client (808) can expect to receive, as well as a list of visual assets and corresponding metadata, and audio assets and corresponding metadata.

[0122] Figure 11 shows some implementation 11 illustrates the fragmentation process in an example. In the example of Figure 11, a packetizer (1102) fragments various adaptation media (1101) into individual packets (1103) suitable for streaming to an immersive client (808), shown as a client endpoint (1104) on the network.

[0123] FIG. 12 illustrates a network sequence that adapts a particular immersive media in a capture format to a suitable streamable delivery format for a particular immersive media client endpoint in some implementations.

[0124] The components and communications depicted in Figure 12 are described as follows: A client (1201) (also referred to as a client endpoint in some embodiments) initiates a media request (1208) to a network delivery interface (1202). The media request (1208) includes information identifying the media requested by the client (1201), either by URN or other standard naming convention. The network delivery interface (1202) teethThe client (1201) responds to the media request (1208) with a profile request (1209) that requests the client (1201) to provide information about currently available resources (including computation, storage, percentage of battery charged, and other information that characterizes the client's current operating status). The profile request (1209) also requests the client to provide one or more neural network models that can be used by the network for neural network inference to extract or interpolate the correct media view that matches the characteristics of the client's presentation system, if such models are available at the client. A response (1210) from the client (1201) to the network delivery interface (1202) provides a client token, an application token, and one or more neural network model tokens (if such neural network model tokens are available at the client). The network delivery interface (1202) then provides a session ID token (1211) to the client (1201). The network distribution interface (1202) then requests the ingest media server (1203) with a ingest media request (1212) that includes a URN or standard nomenclature name for the media identified in the request (1208). The ingest media server (1203) responds to the request (1212) with a response (1213) that includes the ingest media token. The network distribution interface (1202) then provides the media token from the response (1213) in a call (1214) to the client (1201). The network distribution interface (1202) then initiates an adaptation process for the media request (1208) by providing the ingest media token, the client token, the application token, and the neural network model token (1215) to the adaptation interface (1204). The adaptation interface (1204) requests access to the ingested media by providing the ingest media token to the ingest media server (1203) in a call (1216) to request access to the ingested media assets.The ingested media server (1203) responds to the request (1216) with the ingested media access token in response to the adaptation interface (1204) (1217). The adaptation interface (1204) requests the media adaptation module (1205) to adapt the ingested media located in the ingested media access token for the client, application, and neural network inference model corresponding to the session ID token created (1213). The request (1218) from the adaptation interface (1204) to the media adaptation module (1205) includes the necessary token and session ID. The media adaptation module (1205) provides the adapted media access token and session ID to the network delivery interface (1202) (1219). The network delivery interface (1202) provides the adapted media access token and session ID to the package module (1206) (1206) (1206) (1206) (1206) (1206) (1206) (1206). The adaptation interface (1204) responds to the request (1218) from the media adaptation module (1205) with the ingested media access token in response to the adaptation interface (1204) (1206). The adaptation interface (1204) responds to the media adaptation module (1205) (1206 ... The packaging module (1206) provides a response (1221) to the network delivery interface (1302) using the media access token and session ID packaged in the response message (1221). The packaging module (1206) provides the packaged media access token of the packaged asset, the URN, and the session ID in a response (1222) to the packaged media server (1207). teeth , and starts streaming the media asset corresponding to the packaged media access token received in the response message (1221). Execute the request (1223) to The client (1201) performs other requests and provides status updates of messages (1224) to the network delivery interface (1202).

[0125] Figure 13 illustrates the captured media format and assets (902) of Figure 9, which in some embodiments consists of two parts: immersive media and assets in a 3D format (1301) and a 2D format (1302). The 2D format (1302) can be a single-view coded video stream, for example ISO / IEC 14496 Part 10 Advanced Video Coding, or a coded video stream that includes multiple views, for example the multiview compression modification of ISO / IEC 14496 Part 10.

[0126] FIG. 14 is a diagram illustrating the conveyance of neural network model information along a coded video stream. In FIG. 14, a coded video stream (1401) includes a neural network model and one or more SEI messages (1401A) and code 9. In the coded video stream (1402), the one or more SEI messages carry an identifier for the neural network model and its corresponding parameters (1402A) and the coded video bitstream (1402B). In the (1402) scenario, the neural network model and parameters are stored outside the coded video stream, e.g., (901A) in FIG. 9.

[0127] FIG. 15 illustrates the carriage of neural network model information in the captured immersive media and assets 3D format (1501) (corresponding to the immersive media and assets in 3D format (1301) of FIG. 13). The captured immersive media and assets 3D format (1501) references scenes 1 to N represented as 1502). Each scene (1502) references geometry (1503) and processing parameters (1504). The geometry (1503) may include a reference (1503A) to a neural network model. The processing parameters (1504) may include a reference (1504A) to a neural network model. Both (1504A) and (1503A) may reference a network model stored directly with the scene, or an identifier that references a neural network model that exists outside of the captured media, e.g., a network model stored in 901A of FIG. 9.

[0128] It should be noted that the various modules of the present disclosure may be individual devices or components within a device. In some embodiments, the modules are individual devices and may be coupled with other devices. In some embodiments, the modules are processing circuits and may be interconnected with other processing circuits. In some embodiments, the modules are software instruction modules and may be executed by one or more processors.

[0129] BooksSome aspects of the disclosure provide a two-way protocol with two-way messages that enable a client interface (also referred to as a server device) of a media delivery network to communicate directly with a client presentation endpoint (i.e., also referred to as a client device). In some embodiments, the two-way protocol can be implemented at a network layer above the transport layer. In some embodiments, using the two-way protocol, the server device can obtain information about client-specific characteristics and supported capabilities of the client device. In some embodiments, using the two-way protocol, the server device can manage the delivery of media to the client device by establishing a unique session and operational context for the adaptation and delivery of media to the client device. In some embodiments, using the two-way protocol, the server device can respond to conditions during the session if or when the client device needs to rely on the network to supplement the client device's resources, e.g., for computing or storage, in the presentation of a particular media. In some embodiments, the two-way protocol can assist the server device in the adaptation process of the input media to match the capabilities of the client device. In some embodiments, the two-way protocol allows for efficient reuse of a particular media asset for presentation across multiple client devices, if or when such reuse is permitted by, for example, a studio or content creator of the client device. In some embodiments, using a bidirectional protocol, a server device can facilitate near real-time presentation of media to a client device in accordance with existing service level agreements between a network operator (also referred to as a wireless service provider, wireless carrier, mobile network carrier) and the client device.In some embodiments, using a bidirectional protocol, a server device can facilitate near real-time display of media by a client device according to the requirements of the application, e.g., interactive games versus linear passive visual experiences.

[0130] Books According to one aspect of the disclosure, a bidirectional protocol is used to form a bidirectional presentation data stream for a server device in direct communication with a client device, the bidirectional presentation data stream including a series of messages exchanged between the server device and the client device to facilitate timely adaptation of input media to a delivery format suitable for the client device and streaming of the adapted media to the client device for presentation.

[0131] The bidirectional protocol can be used to build a media distribution network that supports a wide variety of client devices requiring asset types in different formats and can reuse assets that have been previously adapted for use by a particular client.

[0132] 16A-16C provide a list of messages that, in some embodiments, can be exchanged between a server device (e.g., shown in FIG. 8 as a client interface module (805)) and a client device (e.g., shown in FIG. 8 as an immersive client (808)). In some embodiments, the list of messages can include messages and information (806) that originate from the server device and are sent to the client device, and can include feedback and status (807) that originate from the client device and are sent to the server device.

[0133] 16A-16B show a first group of messages (messages numbered 1 through 20) sent from a server device to a client device, and FIG. 16C shows a second group of messages (messages numbered 21 through 31) sent from a client device to a server device in accordance with some embodiments.

[0134] Figures 17A-17G show tables of semantic information for a first group of messages in some embodiments, and Figure 18 shows tables of semantic information for a second group of messages in some embodiments.

[0135] Note that the messages in Figures 16A-16C, 17A-17G, and 18 are illustrated. Messages in the two-way protocol may be modified and / or omitted. Additional messages may be added.

[0136] In some implementations, a media distribution network can facilitate the adaptation and streaming of media, such as immersive media content, using messages defined by a two-way protocol. In some examples, the two-way protocol can include specific messages signaling actions to perform. For example, a specific message can cause a client device to send its processing characteristics to a server device, so that upon receiving the processing characteristics of the client device, the server device has enough information to meaningfully adapt the captured media to a format suitable for the client device. In another example, messages can be exchanged between a server device and a client device to allow the server device to send a manifest of scenes for a presentation that an end user of the client device wants to experience. For example, upon receiving such a manifest, the client device can begin the steps necessary to prepare to create the presentation, including requesting each asset for the first scene, the second scene, etc. The server device can respond to each request for an asset by either sending the asset from a media source directly to the client or by informing the client to fetch the asset from a nearby database. In response to a message request, the client device or the server device can respond with a positive response indicating that the request in the message was successfully performed, or a negative response indicating that an error occurred.

[0137] Books Some aspects of the disclosure provide techniques that facilitate the implementation of a two-way protocol. For example, messages in a two-way protocol are divided into two categories: control messages and data messages. In some embodiments, data messages are used to transmit media data for distribution. of The control message includes control information for delivering the data message. ofFor example, the control message may include set-up information for preparing the delivery of the data message, processing information during delivery of the data message, confirmation information after delivery of the data message, etc.

[0138] Additionally, separate communication technologies are used to deliver control messages and data messages. For example, data messages may be delivered using a first communication technology that allows for fast delivery, and control messages may be delivered using a second communication technology that ensures reliable delivery. For example, data messages may be delivered using a low-latency network transport protocol such as User Datagram Protocol (UDP), and control messages may be delivered using a more reliable, connection-based network transport protocol such as Transmission Control Protocol (TCP). R TCP can guarantee ordering, reliability, and consistency, while UDP can reduce overhead and latency and send larger data units.

[0139] In some embodiments, the control messages form a control plane, and the control plane channel is associated with an integrated component of the media distribution network architecture for delivery of the control messages. In an embodiment, the control plane channel can be established according to a TCP protocol for delivery of the control messages. Furthermore, the data messages form a data plane, and the data plane channel is associated with an integrated component of the media distribution network architecture for delivery of the data messages. In an embodiment, the data plane channel can be established according to UDP for delivery of the data messages.

[0140] In this manner, media data can be transmitted over the data plane channel with less latency, and control information such as configuration information, monitoring information, and status notifications to assist in transmission over the data plane channel can be transmitted over the control plane channel to either successfully transmit a data message or detect errors in transmission over the data plane channel and trigger retransmission on the data plane channel.

[0141] Books According to disclosed aspects, bidirectional communication using separate control and data plane channels can be used to build a media distribution network that can support a variety of diverse client devices requiring asset types in different formats and can reuse assets that were previously adapted for use with a particular client device. The separation of bidirectional communication over the control and data plane channels can, in some examples, implement a more robust and efficient media distribution network, with data messages being transmitted over a low latency but unreliable transport layer, while control messages are transmitted over a more reliable transport layer with slower delivery of messages.

[0142] FIG. 19A illustrates a block diagram of a media system (1900) according to some embodiments of the present disclosure. The media system (1900) includes a server device (1901) and a client device (1906). In some examples, the server device (1901) may also be referred to as a client interface module in a media distribution network (e.g., client interface module (805)). In the example of FIG. 19A, communication channels (1902), (1903), (1904), (1905) may be established for bidirectional communication between the server device (1901) and the client device (1906). In some examples, the communication channels refer to physical transmission media. vinegar It also refers to the logical connections on multiplexed media in telecommunications and computer networks. vinegarIt is also possible.

[0143] Specifically, the communication channel (1902) is configured to deliver control messages from the server device (1901) to the client device (1906). Figure 19B shows a list of control messages from the two-way protocol in Figures 16A-16C that can be delivered by the communication channel (1902).

[0144] A communication channel (1903) is configured to deliver control messages from the client device (1906) to the server device (1901). Figure 19C shows a list of control messages from the two-way protocol in Figures 16A-16C that can be delivered by the communication channel (1903).

[0145] In some embodiments, communication channel (1902) and communication channel (1903) can be bidirectional control plane channels. The control messages of Figures 19B and 19C form the control plane of a bidirectional protocol.

[0146] The communication channel (1904) is configured to deliver data messages from the server device (1901) to the client device (1906). FIG. 19D illustrates the two-way protocol from FIGS. 16A-16C that can be delivered by the communication channel (1904). data A list of messages is shown.

[0147] A communication channel (1905) is configured to deliver data messages from the client device (1906) to the server device (1901). FIG. 19E illustrates the two-way protocol from FIGS. 16A-16C that can be delivered by the communication channel (1905). data A list of messages is shown.

[0148] In some embodiments, communication channel (1904) is a unidirectional data plane channel and communication channel (1905) is another unidirectional data plane channel. The data messages of Figures 19D and 19E form the data plane of a bidirectional protocol.

[0149] Note that the messages in Figures 19B-19E are for illustration purposes. Messages in communication channels 1902-1905 may be modified and / or omitted. Additional messages may be added.

[0150] Figure 20 shows a block diagram of a media system (2000) to illustrate the use of a two-way protocol in some embodiments. In some embodiments, the two-way protocol can include the messages of Figures 16A-16C, 17A-17G, and 18. The messages can be divided into control plane and data plane according to Figures 19A-19E.

[0151] The media system (2000) can be used in a variety of applications, including immersive media applications, augmented reality (AR) applications, virtual reality applications, video game applications, sports game animation applications, video conferencing and telepresence applications, and media streaming applications.

[0152] The media system (2000) includes a server device (2010) and a number of media client devices, such as client devices (2060A), (2060B), (2060C) shown in FIG. 20, that may be connected by a network (not shown). In one embodiment, the server device (2010) may include one or more devices with immersive media coding capabilities. In one embodiment, the server device (2010) may include a single computing device, such as a desktop computer, a laptop computer, a server computer, a tablet computer, or the like. In another example, the server device (2010) may include a data center(s), a server farm(s), or the like. The server device (2010) may receive input media content, such as immersive content, video content, audio content, or the like. The client devices (e.g., client devices (2060A), (2060B), (2060C)) each include one or more devices with media presentation capabilities for a media application. In one embodiment, the media client device may include a presentation device such as a desktop computer, a laptop computer, a server computer, a tablet computer, a head mounted display (HMD) device, a lenticular light field display, etc. The media client device, in some embodiments, may present media according to a suitable media presentation format.

[0153] The server device (2010) may be implemented using any suitable technology. In the example of FIG. 20, the server device (2010) includes a processing circuit (2030) and an interface circuit (2011) coupled together. of include.

[0154] The processing circuitry (2030) may include any suitable processing circuitry, such as one or more central processing units (CPUs), one or more graphics processing units (GPUs), an application specific integrated circuit, etc. In the example of FIG. 20, the processing circuitry (2030) is configured to form messages according to a bidirectional protocol and can interpret messages according to the bidirectional protocol. Additionally, the processing circuitry (2030) may generate a media stream carrying the media content. In some implementations, the media stream may be adapted based on messages exchanged between the server device (2010) and the media client device.

[0155] The interface circuit (2011) can interface the server device (2010) with a network. The interface circuit (2011) can include a receiving portion that receives signals from the network and a transmitting portion that transmits signals to the network. For example, the interface circuit (2011) can transmit signals carrying messages to other devices, such as the client device (2060A), the client device (2060B), and the client device (2060C), over the network. The interface circuit (2011) can receive signals carrying messages from media client devices, such as the client devices (2060A), (2060B), and (2060C).

[0156] The network is suitably coupled to the server device (2010) and client devices (e.g., client devices (2060A), (2060B), (2060C)) via wired and / or wireless connections, such as Ethernet connections, fiber optic connections, WiFi connections, cellular network connections, etc.

[0157] The client devices (e.g., client devices (2060A), (2060B), (2060C)) are each configured for media presentation and two-way communication using a two-way protocol.

[0158] Media client devices, such as client devices 2060A, 2060B, and 2060C, may be implemented using any suitable technology. In the example of FIG. 20, client devices 2060A and 2060B are shown, but are not limited to, a head mounted display (HMD) with earphones as user equipment available for use by respective users, such as user A and user B. Client device 2060C is shown, but is not limited to, a lenticular light field display capable of displaying content that can be simultaneously viewed by up to multiple users, each user experiencing their own perspective (i.e., view) of the displayed content.

[0159] In Figure 20, client device (2060A) includes an interface circuit (2061A) and a processing circuit (2070A) coupled together as shown in Figure 20. Client device (2060B) includes an interface circuit (2061B) and a processing circuit (2070B) coupled together as shown in Figure 20. Client device (2060C) includes an interface circuit (2061C) and a processing circuit (2070C) coupled together as shown in Figure 20.

[0160] The interface circuit (2061A) can interface the client device (2060A) with a network. The interface circuit (2061A) can include a receiving section for receiving a signal from the network and a transmitting section for transmitting a signal to the network. For example, the interface circuit (2061A) can receive a signal carrying a message from the server device (2010) and transmit a signal carrying a message to the server device (2010).

[0161] The processing circuitry (2070) may include any suitable processing circuitry, such as a CPU, a GPU, an application specific integrated circuit, etc. The processing circuitry (2070A) may be configured to include various components, such as a media decoder, a rendering, etc.

[0162] Similarly, the interface circuit (2061B) can interface the client device (2060B) with the network. The interface circuit (2061B) can include a receiving section for receiving a signal from the network and a transmitting section for transmitting a signal to the network. For example, the interface circuit (2061B) can receive a signal carrying a message from the server device (2010) and transmit a signal carrying a message to the server device (2010).

[0163] The processing circuitry (2070B) may include any suitable processing circuitry, such as a CPU, a GPU, an application specific integrated circuit, etc. The processing circuitry (2070B) may be configured to include various components, such as a media decoder, a rendering, etc.

[0164] Similarly, the interface circuit (2061C) can interface the client device (2060C) with the network. The interface circuit (2061C) can include a receiving section for receiving a signal from the network and a transmitting section for transmitting a signal to the network. For example, the interface circuit (2061C) can receive a signal carrying a message from the server device (2010) and transmit a signal carrying a message to the server device (2010).

[0165] The processing circuitry (2070C) may include any suitable processing circuitry, such as a CPU, a GPU, an application specific integrated circuit, etc. The processing circuitry (2070C) may be configured to include various components, such as a media decoder, a rendering, etc.

[0166] BooksAccording to one aspect of the disclosure, the server device (2010) and the client device (2060A) can set up a control plane channel (2001) for exchanging (sending and receiving) control messages therebetween; the server device (2010) and the client device (2060B) can set up a control plane channel (2003) for exchanging control messages therebetween; and the server device (2010) and the client device (2060C) can set up a control plane channel (2005) for exchanging control messages therebetween. In one embodiment, the control plane channels (2001), (2003), and (2005) can use TCP and can perform bidirectional transmission of control messages.

[0167] In some embodiments, control A data plane channel can be established in the media system (2000) based on the control messages exchanged on the control plane channels (2001), (2003), (2005). In one embodiment, a data plane channel (2002) can be established between the server device (2010) and the client device (2060A) based on the control messages exchanged on the control plane channel (2001); a data plane channel (2004) can be established between the server device (2010) and the client device (2060B) based on the control messages exchanged on the control plane channel (2003); and a data plane channel (2006) can be established between the server device (2010) and the client device (2060C) based on the control messages exchanged on the control plane channel (2005). In one embodiment, the data plane channels (2002), (2004), (2006) can use UDP and can perform unidirectional transmission of data messages.

[0168] In some embodiments, using the control plane channels (2001), (2003), (2005), the server device (2010) can obtain information regarding the client-specific characteristics and supported features of the client devices (2060A), (2060B), (2060C). R In one embodiment, the server device (2010) can generate messages to each of the client devices (2060A), (2060B), (2060C) to request information from each of the media client devices via the control channel planes (2001), (2003), (2005). This information can include, but is not limited to, the computing resources of the media client device, the storage resources of the media client device, the service level agreement of the media client device with the network service provider, the immersive application requirements of the media client device, the type of the media client device, the model of the media client device, and the neural network model of the client device. The client devices (2060A), (2060B), and (2060C) can provide the requested information in response to messages received from the server device (2010) via the control plane channels (2001), (2003), and (2005). In some embodiments, client devices 2060A, 2060B, 2060C may spontaneously provide client-specific characteristics and supported capabilities without being solicited.

[0169] In some embodiments, using control messages exchanged over the control plane channels (2001), (2003), (2005), the server device (2010) can manage the delivery of media to the media client devices by establishing a unique session and operational context for the adaptation and delivery of media to the media client devices. In some embodiments, the control messages exchanged over the control plane channels (2001), (2003), (2005) can assist the media server device (2010) in the process of adapting the input media to fit the capabilities of the media client devices.

[0170] For example, the server device (2010) can establish a first unique session (e.g., data plane channel (2002)) with the client device (2060A) based on control messages exchanged on the control plane channel (2001). The server device (2010) can generate a first media stream adapted from the input media to fit the capabilities of the client device (2060A). The data plane channel (2002) can provide the first media stream to the client device (2060A).

[0171] The server device (2010) can establish a second unique session (e.g., data plane channel (2004)) with the client device (2060B) based on control messages exchanged on the control plane channel (2003). The server device (2010) can generate a second media stream adapted from the input media to suit the capabilities of the client device (2060B). The data plane channel (2004) can provide the second media stream to the client device (2060B).

[0172] The server device (2010) can establish a second unique session (e.g., data plane channel (2006)) with the client device (2060C) based on control messages exchanged on the control plane channel (2005). The server device (2010) can generate a third media stream adapted from the input media to suit the capabilities of the client device (2060C). The data plane channel (2006) can provide the third media stream to the client device (2060C).

[0173] In some embodiments, using the control plane channel, the server device (2010) can respond to in-session conditions if or when the media client device needs to rely on the network to supplement the client device's resources, such as computing or storage, in displaying a particular media. In one embodiment, the client device (2060B) notifies the server device (2010) via the control plane channel (2003) that the client device (2060B) needs to rely on the network to supplement computing resources, such as computational resources for rendering. The server device (2010) can decide to provide the client device (2060B) with the supplemental computing resources. For example, the server device (2010) can perform computationally intensive media processing on the media data for the client device (2020B).

[0174] In some embodiments, the control plane channel can be used to efficiently reuse a particular media asset for a presentation across multiple client devices, if such reuse is permitted by a content creator, such as a studio or client device. In one embodiment, the server device (2010) can determine whether to reuse a media asset of the client device (2060A) to the client device (2060B). For example, the server device (2010) can obtain information of a media asset rendered from the client device (2060A) to a cache (e.g., storage (2099)) via the control plane channel (2001) and provide the information to the client device (2060B) via the control plane channel (2003). The client device (2060B) can then access the cache of the rendered media asset according to the information. The storage (2099) can be an internal component of the client device (2060A) or an external component of the client device (2060A).

[0175] In another embodiment, the server device (2010) requests the client device (2060A) to send the rendered media assets back to the server device (2010), which can then provide the received media assets to the client device (2060B). In one embodiment, a separate data plane channel (2007) is configured to send the rendered media assets back from the client device (2060A) to the server device (2010) via the control plane channel (2001).

[0176] 21 shows a flow chart outlining a process (2100) according to one embodiment of the present disclosure. The process (2100) may be used in a server device, such as the server device (2010). In various embodiments, the process (2100) is performed by a processing circuit, such as the processing circuit (2030) of the server device (2010). In some embodiments, the process (2100) is implemented with software instructions, such that when the processing circuit executes the software instructions, the processing circuit executes the process (2100). The process starts at (S2101) and proceeds to (S2110).

[0177] At (S2110), the server device exchanges a plurality of control messages with the client device over a control plane channel using the first transport protocol. The plurality of control messages belong to a control plane of an interactive protocol for immersive media delivery. In one example, the interactive protocol for immersive media delivery can be implemented at a network layer above a transport layer, such as an application layer or a presentation layer, in an Open System Interconnection (OSI) model.

[0178] At (S2120), the server device transmits a first plurality of data messages to the client device over a first data plane channel using a second transport protocol, the first plurality of data messages belonging to a data plane of the bidirectional protocol and carrying at least immersive media content, such as 3D graphic data.

[0179] In some embodiments, the first transport protocol is Transmission Control Protocol (TCP) and the second transport protocol is User Datagram Protocol (UDP).

[0180] In some embodiments, the first transport protocol is a connection-based transport protocol and may be bidirectional, and the second transport protocol is a connectionless transport protocol and may be unidirectional.

[0181] In some embodiments, the server device can establish a first data plane channel with the client device according to a plurality of control messages exchanged over the control plane channel.

[0182] In some examples, the server device can receive one or more particular characteristics of the client device over the control plane channel and can adapt the media stream carried in the first plurality of data messages according to the one or more particular characteristics. In some examples, the one or more particular characteristics can include at least one of: computing resources of the client device, storage resources of the client device, a service level agreement of a network service provider at the client device, an immersive application requirement, a type of the client device, a model of the client device, and a neural network model at the client device.

[0183] In some embodiments, the server device can receive a second plurality of data messages from the client device over a second datapoon channel using a second transport protocol, the second plurality of data messages carrying at least one of layer information of a neural network model at the client device and media content rendered by the client device. In embodiments, the server device can transmit media content rendered by the client device to another client device.

[0184] In some embodiments, the client device is a first client device, and the control messages enable the server device to share immersive media content with a second client device. In one embodiment, the server device receives the control messages from the first client device over a control plane channel in response to a request from the server device. , a list of types of assets that can be shared; Cached in immutable storage , those Assets Noyu In another embodiment, the server device can receive a status update for each asset accessible to the first client device over a control plane channel in response to a request from the server device. In another embodiment, the server device can receive a current state of a particular asset type and either a particular server-assigned identifier and a particular asset uniform resource identifier (URI) from the first client device over a control plane channel in response to a request from the server device. The server device can provide a control message to the second client device over a control plane channel between the server device and the second client device, and the control message can enable the second client device to access the immersive media content of the first client device, such as a cached asset.

[0185] Thereafter, the process proceeds to (S2199) and ends.

[0186] The method 2100 may be adapted as appropriate. Steps of the process 2100 may be modified and / or omitted. Additional step(s) may be added. Any suitable order of performance may be used.

[0187] FIG. 22 shows a flow chart outlining a process (2200) according to one embodiment of the present disclosure. The process (2200) can be used in a client device, such as client device (2060A), client device (2060B), or client device (2060C). In various embodiments, the process (2200) is performed by a processing circuit, such as processing circuit (2070A), processing circuit (2070B), or processing circuit (2070C). In some embodiments, the process (2200) is implemented with software instructions, such that when the processing circuit executes the software instructions, the processing circuit executes the process (2200). The process starts at (S2201) and proceeds to (S2210).

[0188] At (S2210), the client device exchanges a plurality of control messages with the server device on a control plane channel using a first transport protocol. The plurality of control messages belong to a control plane of an interactive protocol for immersive media delivery. In one embodiment, the interactive protocol for immersive media delivery can be implemented in a network layer above a transport layer, such as an application layer or a presentation layer, in an Open System Interconnection (OSI) model.

[0189] At (S2220), the client device receives a first plurality of data messages from the server device on a first data plane channel using a second transport protocol, the first plurality of data messages belonging to a data plane of the bidirectional protocol and carrying at least immersive media content.

[0190] At (S2230), the client device may present the immersive media content carried by the first plurality of data messages.

[0191] In some embodiments, the first transport protocol is the Transmission Control Protocol (TCP) and the second transport protocol is the User Datagram Protocol (UDP).

[0192] In some embodiments, the first transport protocol is a connection-based transport protocol and is bidirectional, and the second transport protocol is a connectionless transport protocol and is unidirectional.

[0193] In some embodiments, the client device is configured to receive control messages exchanged over a control plane channel. Therefore , server device Between Configure the first data plane channel.

[0194] In some embodiments, the client device provides, over the control plane channel, to the server device one or more particular characteristics of the client device, the one or more particular characteristics including at least one of: computing resources of the client device, storage resources of the client device, a service level agreement of a network service provider at the client device, an immersive application requirement, a type of the client device, a model of the client device, and a neural network model at the client device.

[0195] In some embodiments, the client device can transmit a second plurality of data messages to the server device over a second data plane channel using a second transport protocol, the second plurality of data messages carrying at least one of layer information of a neural network model of the client device or media content rendered by the client device.

[0196] In some embodiments, the client device is a first client device, and the control messages enable the server device to share immersive media content with a second client device. In one embodiment, the first client device, in response to a request from the server device, sends, over a control plane channel: The types of assets that can be shared, Cached in immutable storage , those Assets Noyu Uniform Resource Identifier (URI) of In another embodiment, the first client device provides a status update for each asset accessible to the first client device over a control plane channel in response to a request from the server device. In another embodiment, the first client device provides a current state of a particular asset type and either a particular server-assigned identifier or a particular asset uniform resource identifier (URI) over the control plane channel in response to a request from the server device. The server device uses the information received from the first client device to control a second client device to access immersive media content, such as cached assets.

[0197] Thereafter, the process proceeds to (S2299) and ends.

[0198] The method 2200 may be adapted as appropriate. Step(s) of the process 2200 may be modified and / or omitted. Additional step(s) may be added. Any suitable order of performance may be used.

[0199] The techniques described above can be implemented as computer software using computer readable instructions and can be physically stored on one or more computer readable media. For example, Figure 23 illustrates a computer system (2300) suitable for implementing certain embodiments of the disclosed subject matter.

[0200] Computer software can be coded using any suitable machine code or computer language that can be subject to assembly, compilation, linking, or similar mechanisms to produce code including instructions that can be executed directly or via implementation, microcode execution, etc., by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc.

[0201] The instructions may be executed on various types of computers or components thereof including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, internet of things, etc.

[0202] 23 for the computer system (2300) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiment of the computer system (2300).

[0203] The computer system (2300) may include certain human interface input devices that may be responsive to input by one or more human users, for example, via tactile input (e.g., keystrokes, swiping, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), or olfactory input (not shown). Human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic images).

[0204] The input human interface devices may include one or more of a keyboard (2301), a mouse (2302), a trackpad (2303), a touch screen (2310), a data glove (not shown), a joystick (2305), a microphone (2306), a scanner (2307), and a camera (2308).

[0205] The computer system (2300) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (2310), data gloves (not shown), or joystick (2305), but may also be haptic feedback devices that do not act as input devices), audio output devices (e.g., speakers (2309), headphones (not shown)), visual output devices (e.g., screens (2310), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without haptic feedback capabilities, some of which may allow two-dimensional visual output or three- or more-dimensional output via such means as stereoscopic output, such as virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown), and printers (not shown).

[0206] The computer system (2300) can also include human accessible storage devices and their accessible media, such as optical media drives (2320) including CD / DVD ROM / RW with media (2321) such as CD / DVD, USB memory (2322), removable head drives or solid state drives (2323), conventional magnetic media such as tape, floppy disks (not shown), specialized ROM / ASIC / PLD based devices such as security dongles, etc.

[0207] Those skilled in the art will also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transitory signals.

[0208] The computer system (2300) may also include an interface (2354) to one or more communication networks (2355). The networks may be, for example, wireless, wired, or optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, and the like. Examples of networks include Ethernet, WLAN, cellular networks including GSM, 3G, 4G, 5G, LTE, and the like, cable TV, satellite TV, and terrestrial broadcast TV, industrial and vehicular including CANBus. Certain networks generally require an external network interface adapter connected to a particular general-purpose data port or peripheral bus (2349) (e.g., a USB port on the computer system (2300)); others are generally integrated into the core of the computer system (2300) by connecting to a system bus (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system) as described below. Using any of these networks, the computer system (2300) may communicate with other entities. Such communications may be unidirectional, receive only (e.g., broadcast television), unidirectional transmit only (e.g., a CAN bus to a particular CAN bus device), or bidirectional, for example to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks may be used in each of these networks and network interfaces, as described above.

[0209] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be connected to a core (2340) of the computer system (2300).

[0210] The cores (2340) may include one or more central processing devices (CPUs) (2341), graphics processing devices (GPUs) (2342), specialized programmable processing devices in the form of field programmable gate areas (FPGAs) (2343), hardware accelerators for specific tasks (2344), graphics adapters (2350), etc. These devices may be connected via a system bus (2348), along with read only memory (ROM) (2345), random access memory (2346), internal mass storage devices such as internal non-user accessible hard drives, SSDs, etc. (2347). In some computer systems, the system bus (2348) is accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be connected directly to the core's system bus (2348) or through a peripheral bus (2349). In an embodiment, a screen (2310) may be connected to the graphics adapter (2350). Peripheral bus architectures include PCI, USB, and the like.

[0211] The CPU (2341), GPU (2342), FPGA (2343), and accelerator (2344) may combine to execute certain instructions that may constitute the computer code described above. The computer code may be stored in a ROM (2345) or a RAM (2346). Transient data may also be stored in the RAM (2346), while permanent data may be stored, for example, in an internal mass storage device (2347). Fast storage and retrieval in any of the memory devices may be enabled through the use of a cache memory that may be closely associated with one or more of the CPU (2341), GPU (2342), mass storage device (2347), ROM (2345), RAM (2346), etc.

[0212] The computer-readable media can have computer code thereon for performing various computer-implemented operations. The media and computer code can be those specially designed and created for the present disclosure, or they can be of the type well known and available to those skilled in the art in the computer software art.

[0213] As an example, and not by way of limitation, a computer system having the architecture (2300), and specifically the core (2340), can provide functionality as a result of a processor (including CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be media associated with a user-accessible mass storage device as described above, as well as a specific storage device of the core (2340) that is non-transitory in nature, such as a core-internal mass storage device (2347) or a ROM (2345). Software implementing various embodiments of the present disclosure may be stored in such devices and executed by the core (2340). The computer-readable media may include one or more memory devices or chips, depending on the particular needs. The software defines data structures stored in RAM (2346) for the core (2340) and specifically the processor (including CPU, GPU, FPGA, etc.) therein, and executes the processes defined by the software. Therefore , to perform certain processes or portions thereof described herein, including modifying such data structures. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (2344)), which may operate in place of or in conjunction with software to perform certain processes or portions thereof described herein. Reference to software includes logic, and vice versa, where appropriate. Reference to computer-readable medium may include circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry embodying logic for execution, or both, as appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0214] While this disclosure describes several exemplary embodiments, there are alterations, permutations, and various equivalents that fall within the scope of the invention. It will thus be appreciated that those skilled in the art will be able to create numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the invention and thus are within its concept and scope.

Claims

1. 1. A method performed by a client device for media streaming, comprising: exchanging control messages between the client device and the server device over a control plane channel using a first transport protocol, the control messages belonging to a control plane of a bidirectional protocol for immersive media content delivery and including information enabling the server device to share media assets stored in the client device with a second client device, the exchanging control messages including providing, upon request from the server device, a status update of each media asset accessible by the client device over the control plane channel, the media assets including immersive media content; receiving, from the server device, a first plurality of data messages over a first data plane channel using a second transport protocol independent of the first transport protocol, the first plurality of data messages belonging to a data plane of the bidirectional protocol; presenting immersive media content carried by the first plurality of data messages; A method comprising:

2. the first transport protocol is a Transmission Control Protocol (TCP) and the second transport protocol is a User Datagram Protocol (UDP); The method of claim 1.

3. The first transport protocol is a connection-based transport protocol, and the second transport protocol is a connectionless transport protocol. The method of claim 1.

4. The method further comprises: establishing the first data plane channel with the server device in accordance with the plurality of control messages exchanged over the control plane channel. The method of claim 1.

5. The method further comprises: providing one or more particular characteristics of the client device to the server device over the control plane channel, The one or more particular characteristics are: Computational resources of the client device; a storage resource of the client device; a service level agreement with a network service provider at said client device; network resource requirements of an immersive application on said client device; The type of the client device; A model of the client device; information about a neural network model provided by the client device; [0036] The method of claim 1.

6. The method further comprises: transmitting a second plurality of data messages to the server device over a second data plane channel using the second transport protocol; The second plurality of data messages include: Layer information of a neural network model at the client device; media content rendered by said client device; [0033] The method of claim 1.

7. The step of exchanging a plurality of control messages includes: In response to a request from the server device, over the control plane channel, a list of types of shareable media assets and uniform resource identifiers (URIs) of the media assets cached in a persistent storage of the client device. The method of claim 1.

8. wherein the step of exchanging a plurality of control messages includes providing, upon request from the server device, over the control plane channel, a current status of a particular asset type and one of a particular server-assigned identifier and a particular asset uniform resource identifier (URI). The method of claim 1.

9. A method for media streaming, performed by a server device, comprising: exchanging control messages between the server device and the client device over a control plane channel using a first transport protocol, the control messages belonging to a control plane of a bidirectional protocol for immersive media content delivery and including information enabling the server device to share media assets stored in the client device with a second client device, the exchanging control messages including receiving a status update for each media asset accessible by the client device over the control plane channel, the media assets including immersive media content; transmitting a first plurality of data messages to the client device over a first data plane channel using a second transport protocol independent of the first transport protocol, the first plurality of data messages belonging to a data plane of the bidirectional protocol; A method comprising:

10. 1. An apparatus for media streaming, comprising: A processing circuit configured to carry out the method according to any one of claims 1 to 9, Device.

Images