Method, computer system, and storage medium for streaming immersive media

Abstract

A method, computer program and computer system for streaming immersive media are provided. Content is ingested in a first two-dimensional format or a first three-dimensional format, whereby the format references a neural network. The ingested content is converted into a second two-dimensional format or a second three-dimensional format based on the referenced neural network. The converted content is streamed to a client endpoint, such as a television, a computer, a head-mounted display, a lenticular light field display, a holographic display, an augmented reality display or a dense light field display.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 127,036 (filed December 17, 2020) filed with the United States Patent and Trademark Office and U.S. Patent Application No. 17 / 407,816 (filed August 20, 2021) filed with the United States Patent and Trademark Office, both of which are incorporated herein by reference in their entirety. Technical Field

[0003] This disclosure generally relates to the field of data processing, and more specifically to video coding. Background Technology

[0004] "Immersive media" generally refers to media that stimulates any or all human sensory systems (visual, auditory, tactile, olfactory, and possibly gustatory) to create or enhance the user's physical experience of the media, exceeding the perceptions of "traditional media" that are distributed across existing commercial networks for timed two-dimensional (2D) video and corresponding audio. Both immersive and traditional media can be characterized as timed or non-timed.

[0005] Time-based media refers to media that is structured and presented according to time. Examples include film features, news reports, and TV series content, all of which are organized according to time periods. Traditional video and audio are often considered time-based media.

[0006] Irregular media are media that are not structured in time; instead, they are structured through logical, spatial, and / or temporal relationships. Examples include video games, where users control an experience created by the gaming device. Another example of irregular media is a still image photograph taken by a camera. Irregular media can contain timed media, such as timed media contained within a continuously looping audio or video clip in a video game scene. Conversely, timed media can contain irregular media, such as video with a fixed still image as a background.

[0007] Devices with immersive media capabilities can refer to devices equipped with the ability to access, interpret, and present immersive media. Such media and devices are inconsistent in terms of the quantity and format of the media, as well as the quantity and type of network resources required for large-scale distribution of such media; that is, the quantity and type of network resources required to achieve distribution equivalent to that of traditional video and audio media over the network are inconsistent. In contrast, traditional devices (such as laptops, televisions, and mobile handheld displays) are similar in their capabilities because all of these devices include rectangular displays and consume 2D rectangular video or still images as their primary media format. Summary of the Invention

[0008] The embodiments relate to a method, system, and computer-readable medium for streaming immersive media. According to one aspect, a method for streaming immersive media is provided. The method may include: ingesting content in a first two-dimensional or first three-dimensional format, whereby the format references a neural network; converting the ingested content into a second two-dimensional or second three-dimensional format based on the referenced neural network; and streaming the converted content to a client endpoint, such as a television, computer, head-mounted display, lenticular light field display, holographic display, augmented reality display, or dense light field display.

[0009] According to another aspect, a computer system for streaming immersive media is provided. The computer system may include one or more processors, one or more computer-readable storage devices, one or more computer-readable tangible storage devices, and program instructions stored on at least one of the one or more storage devices for execution via at least one of the one or more processors, thereby enabling the computer system to perform a method. The method may include: ingesting content in a first two-dimensional or first three-dimensional format, whereby the format references a neural network; converting the ingested content into a second two-dimensional or second three-dimensional format based on the referenced neural network; and streaming the converted content to a client endpoint, such as a television, computer, head-mounted display, lenticular light field display, holographic display, augmented reality display, or dense light field display.

[0010] According to another aspect, a computer-readable medium is provided for streaming immersive media. The computer-readable medium may include one or more computer-readable storage devices and program instructions stored on at least one of the one or more tangible storage devices, the program instructions being executable by a processor. The program instructions, executable by the processor, perform a method, which may include: ingesting content in a first two-dimensional or first three-dimensional format, whereby the format references a neural network; converting the ingested content into a second two-dimensional or second three-dimensional format based on the referenced neural network; and streaming the converted content to a client endpoint, such as a television, computer, head-mounted display, lenticular light field display, holographic display, augmented reality display, or dense light field display. Attached Figure Description

[0011] These and other objects, features, and advantages will become apparent from the following detailed description of illustrative embodiments, taken in conjunction with the accompanying drawings. The various features in the drawings are not drawn to scale because they are shown to clearly facilitate understanding by those skilled in the art in conjunction with the detailed description. In the drawings:

[0012] Figure 1 This is a schematic diagram of the end-to-end process of timed traditional media distribution.

[0013] Figure 2 This is a schematic diagram of a standard media format used for streaming timed traditional media.

[0014] Figure 3 This is a schematic diagram of an embodiment of a data model used to represent and stream timed immersive media.

[0015] Figure 4 This is a schematic diagram of an embodiment of a data model used to represent and stream timed immersive media.

[0016] Figure 5 It is a schematic diagram of the process of acquiring natural scenes and converting them into a representation that can be used as an ingestion format for networks serving heterogeneous client endpoints.

[0017] Figure 6 This is a schematic diagram of the process of creating a representation of a composite scene using 3D modeling tools and formats, which can be used as an ingestion format for networks serving heterogeneous client endpoints.

[0018] Figure 7 It is a system diagram of a computer system.

[0019] Figure 8 This is a diagram illustrating a network that serves multiple heterogeneous client endpoints.

[0020] Figure 9 This is a schematic diagram of a network used to provide adaptation information related to a specific media, expressed in a media ingestion format, prior to, for example, the network process of adapting media for consumption by a specific immersive media client endpoint.

[0021] Figure 10 This is a system diagram of the media adaptation process, which consists of a media renderer-converter that converts source media from its ingested format into a specific format suitable for a particular client endpoint.

[0022] Figure 11 It is a schematic diagram of a network that formats adapted source media into a data model suitable for representation and streaming.

[0023] Figure 12 It is Figure 11 The data model is divided into a system diagram of the media streaming process, which consists of network protocol packets and payloads.

[0024] Figure 13It is a sequence diagram of a network that adapts a specific immersive media ingested in a particular format into a distribution format that is streamable and suitable for a specific immersive media client endpoint.

[0025] Figure 14 yes Figure 10 A schematic diagram of the ingested media format and asset 1002, which consists of an immersive content format and a conventional content format (i.e., 2D video format only), or a combination of immersive and 2D video formats.

[0026] Figure 15 It describes the carrying of neural network model information along with the encoded video stream.

[0027] Figure 16 It describes the carrying of neural network model information along with input immersive media and assets. Detailed Implementation

[0028] This document discloses detailed embodiments of the claimed structures and methods; however, it is to be understood that the disclosed embodiments are merely illustrative of the claimed structures and methods, which can be implemented in various forms. These structures and methods can be implemented in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope to those skilled in the art. Details of well-known features and techniques may be omitted in the description to avoid unnecessarily obscuring the presented embodiments.

[0029] The embodiments generally relate to the field of data processing, and more specifically to video coding. The techniques described herein allow 2D encoded video streams to be represented by scene-specific neural network models, enabling the network to ingest a 2D video source of media, including one or more (typically a small number) views, before actually distributing the formatted media to various client endpoints, and to adapt the 2D media source to one or more streamable "distribution formats" to accommodate various heterogeneous client endpoint devices, the different characteristics and capabilities of heterogeneous client endpoint devices, and the requirements of applications used on the client endpoints. The network model can be directly embedded into the scene-specific encoded video stream of the encoded bitstream via a structured SEI field, or the SEI can be used to represent the use of a specific model stored elsewhere on the distribution network but accessible by the neural network process. The ability to reformat 2D media sources into various streamable distribution formats allows the network to simultaneously serve a variety of client endpoints with varying capabilities and available computing resources, and enables support for emerging immersive client endpoints, such as holograms and light field displays in commercial networks. Furthermore, the ability to adapt scene-specific 2D media sources based on scene-specific neural network models improves the final visual quality. This ability to adapt 2D media sources is particularly important when no immersive media source is available, and when the client cannot support 2D media-based distribution formats. In this scenario, neural network-based methods can be better optimized for specific scenes existing within 2D media by carrying scene-specific neural network models, where the scene-specific neural network model is trained using priors, which are typically similar to objects within the scene or the context of the scene. This improves the network's ability to infer depth-based information relevant to the scene, allowing the network to adapt 2D media to a scene-specific volumetric format suitable for the target client endpoint.

[0030] As previously described, “immersive media” generally refers to media that stimulates any or all human sensory systems (visual, auditory, tactile, olfactory, and possibly gustatory) to create or enhance the user’s physical experience of the media, exceeding the perceptions of “traditional media” that are distributed across existing commercial networks for timed two-dimensional (2D) video and corresponding audio. Both immersive and traditional media can be characterized as timed or non-timed.

[0031] Time-based media refers to media that is structured and presented according to time. Examples include film features, news reports, and TV series content, all of which are organized according to time periods. Traditional video and audio are often considered time-based media.

[0032] Irregular media are media that are not structured in time; instead, they are structured through logical, spatial, and / or temporal relationships. Examples include video games, where users control an experience created by the gaming device. Another example of irregular media is a still image photograph taken by a camera. Irregular media can contain timed media, such as timed media contained within a continuously looping audio or video clip in a video game scene. Conversely, timed media can contain irregular media, such as video with a fixed still image as a background.

[0033] Devices with immersive media capabilities can refer to devices equipped with the ability to access, interpret, and present immersive media. Such media and devices are inconsistent in terms of the quantity and format of the media, as well as the quantity and type of network resources required for large-scale distribution of such media; that is, the quantity and type of network resources required to achieve distribution equivalent to that of traditional video and audio media over the network are inconsistent. In contrast, traditional devices (such as laptops, televisions, and mobile handheld displays) are similar in their capabilities because all of these devices include rectangular displays and consume 2D rectangular video or still images as their primary media format.

[0034] The distribution of any media over the network can employ media delivery systems and architectures that reformat media from an input or network "ingestion" format into a final distribution format. This distribution format is not only suitable for the target client devices and their applications but also facilitates streaming over the network. Media "streaming" broadly refers to the segmentation and grouping of source media, allowing it to be distributed over the network as logically organized and ordered contiguous smaller "chunks" based on either the media's temporal or spatial structure, or both. In such distribution architectures and systems, media may undergo compression or layering processes, ensuring that only the most salient media information is initially distributed to the client. In some cases, a client must receive all the salient media information from a particular portion of the media before it can present any portion of the same media to the end user.

[0035] The ability to reformat input media to match a target client endpoint can be achieved through a neural network process that employs a network model that has some prior knowledge of the specific media being reformatted. For example, a particular model can be adapted to recognize an outdoor park scene (with trees, plants, grass, and other objects common in park scenes), while another different model can be adapted to recognize an indoor dinner scene (with a dining table, server equipment, people sitting at the table, etc.). Those skilled in the art will recognize that a network model adapted to recognize objects from a specific context (e.g., objects in a park scene) will produce better visual results than a network model that is not so adapted. Therefore, the benefit is to provide a scene-specific network model to a neural network process tasked with reformatting input media to match a target client endpoint.

[0036] The mechanism for associating neural network models with specific scenes in 2D media can be achieved by optionally compressing the network model and directly inserting it into the 2D encoded bitstream of the visual scene using supplementary enhancement information (SEI) structured fields. SEI structured fields are typically used to append metadata to encoded video streams in H.264, H.265, and H.266 video compression formats. The presence of an SEI message containing a specific neural network model within the context of a portion of the encoded video bitstream can be used to instruct the network model to be used to interpret and adapt the video content within that portion of the bitstream in which the model is embedded. Alternatively, the SEI message can be used to signal, with the network model's identifier, which neural network model(s) can be used in the absence of the actual model itself.

[0037] The mechanism for associating neural networks suitable for immersive media can be implemented by referencing the appropriate neural network model within the immersive media itself. This referencing can be achieved by directly embedding the network model and its parameters on an object-by-object, scene-by-scene, or some combination of both. Alternatively, instead of embedding one or more neural network models within the media, media objects or scenes can reference specific neural network models through identifiers.

[0038] Another alternative mechanism for referencing a neural network suitable for adapting streaming media to a client endpoint is to have the specific client endpoint itself provide at least one neural network model and corresponding parameters to the adaptation process. Such a mechanism can be implemented by the client providing the neural network model communicating with the adaptation process (e.g., when the client attaches itself to the network).

[0039] After adapting the video to the target client endpoint, an adaptation process within the network can be selected to apply a compression algorithm to the result. Alternatively, the compression algorithm can divide the adapted video signal into multiple layers, corresponding to the most salient to the least salient parts of the visual signal.

[0040] An example of the compression and layering process is the progressive format of the JPEG standard (ISO / IEC 10918 Part 1), which divides an image into multiple layers, such that the entire image is initially presented only with basic shapes and colors, which are not initially in focus, i.e., the lower-order DCT coefficients from the entire image scan. Subsequently, the entire image is presented with additional detail layers, which cause the image to become in focus, i.e., the higher-order DCT coefficients from the image scan.

[0041] The process of dividing media into smaller parts, organizing media into payload portions of continuous network protocol packets, and distributing these protocol packets is called "streaming" media, while the process of converting media into a format suitable for presentation on one of various heterogeneous client endpoints, which operates on one of various heterogeneous applications.

[0042] definition

[0043] Scene graph: A common data structure typically used by vector-based graphics editing applications and modern computer games, which is the logical and usually (but not necessarily) spatial representation of the arrangement of a graphical scene; a collection of nodes and vertices in a graphical structure.

[0044] Node: The basic element of a scene graph, including information related to the logical, spatial, or temporal representation of visual, audio, tactile, olfactory, gustatory, or related processing information; each node should have at most one output edge, zero or more input edges, and at least one edge (input or output) connected to the node.

[0045] Base layer: The nominal representation of an asset, typically designed to minimize the computational resources or time required to render the asset, or to minimize the time required to transmit the asset over a network.

[0046] Enhancement layer: A set of information applied when representing an asset's base layer to enhance the base layer to include features or capabilities not supported in the base layer.

[0047] Attributes: Metadata associated with a node, used to describe a specific characteristic or feature of the node in a typical or more complex form (e.g., from the perspective of another node).

[0048] Container: A serialization format used to store and exchange information to represent all natural scenes, all composite scenes, or a mixture of composite and natural scenes, including scene graphs and all media resources required to render the scene.

[0049] Serialization: The process of converting the state of a data structure or object into a format that can be stored (e.g., in a file or memory buffer) or transmitted (e.g., via a network connection link) and reconstructed later (possibly in a different computing environment). When the resulting bit sequence is reread according to the serialized format, that bit sequence can be used to create a semantically identical clone of the original object.

[0050] A renderer is a selectively hybrid application or process (typically software-based) based on disciplines such as acoustic physics, optical physics, visual perception, audio perception, mathematics, and software development. That is, given an input scene graph and an asset container, the renderer emits typical visual and / or audio signals suitable for presentation on a target device or conforming to desired properties specified by attributes of the rendering target node in the scene graph. For visual-based media assets, the renderer may emit visual signals suitable for the target display or for storage as intermediate assets (e.g., repackaged into another container, i.e., used in a series of rendering processes in the graphics pipeline); for audio-based media assets, the renderer may emit audio signals for presentation in multi-channel speakers and / or binaural headphones, or for repackaging into another (output) container. Popular examples of renderers include Unity and Unreal Engine.

[0051] Evaluation: Produces results (e.g., similar to evaluating the document object model of a webpage), shifting the output from abstract to concrete results.

[0052] Scripting language: An interpreted programming language that can be executed by the renderer at runtime to handle dynamic inputs and variable state changes made to scene graph nodes, affecting the rendering and evaluation of spatial and temporal object topology (including physics forces, constraints, IK, deformation, collision) and energy propagation and transmission (light, sound).

[0053] Shaders: A type of computer program originally used for shading (producing appropriate levels of light, darkness, and color within an image), but now they perform various specialized functions in different areas of computer graphics effects, or perform video post-processing unrelated to shading, or even perform functions that are not related to graphics at all.

[0054] Path tracing: A computer graphics method for rendering 3D scenes, making scene lighting more realistic.

[0055] Timing media: Media ordered by time; for example, media with start and end times based on a specific clock.

[0056] Irregular media: Media organized by spatial, logical, or temporal relationships; for example, in interactive experiences that are based on user actions.

[0057] Neural network model: A set of parameters and tensors (e.g., matrices) defined in well-defined mathematical operations used in visual signals to obtain improved visual outputs, which may include interpolation of new views of the visual signals that are not explicitly provided by the original signals.

[0058] Immersive media can be considered as one or more types of media that, when presented to humans by devices with immersive media capabilities, stimulate any of the five senses (i.e., sight, sound, taste, touch, and hearing) in a more realistic and consistent manner with human understanding of experiences within the natural world—that is, beyond the stimulation achieved using conventional media presented through conventional devices. In this context, the term "conventional media" refers to two-dimensional (2D) visual media (still or moving image frames) and / or their corresponding audio, for which user interaction is limited to pause, play, fast forward, or rewind; "conventional devices" refers to televisions, laptops, monitors, and mobile devices whose capabilities are limited to presenting only conventional media. In consumer-facing applications, presentation devices for immersive media (i.e., devices with immersive media capabilities) are consumer-facing hardware devices, particularly equipped with the ability to utilize specific information embodied by immersive media, enabling the device to create presentations that more closely mimic human understanding of and interaction with the physical world—that is, beyond the capabilities of conventional devices. While conventional devices are limited to presenting only conventional media, immersive media devices are not similarly constrained.

[0059] Over the past decade, numerous devices with immersive media capabilities have been introduced to the consumer market, including head-mounted displays, augmented reality glasses, handheld controllers, haptic gloves, and game consoles. Similarly, holographic displays and other forms of volumetric displays are poised to emerge in the next decade. While these devices are readily available or imminent, the realization of a coherent end-to-end ecosystem for distributing immersive media across commercial networks has failed for several reasons.

[0060] One reason for this is the lack of a single standard representation of immersive media that addresses the two main use cases associated with the current distribution of mass media on commercial networks: 1) real-time distribution of live action events, where content is created and distributed to client endpoints in real-time or near real-time, and 2) non-real-time distribution, which does not require real-time content distribution, where content is physically acquired or created. Correspondingly, these two use cases can be compared to the distribution of existing “broadcast” and “on-demand” formats.

[0061] For real-time distribution, content can be captured by one or more cameras or created using computer-generated techniques. Content captured by cameras is referred to herein as “natural” content, while content created using computer-generated techniques is referred to herein as “composite” content. The media format representing composite content can be a format used in the 3D modeling, visual effects, and CAD / CAM industries, and can include object formats and tools such as meshes, textures, point clouds, structural volumes, amorphous volumes (e.g., for fire, smoke, and fog), shaders, procedurally generated geometry, materials, lighting, virtual camera definitions, and animations. While composite content is computer-generated, composite media formats can be used for both natural and composite content; however, the process of converting natural content into composite media formats (e.g., composite representations) can be time- and computationally intensive, and therefore may be impractical for real-time applications and use cases.

[0062] For real-time distribution of natural content, the content captured by the camera can be distributed in raster format, which is suitable for traditional display devices because many such devices are similarly designed to display raster format. In other words, assuming that traditional displays are uniformly designed to display raster format, the distribution of raster format is best suited for displays capable of displaying only raster format.

[0063] However, displays with immersive media capabilities are not necessarily limited to displaying raster-based formats. Furthermore, some displays with immersive media capabilities cannot render media offered solely in raster-based formats. The availability of displays optimized for creating immersive experiences based on formats other than raster-based is another important reason why a coherent end-to-end ecosystem for distributing immersive media does not exist.

[0064] Another challenge in creating a coherent distribution system for multiple different immersive media devices is the significant variation inherent in current and emerging immersive media-capable devices themselves. For example, some immersive media devices are explicitly designed for single-user use at a time, such as head-mounted displays. Other immersive media devices are designed to be used simultaneously by more than one user; for instance, the "Looking Glass Factory 8K Display" (hereinafter referred to as the "Lens Light Field Display") can display content that can be viewed simultaneously by up to 12 users, each experiencing their own unique perspective (i.e., view) of the displayed content.

[0065] Further complicating the development of coherent distribution systems is the significant variation in the number of unique views each display can generate. In most cases, traditional displays can only create a single view of content. However, lenticular light field displays can support multiple users, each experiencing a unique view of the same visual scene. To achieve this creation of multiple views of the same scene, lenticular light field displays create a specific volumetric viewing frustum, requiring 45 unique views of the same scene as input to the display. This means that 45 slightly different, unique raster representations of the same scene need to be acquired and distributed to the display in a format specific to that particular display (i.e., its viewing frustum). In contrast, the viewing frustum of a traditional display is limited to a single two-dimensional plane, thus there is no way to present content through more than one viewing angle via the display's viewing frustum (regardless of the number of viewers experiencing the display simultaneously).

[0066] Typically, immersive media displays can vary significantly depending on the following characteristics of all displays: the size and volume of the viewing frustum, the number of viewers supported simultaneously, the optical technology used to fill the viewing frustum (which can be point-based, ray-based, or wave-based), the density of light units (points, rays, or waves) occupying the viewing frustum, the availability of computing power and computing type (CPU or GPU), the power source and availability (battery or cable), the amount of local storage or cache, and access to auxiliary resources such as cloud-based computing and storage. In contrast to the uniformity of traditional displays, these characteristics contribute to the heterogeneity of immersive media displays, complicating the development of a single distribution system that can support all displays, including both traditional and immersive types.

[0067] The disclosed subject matter addresses the development of a web-based media distribution system that supports both traditional displays and immersive media displays as client endpoints within a single network context. Specifically, this paper proposes a mechanism that adapts an input immersive media source to a format suitable for the specific characteristics of the client endpoint device (including the application currently running on that client endpoint device). This mechanism for adapting the input immersive media source involves reconciling the characteristics of the input immersive media with the characteristics of the target endpoint client device (including the application running on the client device), and then adapting the input immersive media to a format suitable for both the target endpoint and its application. Furthermore, the adaptation process may include interpolating additional views (e.g., new views) from the input media to create additional views required by the client endpoint. Such interpolation can be performed using neural network processes.

[0068] It should be noted that the remainder of the subject matter disclosed without loss of generality assumes that the process of adapting an input immersive media source to a specific endpoint client device is the same as or similar to the process of adapting the same input immersive media source to a specific application running on a specific client endpoint device. In other words, the problem of adapting the characteristics of an input media source to an endpoint device has the same complexity as the problem of adapting a specific input media source to the characteristics of a specific application.

[0069] Traditional media-enabled devices have achieved widespread consumer adoption because they are also supported by the ecosystem of traditional media content providers (which produce standards-based representations of traditional media) and commercial network service providers (which provide the network infrastructure to connect traditional devices to standard traditional content sources). Beyond their role in distributing traditional media over the network, commercial network service providers facilitate the pairing of traditional client devices with access to traditional content on a content delivery network (CDN). Once paired with access to the appropriate form of content, the traditional client device can request or “pull” traditional content from the content server to the device for presentation to the end user. However, the architecture of the network server “pushing” the appropriate media to the appropriate client is equally suitable without adding additional complexity to the overall architecture and solution design.

[0070] Various aspects are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer-readable media according to various embodiments. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0071] The exemplary embodiments described below relate to the architecture, structure, and components of systems and networks for distributing media, including video, audio, geometric (3D) objects, haptic feedback, associated metadata, or other content for client devices. Specific embodiments relate to systems, structures, and architectures for distributing media content to heterogeneous immersive and interactive client devices.

[0072] Figure 1 This is an example diagram illustrating the end-to-end process of timed traditional media distribution. Figure 1 In this process, timed audio-visual content is captured by a camera or microphone 101A or generated by a computer 101B to create a sequence 102 of 2D images and associated audio. Sequence 102 is input to preparation module 103. The output of 103 is edited content (e.g., for post-production (including language translation, subtitles, and other editing functions)), referred to as the master format, which is prepared to be converted by converter module 104 into a standard intermediate format (Mezzanine Format) (e.g., for video-on-demand media) or a standard contribution format (e.g., for live events). The media is "ingested" by a commercial network service provider, and adaptation module 105 packages the media into various bitrates, temporal resolutions (frame rates), or spatial resolutions (frame sizes), which are packaged into standard distribution formats. The resulting adaptations are stored on content delivery network 106, and various clients 108 pull requests 107 from content delivery network 106 to retrieve the media and present it to end users. It is important to note that the primary format can consist of a mixture of media from 101A or 101B, and format 101A can be obtained in real time, for example, media obtained from live sports events. Furthermore, the client 108 is responsible for selecting the specific adapter 107 best suited to the client configuration and / or current network conditions, but it is equally possible that the network server (…) Figure 1 (Not shown) can determine the appropriate content and then push the appropriate content to the client 108.

[0073] Figure 2 This is an example of a standard media format used for distributing traditional timed media (such as video, audio, and supporting metadata, including timed text for subtitles, for example). Figure 1As indicated by item 106, the media is stored on CDN 201 in a standards-based distribution format. This standards-based format, denoted as MPD 202, consists of multiple parts containing timing periods 203 with start and end times corresponding to a clock. Each period 203 refers to one or more adapter sets 204. Each adapter set 204 is typically used for a single type of media, such as video, audio, or timed text. For any given period 203, multiple video sets 204 may be provided; for example, one video set for video and multiple video sets for audio (e.g., for conversion to various languages). Each video set 204 refers to one or more representations 205 that provide information related to the media's frame resolution (for video), frame rate, and bitrate. Multiple representations 205 can be used to provide access to representations 205, such as those for ultra-high definition video, high definition video, or standard definition video. Each representation 205 relates to one or more fragment files 206, in which the media is actually stored for retrieval by clients (e.g., ...). Figure 1 (as shown in 108) or by a network media server ( Figure 1 Distribution (not shown) (in a "push-based" architecture).

[0074] Figure 3 It is an example representation of a streamable transmission format for timed heterogeneous immersive media. Figure 4 This is an example representation of a streamable format for irregularly shaped, heterogeneous, immersive media. Both diagrams refer to a scene; Figure 3 This relates to scenario 301, which involves timed media. Figure 4 This pertains to scenario 401, which is used for media that is not always in time. For both scenarios, the scenario can be represented by various scenario representations or descriptions.

[0075] For example, in some immersive media designs, a scene can be represented by a scene graph, or as a multi-plane image (MPI) or multi-spherical image (MSI). MPI and MSI techniques are examples of display-agnostic scene representations that help create natural content (i.e., images of the real world captured simultaneously from one or more cameras). On the other hand, scene graph techniques can be used to represent both natural and computer-generated images in the form of synthetic representations; however, creating such a representation is particularly computationally intensive when the content is captured as a natural scene by one or more cameras. That is, creating a scene graph representation of naturally captured content is time- and computationally intensive, requiring complex analysis of the natural images using photogrammetry, deep learning, or both, to create a synthetic representation that can then be used to interpolate a sufficient number of views to fill the visual frustum of the target immersive client's display. Therefore, considering such synthetic representations as candidates for representing natural content is currently impractical because it is practically impossible to create such synthetic representations in real-time for use cases requiring real-time distribution. However, currently, the best candidate representation for computer-generated images is to use scene graphs in conjunction with synthetic models, because computer-generated images are created using 3D modeling processes and tools.

[0076] This dichotomy of optimal representation for both natural and computer-generated content implies that the optimal ingestion format for naturally acquired content differs from the optimal ingestion format for computer-generated content, or from the optimal ingestion format for natural content that is not necessary for real-time distribution applications. Therefore, the goal of the disclosed subject matter is to be robust enough to support multiple ingestion formats for visually immersive media, regardless of whether the visually immersive media is naturally created or computer-generated.

[0077] Below is an example technique for representing scene graphs in a format suitable for representing visually immersive media created using computer-generated techniques, or suitable for representing naturally captured content. For naturally captured content, deep learning or photogrammetry techniques are used to create a corresponding synthetic representation of the natural scene; that is, naturally captured content is not necessary for real-time distribution applications.

[0078] 1. OTOY's

[0079] OTOY's ORBX is one of several scene graph technologies that supports any type of timed or timed visual media, including ray-traced, traditional (frame-based), volumetric, and other types of composite or vector-based visual formats. ORBX is unique among scene graphs because it provides native support for freely available and / or open-source formats for meshes, point clouds, and textures. ORBX is an intentionally designed scene graph aimed at facilitating the interchangeability of multiple vendor technologies operating on it. Furthermore, ORBX offers a rich material system, supports an open shader language, robust camera systems, and Lua scripting support. ORBX is also the foundation for the Immersive Technology Media Format, released by the Immersive Digital Experiences Consortium (IDEA) and licensed without royalties. In the context of real-time media distribution, the ability to create and distribute ORBX representations of natural scenes is a function of the availability of computational resources to perform complex analyses of camera-captured data and to composite the same data into synthetic representations. To date, the availability of computation sufficient for real-time distribution is impractical, but still possible.

[0080] 2. Pixar's general scenario description

[0081] Pixar's Universal Scene Description (USD) is another well-known and mature scene graph, popular in the VFX and professional content production communities. USD is integrated into NVIDIA's Omniverse platform, a toolset that facilitates developers in creating and rendering 3D models using NVIDIA GPUs. A subset of USD is released by Apple and Pixar as USDZ. USDZ is powered by Apple's ARKit.

[0082] 3. Khronos' glTF 2.0

[0083] glTF 2.0 is the latest version of the "Graphics Language Transport Format" specification, written by the Khronos 3D group. This format supports simple scene graph formats, which typically support static (non-timed) objects in the scene, including "png" and "jpeg" image formats. glTF 2.0 supports simple animations, including translation, rotation, and scaling of basic shapes (i.e., geometric objects) described using glTF primitives. glTF 2.0 does not support timed media, therefore it does not support video or audio.

[0084] For example, these known designs only provide scene representations of immersive visual media, and do not limit the ability of the disclosed subject matter to specifying a process for adapting an input immersive media source into a format suitable for the specific characteristics of the client endpoint device.

[0085] Furthermore, any or all of the above example media representations currently employ or can employ deep learning techniques to train and create neural network models capable of selecting or facilitating the selection of specific views based on a specific size of the view frustum to fill the view frustum of a particular display. The view selected for the view frustum of a particular display can be interpolated from existing views explicitly provided in the scene representation (e.g., according to MSI or MPI techniques), or the view selected for the view frustum of a particular display can be rendered directly from these rendering engines based on specific virtual camera positions, filters, or descriptions of virtual cameras used by the rendering engines.

[0086] Therefore, the disclosed subject matter is robust enough to take into account a relatively small but well-known set of immersive media capture formats that can adequately meet the need for real-time or “on-demand” (e.g., non-real-time) distribution of media that is captured naturally (e.g., using one or more cameras) or created using computer-generated techniques.

[0087] The adoption of advanced networking technologies, such as 5G for mobile networks and fiber optic cables for fixed networks, further facilitates the interpolation of views from immersive media ingestion formats using neural network models or network-based rendering engines. In other words, these advanced networking technologies increase the capacity and capability of commercial networks because such advanced network infrastructure can support the transmission and distribution of increasingly large volumes of visual information. Network infrastructure management technologies (such as Multi-Access Edge Computing (MEC), Software-Defined Networking (SDN), and Network Functions Virtualization (NFV)) enable commercial network service providers to flexibly configure their network infrastructure to adapt to changing demands for certain network resources, such as dynamic increases or decreases in demand for network throughput, network speed, round-trip latency, and computing resources. Furthermore, this inherent ability to adapt to dynamic network demands also facilitates the network's ability to adapt immersive media ingestion formats to suitable distribution formats to support a variety of immersive media applications with possible heterogeneous visual media formats for heterogeneous client endpoints.

[0088] Immersive media applications can have varying requirements for network resources. These include: gaming applications, which require significantly lower network latency to respond to real-time updates of the game state; telepresence applications, which have symmetrical throughput requirements for both the uplink and downlink portions of the network; and passive viewing applications, which may have increased downlink resource requirements depending on the type of display on the client endpoint consuming the data. Typically, any consumer-facing application can be supported by a variety of client endpoints, each with different onboard client capabilities for storage, compute, and power. Similarly, different client endpoints have different requirements for specific media representations.

[0089] Therefore, the disclosed subject matter enables a fully equipped network, i.e., a network employing some or all of the characteristics of a modern network, to simultaneously support multiple legacy devices and devices with immersive media capabilities, based on the characteristics specified in the following aspects:

[0090] 1. Provides flexibility in utilizing media ingestion formats, which are useful for both real-time and "on-demand" use cases for media distribution.

[0091] 2. Provide traditional client endpoints and client endpoints with immersive media capabilities with the flexibility to support natural and computer-generated content.

[0092] 3. Supports both scheduled and unscheduled media.

[0093] 4. Provides a process for dynamically adapting the source media ingestion format to a suitable distribution format based on the characteristics and capabilities of the client endpoint and the requirements of the application.

[0094] 5. Ensure that the distribution format can be streamed over IP-based networks.

[0095] 6. Enables the network to simultaneously serve multiple heterogeneous client endpoints, including both legacy devices and devices with immersive media capabilities.

[0096] 7. Provide an exemplary media representation framework that facilitates the organization of distribution media along scene boundaries.

[0097] The improved end-to-end embodiments achievable by the disclosed subject matter are described below. Figures 3 to 16 The detailed description describes the processes and components used to implement them.

[0098] Figure 3 and Figure 4 A single exemplary include distribution format is used, which has been adapted from the ingestion source format to match the capabilities of a specific client endpoint. As described above, Figure 3 The media shown is timed media. Figure 4 The media shown is time-varying media. A particular inclusion format is structurally robust enough to accommodate a wide variety of media attributes, each of which can be layered based on the amount of significant information that contributes to the presentation of the media at each layer. It should be noted that this layering process is already known in the prior art, as illustrated by the progressive development of JPEG and scalable video architectures (such as those specified in ISO / IEC 14496-10 (Scalable Advanced Video Coding)).

[0099] 1. The media included in the streaming media format is not limited to traditional visual and audio media, but may include any type of media information capable of generating signals that can be interacted with machines to stimulate human senses of sight, sound, taste, touch and smell.

[0100] 2. Depending on the media format, the media being streamed can be timed media, untimed media, or a mixture of timed and untimed media.

[0101] 3. A layered representation of media objects is also achieved through a base layer and enhancement layer architecture, enabling streaming of media formats. In one example, separate base and enhancement layers are computed by applying multi-resolution or multi-segmentation analysis techniques to media objects in each scene. This is similar to, but not limited to, the progressively rendered image formats specified by ISO / IEC 10918-1 (JPEG) and ISO / IEC 15444-1 (JPEG 2000), but not limited to raster-based visual formats. In an example embodiment, the progressive representation of geometric objects could be a multi-resolution representation of the objects computed using wavelet analysis.

[0102] In another example of a layered representation of a media format, the enhancement layer applies different properties to the base layer, such as refining the material properties of the surface of the visual object represented by the base layer. In yet another example, the properties could refine the texture of the surface of the base layer object, for example, changing the surface from smooth to porous or from matte to glossy.

[0103] In yet another example of hierarchical representation, the surfaces of one or more visual objects in the scene can be transformed from Lambertian to ray-traceable.

[0104] In another example of layered representation, the network distributes a base layer representation to the client, allowing the client to create a nominal representation of the scene while waiting for additional enhancement layers to be transmitted to refine the resolution or other characteristics of the base representation.

[0105] 4. The resolution of attributes or refinement information in the enhancement layer is not explicitly coupled to the resolution of objects in the base layer of the existing MPEG video and JPEG image standards.

[0106] 5. Includes media format support for any type of information media that can be rendered or actuated by a presentation device or machine, thereby enabling heterogeneous client endpoints to support heterogeneous media formats. In one embodiment of a network distributing media formats, the network first queries the client endpoint to determine the client's capabilities. If the client cannot meaningfully ingest a media representation, the network removes an attribute layer that the client does not support, or adapts the media from its current format to a format suitable for the client endpoint. In one example of such adaptation, the network may convert volumetric visual media assets into a 2D representation of the same visual asset using a network-based media processing protocol. In another example of such adaptation, the network may employ a neural network process to reformat the media into an appropriate format, or alternatively, synthesize the view required by the client endpoint.

[0107] 6. The inventory for a complete or partially complete immersive experience (replay of a live streaming event, game, or on-demand asset) is organized by scenes. This inventory represents the minimum amount of information that the rendering and game engine can currently ingest to create the presentation. The inventory includes a list of individual scenes to be rendered for the entire immersive experience requested by the client. Associated with each scene are one or more representations of geometric objects within the scene, corresponding to a streamable version of the scene's geometry. One embodiment of a scene representation involves a low-resolution version of the scene's geometric objects. Another embodiment of the same scene involves an enhancement layer used for the low-resolution representation of the scene to add additional detail or subdivision to the geometric objects of the same scene. As mentioned above, each scene may have more than one enhancement layer to add detail to the scene's geometric objects in a line-by-line manner.

[0108] 7. Each layer of a media object referenced within a scene is associated with a token (e.g., a URI) that points to the address of a location within the network where the resource can be accessed. Such a resource is analogous to a CDN location where content can be retrieved by a client.

[0109] 8. Tokens used to represent geometric objects can point to locations within the network or within the client. That is, a client can signal to the network that its resources are available for network-based media processing.

[0110] Figure 3 The following embodiment of a media format included for timing media is described. The timing scene list includes a list of scene information 301. Scene 301 relates to a list of components 302, each describing the type and processing information of the media asset included in scene 301. Component 302 relates to assets 303, which further relates to a base layer 304 and an attribute enhancement layer 305.

[0111] Figure 4The following embodiment of a media format for non-timed media is described. Scene information 401 is not associated with a start duration and an end duration based on a clock. Scene information 401 relates to a list of components 402, each describing the type and processing information of the media assets included in scene 401. Component 402 relates to assets 403 (e.g., visual assets, audio assets, and haptic assets), and assets 403 further relate to a base layer 404 and an attribute enhancement layer 405. Furthermore, scene 401 relates to other scenes 401 for non-timed media. Scene 401 also relates to timed media scenes.

[0112] Figure 5 An embodiment of a process 500 for synthesizing a format from natural content is illustrated. Camera unit 501 uses a single camera lens to capture a scene of people. Camera unit 502 captures a scene with five diverging fields of view by mounting five camera lenses around a ring-shaped object. The arrangement of 502 is an exemplary arrangement typically used for capturing omnidirectional content for VR applications. Camera unit 503 captures a scene with seven converging fields of view by mounting seven camera lenses on the inner diameter portion of a sphere. Arrangement 503 is an exemplary arrangement typically used for capturing light fields or light fields of holographic immersive displays. Natural image content 509 is provided as input to a synthesis module 504, which may optionally employ a neural network training module 505, which uses a set of training images 506 to generate an optional acquisition neural network model 508. Another process typically used instead of the training process 505 is photogrammetry. Figure 5 During the process 500 described herein, model 508 is created, and model 508 becomes one of the assets under ingestion format 507 for natural content. Exemplary embodiments of ingestion format 507 include MPI and MSI.

[0113] Figure 6 An embodiment of a process 600 for creating an ingestion format for synthetic media (e.g., computer-generated images) is illustrated. A LiDAR camera 601 acquires a point cloud 602 of the scene. Synthetic content is created on a computer 603 using CGI tools, 3D modeling tools, or another animation process to create 604 CGI assets on the web. A motion acquisition kit 605A with sensors is worn on a person 605 to acquire digital records of the person's movements to generate animated motion acquisition data 606. Data 602, 604, and 606 are provided as input to a compositing module 607, which may also optionally use neural networks and training data to create a neural network model (…). Figure 6 (Not shown).

[0114] The techniques used to represent and stream the heterogeneous immersive media described above can be implemented as computer software that uses computer-readable instructions and is physically stored in one or more computer-readable media. For example, Figure 7 A computer system 700 suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0115] Computer software can be coded using any suitable machine code or computer language. Any suitable machine code or computer language can be assembled, compiled, linked, or similarly processed to create code containing instructions that can be executed directly by a computer's central processing unit (CPU), graphics processing unit (GPU), or through interpretation, microcode execution, or other means.

[0116] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0117] Figure 7 The components of the computer system 700 shown are exemplary in nature and are not intended to impose any limitation on the scope or functionality of computer software implementing embodiments of this disclosure. The configuration of the components should also not be construed as having any dependencies or requirements relating to any one or a combination of components shown in the exemplary embodiments of the computer system 700.

[0118] Computer system 700 may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movement), audio input (e.g., speech, clapping), visual input (e.g., gestures), and olfactory input (not depicted). Human-machine interface devices may also be used to acquire certain media that are not necessarily directly related to human conscious input, such as audio (e.g., speech, music, ambient sound), images (e.g., scanned images, photographic images acquired from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0119] The input human-machine interface device may include one or more of the following (only one of each is shown): keyboard 701, mouse 702, touchpad 703, touch screen 710, data glove (not depicted), joystick 705, microphone 706, scanner 707, and camera 708.

[0120] Computer system 700 may also include certain human-machine interface (HMI) output devices. Such HMI output devices may, for example, stimulate the senses of one or more human users through tactile output, sound, light, and smell / taste. These HMI output devices may include tactile output devices (e.g., tactile feedback of touchscreen 710, data gloves (not depicted), or joystick 705, but may also be tactile feedback devices that are not input devices), audio output devices (e.g., speakers 709, headphones (not depicted)), and visual output devices (e.g., screens 710 including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touchscreen input functionality, each with or without tactile feedback functionality—some of which are capable of outputting two-dimensional or more three-dimensional visual outputs via devices such as stereoscopic image output, virtual reality glasses (not depicted), holographic displays and smoke boxes (not depicted), and printers (not depicted).

[0121] The computer system 700 may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW 720 with media such as CD / DVD 721, finger drives 722, removable hard disk drives or solid-state drives 723, conventional magnetic media such as magnetic tapes and floppy disks (not depicted), devices based on dedicated ROM / ASIC / PLD such as security dongles (not depicted), etc.

[0122] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not cover transmission media, carrier waves, or other transient signals.

[0123] Computer system 700 may also include interfaces to one or more communication networks. These networks may be, for example, wireless networks, wired networks, or optical networks. Further networks may be local area networks, wide area networks, metropolitan area networks, vehicle and industrial networks, real-time networks, latency-tolerant networks, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., cable or wireless wide area digital networks including cable television, satellite television, and terrestrial broadcast television, vehicle and industrial networks including CANBus, etc. Some networks typically require external network interface adapters (e.g., USB ports of computer system 700) to connect to certain general-purpose data ports or peripheral buses (749); other network interfaces are typically integrated into the core of computer system 700 by connecting to the system bus (e.g., an Ethernet interface connected to a PC computer system or a cellular network interface connected to a smartphone computer system). Computer system 700 can use any of these networks to communicate with other entities. Such communication can be one-way receiving (e.g., broadcast television), one-way transmitting (e.g., CANbus connected to certain CANbus devices), or bidirectional, such as connecting to other computer systems using a local area network (LAN) or wide area network (WAN) digital network. As mentioned above, certain protocols and protocol stacks can be used on each of those networks and network interfaces.

[0124] The aforementioned human-machine interface device, human-machine accessible storage device, and network interface can be attached to the kernel 740 of the computer system 700.

[0125] The core 740 may include one or more central processing units (CPUs) 741, graphics processing units (GPUs) 742, dedicated programmable processing units in the form of field-programmable gate areas (FPGAs) 743, hardware accelerators 744 for certain tasks, etc. These devices, along with read-only memory (ROM) 745, random access memory (RAM) 746, and internal mass storage 747 such as internal non-user-accessible hard disk drives (SDs), etc., can be connected via a system bus 748. In some computer systems, the system bus 748 may be accessed via one or more physical connectors to allow for expansion with additional CPUs, GPUs, etc. Peripheral devices may be directly connected to the core's system bus 748 or connected via a peripheral bus 749. Peripheral bus architectures include PCI, USB, etc.

[0126] The CPU 741, GPU 742, FPGA 743, and accelerator 744 can execute certain instructions, which can be combined to form the aforementioned computer code. This computer code can be stored in ROM 745 or RAM 746. Transient data can also be stored in RAM 746, while permanent data can be stored, for example, in internal mass storage 747. Fast storage and retrieval to any storage device can be achieved using a cache, which can be closely associated with one or more CPUs 741, GPUs 742, mass storage 747, ROM 745, RAM 746, etc.

[0127] Computer-readable media may have computer code thereon for performing various computer-implemented operations. The media and computer code may be media and computer code specifically designed and constructed for the purposes of this disclosure, or the media and computer code may be of a type known and available to those skilled in the art of computer software.

[0128] As a non-limiting example, a computer system having architecture 700, particularly kernel 740, can provide functionality by having one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) execute software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage as described above, and some non-transitory memory of kernel 740, such as internal mass storage 747 or ROM 745. Software implementing the various embodiments of this disclosure can be stored in such devices and executed by kernel 740. Depending on specific needs, the computer-readable media may include one or more storage devices or chips. The software can cause kernel 740, particularly the processors therein (including CPUs, GPUs, FPGAs, etc.), to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM 746 and modifying such data structures according to software-defined processes. Additionally or alternatively, the computer system may provide functionality through hard-wired or otherwise embodied logic in circuitry (e.g., accelerator 744), which may replace or operate with software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may include circuitry storing software for execution (e.g., integrated circuits (ICs)), circuitry embodying logic for execution, or both. This disclosure includes any suitable combination of hardware and software.

[0129] Figure 8An exemplary network media distribution system 800 is illustrated, which supports various traditional displays and heterogeneous displays with immersive media capabilities as client endpoints. The content acquisition module 801 uses... Figure 6 or Figure 5 Exemplary embodiments are used to acquire or create media. An ingestion format is created in the content preparation module 802, and then the ingestion format is sent to one or more client endpoints 804 in a network media distribution system using the transmission module 803. A gateway can serve client devices to provide network access to various client endpoints. A set-top box can also be used as a client device to provide access to aggregated content through a network service provider. A wireless demodulator can be used as a mobile network access point for mobile devices, such as mobile handheld devices and displays. In one or more embodiments, a conventional 2D television can be directly connected to a gateway, set-top box, or WiFi router. A laptop computer with a conventional 2D display can be a client endpoint connected to a WiFi router. A head-mounted 2D (raster-based) display can also be connected to a router. A lenticular light field display can be a gateway. The display can include a local computing GPU, storage devices, and a visual rendering unit that uses light-based lenticular optics to create multiple views. A holographic display can be connected to a set-top box and can include a local computing CPU, GPU, storage devices, and a Fresnel-mode wave-based holographic visualization unit. Augmented reality headsets can connect to a wireless demodulator and may include a GPU, storage device, battery, and volumetric visual rendering components. Dense light field displays can connect to a WiFi router and may include multiple GPUs, CPUs, and storage devices, eye-tracking devices, cameras, and dense light field panels.

[0130] Figure 9 An embodiment of an immersive media distribution module 900 is shown, which is capable of serving media distribution applications as previously described. Figure 8 The text describes both traditional displays and heterogeneous displays with immersive media capabilities. Content is created or retrieved within module 901, which further... Figure 5 and Figure 6 This is reflected in the code, used for natural content and CGI content respectively. Then, the content 901 is converted into the ingestion format using the web ingestion format creation module 902. Module 902 also further... Figure 5 and Figure 6The ingested media format is transmitted to the network and stored on storage device 903. Optionally, the storage device may reside in the network of the immersive media content producer and be remotely accessed by an immersive media network distribution module (unnumbered), as depicted by the dashed lines dividing 903. Client and application-specific information may optionally be provided on remote storage device 904, which may optionally reside remotely in an alternative "cloud" network.

[0131] like Figure 9 As depicted, the client interface module 905 serves as the primary source and repository of information for performing the main tasks of the distribution network. In this particular embodiment, module 905 may be implemented in a uniform format with other components of the network. However, by Figure 9 The task described in module 905 forms an essential element of the disclosed subject matter.

[0132] Module 905 receives information related to the characteristics and attributes of client 908, and also collects requests regarding the applications currently running on 908. This information can be obtained from device 904, or, in an alternative embodiment, by directly querying client 908. In the case of directly querying client 908, it is assumed that a bidirectional protocol exists and is operable (bidirectional protocol...). Figure 9 (Not shown), which allows the client to communicate directly with the interface module 905.

[0133] Interface module 905 also initiates communication with media adaptation and segmentation module 910, and media adaptation and segmentation module 910 in... Figure 10 The following describes the process. When the ingested media is adapted and segmented by module 910, the media may optionally be transferred to an intermedia storage device, which is depicted as a media storage device 909 ready for distribution. While the distribution media is prepared and stored in device 909, interface module 905 ensures that the immersive client 908 receives and distributes the media and corresponding description information 906 via its network interface through a "push" request, or the client 908 itself may initiate a "pull" request to obtain the media 906 from storage device 909. The immersive client 908 may optionally employ a CPU (or a CPU, not shown). The media distribution format is stored in the client 908's storage device or cache. Finally, the client 908 visually presents the media through its visualization components.

[0134] During the process of streaming immersive media to the client 908, the interface module 905 will check the status of the client progress through the client progress and status feedback channel 907.

[0135] Figure 10Specific embodiments of the media adaptation process are described, enabling the ingested source media to be appropriately adapted to meet the requirements of client 908. Media adaptation module 1001 includes multiple components that facilitate the adaptation of ingested media into a distribution format suitable for client 908. These components should be considered exemplary. Figure 10 In this process, the adaptation module 1001 receives input network state 1005 to determine the current traffic load on the network; client information 908 (including attribute and feature descriptions, application features and descriptions, and the current application state); and a client neural network model (if available) to help map the geometry of the client's view frustum to the ingested immersive media using interpolation capabilities. The adaptation module 1001 ensures that the output of the adaptation is stored in the client-adapted media storage device 1006 as it is created.

[0136] The adaptation module 1001 employs a renderer 1001B or a neural network processor 1001C to adapt a specific ingested source media into a format suitable for the client. The neural network processor 1001C uses a neural network model 1001A. Examples of such a neural network processor 1001C include depth-view neural network model generators as described in MPI and MSI. If the media is in 2D format, but the client must have a 3D format, the neural network processor 1001C may invoke a process that uses highly correlated images from the 2D video signal to derive a volumetric representation of the scene depicted in the video. An example of such a process could be a neural radiation field process developed at UC Berkeley from one or more images. An example of a suitable renderer 1001B could be a modified version of the OTOY Octane renderer (not shown) that can be modified to interact directly with the adaptation module 1001. Depending on the need for these tools relative to the format of the ingested media and the format required by the client 908, the adaptation module 1001 may optionally employ a media compressor 1001D and a media decompressor 1001E.

[0137] Figure 11 The description includes an adapter media encapsulation module 1103, which ultimately converts media into images residing on the client-side adapter media storage device 1102, originating from... Figure 10 The media adaptation module 1101 adapts the media. The encapsulation module 1103 formats the adaptation media from module 1101 into a robust distribution format, for example, Figure 3 or Figure 4 The exemplary format shown is as follows. Inventory information 1104A provides client 908 with a list of scene data that it can expect to receive, and also provides a list of visual assets and their corresponding metadata, as well as a list of audio assets and their corresponding metadata.

[0138] Figure 12A packet divider module 1202 is depicted, which divides the adapted media 1201 into independent packets 1203 suitable for streaming to the client 908.

[0139] right Figure 13The components and communications shown for sequence diagram 1300 are explained as follows: Client endpoint 1301 initiates a media request 1308 to network distribution interface 1302. Request 1308 includes information for identifying the media requested by the client via URN or other standard naming. Network distribution interface 1302 responds to request 1308 with a profile request 1309, which requests client 1301 to provide information related to its currently available resources (including compute, storage, battery charge percentage, and other information characterizing the client's current operating state). Profile request 1309 also requests the client to provide one or more neural network models that can be used by the network for neural network inference to extract or interpolate appropriate media views to match the characteristics of the client's presentation system (where such models are available at the client). Response 1310 from client 1301 to interface 1302 provides a client token, an application token, and one or more neural network model tokens (where such neural network model tokens are available at the client). Interface 1302 then provides client 1301 with a session ID token 1311. Then, interface 1302 requests the ingest media server 1303 using ingest media request 1312, which includes the URN or standard naming name of the media identified in request 1308. Server 1303 responds to request 1312 with response 1313, which includes an ingest media token. Then, interface 1302 provides the media token from response 1313 to client 1301 in call 1314. Then, interface 1302 initiates an adaptation process for the media requested in 1308 by providing the ingest media token, client token, application token, and neural network model token to adaptation interface 1304. Interface 1304 requests access to the ingested media to request access to the ingested media asset by providing the ingest media token to server 1303 at call 1316. Server 1303 responds to request 1316 with the ingest media access token in response 1317 arriving at interface 1304. Then, interface 1304 requests media adaptation module 1305 to adapt the ingested media located at the ingested media access token for the client, application, and neural network inference model corresponding to the session ID token created at 1313. Request 1318 from interface 1304 to module 1305 contains the required token and session ID. Module 1305 provides the adapted media access token and session ID to interface 1302 in update 1319. Interface 1302 provides the adapted media access token and session ID to encapsulation module 1306 in interface call 1320. Encapsulation module 1306 provides the encapsulated media access token and session ID to interface 1302 in response 1321.In response 1322, module 1306 provides the encapsulated media access token, encapsulated assets, and URN to the encapsulated media server 1307 for a session ID. Client 1301 executes request 1323 to initiate streaming of the media assets corresponding to the encapsulated media access token received in message 1321. Client 1301 executes other requests and provides a status update to interface 1302 in message 1324.

[0140] Figure 14 Depicting Figure 10 The ingested media format and assets 1002 may optionally consist of two parts: immersive media and assets 1401 in 3D format and immersive media and assets 1402 in 2D format. The 2D format 1402 may be a single-view encoded video stream, such as ISO / IEC 14496 Part 10 Advanced Video Coding, or it may be an encoded video stream containing multiple views (e.g., a multi-view compression modification of ISO / IEC 14496 Part 10).

[0141] Figure 15 This diagram depicts the carrying of neural network model information along with the encoded video stream. In this diagram, encoded video stream 1501 includes a neural network model and its corresponding parameters directly carried by one or more SEI messages 1501A. In encoded video stream 1502, one or more SEI messages carry the identifier of the neural network model and its corresponding parameters. In the scenario for 1502, the neural network model and parameters are stored externally to the encoded video stream, for example, stored in... Figure 10 In 1001A.

[0142] Figure 16 The immersive media and assets captured are depicted in 3D format 1601 (originally depicted as...). Figure 14 The information carried under item 1401 in the media 1601 refers to scenes 1 through N, depicted as 1602. Each scene 1602 relates to geometry 1603 and processing parameters 1604. Geometry 1603 may contain a reference 1603A to a neural network model. Processing parameters 1604 may also contain a reference 1604A to a neural network model. Both 1603A and 1604A may relate to a network model or identifier that directly stores the scene, the identifier referring to a neural network model located outside the ingested media, for example, stored in... Figure 10 The network model in 1001A.

[0143] Some embodiments may relate to systems, methods, and / or computer-readable media at any possible level of technical detail. A computer-readable medium may include a computer-readable non-transitory storage medium (or medium) having computer-readable program instructions on it that cause a processor to perform operations.

[0144] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to: electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices (e.g., punched cards or raised structures in recesses on which instructions are recorded), and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as being, in itself, a transient signal, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0145] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or to an external computer or external storage device. The network may include copper cables, fiber optic cables, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives and forwards the computer-readable program instructions from the network for storage in a computer-readable storage medium within the suitable computing / processing device.

[0146] Computer-readable program code / instructions used to perform operations can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and procedural programming languages ​​such as the "C" programming language or similar programming languages. Computer-readable program instructions can run entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., via the Internet provided by an Internet service provider). In some embodiments, electronic circuitry, such as that including programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can be customized using state information from the computer-readable program instructions to run the computer-readable program instructions to perform aspects or operations.

[0147] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute on the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that directs a computer, programmable data processing apparatus, and / or other device to function in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0148] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other equipment to cause a series of operational steps to be performed on the computer, other programmable apparatus or other equipment to produce a computer-implemented process, such that the instructions running on the computer, other programmable apparatus or other equipment perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0149] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer-readable media according to various embodiments. In this regard, each box in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. Methods, computer systems, and computer-readable media may include more, fewer, different, or differently arranged boxes compared to those depicted in the figures. In some alternative implementations, the functions indicated in the boxes may occur in a different order than indicated in the figures. For example, two boxes shown consecutively may actually execute simultaneously or substantially simultaneously, or these boxes may sometimes execute in reverse order, depending on the functions involved. It should also be noted that each box in the block diagrams and / or flowcharts, and combinations of boxes in the block diagrams and / or flowcharts, may be implemented by a hardware-based dedicated system that performs the specified function or action or implements a combination of dedicated hardware and computer instructions.

[0150] It will be apparent that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not a limitation on the implementation method. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to any specific software code—it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0151] Elements, actions, or instructions used herein should not be construed as essential or indispensable unless explicitly stated otherwise. Furthermore, as used herein, the articles “a” and “an” are intended to include one or more items and are used interchangeably with “one or more.” Additionally, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and is used interchangeably with “one or more.” The term “an” or similar language is used where the intent is to include only one item. Furthermore, as used herein, the terms “have,” “possess,” “contain,” etc., are intended to be open-ended terms. Additionally, the phrase “based on” is intended to mean “at least partially based on” unless explicitly stated otherwise.

[0152] Various aspects and embodiments have been described for illustrative purposes, but these aspects and embodiments are not intended to be exhaustive or limited to the disclosed embodiments. While combinations of features are recited in the claims and / or disclosed in the description, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features can be combined in ways not specifically recited in the claims and / or disclosed in the description. Although each dependent claim listed below may be directly dependent on only one claim, the disclosure of possible implementations includes combinations of each dependent claim with each other claim in the claim set. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein is chosen to best explain the principles of the embodiments, practical applications or improvements to techniques found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for streaming immersive media, executable by a processor, the method comprising: The client's capabilities are determined by querying the client endpoint; The content is captured in a first two-dimensional format or a first three-dimensional format, the three-dimensional format referencing a neural network, the content including supplemental enhancement information of a specific neural network, the supplemental enhancement information being used to associate a neural network model of a specific scene with a specific scene existing within the content; Based on the determined capabilities of the client, and using a referenced neural network, the ingested content is converted into a second two-dimensional or second three-dimensional format, wherein the conversion includes: adapting the content to a scene-specific volumetric format associated with the client endpoint based on depth information inferred from the content; interpolating additional views from the input media using a neural network process, and selecting specific views based on a specific size of the view frustum to fill the view frustum of a specific display; and The converted content is streamed to the client endpoint.

2. The method according to claim 1, wherein, The neural network is referenced using metadata used to identify the model's location.

3. The method according to claim 1, wherein, The neural network is referenced by a generic resource identifier corresponding to the metadata used to describe the content.

4. The method according to any one of claims 1 to 3, wherein, Before ingesting the content, the neural network is trained based on priors corresponding to objects within the content.

5. The method according to any one of claims 1 to 3, wherein, The ingested content is transformed based on the characteristics of the client endpoint.

6. The method according to any one of claims 1 to 3, wherein, One or more client endpoints include one or more of a television, computer, head-mounted display, lenticular light field display, holographic display, augmented reality display, and dense light field display.

7. A computer system for streaming immersive media, the computer system comprising: One or more computer-readable non-transitory storage media configured to store computer program code; as well as One or more computer processors are configured to access and operate as instructed by the computer program code, the computer program code comprising: The code is ingested and configured to enable the one or more computer processors to determine the capabilities of the client by querying the client endpoint; content is ingested in a first two-dimensional format or a first three-dimensional format, the three-dimensional format referencing a neural network, the content including supplemental enhancement information for a specific neural network, the supplemental enhancement information being used to associate a neural network model for a specific scene with a specific scene existing within the content; The conversion code is configured to cause the one or more computer processors to convert ingested content into a second two-dimensional or second three-dimensional format based on a referenced neural network, according to the determined capabilities of the client, wherein the conversion includes: adapting the content to a scene-specific volumetric format associated with the client endpoint based on depth information inferred from the content; interpolating additional views from the input media using a neural network process, and selecting specific views based on a specific size of the view frustum to fill the view frustum of a specific display; and Streaming code configured to cause the one or more computer processors to stream the converted content to the client endpoint.

8. The computer system according to claim 7, further comprising: Based on the depth information inferred from the content, the content is adapted into a scene-specific volumetric format associated with the client endpoint.

9. The computer system according to claim 7, wherein, The neural network is referenced using metadata used to identify the model's location.

10. The computer system according to claim 7, wherein, The neural network is referenced by a generic resource identifier corresponding to the metadata used to describe the content.

11. The computer system according to any one of claims 7 to 10, wherein, Before ingesting the content, the neural network is trained based on priors corresponding to objects within the content.

12. The computer system according to any one of claims 7 to 10, wherein, The ingested content is transformed based on the characteristics of the client endpoint.

13. The computer system according to any one of claims 7 to 10, wherein, One or more client endpoints include one or more of a television, computer, head-mounted display, lenticular light field display, holographic display, augmented reality display, and dense light field display.

14. A non-transitory computer-readable medium storing a computer program for streaming immersive media, the computer program being configured to cause one or more computer processors to perform the method for streaming immersive media according to any one of claims 1 to 6.

Images