Storage and / or streaming format for multi-view audio and video based on graph-based object-centric representation

Abstract

In one example, a method for streaming multi-perspective audio and video (MPAV) from a server device includes generating a graph representation of a plurality of videos corresponding to the MPAV, the graph representation including a plurality of nodes interconnected via a plurality of edges, each of the nodes representing a respective one of the videos, each of the edges representing an allowed switch between a corresponding pair of the videos during playback; encoding the plurality of videos into a corresponding plurality of bitstreams; packaging the plurality of bitstreams into a storage container based on the graph representation, the storage container being accessible via the server device; providing metadata configured to enable a player device to reconstruct the graph representation; and in response to a request generated at the player device based on the reconstructed graph representation, streaming a selected subset of the plurality of bitstreams.

Description

[0001] 1. Cross-references to related applications This application claims priority to U.S. Provisional Application Serial No. 63 / 640,663, filed April 30, 2024, and U.S. Provisional Application Serial No. 63 / 707,838, filed October 16, 2024, each of which is incorporated herein by reference in its entirety. 2. Technical Field Various example embodiments relate to video streaming, and more specifically, but not exclusively, to the storage and streaming of multi-view audio and video (MPAV). 3. Background Technology The success of video streaming has sparked interest in newer forms of video content, such as video / audio content generated using 360-degree cameras, multi-angle camera arrays, and / or light field cameras. The immersive experiences enabled by these formats can enhance user satisfaction with broadcast performances (e.g., theater and concerts) and events (e.g., graduations and award ceremonies), as well as online meetings. With MPAV capture, users are not merely passively consuming content, but can interactively traverse it along many different paths from their chosen perspectives, allowing different users to observe the same content from various viewpoints. To support MPAV applications at internet scale, it is expected that client player devices can download the content, and that content delivery networks (CDNs) can generate viewpoints on demand, while ensuring low latency and supporting a wide range of devices used to capture and consume the corresponding MPAV content. Summary of the Invention

[0004] Some embodiments provide an object-centric graph representation configured to describe multiple functional states of an object or product. In the graph representation, each node represents a video, and directional edges represent permitted video transitions. Videos belonging to each functional state can be grouped into subgraphs, and each subgraph can connect to other subgraphs to allow navigation through different functional states. Additional intra-video and inter-video attributes can be stored as metadata corresponding to nodes and edges. In some examples, the graph representation is configured to cover multiple objects, products, and / or product models. Different aspects of the graph representation are supported using different formats compatible with ISOBMFF and / or MPEG-DASH Media Presentation Description (MPD). Various hybrid formats can be advantageously used to accommodate different trade-offs regarding bitrate and latency. For some use cases, time-aware graph-based formats are constructed.

[0005] According to an example embodiment, a method for streaming multi-view audio and video (MPAV) from a server device includes: generating a graph representation of a plurality of videos corresponding to the MPAV, the graph representation including a plurality of nodes interconnected via a plurality of edges, each of the nodes representing a corresponding video in the video, each of the edges representing a permissible switch between a corresponding pair of the videos during playback; encoding the plurality of videos into a plurality of corresponding bitstreams; packaging the plurality of bitstreams into a storage container based on the graph representation, the storage container being accessible via the server device; providing metadata configured to enable a player device to reconstruct the graph representation; and streaming a selected subset of the plurality of bitstreams in response to a request generated at the player device based on the reconstructed graph representation.

[0006] According to another example embodiment, a non-transitory computer-readable medium is provided, the non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations including the methods described above.

[0007] According to yet another example embodiment, an apparatus for streaming multi-view audio and video (MPAV) includes: at least one processor; and at least one memory including program code, wherein the at least one memory and the program code are configured, together with the at least one processor, to cause the apparatus to perform at least the following operations: generating a graph representation of a plurality of videos corresponding to the MPAV, the graph representation including a plurality of nodes interconnected via a plurality of edges, each of the nodes representing a corresponding video in the video, each of the edges representing a permissible switch between a corresponding pair of videos during playback; encoding the plurality of videos into a plurality of corresponding bitstreams; packing the plurality of bitstreams into a storage container based on the graph representation; providing metadata configured to enable a player device to reconstruct the graph representation; and streaming a selected subset of the plurality of bitstreams in response to a request generated at the player device based on the reconstructed graph representation.

[0008] According to yet another example embodiment, a method for streaming multi-view audio and video (MPAV) using a playback device includes: receiving metadata configured to enable the player device to reconstruct a graph representation of a plurality of videos corresponding to the MPAV, the graph representation including a plurality of nodes interconnected via a plurality of edges, each of the nodes representing a corresponding video in the video, each of the edges representing a permissible switch between a corresponding pair of the videos during playback; generating a streaming request for a server device based on the reconstructed graph representation and further based on interactive user input, the streaming request identifying a selected subset of a plurality of bitstreams packaged into a storage container accessible via the server device, the plurality of bitstreams encoding the plurality of videos corresponding to the MPAV; and receiving the selected subset of the plurality of bitstreams transmitted by the server device in response to the streaming request.

[0009] According to yet another example embodiment, a non-transitory computer-readable medium is provided, the non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations including the methods described above.

[0010] According to yet another example embodiment, an apparatus for streaming multi-view audio and video (MPAV) includes: at least one processor; and at least one memory including program code, wherein the at least one memory and the program code are configured, together with the at least one processor, to cause the apparatus to at least: receive metadata configured to enable a player device to reconstruct a graph representation of a plurality of videos corresponding to the MPAV, the graph representation including a plurality of nodes interconnected via a plurality of edges, each of the nodes representing a corresponding video in the video, each of the edges representing a permissible switch between a pair of corresponding videos during playback; generate a streaming request for a server device based on the reconstructed graph representation and further based on interactive user input, the streaming request identifying a selected subset of a plurality of bitstreams packaged into a storage container accessible via the server device, the plurality of bitstreams encoding the plurality of videos corresponding to the MPAV; and receive the selected subset of the plurality of bitstreams transmitted by the server device in response to the streaming request. Attached Figure Description

[0011] Other aspects, features, and advantages of the various disclosed embodiments will become more fully apparent by way of example, in conjunction with the following detailed description and accompanying drawings, in which: Figure 1This is a flowchart illustrating the workflow of an object / product-centric MPAV approach based on some examples.

[0012] Figures 2A to 2F A collection of videos illustrating a dishwasher as an example is presented in image format.

[0013] Figures 3A to 3C The illustration is based on some examples. Figures 2A to 2F A diagram representing a collection of videos.

[0014] Figures 4A to 4F A collection of six videos depicting a cabinet, illustrated in image form, is presented as an example.

[0015] Figures 5A to 5C The illustration is based on some examples. Figures 4A to 4F A diagram representing a collection of videos.

[0016] Figure 6 It is a block diagram illustrating a diagram representation of an example, including three sub-diagrams.

[0017] Figures 7A to 7C The possible differences within a video group based on an example are illustrated in the image format.

[0018] Figure 8 This is a block diagram illustrating a hierarchy defined in ISO BMFF based on an example.

[0019] Figure 9 This is a block diagram illustrating the overall structure of an ISO BMFF file for timestamped data, based on some examples.

[0020] Figure 10 This is a block diagram illustrating a movie bitstream as an example.

[0021] Figure 11 This is a block diagram illustrating a communication system configured to support MPAV transmission using DASH and ISO BMFF, based on some examples.

[0022] Figure 12 The illustration shows some examples. Figure 11 A block diagram of the data structure of the MPAV container used in the communication system.

[0023] Figure 13 This is a block diagram illustrating a high-level hierarchical data model that can be used in DASH, based on some examples.

[0024] Figure 14 This is a block diagram illustrating a media streaming architecture based on some examples.

[0025] Figure 15 This is a block diagram illustrating file encapsulation formats based on some examples.

[0026] Figure 16 This is a block diagram illustrating the configuration based on an example track reference.

[0027] Figure 17 This is a block diagram illustrating the fragment encapsulation format and object representation information signaling based on some examples.

[0028] Figure 18 This is a block diagram illustrating a DASH configuration for indicating two groups that can be switched between each other, based on an example.

[0029] Figure 19 This is a block diagram illustrating a DASH configuration for indicating combinations of adaptive sets, based on an example.

[0030] Figures 20A to 20B This is a block diagram illustrating a hybrid ISO BMFF and MPEG-DASH format based on an example.

[0031] Figures 21A to 21B This is a block diagram illustrating a hybrid ISO BMFF and MPEG-DASH format based on an example.

[0032] Figure 22 This is a block diagram illustrating a graph representing four MPD time periods, based on an example.

[0033] Figure 23 The multi-view video is illustrated using images based on an example.

[0034] Figure 24 The illustration shows a representation that can be used based on an example. Figure 23 A block diagram of the four MPD time periods of the multi-view video.

[0035] Figure 25 A multi-view video based on another example is illustrated using images.

[0036] Figure 26 The illustration shows a representation that can be used based on an example. Figure 25 A block diagram of the four MPD time periods of the multi-view video.

[0037] Figure 27 A multi-view video based on another example is illustrated using images.

[0038] Figure 28 The illustration shows a representation that can be used based on an example. Figure 27 A block diagram of five MPD time periods of a multi-view video.

[0039] Figure 29 It is a block diagram of an example computing device based on various examples, one or more instances of which can be used to implement the various methods and processes disclosed herein. Detailed Implementation

[0040] Multi-view audio and video (MPAV) has a wide range of applications, including but not limited to sporting events, concerts, parties, and educational content, and involves multiple cameras (or views) capturing (or representing) the same large scene. The aforementioned U.S. Patent Application No. 63 / 640,663 relates to several event-based applications of MPAV (primarily focused on the timeline of corresponding events) and discloses a low-latency, seamless perspective switching framework suitable for such applications. However, MPAV also has additional applications (e.g., in social e-commerce) where the primary concern is not the timeline, but rather various aspects and / or attributes of objects. Therefore, the various embodiments disclosed herein aim to provide a framework for object- and / or product-centric applications of MPAV.

[0041] In representative use cases, event-based MPAV user-generated content (UGC) captures videos around events in a largely unplanned manner, such as videos originating from unorganized crowds (e.g., participants in a concert or birthday party). In contrast, object-centric applications of MPAV (such as "product-centric" videos designed for social e-commerce) have a clear objective: to describe and / or promote objects and / or products, for example, with the intent to allow end-users to explore objects in a more interactive way without timeline guidance or constraints. In other words, the storyline of videos provided for these purposes can be planned in advance (before capture), managed during capture, edited in the editing phase after the initial content capture, and / or supplemented by recapture or adding more footage. More product information can also be added after the initial capture. Utilizing this object-centric interactive format, end-users can beneficially gain more information about the product, interact with the product, and even interact with the content creator (e.g., in at least some use cases, such as during live shopping). Therefore, at least some embodiments are designed to beneficially enrich product information for end-users to better inform their purchasing decisions.

[0042] Figure 1 This is a block diagram illustrating the workflow of an object / product-centric MPAV method (100) based on some examples. In the example shown, the method (100) includes a capture block (110), an edit block (120), a dispatch block (130), and a playback block (140). Below are examples briefly describing the basic functionality of the blocks (110-140).

[0043] Within the capture block (110), content creators can interact with artificial intelligence (AI) based... Capture Planning Assistance (CPA) tools work together to plan how to capture video for the target product, target audience, allocated video capture budget, reasonable capture hardware resources, available capture personnel configuration, and expected capture environment. In various examples, CPA tools provide advice during the planning phase, such as how to set up the camera and prepare storylines and / or storyboards for the capture. In some examples, CPA tools are also used to perform real-time analysis on each video being recorded, for example, to provide feedback and insert labels and / or feature vectors for each shot.

[0044] In the edit block (120), content creators interact with AI-based... Content editing assistance The CEA (Content Execution Analysis) tool works collaboratively to analyze footage, synchronize video across multiple related cameras, and segment scenes from multiple videos. In various examples, the CEA tool also provides suggestions on how to promote the product based on different promotional and / or presentation styles, selectable from a variety of such styles. In some examples, the CEA tool provides linking information (e.g., for object / product representations in relation to corresponding diagrams), including suggestions on how to connect different scenes and present the product in different product poses / functional states and / or product usage environments. In some cases, the CEA tool is configured to generate product representations so that end users can navigate through different segments of footage and understand the product under different usage conditions (e.g., including but not limited to lighting, temperature, static / slow-motion / fast-motion), types (e.g., color, size), viewing angles, and / or product states (open access to product parts, functional use of different parts of the product, etc.). Various corresponding types of multimedia formats are supported in various examples, including video, audio, images, and 3D models.

[0045] In the distribution block (130), an efficient distribution system is configured to transmit product representations and support the intended interactivity. In some examples, both the ISO BMFF encapsulation format and the MPEG-DASH format are used together to implement different trade-offs between various priorities, such as (a) low bitrate priority, (b) low latency priority, and (c) interactivity priority. In some examples, an AI-based Content Delivery Assistance (CDA) tool is used to assist in decisions regarding the packaging of footage across different systems to guide the target audience towards achieving target sales goals. In some examples, the distribution system employed in the distribution block (130) is configured to simultaneously support live streaming and offline streaming capabilities.

[0046] In the playback block (140), the corresponding playback system processes the received bitstream so that the end user can navigate between different video components of the MPAV and learn more details about the product. In some examples, the playback system is configured to support both live video and offline pre-recorded video simultaneously, which is beneficial for achieving promotional goals. In some examples, the playback system runs an application (app) that provides a simplified shopping experience, for example, starting with exploring products and ending with ordering from a virtual shopping cart.

[0047] Some embodiments disclosed herein focus specifically on the implementation of the editing block (120) and the distribution block (130) of the method (100). Example features of such embodiments include, but are not limited to: • Object / product-centric representations based on one or more graphs.

[0048] • Store intra-video attributes and / or inter-video attributes as metadata associated with the nodes and edges of the graph.

[0049] • The representation is implemented using ISO BMFF and MPEG DASH. Corresponding additional descriptors / boxes are implemented according to the framework of these two standards to enable / complete the relevant operations.

[0050] • Guides the trade-off between latency and bit rate, and design principles that combine bitstream level (ISO BMFF) with streaming level (MPEG DASH).

[0051] • Time-aware graph representation to enable additional applications and use cases.

[0052] Graph-based representation of objects or products A graph is a fundamental construct in computer science, used as an abstract representation of interconnected objects. The two main components of a graph are vertices representing objects and edges representing the links between objects. The following terminology is commonly used when referring to graphs: • Vertex. A vertex, also known as a node, represents an entity within a graph. In various use cases, vertices can represent all sorts of things, such as a city in a transportation network or a user in a social media platform.

[0053] • Edge. An edge represents a connection or relationship between two vertices in a graph. It can be directed (from one vertex to another) or undirected (connecting two vertices without a specific direction).

[0054] • Adjacency. Adjacency refers to the relationship between vertices that are directly connected by an edge. For example, if vertex A is connected to vertex B by an edge, then A and B are considered adjacent.

[0055] • Path. A path in a graph is a sequence of vertices connected by edges. It represents a route or journey from one vertex to another. A path can be simple (without repeating vertices) or cyclic (with repeating vertices).

[0056] • Directed graph. A directed graph, also known as a digraph, is a graph in which edges have directions. This means that the relationships between vertices are unidirectional, indicating a specific direction of traversal.

[0057] Object / product representation formats are crucial elements supporting user interaction experiences. According to the example embodiment, graph representations are used to connect different scenes. Each node in the graph represents an audio / video scene, and directed edges indicate the ability to switch from one scene to one or more other scenes. A node can have multiple outward edges to allow multiple videos to play simultaneously or be selected and switched by the end user. The example embodiment uses a directed graph and can be applied to multiple scenes. Some example scenes are illustrated and described below with reference to Figures 2 through 5.

[0058] Figures 2A to 2F A set of six videos (202-212) depicting a dishwasher according to an example are illustrated in image form. The first subset, including videos (202, 204, 206), shows the dishwasher's first functional state (door closed) from different corresponding angles. The second subset, including videos (208, 210, 212), shows the dishwasher's second functional state (door open) from different corresponding angles.

[0059] Figures 3A to 3C This is a block diagram illustrating a graphical representation of a set of videos (202-212) based on some examples. The diagram treats each video (202-212) as a corresponding node, such as... Figure 3A As shown. Therefore, there are six nodes, which are labeled (af). Node (a) represents video (202). Node (b) represents video (204), and so on. Figure 3B The diagram illustrates a first example graph (310) constructed using nodes (af). Graph (310) is a directed graph that defines the presentation order of the videos (202-212) according to the cyclical playback list, for example, starting from node (a) and proceeding through nodes (bf) in sequence, and then restarting at block (a). Figure 3CA second example diagram (320) constructed using nodes (af) is illustrated. In diagram (320), nodes (af) are grouped according to a first functional state and a second functional state (door closed or door open) to form a first node group and a second node group, namely, group (ac) and group (df). Playback within each of the first and second groups can be freely switched, as indicated by the corresponding unlabeled bidirectional arrows. However, in diagram (320), switching between groups is only allowed between nodes (a) and nodes (e), as indicated by the bidirectional arrow (322).

[0060] Figures 4A to 4F A set of six videos (402-412) depicting a cabinet according to another example is illustrated in image form. The first subset, including videos (402, 404), shows the cabinet in a first functional state (doors and drawers closed) from two different angles. The second subset, including videos (406, 408), shows the cabinet in a second functional state (doors open and drawers closed) from two different angles. The third subset, including videos (410, 412), shows the cabinet in a third functional state (doors open and drawers open) from two different angles.

[0061] Figures 5A to 5C This is a block diagram illustrating a graphical representation of a set of videos (402-412) based on some examples. The diagram treats each video (402-412) as a corresponding node, such as... Figure 5A As shown. Therefore, there are six nodes, which are labeled (af). Node (a) represents video (402). Node (b) represents video (404), and so on. Figure 5B The diagram illustrates a first example graph (510) constructed using nodes (af). Graph (510) is a directed graph that defines the presentation order of the videos (402-412) according to the cyclical playback list, for example, starting from node (a) and proceeding through nodes (bf) in sequence, and then restarting at block (a). Figure 5C The diagram illustrates a second example diagram (520) constructed using nodes (af). In diagram (520), nodes (af) are grouped according to a first functional state, a second functional state, and a third functional state to form a first node group, a second node group, and a third node group, namely, group (ab), group (cd), and group (ef). Playback within each of the first, second, and third groups can be freely switched, as indicated by the corresponding unlabeled bidirectional arrows. However, in diagram (520), switching between groups is only allowed between nodes (a), (c), and (e), as indicated by bidirectional arrows (522, 524, 526).

[0062] Figure 6This is a block diagram illustrating a graphical representation (600) comprising three subgraphs (610-630) according to an example. The graphical representation (600) is configured to simultaneously support timestamped presentation with multiple products by linking the subgraphs (610-630) together. Subgraph (610) has nodes (a0-f0) and represents the first product. Subgraph (620) has nodes (a1-f1) and represents the second product. Subgraph (630) has nodes (a2-f2) and represents the third product. Subgraph (610) allows for cyclic switching between views of the first product (0) (e.g., having only one functional state, viewed from six different viewing angles). Subgraph (620) has three node groups corresponding to the three functional states of the second product (1) and allows for view switching between nodes (a1), (c1), and (e1). Subgraph (630) also has three node groups, which correspond to the three functional states of the third product (2) and allow view switching between nodes (a2), (c2), and (e2). View switching between different products (0, 1, 2) is limited to bidirectional switching between nodes (a0) and (a1) and between nodes (a1) and (a2). Direct switching between nodes (a0) and (a2) is not allowed.

[0063] In some examples, each node in the graph represents a corresponding scene and can include multiple distinct attributes and / or information that the end user can use to determine what content to view next. Supported features include: • Multiple content types: audio, video, images, graphics, web links.

[0064] • Content playback - composite layout.

[0065] • Content information: viewing angle, function status, duration, dimension, frame rate.

[0066] The corresponding information can be encapsulated in metadata and formatted using one of the formats described below (e.g., MPEG DASH level or ISO BMFF level) so that users can select the node to switch to.

[0067] In some examples, each edge in the graph provides a description of the corresponding transition from one node to another to assist with node (scene / view) switching. Supported features include: • View transition effects. Transition effects are specified for the “edges” in the graph and indicate the transition effect from one video to another. In some cases, transition effects can be regular scene transitions, such as fade-in / fade-out or dissolve. For view transitions, transition effects can be image distortion or warping. Transitions can generally be classified into two distinct groups: (1) within a function state; and (2) between function states.

[0068] • Navigating via Graphs (Director's Cut). While graphs offer many degrees of freedom (edges) to allow users to explore nodes, it's also possible to define a "default" viewing path through the graph so that end users can sit back and enjoy the experience without actively navigating it themselves.

[0069] The table below provides an example illustrating some view transformation effects defined in ISO / IEC 23090-2 - Information technology - Encoded representation of immersive media - Part 2: Omnidirectional media formats.

[0070] Table 15—Types of Viewpoint Change Effects

[0071] Table 15 — Viewpoint shift effect types

[0072] Other viewpoint shift effect types can also be used in other examples.

[0073] Figures 7A to 7C The possible differences within a video group (702-706) based on an example are illustrated in image form. Typically, within a video group, there may be identical functional states, where the differences between videos lie in (1) regional / local differences in the spatial domain and / or (2) differences in the temporal domain. In the example shown, for video group (702-706), the base videos are identical, and the differences lie in the lip color (and the text and shopping cart icon synthesized later using different corresponding information, as described in more detail below, for example).

[0074] from Figures 7A to 7C It is evident that a large amount of public information exists within the video (702-706). Therefore, more efficient methods can be employed to compress these videos (702-706) to support their lower bitrate transmission. In various examples, at least the following types of spatial domain efficient compression can be considered: (A) Regional differences under the same frame ratio. For this scenario, the following three different types of video can be used: • Public base video. This is the video that shows the common shared pixels among all videos in this group.

[0075] • Masked Video. This is a mask indicating the pixel locations where different videos in the group differ from each other. In various examples, the differences can be color differences, texture differences, etc. Masked video allows for the replacement of corresponding pixel values ​​in a common base video for different videos in the group. In some examples, the resolution of the masked video can be smaller than the resolution of the common base video. In different implementations, the values ​​in the masked video can be binary (for full masking) or multi-valued (for gradual blending).

[0076] • Region Video. This is the video used for the masked area. In some examples, the region video is a low-resolution video configured to represent information corresponding to the mask pixels. The dimensions of the region video and the mask video will typically match each other.

[0077] The three types of video listed above can be encoded as separate bitstreams, or they can be stitched together using the tile / strip-based structure used in AVC / HEVC video bitstreams. Metadata is typically required to indicate the mask offset and scaling parameters.

[0078] (B) The content of the regions is the same, but the aspect ratios of the corresponding frames are different. In some examples, the difference between the videos is that some videos are enlarged / shrunk versions of other videos. For this scenario, the following three different types of videos can be used: • Public base video. This video shows all the pixels present in the video group.

[0079] • Masked Video. This is a mask configured to represent the area to be scaled from the public underlying video. In some examples, this information can be in metadata format (e.g., when using a rectangular area), or it can have any shape that the mask is needed to describe.

[0080] • Regional video used for the masked area. In some examples, this video can be a scaled version of the masked area, or it can be an empty video with metadata indicating the scaling method applied.

[0081] The three types of video listed above can be encoded as separate bitstreams, or they can be stitched together using the tile / strip-based structure used in AVC / HEVC video bitstreams. Metadata is typically required to indicate the mask offset and scaling parameters.

[0082] (C) Cropping a region from the same video. This scenario can be considered a special case of (B), where the scaling factor is 1.

[0083] In various examples, compression methods similar to the efficient compression methods described above can also be utilized in the temporal domain along the timeline by adding metadata to indicate the time offset relative to the public underlying video. The following examples of corresponding end-to-end architectures and distribution formats will describe how to implement the aforementioned efficient compression.

[0084] Relevant elements of ISO BMFF and MPEG DASH The various embodiments disclosed herein relate to providing an end-to-end interactive video streaming system, the system comprising: • Components located on the server side and the content storage formats used therein; • Components that support network distribution protocols; and • Components that support interactive playback solutions on the terminal client side.

[0085] Some embodiments use the MPEG-DASH and ISO BMFF standards and / or their extensions, and additionally provide new tools to support new features. One objective of this embodiment is to enable the rapid and / or cost-effective deployment of interactive, immersive volumetric video experiences by building upon and extending certain legacy technologies / infrastructure to achieve seamless interactive view switching for video streaming. Some embodiments may benefit from at least some of the features disclosed in one or more of the following documents: (1) ISO / IEC 14496-12:2020, Information technology—Encoding of audiovisual objects—Part 12: ISO Basic Media File Format; (2) ISO / IEC 14496-15:2021, Information technology—Encoding of audiovisual objects—Part 15: Structured video with Network Abstraction Layer (NAL) units carrying ISO Basic Media File Format; (3) ISO / IEC 23000-19:2018, Information technology—Multimedia Application Format (MPEG-A)—Part 19: Common Media Application Format (CMAF) for segmented media; and (4) ISO / IEC 23009-1(2019), Information technology—Dynamically Adaptive Streaming Based on HTTP (DASH)—Part 1: Media Presentation Description and Segment Format. Each of these documents is incorporated herein by reference in its entirety.

[0086] The embodiments disclosed below provide solutions for building end-to-end interactive video streaming systems. For storage formats, various embodiments are based on the ISO Basic Media File Format (BMFF), commonly known as ISO BMFF (ISO / IEC 14496-12), and its extension ISO / IEC 14496-15, which provides the carrying of network abstraction layer (NAL) unit structured video (such as AVC, HEVC, VVC, etc.). In this document, the acronym ISO stands for International Organization for Standardization. For data transmission, various embodiments are based on HTTP-based Dynamic Adaptive Streaming (DASH) (ISO / IEC 230090-1). In some examples, a similar approach is applied to the Common Media Application Format (CMAF). Note that the disclosed video streaming systems can also support other multi-view codec solutions, such as MVC, SHVC, and multi-layer VVC. For illustrative purposes and without any implied limitation, the example embodiments are described with reference to the HEVC codec. Based on the provided description, those skilled in the art will be able to create and use other embodiments compatible with other suitable codecs without any unnecessary experimentation.

[0087] Figure 8 This is a block diagram illustrating the hierarchical structure defined in ISO BMFF according to an example. In this document, the acronym BMFF stands for Basic Media File Format. These files have: (i) a logical structure, e.g., a movie containing a set of time-parallel tracks; (ii) a temporal structure, e.g., tracks containing sequences of time samples, and these sequences are mapped to the timeline of the entire movie via an optional edit list; and (iii) a temporal structure, e.g., tracks containing sequences of time samples, and these sequences are mapped to the timeline of the entire movie via an optional edit list. These structures do not need to be coupled.

[0088] Each media stream is contained in a dedicated track for that media type (audio, video, etc.) and is further parameterized by sample entries. A sample entry contains the 'name' of the exact media type (e.g., the type of decoder required to decode the stream) and any parameterization required by that decoder. This name is also in the form of a four-character code. Sample entry formats are defined not only for MPEG-4 media but also for media types used by other organizations using this file format family. They are registered with the MP4 registry. Tracks (or subtracks) can be identified as alternatives to each other, and declarations in the form of track selection data are supported to identify which aspects of a track can be used to determine which alternative to present.

[0089] Data is stored in basic structures called “boxes.” There is no data outside the boxes. Each box has a size (or length), type (four printable characters, 4CC), possible versions, and a flag, and contains data. Boxes can be nested and arranged hierarchically, meaning one box can be inside another. The following pseudocode provides an example.

[0090]

[0091] • Size is an integer that specifies the number of bytes in the box, including all its fields and the boxes it contains; if size is 1, the actual size is in the largesize field; if size is 0, the box is the last one in the file and its contents are extended to the end of the file (typically only used for media data boxes). • The Type field identifies the box type; standard boxes use a compact type, which is typically four printable characters for easy identification and is shown in the box below. User extensions use an extended type; in this case, the type field is set to 'uuid'.

[0092] In addition to the term "box," the "FullBox" definition, which contains more information, is as follows:

[0093] Two different definitions are used to distinguish between still images and videos: • Item: Static images are stored as items. All image items are encoded independently and do not depend on any other items during their decoding. Any number of image items can be included in the same file.

[0094] • Track: Image sequences are stored as tracks. Image sequence tracks are used when there are encoding dependencies between images or when image playback includes timestamps. Unlike video tracks, the timing in image sequence tracks is suggestive.

[0095] In ISO BMFF files, timestamped data is divided into samples, which are described using a logical concept called a "track". Track boxes, identified by 4CC 'trak', are stored within movie boxes 'moov' and contain multiple boxes that provide information about where to find samples in the file and how to interpret them. The samples themselves are stored in media data boxes identified by 4CC 'mdat'. Each track contains at least one sample entry within a sample description box 'stsd'. The sample entry describes the encoding and encapsulation format used in the track sample. Sample entries are also identified by 4CC and can contain more boxes for further configuration. In some examples, untimestamped data is stored in a meta box 'meta', which is represented by a sample described by a logical concept called an item. Untimestamped sample data can be stored in media data boxes or in item data boxes 'idat', which are stored in meta boxes.

[0096] Figure 9 This is a block diagram illustrating the overall structure of an ISO BMFF file (900) for timestamped data, based on some examples. Figure 9 The illustration provided further details of the various boxes in the description document (900). This subsection below provides more detailed descriptions of several related boxes.

[0097] Figure 10 This is a block diagram illustrating a movie bitstream (1000) based on an example. The bitstream (1000) has two main components: (1) a metadata bitstream (1010) with 4CC as “meta”, and (2) a media bitstream (1020) with 4CC as “moov”. Each track of the media bitstream (1020) includes a sequence of timestamped samples. Each sample has a decoding time and may also have a synthesis (display) time offset. Edit lists can be used to overwrite the implicit direct mapping in the media timeline to the timeline of the entire movie. In some cases, samples within a track have different characteristics or need to be specially identified. One of the most common and important characteristics is synchronization points (e.g., video I-frames). These points are identified by special tables in each track. More generally, the nature of dependencies between track samples can also be recorded. There is also a concept of named parameterized sample groups. Each sample in a track can be associated with a single group description of a given group type, and many group types can exist.

[0098] ISO BMFF defines the top box as follows:

[0099] The “meta” box contains the following boxes:

[0100] The “moov” box contains the following boxes:

[0101] A “trak” is a container box for a single track used for rendering. A rendering consists of one or more tracks. Each track carries its own temporal and spatial information. Each track will contain its associated MediaBox. Tracks serve multiple purposes, including: (a) containing media data (media tracks), and (b) containing packet information for streaming protocols (e.g., cue tracks). At least one media track exists within a MovieBox, and all media tracks that contribute to the cue track are retained in the file, even if the media data in those tracks is not referenced by the cue track. After all cue tracks are deleted, the entire un-cue rendering is preserved. The following pseudocode provides an example of a “trak”:

[0102] "tkhd" is a box that specifies the properties of a single track. A track can contain only one TrackHeaderBox. The following pseudocode provides an example of "tkhd":

[0103] track_ID is an integer that uniquely identifies the track throughout its entire lifetime. track_ID is never reused and cannot be zero.

[0104] Without an edit list, track rendering begins at the start of the entire rendering. An 'empty' edit is used to offset the start time of a track. Tracks marked with the `track_in_movie` flag set to 1 are tracks the file writer intends to render directly. Therefore, tracks used as input to another track before or after decoding, but not rendered by themselves, should have the `track_in_movie` flag set to 0. Tracks with the `track_in_movie` flag set are candidates for playback, regardless of whether they are media tracks or receive cue tracks. Tracks can be used as input and will also have the `track_in_movie` flag set to 1.

[0105] If a track is a member of a group that provides alternatives for presentation (either by using the alternative_group field in the track header or by using a TrackGroupBox with a group type that defines the alternatives), then either only the track_in_movie flag of the preferred or default selected track should be set to 1, or the track_in_movie flag of all tracks should be set to 1 (if no default value is indicated).

[0106] "tref" is a box containing a collection of TrackReferenceTypeBoxes, each of which indicates, by its type, that one or more references of that type exist for the track it belongs to. Each reference type appears at most once. Within each TrackReferenceTypeBox is an array of track_IDs. A given value appears at most once within a given array. Other structures in the file format are indexed using these arrays. Index values ​​start from 1. A TrackBox can contain only one TrackReferenceBox.

[0107] If the "tref" box does not exist, the track does not reference any other track in any way. The size of the reference array is determined by the number of reference type boxes to fill, for example, as shown in the following pseudocode example:

[0108] “track_IDs” is an array of integers that provides the track identifier of the referenced track or the track_group_id value of the referenced track group. Each track_IDs[i] (where, i `track_group_id` (a valid index of the `track_IDs[]` array) is an integer that provides a reference from the current track to a track whose `track_ID` equals `track_IDs[i]` or to a track group whose `track_group_id` equals `track_IDs[i]` and `TrackGroupTypeBox`'s `(flags&1)` equals 1. When referencing a `track_group_id` value, the track reference is applied individually to each track in the referenced track group unless otherwise specified in the semantics of the specific track reference type. The value 0 does not exist. Duplicate values ​​are not present in the array. However, `track_ID` may appear in the array and may also be a member of one or more track groups whose `track_group_id` also appears in the array. This means that when forming a list of tracks, duplicate `track_ID`s may exist after replacing `track_group_ID` with the `track_ID` of the tracks in those groups. `track_group_ID` is not used when the semantics of the reference require that the reference must point to a single track.

[0109] The group list box (grpl) is defined as follows: Box type: 'grpl' Container: MetaBox Necessity: No Quantity: zero or one The GroupsListBox contains the entity groups specified for the file. This box contains a complete set of boxes, each called EntityToGroupBox, where a four-character code indicates the defined grouping type.

[0110] When a GroupsListBox exists in a file-level MetaBox, no ItemInfoBox in any file-level MetaBox should have an item_ID value equal to the track_ID value in any TrackHeaderBox.

[0111]

[0112] The EntityToGroupBox is defined as follows:

[0113] group_id is a non-negative integer assigned to a specific group. It should not be equal to any group_id value of any other EntityToGroupBox, should not be equal to any item_ID value of the hierarchy level (file, movie, or track) containing the GroupsListBox, or should not be equal to any track_ID value (when the GroupsListBox is contained at the file level).

[0114] num_entities_in_group specifies the number of entity_id values ​​mapped to this entity group.

[0115] When an item with an item_ID equal to entity_id exists at the level containing the GroupsListBox (file, movie, or track), entity_id is resolved to an item; when a track with an item_ID equal to entity_id exists and the GroupsListBox is contained at the file level, entity_id is resolved to a track.

[0116] Figure 11 This is a block diagram illustrating a communication system (1100) configured to support MPAV transmission using DASH and ISO BMFF, according to some examples. The communication system (1100) includes a server (1102) and a client (1106) connected via a communication channel (1104). The server (1102) includes a data storage device in which one or more MPAV containers (1200) are stored. In operation, the client (1106) requests and receives, via the communication channel (1104), relevant portions of the data stored in the MPAV containers (1200), as described in more detail below.

[0117] Figure 12 This is a block diagram illustrating the data structure of an MPAV container (1200) used in a communication system (1100) according to some examples. The MPAV container (1200) comprises multiple segments logically organized (e.g., using appropriate indexes) to form three sets of planes in a 3D space, the dimensions of which are adaptive sets, representations, and segment times. There are two types of segments: initialization segments (1332) and media segments (1334). The end user at the client (1106) typically requests an initialization segment (1332) for each time slot (1310). In one example, the initialization segment (1332) contains camera information and metadata related to the components of the media segment (e.g., video, audio, captions, etc.), allowing the end user to know which view(s)(s) to use. A single media segment (1334) contains audio and / or video content for different corresponding views at different corresponding bitrates within the corresponding time slot (1310). The end user typically requests only the media segment(s)(s) required via the client (1106).

[0118] Figure 13 This is a block diagram illustrating a high-level hierarchical data model (1300) that can be used in DASH, based on some examples. In one dimension (illustratively the horizontal domain), the model (1300) includes the time series corresponding to the media presentation. In another dimension (illustratively the vertical domain), the model (1300) includes options provided in the media presentation that are to be selected by the DASH client in a static or dynamic manner.

[0119] In some examples, model (1300) depends on the following definitions of various levels used in the DASH hierarchy: • Media Presentation Description (MPD): The DASH media presentation is described by the MPD document (1302). This document describes the sequence of time periods (610) that constitute the media presentation.

[0120] • Period: A period (1310) typically represents a time period in which a consistent encoded version of the media content is available. Within a period (1310), the material is arranged into an adaptive set (1320).

[0121] • Adaptation Set: An adaptation set (1320) represents a set of interchangeable encoded versions of one or more components of media content and contains a set of representations (1330). In some examples, the concept of preselection (1312) is used to combine different adaptation sets (1330) into a single decoding instance and achieve a better user experience.

[0122] • Representation: Representation (1330) describes a transmissible encoded version of one or more media content components and includes one or more media streams (e.g., each media content component in a multiplexed application corresponds to a media stream). An example way to use representation (1330) is to apply different bit rates. Within representation (630), content can be divided into time-series segments (1332, 1334, 1336) to achieve appropriate accessibility and transmissibility.

[0123] • Segment: In some examples, a segment is the largest unit of data that can be retrieved in a single HTTP request. To access a segment, a URL is provided for each segment. For segmented representation, two types of segments are used: ○ Initialize fragment (1332), which contains static metadata representing (1330); and ○ Media clip (1334), which contains media samples and advances the timeline.

[0124] • Self-initializing Segment: Some representations (1330) can also be organized into a single self-initializing segment (1336) containing both initialization information and media data.

[0125] In the MPAV scenario under consideration, a crucial component on the server side is the MPD (MPA Descriptor). When a user joins a session to begin MPAV playback, the MPD is sent to the end user. The MPD file is configured to contain sufficient information for the end user to select the viewpoint / bitstream to request for download. As discussed above, the MPD has a hierarchical structure, and each level can be mapped to an MPAV streaming system, for example, as shown below.

[0126] • Adaptation Set: Each adaptation set represents an MPAV with a specific compositing configuration viewed from a single perspective. For example, if there are 50 perspectives, there will be 50 corresponding adaptation sets. The end user can decide which adaptation set to download by selecting the perspective with the specific compositing to render. Note that the end user can request one adaptation set for a single perspective playback, or multiple adaptation sets for prefetching or for displaying multiple perspectives simultaneously.

[0127] • Representation: Each representation is associated with a specific encoded MPAV bitrate. Different representations can be selected based on the current network conditions between the server and the client.

[0128] • Segment: There are two types of segments: (1) Initialization Segment and (2) Media Segment. The end user requests an initialization segment for each segment. The initialization segment contains all relevant viewpoint information, as well as metadata related to the components of the media segment (video, audio, subtitles, etc.), so that the end user knows which viewpoint(s) and composition to use. A media segment contains audio and video content at different bitrates in different views within a given time period. The end user can request the desired media segment.

[0129] End-to-end architecture and sample distribution format Figure 14 This is a block diagram illustrating a streaming architecture (1400) based on some examples. The architecture (1400) comprises three main functional units: a content preparation component (1402), a server (e.g., a server (1102), which may be a component of the transport component (1404)), and an interactive player (1406). The content preparation component (1402) receives the following as inputs: (i) a collection of videos (1410) representing the target object, and (ii) an object representation (1422) using a directed graph. In some examples, the object representation (1422) is generated using an object representation generator (1420) based on the videos (1410) and the corresponding metadata. In some examples, the object representation generator (1420) is configured with multiple groups (or subgraphs). When multiple groups exist, the criteria used to form groups can be based on functional states. When switching between groups is available, the switching information includes the allowed switching paths between members and the destination group. For each group (subgraph), the corresponding information indicates the members and the nodes that are allowed to switch.

[0130] The video (1410) can be packaged at the bitstream level (ISO BMFF) or at the streaming level (MPEG-DASH) based on system design preferences and object representation or graph (1422). In some examples, placing relevant information at these two different levels is a design option based on a trade-off of several factors, such as playback computation / latency, network bandwidth, server storage capacity, and overall interactive experience. The prepared bitstream and DASH MPD can be stored on the server side (1102).

[0131] At the server and client playback sides (1102, 1106), the end user can reconstruct the graph (1422) based on the received metadata in the ISO BMFF and / or MPEG DASH MPD. Using the graph (1422), the end user can navigate to different graph nodes according to the graph design (the edges of the graph). In addition, based on the conditions of the network channel (1104, 1404), the player (1406) will select an appropriate bitrate for the selected video node to achieve smooth playback.

[0132] Figure 15 This is a block diagram illustrating a file encapsulation format (1500) based on some examples. In format (1500), object representation information (1422) is stored as metadata in the same file, where multiple scenes (af) are encapsulated in tracks and / or projects. When multiple scenes (af) are available, the corresponding encoded bitstreams (1502) of the scenes are input into the file encapsulation format (1500), such as... Figure 15 As indicated. In some examples, each video bitstream is stored in a track, and each image bitstream is stored in a project.

[0133] Object representation entity grouping contains object representation information (1422) associated with media carried in other tracks or projects. This object representation information includes the grouping of videos or images, how group members play, and information on switching to other groups. An example is provided below: Box type: 'oreg' Container: GroupsListBox('grpl') Necessity: No Quantity: Zero or one An EntityToGroupBox with grouping_type equal to 'oreg' describes object representation information associated with media carried in other tracks or projects (1422). The following pseudocode provides an example:

[0134] The following description explains the semantics: object_id is a non-negative integer identifier of the source object associated with the group entity.

[0135] The grouping_type indicates the type (e.g., the criteria used to form groups) and is set to one of the following values: • 0: This group is formed based on the functional state.

[0136] • 1: This group is formed based on different masks.

[0137] • 2: This group is formed based on perspective.

[0138] num_groups specifies the number of groups.

[0139] group_id is a non-negative integer identifier for the group, and it should not be equal to any group_id value from the same source object.

[0140] A playback_info_present_flag value of 1 indicates that PlaybackInfoStruct() exists. A playback_info_present_flag value of 0 indicates that PlaybackInfoStruct() does not exist. When PlaybackInfoStruct() does not exist, this means that the player can start playback from the entity indicated by anchor_entity_idx and freely switch between entities mapped to that group.

[0141] A `switching_info_present_flag` value of 1 indicates that `SwitchingListStruct()` exists. A `switching_info_present_flag` value of 0 indicates that `SwitchingListStruct()` does not exist. When `SwitchingListStruct()` does not exist, this means there is no recommended behavior when switching between groups, and the player can freely switch between this group and any other group as needed.

[0142] num_entities_group specifies the number of entities mapped to this group, as indicated by the entity_idx value.

[0143] entity_idx indicates the identifier of the item (item_ID) or track (track_ID) mapped to this group.

[0144] The anchor_entity_idx indicates the identifier of the item (item_ID) or track (track_ID) that is allowed to switch from and to the group. It should be set to the identifier of one of the entities mapped to the group.

[0145]

[0146] The playback_mode indicator specifies the playback mode for the entities that can be toggled in the group.

[0147] • 0: Indicates that the switchable entity can freely switch to other entities in the group that are allowed to switch.

[0148] • 1: Indicates sequential playback according to the order of the entity_idx of the switchable entities.

[0149] A `disallow_info_present_flag` value of 1 indicates that some switching is not allowed between entities in the group. A `disallow_info_present_flag` value of 0 indicates that switching is allowed between any entity in the group.

[0150] `num_diallowed_switch` indicates the number of disallowed switching relationships between entities in the group.

[0151] src_entity_idx and dest_entity_idx indicate the identifiers of the source and destination entities that are not allowed to be switched, respectively.

[0152]

[0153] num_switch_group indicates the number of switch groups that may come from the group associated with SwitchingListStruct().

[0154] destination_group_id indicates the identifier of the destination group to which the current group is switched from the group associated with SwitchingListStruct().

[0155] The `group_transition_effect_present_flag` flag specifies that the `group_transition_effect_type` exists. A `group_transition_effect_present_flag` value of 1 indicates that the `group_transition_effect_type` exists.

[0156] group_transition_effect_type indicates the type of transition effect when switching to the destination group, as described in Table 1.

[0157] Table 1: Group conversion effect types.

[0158]

[0159] When three bitstreams are carried in separate corresponding tracks within the same file, the player (1406) needs to be able to identify the types of bitstreams in the track and appropriately synthesize the bitstreams to obtain the correct output. In some examples, this functionality is implemented using the following track references and corresponding new 4CCs: - 'altv': Refers to a track containing video that provides new textures on a specific area of ​​the current track.

[0160] - 'mask': The reference track contains a mask that indicates which areas of the video should be replaced.

[0161] Figure 16This is a block diagram illustrating a configuration (1600) for a track reference based on an example. Configuration (1600) graphically indicates how the aforementioned track references are organized according to the type of video bitstream in the track. The common base video track (1610) contains 'altv' track references to refer to video tracks (1620, 1630, 1640) that provide new textures on specific regions. Each of the new texture video tracks (1620, 1630, 1640) also contains a corresponding 'mask' track reference to indicate which regions in the masked video track (1650) should be replaced.

[0162] Figure 17 This is a block diagram illustrating the object representation information signaling in the DASH MPD (1702) according to some example fragment encapsulation formats (1700). When multiple scene bitstreams (1704) are available, the encoded bitstreams are encapsulated into fragments based on ISO BMFF. Each bitstream can be carried in a fragment with a specific representation. To provide a grouped experience, prefetching and seamless playback between group entities will be supported. To achieve this, the object representation information (1422) of the group is indicated in the DASH MPD (1702).

[0163] When each bitstream is carried in a specific representation of an adaptive set, it is necessary to indicate the group of adaptive sets carrying bitstreams associated with the same object. To group different adaptive sets from the same source object, preselection can be used to achieve this grouping. Preselection can be signaled using preselection elements within time-segment elements in MPD. The `@preselectionComponents` attribute of the preselection element specifies the IDs of the AdaptationSet belonging to that preselection as a space-separated list. The first ID defines the AdaptationSet containing the anchor video of the object representation.

[0164] To identify each pre-selected property used for object representation, an ObjectRepresentation descriptor can be used at the pre-selection level. An ObjectRepresentation descriptor is an EssentialProperty element with its @schemeIdUri set to "urn:dolby:2024:iUGC:objectRepresentation." For grouping types and toggleable group information, the ObjectRepresentation descriptor is transmitted via signals. The @value of the ObjectRepresentation descriptor does not exist. The ObjectRepresentation descriptor includes attributes as specified in Table 2.

[0165] Table 2: Attributes of the ObjectRepresentation descriptor.

[0166]

[0167] To indicate the next playable destination representation within the same preselection, the representation contains @associationType and @associationId. @associationType is set to 'swit', and @associationId contains a space-separated list of playable Representation@id values.

[0168] Figure 18 This is a block diagram illustrating a DASH configuration (1800) for indicating two groups that can be switched between each other within an MPD, based on an example. Each group is generated from the same source object, and each entity is associated with a different corresponding functional state. Each functional state video track is presented as a separate adaptive set. A first preselection (1802) groups three adaptive sets (1806, 1808, 1810). A second preselection (1804) groups three adaptive sets (1812, 1814, 1816). The preselection element indicates the grouping of adaptive sets associated with the same object source. Furthermore, the next switchable representation is indicated using @associationType and @associationId.

[0169] When an adaptive set carries multiple bitstreams, it contains multiple content components. Each ContentComponent@Id corresponds to a track ID in the segment. In this case, preselection exists, and the @preselectionComponents of the preselected elements specify the IDs of the ContentComponents belonging to that preselection as a space-separated list. The first ID defines the ContentComponent containing the anchor video represented by that object. In some examples, the ObjectRepresentation descriptor is used at the preselection level to identify the characteristics of each preselection. In some examples, the following constraints may be used: - The `@associationType` and its `@associationId` should not be set to 'swit' in the representation.

[0170] In some examples, the object containing PlaybackInfoStruct() in the initialization fragment represents the existence of an entity group ('oreg').

[0171] The example pseudocode provided below describes an MPD example that uses preselected elements to indicate grouping of multiple components when an adaptive set carries six video components (three of which are associated with object 1 and another three with object 2).

[0172]

[0173] When each video bitstream is carried within a segment, appropriate information needs to be provided so that the player (1406) can prefetch the appropriate segments and synthesize them to obtain the correct output. To achieve this, preselection elements in MPD can be used. The `@preselectionComponents` attribute of the preselection element specifies the IDs of the AdaptationSets associated with the same common underlying video as a space-separated list. The first ID defines the AdaptationSet containing the common underlying video.

[0174] To identify the preselected characteristic, the ObjectRepresentation descriptor described above, which includes the @object_id, @group_id, and @grouping_type attributes, can be used.

[0175] To indicate the association between the masked video and the new texture representation, each new texture representation contains @associationType and @associationId. @associationType is set to "mask", and @associationId contains an identifier for the representation that includes the masked video.

[0176] Figure 19 This is a block diagram illustrating a DASH configuration (1900) for indicating combinations of AdaptationSets according to some examples, where AdaptationSets are configured to provide an experience where a portion of a common base video can be replaced with a new texture. In the examples shown, in the first experience, a portion of the common base video can be replaced with a pink video texture. In the second experience, a portion of the common base video can be replaced with a red video texture. In the third experience, a portion of the common base video can be replaced with a purple video texture.

[0177] The DASH configuration (1900) includes preselections (1902, 1904, 1906) for video compositing with the masked video. When the adaptive set carries these multiple bitstreams, the adaptive set contains multiple content components. Each ContentComponent@Id corresponds to a track ID in the clip. In this case, preselections exist, and the @preselectionComponents of the preselection element specifies the ID of the ContentComponents belonging to that preselection. The first ID (=1) defines the ContentComponent containing the base video. The corresponding ObjectRepresentation descriptor for each of the preselections (1902, 1904, 1906) identifies the characteristics of each preselection. In the example shown, there should be no @associationType set to 'mask' and its @associationId in the representation. To identify the role of each component, a role element can be used, with its @schemeIdUri set to "urn:dolby: 2024:iUGC:role:2024", and the following value can be used: 'main': The content component contains the main video: - 'altv': Content components include videos that provide new textures on specific areas of the main video.

[0178] - 'mask': The content component contains a mask that indicates which areas of the video should be replaced.

[0179] The following pseudocode snippet provides a way to define using preselected elements. Figure 19 The illustration shows three MPD examples of experiences that are reused at the file container level.

[0180]

[0181] The following pseudocode provides an example of a statically rendered MPD with two videos, where one video represents a magnified portion of the other.

[0182]

[0183] In some examples, the object representation (1422) can be deployed at both the ISO BMFF level and the DASH MPD level. Encapsulating video in ISO BMFF can be used to achieve low-latency view switching, but this typically comes at the cost of increased overall bitrate overhead. However, encapsulating video in DASH-MPD can be used to reduce bitrate overhead, but this typically comes at the cost of higher introduced latency.

[0184] In some examples, system designers can measure the scene transition probabilities between various videos and then cluster them into multiple groups based on the measurements. For each group, the intra-group transition probabilities between corresponding videos are relatively high. These groups result in relatively low inter-group transition probabilities. Based on this grouping, a hybrid mode can be implemented, where each group is stored in an ISO BMFF format, and different groups are placed as different pre-selections in their corresponding MPDs. Various embodiments of the hybrid mode are discussed in more detail below.

[0185] Examples of mixed formats Since the object representation (1422) is based on a directed graph, the video a user can switch from from the currently viewed video (node) is limited to the set of videos represented by directed connected nodes. To achieve seamless switching with low latency, one possible solution is to package all videos directly connected to the current video into a multi-track ISO BMFF bitstream. In some examples, for each node, this type of multi-track ISO BMFF bitstream can be constructed to include all directed linked videos, and these linked videos can be downsampled in a lower resolution format and placed on other tracks. In this case, when the user performs a switch at the current playback segment, the system can immediately switch from one track to another at the current segment. At the next playback segment, the system can request the next ISO BMFF bitstream centered on that new master node and with new directly linked videos. In other words, the playback segment moves between nodes linked by directed edges. At the MPD level, each node can be listed as an AdaptationSet.

[0186] Figures 20A to 20B This is a block diagram illustrating a hybrid ISO BMFF and MPEG-DASH format (2000) based on an example. More specifically, Figure 20A The diagram (520) illustrates the basis of the hybrid ISO BMFF and MPEG-DASH formats (2000). References have been made. Figure 5C Figure (520) is described. In Figure (520), nodes (af) are grouped to form a first node group, a second node group, and a third node group, namely, group (ab), group (cd), and group (ef). Figure 20B The left-hand subplot illustrates the mixed format at the MPEG-DASH level (2000). Figure 20BThe right-hand panel illustrates the mixed format (2000) of the ISO BMFF level. The MPEG-DASH level is configured to transmit connections between AdaptationSets (for each node) using signaling. The ISO BMFF level is configured to transmit directed connections to the corresponding master node (to the nearest neighbor node) using signaling.

[0187] In the example shown, the hybrid format (2000) is implemented using DASHMPD (2002), which includes six adaptive sets (2004-2014). The adaptive set (2004) is centered at node (a) of Figure (520) and supports the transitions from the corresponding video (a) to videos (b, c, e) allowed by Figure (520). The bitstream set (2020) corresponding to DASHMPD (2002) includes a multitrack bitstream (2024), where the bitrate of the track corresponding to video (a) is higher than the bitrate of the track corresponding to video (b, c, e). The adaptive set (2006) is centered at node (c) of Figure (520) and supports the transitions from the corresponding video (c) to videos (a, e, d) allowed by Figure (520). Set (2020) includes a multitrack bitstream (2026) where the bitrate of the track corresponding to video (c) is higher than the bitrate of the track corresponding to video (a, e, d). Adaptive set (2008) is centered at node (e) of graph (520) and supports transitions from the corresponding video (e) to video (a, c, f) permitted by graph (520). Set (2020) includes a multitrack bitstream (2028) where the bitrate of the track corresponding to video (e) is higher than the bitrate of the track corresponding to video (a, c, f). Adaptive set (2010) is centered at node (b) of graph (520) and supports transitions from the corresponding video (b) to video (a) permitted by graph (520). Set (2020) includes a multitrack bitstream (2030) where the bitrate of the track corresponding to video (b) is higher than the bitrate of the track corresponding to video (a). The adaptive set (2012) is centered at node (d) of graph (520) and supports the transition from the corresponding video (d) to video (c) allowed by graph (520). The set (2020) includes a multitrack bitstream (2032) in which the bitrate of the track corresponding to video (d) is higher than the bitrate of the track corresponding to video (c). The adaptive set (2014) is centered at node (f) of graph (520) and supports the transition from the corresponding video (f) to video (e) allowed by graph (520). The set (2020) includes a multitrack bitstream (2034) in which the bitrate of the track corresponding to video (f) is higher than the bitrate of the track corresponding to video (e).

[0188] In some examples, a node in a graph can have a relatively large number of neighboring nodes, and the graph allows transformations to these neighboring nodes. In such examples, it might be desirable to impose a bitrate constraint on each of the multitrack ISO BMFF bitstreams to ensure the quality of the received video. This can be achieved, for example, by limiting the total number of nodes encapsulated into the corresponding multitrack ISO BMFF bitstream. In one example, the number of encapsulated nodes is limited to three, thus allowing for one master node and at most two neighboring nodes in the corresponding bitstream.

[0189] Figures 21A to 21B The diagram illustrates a block diagram of the hybrid ISO BMFF and MPEG-DASH format (2100) according to the aforementioned example. The hybrid format (2100) is generally similar to the hybrid format (2000), but is limited by a total number of encapsulation nodes of three. Therefore, in the bitstream set (2120) corresponding to MPD (2002), the three bitstreams (2024, 2026, 2028) each with four corresponding tracks are replaced with bitstreams (2124, 2126, 2128) each with three corresponding tracks. More specifically, for the adaptive set (2004), the ISO BMFF bitstream 2124 has only three tracks carrying video (a, b, c). For the adaptive set (2006), the ISO BMFF bitstream 2126 has only three tracks carrying video (a, c, d). For the adaptive set (2008), the ISO BMFF bitstream 2128 has only three tracks carrying video (c, e, f).

[0190] Using the hybrid format (2100), when a user requests to switch to a node whose video is not carried in the corresponding ISO BMFF bitstream, the player (1406) operates to send a request to obtain another ISO BMFF bitstream that actually contains the video requested. In this case, additional latency is introduced.

[0191] When the number of tracks in each ISO BMFF bitstream is limited to one, the corresponding format becomes a pure MPD-based solution. In other words, each AdaptationSet contains one node. This streaming format has the lowest bit rate but the highest latency.

[0192] When the number of AdaptationSets becomes one, all nodes are stored across all tracks within a single ISO BMFF format. This results in a pure ISO BMFF storage format with the highest bit rate but the lowest latency.

[0193] To minimize view switching latency, node switching probabilities can be collected starting from the content publication date. Once sufficient statistics are collected and the measured node switching probabilities reach a (quasi) quiescent state, the trade-off between latency and bit rate can be adjusted.

[0194] The total number of nodes in the graph is expressed as N It is possible to build a N × N A matrix is ​​used to calculate the node switching probability. For example, for each master node, within the maximum allowed... n In the case of 1 neighboring node, it will have The first n The node videos with varying node switching probabilities are packaged together. This will be done from the node... i To the node j The switching probability is expressed as Then, the matrix of switching probabilities for all nodes (represented as...) This can be expressed as follows: (1) For each master node i It can collect all outward probabilities as a vector. : (2) Through the Sort all elements in the given list to directly determine the element with the highest outward probability. n Each node is then used to implement a hybrid format with a limited number of orbits.

[0195] Time-aware function status In previous chapters, we discussed how functional states can sometimes be arranged into lower-dimensional functional states by flattening other multidimensional variables. For example, a functional state might consider two variables, such as viewing angle and different models. However, in some use cases, it is impossible to flatten functional states without making unacceptable compromises to the content. One such use case involves incorporating a timeline (e.g., limited to MPEG DASH designs). Since both DASH MPD and ISO BMFF can handle the temporal aspect of events, some embodiments described in this chapter specifically address this particular use case.

[0196] Figure 22 This is a block diagram (2200) illustrating a graph (0-3) representing four MPD time periods (0-3) according to an example. Within each time period (0-3), the corresponding subgraph is flattened into a column of nodes (in... Figure 22(Vertical arrangement in the middle), where allowed intra-subgraph transitions and inter-subgraph transitions are indicated by arrows. In order to properly implement the switching corresponding to graph (2200), the dependencies between time periods (0-3) need to be explicitly described in the MPD so that the player (1406) can reconstruct the graph and its subgraphs and enable the user to navigate between time periods (0-3).

[0197] When each bitstream is carried in a specific representation of an adaptation set, it is necessary to indicate the next valid switch across time periods (0-3). To achieve this, a switch descriptor is used at the AdaptationSet level. In some examples, the switch descriptor indicates the next valid switch AdaptationSets in other time periods.

[0198] In one example, the switching descriptor is an EssentialProperty element with its @schemeIdUri set to "urn:dolby: 2024:iUGC:next-swiching". The @values ​​of the switching descriptor exist and are specified in Table 3 provided below. The @values ​​of the EssentialProperty element using the switching descriptor scheme (urn:dolby: 2024:iUGC:next-swiching) are space-separated lists of values, and each value includes the @id "." of the time period and the value of AdaptationSet@id in the indicated time period.

[0199] Table 3: The @value attribute of the toggle descriptor.

[0200]

[0201] The following pseudocode provides an MPD example of three adaptive sets (1, 2, 3) in time period P0 and two adaptive sets (4, 5) in time period P1. The next switchable adaptive set is indicated by the switching descriptor in each adaptive set. This particular example indicates that adaptive set (1, 2) in P0 can switch to adaptive set (4) in time period P1, and adaptive set (3) in time period P0 can switch to adaptive set (5) in P1.

[0202]

[0203] The next pseudocode provides an MPD example of how the adaptive set (1) in time period P0 can be switched to adaptive sets in multiple time periods P1, P2 and P3.

[0204]

[0205] Figures 23 to 28 Three different example use cases are illustrated in the image to further illustrate the above concepts. Figures 23 to 24 The first use case among these use cases is illustrated. Figures 25 to 26 The second use case among these use cases is illustrated. Figures 27 to 28 The diagram illustrates the third use case among these use cases.

[0206] Figure 23 A multi-view video (2300) according to an example is illustrated in image form. The multi-view video (2300) includes a main view (2302) and three detail views (2304, 2306, 2308). The narrative of the main view (2302) is configured to operate on a first timeline (2312). During various parts of the main narrative, certain aspects of the main view (2302) are provided as one or more of the detail views (2304, 2306, 2308) configured to operate on a second timeline (2314), which is an extended (extended) timeline compared to the first timeline (2312). The availability of the extended second timeline (2314) allows one person watching the multi-view video (2300) to gain a deeper understanding of a topic / aspect of interest through selected detailed perspectives in the detailed views (2304, 2306, 2308) operating on the second timeline (2314), while another person may prefer to remain on the shorter, more condensed first timeline (2312). In some examples, each of the detailed views (2304, 2306, 2308) can be configured to operate on a different corresponding extended timeline.

[0207] When a user switches to a view of a longer timeline (2314) and then either reaches the end of that view segment or switches back to the main view (2302) with the associated timeline (2312), playback requires switching back to the main view and the previous playback point on the timeline (2312).

[0208] Figure 24 This is a block diagram (2400) illustrating a representation (based on an example) that can be used for four MPD time periods (0-3) of a multi-view video (2300). The graphical representation (2400) of the multi-view video (2300) has the following nodes: • Node (a n ) corresponds to the main view (2302) with the first (longest) timeline (2312).

[0209] • Node (b n (2304) corresponds to a detailed view with a second timeline (2314).

[0210] • Node (c n (2306) corresponds to a detailed view with a different third timeline.

[0211] • Node (d) n (2308) corresponds to a detailed view with a different fourth timeline. The index is based on the corresponding time period in the time period (0-3). n It can be n =0, 1, 2, 3. The main view (2302) is available in each time period (0-3). The detailed view (2304) is not available during time period (1), but is available during other time periods. The detailed view (2306) is not available during time period (2), but is available during other time periods. The detailed view (2308) is not available during time period (1), but is available during other time periods. The arrows indicate the switching between views (2302-2308) as allowed according to the figure (2400). Because different nodes in the figure (2400) may have different timelines, the first (longest) timeline (2312) is divided into multiple time periods, which allows videos with different timelines to still be time-aligned with each of these time periods.

[0212] Note that the playback delay for each track in the ISO BMFF bitstream can be set via a corresponding box. Therefore, the start time of each segment of each node can have its own time offset via the corresponding setting in the box. The following provides a box definition based on an example: The “elst” box contains an explicit timeline mapping. Each entry defines a portion of the track timeline: either by mapping a portion of the composite timeline, or by indicating 'empty' time (presenting a portion of the timeline that is not mapped to the media, i.e., an 'empty' edit), or by defining a 'stay', i.e., a single point in time in the media that remains for a certain period of time.

[0213] The next pseudocode provides an example of the "elst" box:

[0214] Figure 25A multi-view video (2500) based on another example is illustrated in image form. In the example shown, the multi-view video (2500) presents a comparison of two models of a product (guitar). The content can be viewed linearly (2510), i.e., the first model is demonstrated first, followed by the second model. However, to better compare the first and second models, the content creator might want to give the user the ability to switch between the two products, for example, by specifically implementing switching between the same points in the timelines of the first and second demonstrations. As a non-limiting example, the two instrument models could be used to play the same piece of music twice. The timelines of the performances could be aligned so that the continuity of the music playback is not interrupted even if the user switches between the two instruments.

[0215] It is important to note that, in the scenarios outlined above, the content is, in a sense, a single perspective, and the multi-node feature is achieved by implementing instantaneous switching between two corresponding points in time. For this specific use case, it is also important to note that these two points in time are relatively far apart; that is, when the replay of the first perspective (the first point in time on the linear narrative timeline) occurs, the replay point of the second perspective (the second point in time on the linear narrative timeline) will change accordingly.

[0216] Figure 26 This is a block diagram (2600) illustrating a representation (based on an example) that can be used for four MPD time periods (0-3) of a multi-view video (2500). The graphical representation (2600) of the multi-view video (2500) has the following nodes: • Node (a n This corresponds to the first model (OOO-28).

[0217] • Node (b n This corresponds to the second model (OOO-42). The index is based on the corresponding time period in the time period (0-3). n It can be n = 0, 1, 2, 3. Since there are multiple shared (or common) scenarios between the two models within some time periods, two separate bitstreams can be used, but are not required. In at least some examples, it is sufficient to have a single bitstream representing both the first and second models. In the latter case, whenever it is necessary to show the different characteristics between the first and second models within some selected time periods, graph (2600) branches out into multiple nodes. For example, in time period (1), graph (2600) branches out into nodes (a1, b1). Similarly, in time period (3), graph (2600) branches out into nodes (a3, b3).

[0218] Figure 27 A multi-view video (2700) according to another example is illustrated in image form. In the example shown, an application is used to create a navigation interface with multiple (e.g., two) video tiles that enable previewing segments of the long form (2710) of the video (2700), for example, where multiple products are shown in time series. In this use case, a viewer can use the navigation interface (2720) to navigate through various selected portions of the long form (2710) of the video (2700) while also continuously watching the linear replay of the long form narrative in a dedicated master tile (2712). The viewer can jump to selected segments of the long form (2710) of the video (2700) by interacting (2730) with multiple tiles (2722, 2724) in the navigation interface (2720).

[0219] Figure 28 This is a block diagram (2800) illustrating a representation (based on an example) of five MPD time periods (0-4) that can be used for a multi-view video (2700). The graphical representation (2800) of the multi-view video (2700) has the following nodes: • Node (a n ) corresponds to the navigation interface (2720) video.

[0220] • Node (b n ) corresponds to long format video (2710). The index is based on the corresponding time period in the time period (0-4). n It can be n = 0, 1, 2, 3, 4.

[0221] At time segment (0), the user will only see the navigation interface video and can choose which scene to jump to. Note that in some examples, time segment (0) can be skipped and the long-form video (2710) can be shown directly. Using the interaction (2730), the user can jump to different times along the overall timeline or retrieve videos from different timelines.

[0222] Example hardware Figure 29 This is a block diagram of an example computing device (2900) based on various examples. In some examples, the computing device (2900) is configured to perform at least some of the methods and procedures described above. In some examples, two or more instances of the computing device (2900) are used to perform various methods and procedures.

[0223] Figure 29The computing device (2900) is illustrated as having multiple components, but any one or more of these components may be omitted or repeated as appropriate for application and setup. In some embodiments, some or all of the components included in the computing device (2900) may be attached to one or more motherboards and encapsulated in a housing. In some embodiments, some of these components may be manufactured onto a single system-on-a-chip (SoC) (e.g., the SoC may include one or more electronic processing devices (2902) and one or more storage devices (2904)). Additionally, in various embodiments, the computing device (2900) may not include... Figure 29 One or more of the components shown may be included, but may include interface circuitry for coupling to the one or more components using any suitable interface, such as a Universal Serial Bus (USB) interface, a High Definition Multimedia Interface (HDMI) interface, a Controller Area Network (CAN) interface, a Serial Peripheral Interface (SPI) interface, an Ethernet interface, a wireless interface, or any other suitable interface. For example, the computing device (2900) may not include a display device (2910), but may include display device interface circuitry (e.g., connector and driver circuitry) to which an external display device (2910) may be coupled.

[0224] The computing device (2900) includes a processing device (2902) (e.g., one or more processing devices). The terms "electronic processor device" and "processing device" are interchangeable to refer to any device or part of a device that processes electronic data from registers and / or memory to transform that electronic data into other electronic data that can be stored in registers and / or memory. In various embodiments, the processing device (2902) may include one or more digital signal processors (DSPs), application-specific integrated circuits (ASICs), central processing units (CPUs), graphics processing units (GPUs), server processors, or any other suitable processing device.

[0225] The computing device (2900) also includes a storage device (2904) (e.g., one or more storage devices). In various embodiments, the storage device (2904) may include one or more memory devices, such as random access memory (RAM) devices (e.g., static RAM (SRAM) devices, magnetic RAM (MRAM) devices, dynamic RAM (DRAM) devices, resistive RAM (RRAM) devices, or conductive bridged RAM (CBRAM) devices), hard disk drive-based memory devices, solid-state memory devices, network drives, cloud drives, or any combination of memory devices, etc. In some embodiments, the storage device (2904) may include memory that shares a die with the processing device (2902). In such embodiments, the memory may be used as cache memory and may include, for example, embedded dynamic random access memory (eDRAM) or spin-transfer torque magnetic random access memory (STT-MRAM). In some embodiments, the storage device (2904) may include a non-transitory computer-readable medium having instructions thereon that, when executed by one or more processing devices (e.g., processing device (2902)), cause the computing device (2900) to perform any suitable method or portion thereof of the disclosed methods.

[0226] The computing device (2900) further includes an interface device (2906) (e.g., one or more interface devices (2906)). In various embodiments, the interface device (2906) may include one or more communication chips, connectors, and / or other hardware and software to manage communication between the computing device (2900) and other computing devices. For example, the interface device (2906) may include circuitry for managing wireless communication to enable data transmission to and from the computing device (2900). The term "wireless" and its derivatives can be used to describe circuits, devices, systems, methods, techniques, communication channels, etc., that can transmit data via modulated electromagnetic radiation through a non-solid medium. This term does not imply that the associated devices do not contain any wires, although in some embodiments they may not contain wires. The circuitry included in the interface device (2906) for managing wireless communications can implement any of a variety of wireless standards or protocols, including but not limited to IEEE standards (including Wi-Fi (IEEE 802.11 series, IEEE 802.16 standards), Long Term Evolution (LTE) projects, and any amendments, updates, and / or revisions (e.g., Advanced LTE project, Ultra Mobile Broadband (UMB) project (also known as “3GPP2”), etc.). In some embodiments, the circuitry included in the interface device (2906) for managing wireless communications can operate according to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed ​​Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE networks. In some embodiments, the circuitry included in the interface device (2906) for managing wireless communications can operate according to GSM Evolved Enhanced Data (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). In some embodiments, the circuitry included in the interface device (2906) for managing wireless communications may be based on Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications System (DECT), Evolved Data Optimization (EV-DO) and its derivative protocols, as well as any other wireless protocol designated as 3G, 4G, 5G, etc. In some embodiments, the interface device (2906) may include one or more antennas (e.g., one or more antenna arrays) configured to receive and / or transmit wireless signals.

[0227] In some embodiments, the interface device (2906) may include circuitry for managing wired communications, such as electrical communication protocols, optical communication protocols, or any other suitable communication protocols. For example, the interface device (2906) may include circuitry supporting communications based on Ethernet technology. In some embodiments, the interface device (2906) may support both wireless and wired communications, and / or may support multiple wired communication protocols and / or multiple wireless communication protocols. For example, a first set of circuitry for the interface device (2906) may be dedicated to shorter-range wireless communications such as Wi-Fi or Bluetooth, and a second set of circuitry for the interface device (2906) may be dedicated to longer-range wireless communications such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some other embodiments, a first set of circuitry for the interface device (2906) may be dedicated to wireless communications, and a second set of circuitry for the interface device (2906) may be dedicated to wired communications.

[0228] The computing device (2900) also includes a battery / power circuit (2908). In various embodiments, the battery / power circuit (2908) may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device (2900) to an energy source (e.g., AC line power) separate from the computing device (2900).

[0229] The computing device (2900) also includes a display device (2910) (e.g., one or more separate display devices). In various embodiments, the display device (2910) may include any visual indicator, such as a head-up display, computer monitor, projector, touch screen display, liquid crystal display (LCD), light-emitting diode display, or flat panel display, etc.

[0230] The computing device (2900) also includes additional input / output (I / O) devices (2912). In various embodiments, the I / O devices (2912) may include one or more data / signal transmission interfaces, audio I / O devices (e.g., microphones or microphone arrays, speakers, headsets, earpieces, sirens, etc.), audio codecs, video codecs, printers, sensors (e.g., thermocouples or other temperature sensors, humidity sensors, pressure sensors, vibration sensors, etc.), image capture devices (e.g., one or more cameras), human-machine interface devices (e.g., keyboards, cursor control devices such as mice, styluses, trackballs, or touchpads), etc.

[0231] According to specific embodiments, various components of the interface device (2906) and / or I / O device (2912) can be configured to output suitable control signals, receive suitable control / telemetry signals, and receive and transmit data streams. In some examples, the interface device (2906) and / or I / O device (2912) includes one or more analog-to-digital converters (ADCs) for converting received analog signals into a form suitable for operation performed by the processing device (2902) and / or storage device (2904). In some additional examples, the interface device (2906) and / or I / O device (2912) includes one or more digital-to-analog converters (DACs) for converting digital signals provided by the processing device (2902) and / or storage device (2904) into an analog form suitable for transmission over a communication channel.

[0232] Based on the exemplary embodiments disclosed above, for example, in the summary section and / or references Figures 1 to 29 Any one or any combination of some or all of the above provides an apparatus for streaming multi-view audio and video (MPAV), the apparatus comprising: at least one processor; and at least one memory including program code, wherein the at least one memory and the program code are configured, together with the at least one processor, to cause the apparatus to perform at least the following operations: generating a graph representation of a plurality of videos corresponding to the MPAV, the graph representation including a plurality of nodes interconnected via a plurality of edges, each of the nodes representing a corresponding video in the video, each of the edges representing a permissible switch between a corresponding pair of the videos during playback; encoding the plurality of videos into a plurality of corresponding bitstreams; packing the plurality of bitstreams into a storage container based on the graph representation; providing metadata configured to enable a player device to reconstruct the graph representation; and streaming a selected subset of the plurality of bitstreams in response to a request at the player device based on the reconstructed graph representation.

[0233] According to another example embodiment disclosed above, for example, in the summary section and / or references Figures 1 to 29Any one or any combination of some or all of the above provides a method for streaming multi-view audio and video (MPAV) from a server device, the method comprising: generating a graph representation of a plurality of videos corresponding to the MPAV, the graph representation including a plurality of nodes interconnected via a plurality of edges, each of the nodes representing a corresponding video in the video, each of the edges representing a permissible switch between a corresponding pair of the videos during playback; encoding the plurality of videos into a plurality of corresponding bitstreams; packaging the plurality of bitstreams into a storage container based on the graph representation, the storage container being accessible via the server device; providing metadata configured to enable a player device to reconstruct the graph representation; and streaming a selected subset of the plurality of bitstreams in response to a request generated at the player device based on the reconstructed graph representation.

[0234] In some embodiments of the above method, the graph representation includes a directed graph.

[0235] In some embodiments of any of the methods described above, the graph representation includes multiple node groups, each of which represents a different corresponding functional state of the object represented by the MPAV.

[0236] In some embodiments of any of the methods described above, the video group corresponding to the node group includes: a base video showing common shared pixels among all videos in the video group; a mask video indicating pixel locations where different videos in the video group differ from each other; and one or more region videos, each of which shows a different corresponding video for the pixel location indicated by the mask video.

[0237] In some embodiments of any of the methods described above, the metadata provides a description of the corresponding transformation effect for each edge of the graph representation.

[0238] In some embodiments of any of the methods described above, the metadata provides a description of a default navigation path represented by the graph, which is to be used when the end user of the player device does not actively navigate through the graph representation.

[0239] In some embodiments of any of the methods described above, the streaming includes: providing the player device with a media presentation description (MPD) of the plurality of bitstreams; providing the player device with a corresponding initialization segment from the storage container for a given time period, the corresponding initialization segment being configured to notify one or more videos at the player device to select a requested media segment from the video for rendering; receiving from the player device a request to identify the selection; and transmitting one or more bitstreams to the player device, the one or more bitstreams carrying the media segment selected in the storage container based on the identified selection.

[0240] In some embodiments of any of the methods described above, for the time period, the storage container has multiple media segments logically organized according to the graph representation, and the multiple media segments are further logically organized according to one or more of different bit rates, different resolutions, different codec types, and different frame rates.

[0241] In some embodiments of any of the methods described above, the streaming further includes switching from a first corresponding media segment sequence to a different second corresponding media segment sequence when the selection identified by the request indication changes.

[0242] In some embodiments of any of the methods described above, the MPD includes information corresponding to two or more streaming content periods.

[0243] In some embodiments of any of the methods described above, the corresponding initialization fragment has a box format compatible with at least one of the MPEGDASH and ISO BMFF specifications.

[0244] In some embodiments of any of the methods described above, the MPD includes information about two groups of nodes that can switch with each other according to the graph representation.

[0245] In some embodiments of any of the methods described above, the method further includes encapsulating the corresponding plurality of bitstreams using a file format, wherein different bitstreams in the bitstreams are stored in different corresponding tracks, and each image bitstream is stored in a different corresponding project.

[0246] In some embodiments of any of the methods described above, the method further includes encapsulating the corresponding plurality of bitstreams using fragment encapsulation, wherein each fragment is carried in a corresponding representation of an adaptive set.

[0247] In some embodiments of any of the methods described above, the graph representation includes multiple groups of nodes formed by measuring the transition probabilities between different nodes of the graph representation during playback by multiple different users.

[0248] In some embodiments of any of the above methods, for each of the node groups, the intra-group transition probability is greater than a threshold; and the inter-group transition probability is less than the threshold.

[0249] In some embodiments of any of the methods described above, a set of bitstreams corresponding to the node groups represented by the graph are stored in the storage container using ISO BMFF format; and wherein different groups are indicated as different preselections in the corresponding MPD.

[0250] In some embodiments of any of the above methods, the preselection in the corresponding MPD includes multiple adaptive sets; wherein different adaptive sets in the adaptive sets are centered on different corresponding nodes in the graph representation; and wherein the multi-track bitstream corresponding to the adaptive set includes a first track and one or more second tracks, the first track carrying the bitstream of the corresponding center node at a first bit rate, each of the one or more second tracks carrying the corresponding bitstream of a node in the graph representation that is allowed to transition from the center node at a second bit rate, the first bit rate being greater than the second bit rate.

[0251] In some embodiments of any of the methods described above, the graph representation includes multiple subgraphs, each of which corresponds to a different time period of the corresponding DASH MPD.

[0252] In some embodiments of any of the methods described above, a switching descriptor of the corresponding adaptive set of the corresponding MPD is used to indicate a valid switch between the subgraphs.

[0253] In some embodiments of any of the methods described above, the subgraphs in the plurality of subgraphs include a first node characterized by a first viewing timeline and a second node characterized by a second viewing timeline, the first timeline and the second timeline having different corresponding time scales.

[0254] In some embodiments of any of the above methods, the subgraphs in the plurality of subgraphs include a first node corresponding to a first object presented in the MPAV and a second node corresponding to a different second object presented in the MPAV.

[0255] In some embodiments of any of the methods described above, a subgraph in the plurality of subgraphs includes a first node corresponding to the long-form video of the MPAV and a second node corresponding to a navigation interface for navigating along the long-form video.

[0256] A non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations including any of the methods described above.

[0257] According to yet another example embodiment disclosed above, for example, in the summary section and / or references Figures 1 to 29 A device is provided for streaming multi-view audio and video (MPAV), comprising: at least one processor; and at least one memory including program code, wherein the at least one memory and the program code are configured, together with the at least one processor, to cause the device to at least: receive metadata configured to enable a player device to reconstruct a graph representation of a plurality of videos corresponding to the MPAV, the graph representation including a plurality of nodes interconnected via a plurality of edges, each of the nodes representing a corresponding video in the video, each of the edges representing a permissible switch between a corresponding pair of the videos during playback; generate a streaming request for a server device based on the reconstructed graph representation and further based on interactive user input, the streaming request identifying a selected subset of a plurality of bitstreams packaged into a storage container accessible via the server device, the plurality of bitstreams encoding the plurality of videos corresponding to the MPAV; and receive the selected subset of the plurality of bitstreams transmitted by the server device in response to the streaming request.

[0258] According to yet another example embodiment disclosed above, for example, in the summary section and / or references Figures 1 to 29A method for streaming multi-view audio and video (MPAV) using a player device is provided, comprising: receiving metadata configured to enable the player device to reconstruct a graph representation of a plurality of videos corresponding to the MPAV, the graph representation including a plurality of nodes interconnected via a plurality of edges, each of the nodes representing a corresponding video in the video, each of the edges representing a permissible switch between a corresponding pair of the videos during playback; generating a streaming request for a server device based on the reconstructed graph representation and further based on interactive user input, the streaming request identifying a selected subset of a plurality of bitstreams packaged into a storage container accessible via the server device, the plurality of bitstreams encoding the plurality of videos corresponding to the MPAV; and receiving the selected subset of the plurality of bitstreams transmitted by the server device in response to the streaming request.

[0259] In some embodiments of the above method, the method further includes: decoding the selected subset of the plurality of bitstreams to generate a decoded video signal; and rendering the decoded video signal on a display device.

[0260] In some embodiments of any of the methods described above, the graph representation includes a directed graph.

[0261] In some embodiments of any of the methods described above, the graph representation includes multiple node groups, each of which represents a different corresponding functional state of the object represented by the MPAV.

[0262] In some embodiments of any of the methods described above, the video group corresponding to the node group includes: a base video showing common shared pixels among all videos in the video group; a mask video indicating pixel locations where different videos in the video group differ from each other; and one or more region videos, each of which shows a different corresponding video for the pixel location indicated by the mask video.

[0263] In some embodiments of any of the methods described above, the metadata provides a description of the corresponding transformation effect for each edge of the graph representation.

[0264] In some embodiments of any of the methods described above, the metadata provides a description of a default navigation path represented by the graph, which is to be used when the end user of the player device does not actively navigate through the graph representation.

[0265] In some embodiments of any of the methods described above, the method further includes: receiving a media presentation description (MPD) of the plurality of bitstreams from the server device; receiving, for a period of time, a corresponding initialization fragment from the storage container, the corresponding initialization fragment being configured to notify one or more videos at the player device to select a requested media segment from the video for rendering; sending a request to the server device to identify the selection; and receiving one or more bitstreams from the server device, the one or more bitstreams carrying a media segment selected in the storage container based on the identified selection.

[0266] In some embodiments of any of the methods described above, the MPD includes information corresponding to two or more streaming content periods.

[0267] In some embodiments of any of the methods described above, the corresponding initialization fragment has a box format compatible with at least one of the MPEGDASH and ISO BMFF specifications.

[0268] In some embodiments of any of the methods described above, the MPD includes information about two groups of nodes that can switch with each other according to the graph representation.

[0269] In some embodiments of any of the methods described above, the plurality of bitstreams are encapsulated using a file format, wherein different bitstreams in the bitstreams are stored in different corresponding tracks, and each image bitstream is stored in a different corresponding project.

[0270] In some embodiments of any of the methods described above, the plurality of bitstreams are encapsulated using fragment encapsulation, wherein each fragment is carried in a corresponding representation of an adaptive set.

[0271] In some embodiments of any of the methods described above, a set of bitstreams corresponding to the node groups represented by the graph are stored in the storage container using ISO BMFF format; and wherein different groups are indicated as different preselections in the corresponding MPD.

[0272] In some embodiments of any of the above methods, the preselection in the corresponding MPD includes multiple adaptive sets; wherein different adaptive sets in the adaptive sets are centered on different corresponding nodes in the graph representation; and wherein the multi-track bitstream corresponding to the adaptive set includes a first track and one or more second tracks, the first track carrying the bitstream of the corresponding center node at a first bit rate, each of the one or more second tracks carrying the corresponding bitstream of a node in the graph representation that is allowed to transition from the center node at a second bit rate, the first bit rate being greater than the second bit rate.

[0273] In some embodiments of any of the methods described above, the graph representation includes multiple subgraphs, each of which corresponds to a different time period of the corresponding MPD.

[0274] In some embodiments of any of the methods described above, a switching descriptor of the corresponding adaptive set of the corresponding MPD is used to indicate a valid switch between the subgraphs.

[0275] In some embodiments of any of the methods described above, the subgraphs in the plurality of subgraphs include a first node characterized by a first viewing timeline and a second node characterized by a second viewing timeline, the first timeline and the second timeline having different corresponding time scales.

[0276] In some embodiments of any of the above methods, the subgraphs in the plurality of subgraphs include a first node corresponding to a first object presented in the MPAV and a second node corresponding to a different second object presented in the MPAV.

[0277] In some embodiments of any of the methods described above, a subgraph in the plurality of subgraphs includes a first node corresponding to the long-form video of the MPAV and a second node corresponding to a navigation interface for navigating along the long-form video.

[0278] A non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations including any of the methods described above.

[0279] Regarding the processes, systems, methods, heuristics, etc., described herein, it should be understood that although the steps of such processes, etc., have been described as being performed in a specific ordered sequence, such processes can be practiced using the described steps performed in a different order than that described herein. Furthermore, it should be understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. In other words, the process descriptions herein are provided for the purpose of illustrating certain embodiments and should in no way be construed as limiting the claims.

[0280] Accordingly, it should be understood that the above description is intended to be illustrative rather than restrictive. Many embodiments and applications beyond the examples provided will become apparent upon reading the above description. The scope should not be determined by reference to the above description, but rather by reference to the appended claims and the full scope of their equivalents. It is anticipated and desired that the techniques discussed herein will evolve in the future, and the disclosed systems and methods will be incorporated into such future embodiments. In conclusion, it should be understood that modifications and changes are possible with this application.

[0281] All terms used in the claims are intended to be given the broadest reasonable interpretation and common meaning as understood by one skilled in the art described herein, unless expressly indicated otherwise herein. In particular, the use of singular articles such as “a,” “the,” and “the” should be understood to refer to one or more of the indicated elements, unless the claims expressly limit this to the contrary.

[0282] An abstract of this disclosure is provided to allow the reader to quickly determine the nature of this technical disclosure. This abstract is submitted on the premise that it will not be used to interpret or limit the scope or meaning of the claims. Furthermore, as can be seen in the foregoing detailed description, various features are combined together in various embodiments for the purpose of making this disclosure a coherent whole. The method of this disclosure should not be construed as reflecting an intention to incorporate more features than expressly recited in each claim in the claimed embodiments. Rather, as reflected in the appended claims, the inventive subject matter lies in fewer than all the features of a single disclosed embodiment. Therefore, the appended claims are thus incorporated into the detailed description, each claim being an independent subject matter claimed.

[0283] While this disclosure includes references to illustrative embodiments, this specification is not intended to be limiting. Various modifications to the described embodiments, as well as other embodiments within the scope of this disclosure, will be apparent to those skilled in the art to which this disclosure pertains, and are considered to be within the principles and scope of this disclosure, such as those expressed in the claims.

[0284] Some embodiments can be implemented as circuit-based processes, including possible implementations on a single integrated circuit.

[0285] Some embodiments may be embodied in the form of methods and apparatus for practicing these methods. Some embodiments may also be embodied in the form of program code recorded in a tangible medium, such as a magnetic recording medium, optical recording medium, solid-state storage, floppy disk, CD-ROM, hard disk drive, or any other non-transitory machine-readable storage medium, wherein when the program code is loaded into and executed by a machine (such as a computer), the machine becomes an apparatus for practicing the patented invention(s). Some embodiments may also be embodied in the form of program code, for example, stored in a non-transitory machine-readable storage medium (including being loaded into and / or executed by the machine), wherein when the program code is loaded into and executed by a machine (such as a computer or processor), the machine becomes an apparatus for practicing the patented invention(s). When implemented on a general-purpose processor, the program code segments are combined with the processor to provide a unique device that operates similarly to a particular logic circuit.

[0286] Unless otherwise expressly stated, each value and range should be interpreted as approximate as if preceded by the words “about” or “approximately”.

[0287] The use of figure numbers and / or reference numerals in the claims is intended to identify one or more possible embodiments of the claimed subject matter to facilitate the interpretation of the claims. Such use should not be construed as necessarily limiting the scope of these claims to the embodiments shown in the corresponding figures.

[0288] Although the elements in the method claims (if any) are recited in a particular order with corresponding markings, these elements are not necessarily intended to be limited to being recited in a particular order unless the recitation of the claims otherwise implies a particular order for implementing some or all of these elements.

[0289] References to “one embodiment” or “an embodiment” herein mean that a particular feature, structure, or characteristic described in connection with an embodiment may include at least one embodiment of this disclosure. In this specification, the phrase “in one embodiment” appearing in various places does not necessarily refer to the same embodiment, nor is it necessarily a separate or alternative embodiment that is mutually exclusive with other embodiments. The same applies to the term “implementation.”

[0290] Unless otherwise specified herein, the use of ordinal adjectives such as “first,” “second,” “third,” etc., to refer to objects among multiple similar objects merely indicates different instances of such similar objects being mentioned, and is not intended to imply that such similar objects must be in a corresponding order or sequence in time, space, ranking, or any other way.

[0291] Unless otherwise specified herein, the conjunction “if” may also be interpreted, in addition to its simple meaning, as meaning “when…”, “at…”, “in response to determination”, or “in response to detection,” depending on the specific context. For example, the phrases “if it is determined…” or “if [the stated condition] is detected” may be interpreted as meaning “after it is determined…”, “in response to determining…”, “after [the stated condition or event] is detected”, or “in response to [the stated condition or event] being detected.”

[0292] For the purposes of this description, the terms “couple,” “coupling,” “coupled,” “connect,” “connecting,” or “connected” refer to any manner known in the art or subsequently developed that allows energy to be transferred between two or more elements, and envision the insertion of one or more additional elements, although this is not required. Conversely, the terms “direct coupling,” “direct connection,” etc., imply the absence of such additional elements.

[0293] As used in this document with respect to components and standards, the term "compatible" means that the component communicates with other components in a manner wholly or partially specified in the standard, and is recognized by other components as capable of communicating with other components in a manner specified in the standard. Compatible components do not need to operate internally in a manner specified in the standard.

[0294] The functions of the various elements shown in the figure, including any functional blocks labeled “processor” and / or “controller”, can be provided by using dedicated hardware and hardware capable of executing the software, associated with appropriate software. When provided by a processor, the functionality can be provided by a single dedicated processor, a single shared processor, or multiple separate processors (some of which may be shared). Furthermore, the explicit use of the terms “processor” or “controller” should not be construed as exclusively referring to hardware capable of executing software, and may implicitly include, but is not limited to, digital signal processor (DSP) hardware, network processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), read-only memory (ROM) for storing software, random access memory (RAM), and non-volatile storage devices. Other conventional and / or custom hardware may also be included. Similarly, any switches shown in the figure are conceptual only. Their functionality can be performed by the operation of program logic, by dedicated logic, by the interaction of program control and dedicated logic, or even manually, with the specific technology chosen by the implementer, as understood more specifically from the context.

[0295] As used herein, the terms “circuit” and “circuitry” may refer to one or more of the following: (a) a purely hardware circuit implementation (such as implementations in purely analog and / or digital circuits); (b) a combination of hardware circuitry and software, such as (if applicable): (i) a combination of analog and / or digital hardware circuitry with software / firmware, and (ii) any portion of a hardware processor with software (including digital signal processors), software, and memory, which work together to enable a device such as a mobile phone or server to perform various functions; and (c) a hardware circuitry and / or processor, such as a microprocessor or a portion thereof, which requires software (e.g., firmware) to operate, but may be absent when operation is not required. This definition of circuitry applies to all uses of the term in this application (including all claims). As a further example, as used herein, the term “circuit” also covers implementations of only one hardware circuit or one or more processors, or a portion thereof, and its accompanying software and / or firmware. The term "circuit" may also cover, for example, baseband integrated circuits or processor integrated circuits for mobile devices, or similar integrated circuits in servers, cellular network devices or other computing or networking devices, where applicable to certain elements of the claims.

[0296] Those skilled in the art will recognize that any block diagram herein represents a conceptual view of an illustrative circuit embodying the principles of this disclosure. Similarly, it will be appreciated that any flow chart, flow diagram, state transition diagram, pseudocode, etc., represents various processes that can be substantially represented in a computer-readable medium and thus executed by a computer or processor, whether or not such computer or processor is explicitly shown.

[0297] The "Summary" in this specification is intended to introduce some exemplary embodiments, and additional embodiments are described in the "Detailed Description" and / or with reference to one or more accompanying drawings. The "Summary" is not intended to identify essential elements or features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

Claims

1. A method for streaming multi-view audio and video (MPAV) from a server device, the method comprising: Generate a graph representation of multiple videos corresponding to the MPAV, the graph representation including multiple nodes interconnected via multiple edges, each of the nodes representing a corresponding video in the video, and each of the edges representing a allowed switch between a corresponding pair of videos during playback; The multiple videos are encoded into multiple corresponding bitstreams; Based on the graph representation, the multiple bit streams are packaged into a storage container, which can be accessed via the server device; Provide metadata configured to enable the player device to reconstruct the graph representation; as well as In response to a request generated at the player device based on the reconstructed graph representation, a selected subset of the plurality of bit streams is streamed.

2. The method as described in claim 1, wherein, The graph representation includes directed graphs.

3. The method as described in claim 1, wherein, The diagram represents multiple node groups, each of which represents a different corresponding functional state of the object represented by the MPAV.

4. The method of claim 3, wherein, The video groups corresponding to the node groups include: A base video, which shows the common shared pixels among all videos in the video group; Masked video, the masked video indicating the pixel locations where different videos in the video group differ from each other; and One or more regional videos, each of which shows a different corresponding video for the pixel location indicated by the masked video.

5. The method of claim 1, wherein, For each edge in the graph representation, the metadata provides a description of the corresponding transformation effect.

6. The method of claim 1, wherein, The metadata provides a description of the default navigation path represented by the graph, which is to be used when the end user of the player device does not actively navigate through the graph representation.

7. The method of claim 1, wherein, The streaming includes: Provide the media presentation description (MPD) of the plurality of bitstreams to the player device; For a given time period, a corresponding initialization segment from the storage container is provided to the player device, the corresponding initialization segment being configured to notify one or more videos at the player device to select a requested media segment from the video for rendering; Receive a request from the player device to identify the selection; and One or more bitstreams are transmitted to the player device, the one or more bitstreams carrying the media segment selected in the storage container based on the identified selection.

8. The method of claim 7, wherein, For the specified time period, the storage container has multiple media segments, which are logically organized according to the diagram representation, and are further logically organized according to one or more of different bit rates, different resolutions, different codec types, and different frame rates.

9. The method of claim 8, wherein, The streaming further includes switching from a first corresponding media segment sequence to a different second corresponding media segment sequence when the selection identified by the request indication changes.

10. The method of claim 7, wherein, The MPD includes information corresponding to two or more streaming content time periods.

11. The method of claim 7, wherein, The corresponding initialization fragment has a box format compatible with at least one of the MPEG DASH and ISOBMFF specifications.

12. The method of claim 7, wherein, The MPD includes information about two groups of nodes that can switch between each other according to the graph representation.

13. The method of claim 1, further comprising encapsulating the corresponding plurality of bitstreams using a file format, wherein, The different bitstreams in the bitstream are stored in different corresponding tracks, and each image bitstream is stored in a different corresponding project.

14. The method of claim 1, further comprising encapsulating the corresponding plurality of bitstreams using fragment encapsulation, wherein, Each fragment is carried in the corresponding representation of the adaptive set.

15. The method of claim 1, wherein, The graph representation includes multiple groups of nodes formed by measuring the transition probabilities between different nodes of the graph representation during playback by multiple different users.

16. The method of claim 15, in, For each of the nodes in the group, the intra-group transition probability is greater than a threshold; and Wherein, the inter-group transition probability is less than the threshold.

17. The method as described in claim 1, in, A set of bitstreams corresponding to the node group represented by the graph is stored in the storage container using ISO BMFF format; and Different groups are indicated as different pre-selections in the corresponding MPD.

18. The method of claim 17, in, The pre-selection in the corresponding MPD includes multiple adaptive sets; Wherein, the different adaptive sets in the adaptive set are centered on the different corresponding nodes represented by the graph; and The multi-track bitstream corresponding to the adaptive set includes a first track and one or more second tracks. The first track carries the bitstream of the corresponding central node at a first bit rate, and each of the one or more second tracks carries the corresponding bitstream of a node in the graph representation that is allowed to transition from the central node at a second bit rate. The first bit rate is greater than the second bit rate.

19. The method of claim 1, wherein, The diagram represents multiple sub-graphs, each of which corresponds to a different time period of the corresponding MPD.

20. The method of claim 19, wherein, The switching descriptor of the corresponding adaptive set of the corresponding MPD is used to indicate the effective switching between the subgraphs.

21. The method of claim 19, wherein, The subgraphs in the plurality of subgraphs include a first node represented by a first viewing timeline and a second node represented by a second viewing timeline, the first timeline and the second timeline having different corresponding time scales.

22. The method of claim 19, wherein, The subgraphs in the plurality of subgraphs include a first node corresponding to a first object presented in the MPAV and a second node corresponding to a different second object presented in the MPAV.

23. The method of claim 19, wherein, The subgraphs in the plurality of subgraphs include a first node corresponding to the long form video of the MPAV and a second node corresponding to a navigation interface for navigating along the long form video.

24. A non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations including the method of claim 1.

25. An apparatus for streaming multi-view audio and video (MPAV), the apparatus comprising: At least one processor; as well as At least one memory containing program code, The at least one memory and the program code are configured, together with the at least one processor, to cause the device to perform at least the following operations: Generate a graph representation of multiple videos corresponding to the MPAV, the graph representation including multiple nodes interconnected via multiple edges, each of the nodes representing a corresponding video in the video, and each of the edges representing a allowed switch between a corresponding pair of videos during playback; The multiple videos are encoded into multiple corresponding bitstreams; Based on the graph representation, the multiple bit streams are packaged into a storage container; Provides metadata configured to enable the player device to reconstruct the graph representation; and In response to a request generated at the player device based on the reconstructed graph representation, a selected subset of the plurality of bit streams is streamed.

26. A method for streaming multi-view audio and video (MPAV) using a playback device, the method comprising: Receive metadata, which is configured to enable the player device to reconstruct a graph representation of multiple videos corresponding to the MPAV, the graph representation including multiple nodes interconnected via multiple edges, each of the nodes representing a corresponding video in the video, and each of the edges representing a allowed switch between a corresponding pair of videos during playback; Based on the reconstructed graph representation and further based on interactive user input, a streaming request for a server device is generated, the streaming request identifying a selected subset of multiple bitstreams packaged into a storage container accessible via the server device, the multiple bitstreams encoding the multiple videos corresponding to the MPAV; as well as Receive the selected subset of the plurality of bit streams transmitted by the server device in response to the streaming request.

27. The method of claim 26, further comprising: The selected subset of the plurality of bitstreams is decoded to generate a decoded video signal; as well as The decoded video signal is rendered on a display device.

28. The method of claim 26, wherein, The graph representation includes directed graphs.

29. The method of claim 26, wherein, The diagram represents multiple node groups, each of which represents a different corresponding functional state of the object represented by the MPAV.

30. The method of claim 29, wherein, The video groups corresponding to the node groups include: A base video, which shows the common shared pixels among all videos in the video group; Masked video, the masked video indicating the pixel locations where different videos in the video group differ from each other; and One or more regional videos, each of which shows a different corresponding video for the pixel location indicated by the masked video.

31. The method of claim 26, wherein, For each edge in the graph representation, the metadata provides a description of the corresponding transformation effect.

32. The method of claim 26, wherein, The metadata provides a description of the default navigation path represented by the graph, which is to be used when the end user of the player device does not actively navigate through the graph representation.

33. The method of claim 26, further comprising: Receive the Media Presentation Description (MPD) of the plurality of bitstreams from the server device; For a given time period, a corresponding initialization segment is received from the storage container from the server device, the corresponding initialization segment being configured to notify one or more videos to be selected for rendering at the player device; Send a request to the server device to identify the selection; as well as Receive one or more bit streams from the server device, the one or more bit streams carrying the media segment selected in the storage container based on the identified selection.

34. The method of claim 33, wherein, The MPD includes information corresponding to two or more streaming content time periods.

35. The method of claim 33, wherein, The corresponding initialization fragment has a box format compatible with at least one of the MPEG DASH and ISO BMFF specifications.

36. The method of claim 33, wherein, The MPD includes information about two groups of nodes that can switch between each other according to the graph representation.

37. The method of claim 26, wherein, The multiple bitstreams are encapsulated using a file format, wherein different bitstreams within the bitstreams are stored in different corresponding tracks, and each image bitstream is stored in a different corresponding project.

38. The method of claim 26, wherein, The multiple bitstreams are encapsulated using fragment encapsulation, where each fragment is carried in a corresponding representation of an adaptive set.

39. The method of claim 26, in, A set of bitstreams corresponding to the node group represented by the graph is stored in the storage container using ISO BMFF format; and Different groups are indicated as different pre-selections in the corresponding MPD.

40. The method of claim 39, in, The pre-selection in the corresponding MPD includes multiple adaptive sets; Wherein, the different adaptive sets in the adaptive set are centered on the different corresponding nodes represented by the graph; and The multi-track bitstream corresponding to the adaptive set includes a first track and one or more second tracks. The first track carries the bitstream of the corresponding central node at a first bit rate, and each of the one or more second tracks carries the corresponding bitstream of a node in the graph representation that is allowed to transition from the central node at a second bit rate. The first bit rate is greater than the second bit rate.

41. The method of claim 26, wherein, The diagram represents multiple sub-graphs, each of which corresponds to a different time period of the corresponding MPD.

42. The method of claim 41, wherein, The switching descriptor of the corresponding adaptive set of the corresponding MPD is used to indicate the effective switching between the subgraphs.

43. The method of claim 41, wherein, The subgraphs in the plurality of subgraphs include a first node represented by a first viewing timeline and a second node represented by a second viewing timeline, the first timeline and the second timeline having different corresponding time scales.

44. The method of claim 41, wherein, The subgraphs in the plurality of subgraphs include a first node corresponding to a first object presented in the MPAV and a second node corresponding to a different second object presented in the MPAV.

45. The method of claim 41, wherein, The subgraphs in the plurality of subgraphs include a first node corresponding to the long form video of the MPAV and a second node corresponding to a navigation interface for navigating along the long form video.

46. ​​A non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations including the method of claim 26.

47. An apparatus for streaming multi-view audio and video (MPAV), the apparatus comprising: At least one processor; as well as At least one memory containing program code, The at least one memory and the program code are configured, together with the at least one processor, to cause the device to perform at least the following operations: Receive metadata configured to enable the player device to reconstruct a graph representation of multiple videos corresponding to the MPAV, the graph representation including multiple nodes interconnected via multiple edges, each of the nodes representing a corresponding video in the video, and each of the edges representing a permissible switch between a corresponding pair of videos during playback; Based on the reconstructed graph representation and further based on interactive user input, a streaming request is generated for a server device. This streaming request identifies a selected subset of multiple bitstreams packaged into a storage container accessible via the server device, wherein the multiple bitstreams encode multiple videos corresponding to the MPAV. Receive the selected subset of the plurality of bit streams transmitted by the server device in response to the streaming request.

Images