Method, apparatus and computer readable medium for virtual reality

Abstract

The present invention relates to a method, apparatus and computer-readable medium for virtual reality. In one solution, a bitstream defining a presentation is generated, the presentation including omnidirectional visual media content and a first visual media component and a second visual media component; a first presentation timeline associated with the first visual media component is indicated in the bitstream; a second presentation timeline associated with the second visual media component is indicated in the bitstream; a switching mode to the second presentation timeline associated with the second visual media component is indicated in the bitstream; and the switching mode is indicated in the bitstream as to whether it is about the first presentation timeline or about the second presentation timeline.

Description

[0001] This application is a divisional application of the patent application with application number 201911281912.X filed on December 13, 2019 and invention name “Method, device and computer program product for virtual reality”. Technical Field

[0002] This solution often involves virtual reality. Background Art

[0003] Since the beginning of photography and cinematography, the most common types of image and video content have been captured by cameras with relatively narrow fields of view and displayed as rectangular scenes on flat displays. In this application, such content is referred to as "flat content" or "flat images" or "flat video". Cameras are primarily directional, so they only capture a limited angular field of view (the field of view they are pointing towards).

[0004] Recently, new image and video capture devices have become available. These devices are able to capture all the visual and audio content around, i.e., they can capture the entire angular field of view (sometimes also called a 360-degree field of view). More precisely, they can capture a spherical field of view (i.e., 360 degrees in all spatial directions). In addition, new types of output technologies have been invented and produced, such as head-mounted displays. These devices allow a person to see the visual content around him / her, giving a feeling of being "immersed" in a scene captured by a 360-degree camera. The new capture and display paradigm, in which the field of view is spherical, is often referred to as virtual reality (VR), and is believed to be the common way people will experience media content in the future. Summary of the invention

[0005] An improved method for encoding and decoding and a technical device for implementing the method have now been invented. Various aspects of the invention include a method, an apparatus and a computer-readable medium including a computer program stored therein, characterized by what is stated in the independent claims. Various embodiments of the invention are disclosed in the dependent claims.

[0006] According to a first aspect, a method is provided, the method comprising: generating a bitstream defining a presentation, the presentation comprising omnidirectional visual media content and a first visual media component and a second visual media component; indicating in the bitstream a first presentation timeline associated with the first visual media component; indicating in the bitstream a second presentation timeline associated with the second visual media component; indicating in the bitstream a switching mode to the second presentation timeline associated with the second visual media component; and indicating in the bitstream whether the switching mode is with respect to the first presentation timeline or with respect to the second presentation timeline.

[0007] According to a second aspect, a device is provided, comprising a processor, a memory, and a computer program code residing in the memory, wherein the computer code, when executed by the processor, is configured to cause the device to: generate a bitstream defining a presentation, the presentation comprising omnidirectional visual media content and a first visual media component and a second visual media component; indicate in the bitstream a first presentation timeline associated with the first visual media component; indicate in the bitstream a second presentation timeline associated with the second visual media component; indicate in the bitstream a switching mode to the second presentation timeline associated with the second visual media component; and indicate in the bitstream whether the switching mode is with respect to the first presentation timeline or with respect to the second presentation timeline.

[0008] According to a third aspect, a method is provided, the method comprising: determining a presentation from a bitstream, the presentation comprising omnidirectional visual media content and a first visual media component and a second visual media component; decoding a first presentation timeline associated with the first visual media component from the bitstream; decoding a second presentation timeline associated with the second visual media component from the bitstream; a switching mode from the bitstream to the second presentation timeline associated with the second visual media component; and decoding from the bitstream whether the switching mode is about the first presentation timeline or about the second presentation timeline.

[0009] According to a fourth aspect, a device is provided, comprising a processor, a memory, and a computer program code residing in the memory, wherein the computer code, when executed by the processor, is configured to cause the device to: determine a presentation from a bitstream, the presentation comprising omnidirectional visual media content and a first visual media component and a second visual media component; decode a first presentation timeline associated with the first visual media component from the bitstream; decode a second presentation timeline associated with the second visual media component from the bitstream; a switching mode from the bitstream to the second presentation timeline associated with the second visual media component; and whether the switching mode decoded from the bitstream is about the first presentation timeline or about the second presentation timeline.

[0010] According to a fifth aspect, a computer program product embodied on a computer-readable medium and comprising a computer program code is provided, wherein the computer program code is configured to, when executed on at least one processor, cause an apparatus or system to: generate a bitstream defining a presentation, the presentation comprising omnidirectional visual media content and a first visual media component and a second visual media component; indicate in the bitstream a first presentation timeline associated with the first visual media component; indicate in the bitstream a second presentation timeline associated with the second visual media component; indicate in the bitstream a switching mode to the second presentation timeline associated with the second visual media component; and indicate in the bitstream whether the switching mode is with respect to the first presentation timeline or with respect to the second presentation timeline.

[0011] According to a sixth aspect, there is provided a computer program product embodied on a computer-readable medium and comprising a computer program code, the computer program code being configured to, when executed on at least one processor, cause a device or system to: determine a presentation from a bitstream, the presentation comprising omnidirectional visual media content and a first visual media component and a second visual media component; decode a first presentation timeline associated with the first visual media component from the bitstream; decode a second presentation timeline associated with the second visual media component from the bitstream; a switching mode from the bitstream to the second presentation timeline associated with the second visual media component; and whether the switching mode decoded from the bitstream is about the first presentation timeline or about the second presentation timeline.

[0012] According to one embodiment, an indication is indicated in a bitstream / decoded from a bitstream of a main presentation timeline or an indication for creating a player presentation timeline; and an indication is indicated in a bitstream / decoded from a bitstream whether a switching mode is about a first presentation timeline or about a second presentation timeline or about the main presentation timeline or the player presentation timeline.

[0013] According to one embodiment, a retrieval mechanism for the second visual media component is indicated in / decoded from the bitstream.

[0014] According to one embodiment, information about the switching mode is included in a container format.

[0015] According to one embodiment, a computer program product is embodied on a non-transitory computer-readable medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Hereinafter, various embodiments of the present invention will be described in more detail with reference to the accompanying drawings, in which

[0017] Figure 1 An example of the relationship of spherical coordinates to azimuth and elevation angles to the X, Y, and Z coordinate axes is shown;

[0018] Figure 2 An example of converting a spherical image to a packed image is shown;

[0019] Figure 3 An example of OMAF video processing steps is shown;

[0020] Figure 4 An example of seamless switching is shown;

[0021] Figure 5 An example of switching to zero time is shown;

[0022] Figure 6 An example of switching to an offset time is shown;

[0023] Figure 7 is a flow chart illustrating a method according to an embodiment;

[0024] Figure 8 is a flow chart illustrating a method according to another embodiment; and

[0025] Fig. 9 An apparatus according to an embodiment is shown. DETAILED DESCRIPTION

[0026] In the following, several embodiments of the present invention will be described in the context of immersive multimedia (i.e. virtual reality) such as omnidirectional content. The consumption of omnidirectional content by the end user is more complex than the consumption of 2D content. This is due to the higher degree of freedom that the end user can use. This freedom also leads to more uncertainty. The situation becomes even more complex when layers of content are rendered, such as in the case of an overlay.

[0027] Available media file format standards include the International Organization for Standardization (ISO) Base Media File Format (ISO / IEC14496-12, which can be abbreviated as ISOBMFF), the Moving Picture Experts Group (MPEG)-4 file format (ISO / IEC 14496-14, also known as the MP4 format), the NAL (Network Abstraction Layer) unit structured video file format (ISO / IEC 14496-15), and the High Efficiency Video Coding standard (HEVC or H.265 / HEVC).

[0028] Some concepts, structures and specifications of ISOBMFF are described below as an example of a container file format based on which embodiments can be implemented. Aspects of the present invention are not limited to ISOBMFF, but the description gives a possible basis on which the present invention can be partially or completely implemented.

[0029] The basic building block of the ISO base media file format is called a box. Each box has a header and a payload. The box header indicates the type of box and the size of the box in bytes. The box type can be identified by an unsigned 32-bit integer, interpreted as a four-character code (4CC). Boxes can enclose other boxes, and the ISO file format specifies which box types are allowed within a certain type of box. Furthermore, the presence of certain boxes in each file may be mandatory, while the presence of other boxes may be optional. In addition, for some box types, multiple boxes may be allowed to be contained in a single file. Therefore, the ISO base media file format can be considered to specify a hierarchy of boxes.

[0030] According to the ISO base media file format, a file includes media data and metadata encapsulated in a box.

[0031] In a file that conforms to the ISO base media file format, media data may be provided in one or more instances of MediaDataBox ("mdat"), while MovieBox ("moov") may be used to contain metadata for timed media. In some cases, both "mdat" and "moov" boxes may need to be present in order for a file to be operable. A "moov" box may include one or more tracks, each of which may reside in a corresponding TrackBox ("trak"). Each track is associated with a handler that specifies the track type, identified by a four-character code. Video, audio, and image sequence tracks may be collectively referred to as media tracks, and they contain elementary media streams. Other track types include hint tracks and timed metadata tracks.

[0032] A track consists of samples, such as audio or video frames. For video tracks, media samples may correspond to coded pictures or access units.

[0033] A media track refers to samples (which may also be referred to as media samples) formatted according to a media compression format (and its encapsulation of the ISO base media file format). A hint track refers to hint samples, containing cookbook instructions for building packets for transmission over the indicated communication protocol. A timed metadata track may refer to samples describing the indicated media and / or hint samples.

[0034] The "trak" box includes a SampleDescriptionBox in its box hierarchy, which provides details about the type of encoding used and any initialization information required for that encoding. The SampleDescriptionBox contains an entry count and up to the number of sample entries indicated by the entry count. The format of the sample entries is specific to the track type, but is derived from the generic classes (e.g. VisualSampleEntry, AudioSampleEntry). Which type of sample input format is used to derive the sample input format specific to that track type is determined by the track's media handler.

[0035] Movie segmentation can be used, for example, when recording content to an ISO file, for example, to avoid data loss when the recording application crashes, memory space is insufficient, or other unexpected events occur. Without movie segmentation, data loss may occur because the file format may require that all metadata (e.g., movie boxes) be written to one continuous area of ​​the file. In addition, when recording a file, there may not be enough storage space (e.g., random access memory (RAM)) to buffer the movie box for the available storage size, and recalculating the contents of the movie box at the end of the movie may be too slow. In addition, movie segmentation can enable the use of a conventional ISO file parser to record and playback files simultaneously. In addition, progressive downloading (e.g., receiving and playback files simultaneously when using movie segmentation) may require a shorter initial buffering duration, and the initial movie box is smaller than a file with the same media content and structured without movie segmentation.

[0036] The movie segmentation feature may enable the metadata that may otherwise reside in the movie box to be split into multiple strips. Each strip may correspond to a specific time period of a track. In other words, the movie segmentation feature may enable the interleaving of file metadata and media data. Thus, the size of the movie box may be limited and the above-mentioned use cases may be achieved.

[0037] In some examples, the media samples of a movie segment may reside in an mdat box. However, for the metadata of the movie segment, a moof box may be provided. The moof box may include information that previously existed in a moov box for a certain playback time duration. The moov box may still represent a valid movie by itself, but in addition, it may include an mvex box indicating that a movie segment will follow in the same file. The movie segment may extend the presentation associated with the moov box in time.

[0038] There can be a track segment set within a movie segment, including any position from zero to multiple per track. A track segment can also include any position from zero to multiple track runs, and each track document is a continuous sample run for the track (thus similar to a chunk). Within these structures, many fields are optional and can be default. The metadata that can be included in the moof box can be limited to a subset of metadata that can be included in the moov box and can be encoded differently in some cases. Details about the boxes that can be included in the moof box can be found in the ISOBMFF specification. A self-contained movie segment can be defined as a moof box and an mdat box that are continuous in file order, and the mdat box contains samples of the movie segment (the moof box provides metadata for it), and does not contain samples of any other movie segment (i.e., any other momo box).

[0039] A media segment may include one or more self-contained movie segments. The media segment may be used for delivery (eg, streaming delivery) in, for example, MPEG-DASH.

[0040] The ISO base media file format contains three mechanisms for timed metadata that can be associated with specific samples: sample groups, timed metadata tracks, and sample auxiliary information. Derived specifications may provide similar functionality through one or more of these three mechanisms.

[0041] Based on the grouping criteria, sample grouping in the ISO base media file format and its derivatives can be defined as assigning each sample in a track as a member of a sample group based on the grouping condition. Sample groups in a sample grouping are not limited to immediately adjacent samples and can contain non-adjacent samples. Since there may be multiple sample groupings for samples in a track, each sample grouping can have a type field to indicate the type of grouping. Sample grouping can be represented by two linked data structures: (1) SampleToGroupBox (sbgp box) represents the assignment of samples to sample groups; and (2) SampleGroupDescriptionBox (sgpd box) contains sample group entries for each sample group that describe the properties of the group. Based on different grouping criteria, there can be multiple instances of SampleToGroupBox and SampleGroupDescriptionBox. These can be distinguished by the type field used to indicate the type of grouping. SampleToGroupBox may include a grouping_type_parameter field, which can be used, for example, to indicate the subtype of the grouping. A default sample group entry can be provided in the SampleGroupDescriptionBox, which applies to all samples that are not mapped in any SampleToGroupBox in the same sample grouping.

[0042] In ISOMBFF, the edit list provides a mapping between the presentation timeline and the media timeline. In particular, the edit list provides a linear offset for the presentation of samples in a track, provides an indication of idle time, and provides specific samples to be retained for a specific period of time. The presentation timeline can be modified accordingly to provide loops, such as looping video for various areas of a scene. An example of a box that includes an edit list, namely EditListBox, is provided below:

[0043]

[0044] In ISOBMFF, an EditListBox can be contained in an EditBox, and the EditBox is contained in a TrackBox ("trak").

[0045] In this example of an edit list box, a flag specifies the repetition of an edit list. For example, setting a particular bit within the box flags (the least significant bit, i.e., flags&1 in ANSI-C notation, where & indicates a bitwise AND operation) to 0 indicates that the edit list is not repeated, while setting a particular bit (i.e., flags&1 in ANSI-C notation) to 1 indicates that the edit list is repeated. Box flag values ​​greater than 1 can be defined as reserved for future expansion. Thus, when the edit list box indicates that zero or one sample is played, (flags&1) should be equal to zero. When an edit list is repeated, the media generated by the edit list at time 0 is immediately followed by the media with the maximum time generated by the edit list, thereby seamlessly repeating the edit list.

[0046] In ISOBMFF, a Track group can group tracks based on some characteristics, or the tracks in a group have a specific relationship. However, Track grouping does not allow any image items in the group.

[0047] The syntax of TrackGroupBox in ISOBMFF is as follows:

[0048] aligned(8)class TrackGroupBox extends Box('trgr'){

[0049] }

[0050] aligned(8)class TrackGroupTypeBox(unsigned int(32)track_group_type)

[0051] extends FullBox(track_group_type,version=0,flags=0)

[0052] {

[0053] unsigned int(32)track_group_id;

[0054] / / You can specify the remaining data for a specific track_group_type

[0055] }

[0056] track_group_type indicates the grouping_type, and shall be set to one of the following values, or a registered value, or a value from a derived specification or registration:

[0057] "msrc" indicates that the track belongs to a multi-source presentation.

[0058] Within a TrackGroupTypeBox of track_group_type 'msrc', tracks with the same track_group_id value are mapped as originating from the same source. For example, a recording of a video phone call might have both audio and video from two participants, and the track_group_id values ​​associated with the audio and video tracks of one participant are different from the track_group_id values ​​associated with the tracks of the other participant.

[0059] A track_group_id and track_group_type pair identifies a track group in the file. Tracks containing a specific TrackGroupTypeBox with the same track_group_id and track_group_type values ​​belong to the same track group.

[0060] Entity grouping is similar to track grouping, but enables grouping of both track and image items in the same group.

[0061] The syntax of EntityToGroupBox in ISOBMFF is as follows.

[0062] aligned(8)class EntityToGroupBox(grouping_type,version,flags)

[0063] extends FullBox(grouping_type,version,flags){

[0064] unsigned int(32)group_id;

[0065] unsigned int(32)num_entities_in_group;

[0066] for(i=0;i <num_entities_in_group;i++)

[0067] unsigned int(32)entity_id;

[0068] }

[0069] group_id is a non-negative integer assigned to a specific grouping that must not be equal to any group_id value of any other EntityToGroupBox, any item_ID value of the hierarchy level (file, movie, or track) containing the GroupsListBox, or any track_ID value (when the GroupsListBox is contained at the file level).

[0070] num_entities_in_group specifies the number of entity_id values ​​that map to this entity group.

[0071] When an item with item_ID equal to entity_id exists in the hierarchy level (file, movie or track) containing the GroupsListBox, then entity_id is resolved to the item; when a track with track_ID equal to entity_id exists and the GroupsListBox is contained in the file level, then entity_id is resolved to the track.

[0072] Files conforming to ISOBMFF may contain any non-timed object (called an item, meta-item, or metadata item) in a meta box (four-character code: "meta"). Although the name of the meta box refers to metadata, an item may contain metadata or media data in general. A meta box may be located at the top level of a file, within a movie box (four-character code: "moov"), and within a track box (four-character code: "trak"), but at most one meta box may occur at each level of the file level, the movie level, or the track level. A meta box may be required to contain an "hdlr" box indicating the structure or format of the contents of the "meta" box. A meta box may list and characterize any number of items that may be referenced, and each item may be associated with a file name and uniquely identified by the file by an item identifier (item_id) that is an integer value. Metadata items may be stored, for example, in an "idat" box of a meta box or in an "mdat" box or reside in a separate file. If the metadata is located outside the file, its location may be declared by a DataInformationBox (four-character code: "dinf"). In the specific case where metadata is formatted using XML syntax and requires to be stored directly in a MetaBox, the metadata can be encapsulated into an XMLBox (four-character code: 'xml') or a BinaryXMLBox (four-character code: 'bxml'). Items can be stored as a contiguous byte range or as multiple extents, each of which is a contiguous byte range. In other words, items can be stored fragmented into extents, for example to enable interleaving. An extent is a contiguous byte subset of a resource; a resource can be formed by concatenating extents.

[0073] ItemPropertiesBox enables any item to be associated with an ordered set of item properties. Item properties can be viewed as small data records. ItemPropertiesBox consists of two parts: an ItemPropertyContainerBox that contains an implicitly indexed list of item properties; and one or more ItemPropertyAssociationBoxes that associate items with item properties.

[0074] High Efficiency Image File Format (HEIF) is a standard developed by the Moving Picture Experts Group (MPEG) for storing images and image sequences. In particular, the standard facilitates file encapsulation of data encoded according to the High Efficiency Video Coding (HEVC) standard. HEIF includes features built on top of the ISO Base Media File Format (ISOBMFF) used.

[0075] In the design of HEIF, ISOBMFF structure and features are largely used. The basic design of HEIF includes storing still images as projects and image sequences as tracks.

[0076] In the context of HEIF, the following boxes may be included in the root-level 'meta' box and may be used as described below. In HEIF, the 'meta' box has a handler value of 'pict' in the handler box. Resources containing coded media data (whether within the same file or in an external file identified by a uniform resource identifier) ​​are resolved via a 'data information' ('dinf') box, while an 'item location' ('iloc') box stores the location and size of each item in the referenced file. The 'item reference' ('iref') box records the relationship between items using typed references. If there is an item in a collection of items that is in some way considered to be the most important item compared to the others, that item is signaled by a 'primary item' ('pitm') box. In addition to the boxes mentioned here, the 'meta' box is also flexible to include other boxes necessary to describe the item.

[0077] Any number of image items may be included in the same file. Given a collection of images stored using the 'meta' box method, it is sometimes necessary to define certain relationships between the images. Examples of such relationships include indicating a cover image for the collection, providing thumbnails for some or all images in a collection, and associating some or all images in a collection with auxiliary images such as an alpha plane. A cover image in a collection of images is indicated using the 'pitm' box. Thumbnails or auxiliary images are linked to the primary image item using an item reference of type 'thmb' or 'auxl', respectively.

[0078] Entities can be defined as tracks or projects. Entity groups are groupings of projects and can also group tracks. Entity groups can be used instead of project references when the grouped entities do not have clear dependencies or directional references. Entities in an entity group share specific characteristics or have specific relationships, as indicated by the grouping type.

[0079] Entity Groups are groupings of projects and tracks. Entities in an entity group share specific characteristics or have specific relationships, as indicated by the grouping type.

[0080] Entity groups are indicated in the GroupsListBox. Entity groups specified in the GroupsListBox of the file-level MetaBox refer to tracks or file-level items. Entity groups specified in the GroupsListBox of the movie-level MetaBox refer to movie-level items. Entity groups specified in the GroupsListBox of the track-level MetaBox refer to track-level items for that track.

[0081] GroupsListBox contains EntityToGroupBoxes, each of which specifies an entity group. The syntax of EntityToGroupBox is as follows:

[0082]

[0083] When an item with item_ID equal to entity_id exists in the hierarchy level (file, movie or track) containing the GroupsListBox, entity_id is resolved to the item; when a track with track_ID equal to entity_id exists and the GroupsListBox is contained in the file level, entity_id is resolved to the track.

[0084] A TrackGroupBox contained in a Trackbox enables the indication of groups of tracks where each group shares specific characteristics, or where the tracks within a group have specific relationships. The box contains zero or more boxes, and the specific characteristics or relationships are indicated by the box types of the contained boxes. The contained boxes include identifiers that can be used to infer tracks that belong to the same track group. Tracks that contain boxes of the same type as contained in a TrackGroupBox and have the same identifier values ​​within those contained boxes belong to the same track group. The syntax for defining contained boxes via TrackGroupTypeBox is as follows:

[0085]

[0086] The Matroska file format is capable of (but not limited to) storing any of the video, audio, picture or subtitle tracks in one file. Matroska can be used as the base format for exported file formats (such as WebM). Matroska uses the Extensible Binary Meta Language (EBML) as its basis. EBML is inspired by XML principles and specifies a binary and octet (byte) aligned format. EBML itself is a general description of binary tagging technology. Matroska files consist of "elements" that make up the EBML "document". Elements contain element IDs, descriptors of element sizes, and the binary data itself. Elements can be nested. Matroska's segmentation element is a container for other top-level (level 1) elements. Matroska files can include a segment (but are not limited to being composed of it). The multimedia data in a Matroska file is organized in clusters (or cluster elements), each of which can contain several seconds of multimedia data. Clusters include BlockGroup elements, which in turn include Block Elements. A Cues Element includes metadata that may assist in random access or seeking, and may include a file pointer or a corresponding timestamp for a seek point.

[0087] A Uniform Resource Identifier (URI) can be defined as a string of characters that identifies a resource name. Such an identification enables interaction with the representation of a resource over a network using a specific protocol. URIs are defined by a scheme that specifies the specific syntax of URIs and the associated protocols. Uniform Resource Locators (URLs) and Uniform Resource Names (URNs) are forms of URIs. A URL can be defined as a URI that identifies a Web resource and specifies the means to operate on or obtain a representation of the resource, specifying its primary access mechanism and network location. A URN can be defined as a URI that identifies a resource by a name in a specific namespace. URNs can be used to identify a resource without implying its location or how it is accessed.

[0088] The Hypertext Transfer Protocol (HTTP) has been widely used to deliver real-time multimedia content over the Internet (e.g., video streaming applications). Several protocols for adaptive streaming over HTTP (e.g., Smooth Streaming, Adaptive HTTP Live Streaming and Dynamic Streaming) and has carried out standardization projects. Adaptive HTTP streaming (AHS) was first standardized in version 9 of the 3rd Generation Partnership Project (3GPP) Packet Switched Streaming (PSS) service (3GPP TS 26.234 version 9: "Transparent end-to-end packet switched streaming service (PSS)"; Protocols and codecs "). MPEG used 3GPP AHS version 9 as the starting point for the MPEG DASH standard (ISO / IEC 23009-1: "Dynamic Adaptive Streaming over HTTP (DASH) - Part 1: Media presentation description and segmentation format", International Standard, Second Edition, 2014). MPEG DASH and 3GP-DASH are technically close to each other, so they can be collectively referred to as DASH.

[0089] Some concepts, structures and specifications of DASH are described below as an example of a container file format based on which embodiments may be implemented. Aspects of the invention are not limited to DASH, but rather a description of a possible basis is given on which the invention may be partially or fully implemented.

[0090] In DASH, multimedia content can be stored on an HTTP server or delivered using HTTP. The content can be stored on the server in two parts: a media presentation description (MPD), which describes the list of available content, its various alternatives, their URL addresses and other characteristics; and segments, which contain the actual multimedia bitstream in a single or multiple files in the form of blocks. MDP provides the client with the necessary information to establish dynamic adaptive streaming over HTTP. MPD contains information describing the media presentation (such as the HTTP Uniform Resource Locator (URL) of each segment) to make a GET segment request. In order to play the content, the DASH client can obtain the MPD, for example, by using HTTP, email, thumb drive, broadcast or other transmission methods. By parsing the MPD, the DASH client can understand the program timing, media content availability, media type, resolution, minimum and maximum bandwidth, and various encoding alternatives for multimedia components, accessibility characteristics and the presence of required digital rights management (DRM), media component location on the network, and other content characteristics. Using this information, the DASH client can select the appropriate encoding alternative and start streaming the content by obtaining the segments using, for example, HTTP GET requests. The client can continue to fetch subsequent segments after doing appropriate buffering to allow for variations in network throughput, and also monitor network bandwidth fluctuations. The client can decide how to adapt to the available bandwidth by fetching segments of different alternatives (with lower or higher bit rates) to maintain an adequate buffer.

[0091] In the context of DASH, the following definitions may be used: A media content component or media component may be defined as a continuous component of media content with an assigned media component type that may be individually encoded as a media stream. Media content may be defined as a media content period or a continuous sequence of media content periods. A media content component type may be defined as a single type of media content such as audio, video, or text. A media stream may be defined as an encoded version of a media content component.

[0092] In DASH, a hierarchical data model is used to construct a media presentation as shown below. A media presentation consists of a series of one or more periods, each period contains one or more groups, each group contains one or more adaptation sets, each adaptation set contains one or more representations, and each representation consists of one or more segments. A group can be defined as a collection of adaptation sets that are not expected to be presented simultaneously. An adaptation set can be defined as a collection of interchangeable encoded versions of one or several media content components. A representation is one of the alternatives for media content or a subset thereof, which may differ in encoding choices such as bit rate, resolution, language, codec, etc. A segment contains media data of a certain duration, as well as metadata for decoding and presenting the included media content. A segment is identified by a URI and can be requested by an HTTP GET request. A segment can be defined as a unit of data associated with an HTTP-URL and optionally positioned as a byte range specified by an MPD.

[0093] The DASH MPD conforms to the Extensible Markup Language (XML) and is therefore specified by elements and attributes as defined in XML. The MPD may be specified using the following conventions: Elements in an XML document may be identified by an uppercase first letter and may be displayed in bold as Element. To indicate that an element Element1 is contained within another element Element2, Element2.Element1 may be written. If the name of an element consists of two or more compound words, camel-casing may be used, such as ImportantElement. Elements may appear only once or may be preceded and followed by a single character. <minoccurs> ... <maxoccurs>Defines minimum and maximum occurrences. Attributes in XML documents can be identified by a lowercase first letter or by preceding it with the "@" symbol, for example @attribute. To refer to a specific attribute @attribute contained in an element Element, you can write Element@attribute. If the name of an attribute consists of two or more compound words, you can use camel case format after the first word, for example @veryImportantAttribute. Attributes can be assigned the states mandatory (M), optional (O), optional with default value (OD), and conditionally mandatory (CM) in XML.

[0094] In DASH, all descriptor elements can be constructed in the same way, that is, they contain the @schemeIdUri attribute (which provides a URI for identifying the scheme), the optional attribute @value, and the optional attribute @id. The semantics of the element are specific to the scheme adopted. The URI identifying the scheme can be a URN or a URL. Some descriptors are specified in MPEG-DASH (ISO / IEC 23009-1), while descriptors may be specified in addition or alternatively in other specifications. When specified in specifications other than MPEG-DASH, the MPD does not provide any specific information about how to use descriptor elements. It is up to the application or specification that adopts the DASH format to instantiate the description element with the appropriate scheme information. The application or specification that uses one of these elements defines a scheme identifier (Scheme Identifier) ​​in the form of a URI, and the value space used for the element when the scheme identifier is used. The scheme identifier appears in the @schemeIdUri attribute. In the case where a simple set of enumerated values ​​is required, a text string can be defined for each value, and the string can be included in the @value attribute. If structured data is required, any extension elements or attributes can be defined in a separate namespace. The @id value may be used to refer to a unique descriptor or to a set of descriptors. In the latter case, descriptors having the same value for the attribute @id may be required to be synonymous, i.e., treatment of one of the descriptors having the same value for @id is sufficient. Two elements of type DescriptorType are equivalent if the element name, the value of the @schemeIdUri, and the value of the @value attribute are equivalent. If @schemeIdUri is a URN, equivalence may refer to lexical equivalence as defined in clause 5 of RFC2141. If @schemeIdUri is a URL, equivalence may refer to equality on a character-by-character basis as defined in clause 6.2.1 of RFC3986. If the @value attribute is not present, equivalence may be determined solely by the equivalence of the @schemeIdUri. Attributes and elements in the extended namespace may not be used to determine equivalence. The @id attribute may be ignored in equivalence determinations.

[0095] MPEG-DASH specifies the descriptors EssentialProperty and SupplementalProperty. For the element EssentialProperty, the media presentation author indicates that successful processing of the descriptor is essential for the correct use of the information in the parent element containing this descriptor, unless the element shares the same @id with another EssentialProperty element. If the EssentialProperty elements share the same @id, it is sufficient to process one of the EssentialProperty elements with the same @id value. At least one EssentialProperty element shall be processed for each different @id value. If the scheme or value of the EssentialProperty descriptor is not recognized, the DASH client shall ignore the parent element containing the descriptor. Multiple EssentialProperty elements with the same @id value and with different @id values ​​can be present in the MPD.

[0096] For the element SupplementalProperty, the media presentation author indicates that the descriptor contains supplemental information that the DASH client can use for optimized processing. If the scheme or value for the SupplementalProperty descriptor is not recognized, the DASH client should ignore the descriptor. Multiple SupplementalProperty elements can be present in the MPD.

[0097] MPEG-DASH specifies the Viewpoint element, which is formatted as a property descriptor. The @schemeIdUri attribute of the Viewpoint element is used to identify the adopted view scheme. Adaptation sets containing non-equivalent Viewpoint element values ​​contain different media content components. Viewpoint elements can be applied equally to non-video media content types. Adaptation sets with equivalent Viewpoint element values ​​are intended to be presented together. This treatment should be applied equally to recognized and unrecognized @schemeIdUri values.

[0098] SRD (Spatial Relationship Description) is specified in the normative Annex H of MPEG-DASH. The following contains some excerpts from the SRD specification.

[0099] The SRD scheme allows Media Presentation Description authors to express spatial relationships between spatial objects. Spatial objects are represented by either adaptation sets or subrepresentations. For example, a spatial relationship can represent a spatial portion of another full-frame video (such as a region of interest or tile).

[0100] SupplementalProperty and / or EssentialProperty descriptors with @schemeIdUri equal to "urn:mpeg:dash:srd:2014" are used to provide spatial relationship information associated with the contained spatial objects. The SRD shall only be included in these two MPD elements (adaptation set and sub-representation).

[0101] Sub-representation level SRD can be used to represent spatial objects of one representation (e.g. HEVC tile stream). In this case, SRD descriptors can appear in adaptation sets as well as at the sub-representation level.

[0102] The @value of a SupplementalProperty or EssentialProperty element using the SRD scheme is a comma-delimited list of SRD parameter values. The SRD parameters source_id, object_x, object_y, object_width, and object_height are required to be present, and the SRD parameters total_width, total_height, and space_set_id are conditionally or optionally present.

[0103] source_id is a non-negative integer expressed in decimal that provides an identifier for the source of the content. The source_id parameter provides a unique identifier for the source of the content within the period. It implicitly defines the coordinate system associated with this source. This coordinate system uses an arbitrary origin (0; 0); the x-axis runs from left to right, and the y-axis runs from top to bottom. All SRDs that share the same source_id value have the same origin and axis directions. Spatial relationships between spatial objects that use SRDs with different source_id values ​​are undefined.

[0104] For a given source_id value, a reference space is defined that corresponds to a rectangular region that contains the entire source content, with its upper left corner at the origin of the coordinate system. The total_width and total_height values ​​in the SRD provide the size of this reference space in arbitrary units. total_width is a non-negative integer expressed in decimal notation that represents the width of the reference space in arbitrary units. total_height is a non-negative integer expressed in decimal notation that represents the height of the reference space in arbitrary units. It is allowed for spatial objects in the MPD that do not cover the entire content source, for example when the entire source content is represented by two separate videos.

[0105] object_x is a non-negative integer expressed in decimal representing the horizontal position of the upper left corner of the spatial object in arbitrary units. object_y is a non-negative integer expressed in decimal representing the vertical position of the upper left corner of the spatial object in arbitrary units. object_width is a non-negative integer expressed in decimal representing the width of the spatial object in arbitrary units. object_height is a non-negative integer expressed in decimal representing the height of the spatial object in arbitrary units. The object_x and object_y parameters (object_width and object_height, respectively) represent the 2D position (2D size, respectively) of the associated spatial object in the coordinate system associated with the source. As defined above, the values ​​of the object_x, object_y, object_width, and object_height parameters are relative to the values ​​of the total_width and total_height parameters. The position (object_x, object_y) and size (object_width, object_height) of SRDs sharing the same source_id value can be compared after taking into account the size of the reference space, i.e., the object_x and object_width values ​​are divided by the total_width value and the object_y and object_height values ​​are divided by the total_height value of their respective descriptors. Different total_width and total_height values ​​can be used in different descriptors to provide position and size information in different units for the same reference space.

[0106] spatial_set_id is a non-negative integer expressed in decimal that provides an identifier for a set of spatial objects. If not present, the spatial object associated with this descriptor does not belong to any spatial set and no spatial set information is provided. MPD authors can use the spatial_set_id parameter to indicate that certain spatial objects within a given source_id have a specific spatial relationship. For example, an MPD author can group all adaptation sets corresponding to tiles at the same resolution level. In this way, DASH clients can use the spatial_set_id parameter to quickly select spatially related spatial objects.

[0107] An initialization segment may be defined as a segment containing the metadata necessary for presenting a media stream encapsulated in a media segment. In an ISOBMFF-based segment format, an initialization segment may include a movie box ("moov") which may not include any metadata for the samples, i.e., any metadata for the samples is provided in a "moof" box.

[0108] A media segment contains media data of a certain duration for playback at normal speed, and such a duration is called a media segment duration or segment duration. The content producer or service provider can select the segment duration according to the desired characteristics of the service. For example, a relatively short segment duration can be used in a real-time service to achieve a shorter end-to-end delay. The reason is that the segment duration can be a lower limit on the end-to-end delay perceived by the DASH client, because the segment is a discrete unit for generating media data for DASH. Content generation can be accomplished by making the entire media data segment available to the server. In addition, many client implementations use segments as units of GET requests. Therefore, in the layout of a live service, a DASH client can request a segment only when a media segment of the entire duration is available and has been encoded and encapsulated into a segment. For on-demand services, different strategies for selecting segment durations can be used.

[0109] Segments can be further divided into subsegments, for example so that segments can be downloaded in multiple parts. Segments can be required to contain complete access units. Subsegments can be indexed by a Segment Index box, which contains information for mapping the presentation time range and byte range of each subsegment. The Segment Index box can also describe subsegments and stream access points in a segment by signaling their duration and byte offset. The DASH client can use the information obtained from the Segment Index box to make HTTP GET requests for specific subsegments using byte range HTTP requests. If relatively long segment durations are used, subsegments can be used to keep the size of the HTTP response reasonable and flexible to adapt to the bit rate. The index information for a segment can be placed in a single box at the beginning of the segment, or it can be spread across many index boxes for the segment. There can be different methods of spreading, such as layering, daisy chaining, and mixing. This technique can avoid adding large boxes at the beginning of the segment and thus prevent possible initial download delays.

[0110] A subrepresentation is embedded in a regular representation and is described by a subrepresentation element. A subrepresentation element is contained within a representation element. A subrepresentation element describes the characteristics of one or several media content components embedded in a representation. For example, it may describe the exact characteristics of an embedded audio component (e.g. codec, sampling rate, etc.), embedded subtitles (e.g. codec), or it may describe some embedded lower quality video layer (e.g. some lower frame rate or something else). Subrepresentations and representations share some common attributes and elements. If the @level attribute is present in a subrepresentation element, the following applies:

[0111] Subrepresentations provide the ability to access lower quality versions of the representations that contain them. In this case, subrepresentations allow, for example, extraction of an audio track in a multiplexed representation, or if a lower frame rate is provided, efficient fast forward or rewind operations;

[0112] The initialization segment and / or media segment and / or index segment should provide enough information so that the data can be easily accessed via an HTTP partial GET request. The details of providing such information are defined by the media format used.

[0113] When using ISOBMFF segmentation, the following applies:

[0114] The Initialization section contains the Level Assignment box.

[0115] For each subsegment there is a "Subsegment Index" box ("ssix").

[0116] The attribute @level specifies the level in the subsegment index associated with the described subrepresentation. The information in the representation, subrepresentation, and level assignment ("leva") boxes contains information about the assignment of media data to levels.

[0117] The media data should have an order such that each level provides enhancement compared to lower levels.

[0118] If the @level attribute is not present, child presentation elements are used only to provide a more detailed description of the media stream embedded in the presentation.

[0119] ISOBMFF includes a so-called level mechanism for specifying file subsets. Levels follow a dependency hierarchy so that a sample mapped to level n can depend on any sample at level m, where m<=n, and does not depend on any sample at level p, where p>n. For example, levels can be specified based on temporal sublayers (e.g., TemporalId for HEVC). Levels can be declared in a level assignment ("leva") box contained in a Movie Extends ("mvex") box. Levels cannot be specified for the initial movie. When a level assignment box is present, the box applies to all movie fragments after the initial movie. For the context of a level assignment box, a fraction is defined as consisting of one or more movie fragment boxes and associated media data boxes, and may include only the initial part of the last media data box. In a fragment, data for each level appears consecutively. Data for levels within a fragment appear in ascending order of level values. All data in a fragment is assigned a level. The level assignment box provides a mapping from features (e.g., scalability layers or temporal sublayers) to levels. Features can be specified by tracks, subtracks within tracks, or sample groupings of tracks. For example, a temporal level sample grouping can be used to indicate the mapping of a picture to a temporal level, which is equivalent to a temporal sublayer in HEVC. That is, a HEVC picture of a specific TemporalId value can be mapped to a specific temporal level using a temporal level sample grouping (and the same operation can be repeated for all TemporalId values). The level allocation box can then refer to the temporal level sample grouping in the indicated mapping to the level.

[0120] The subsegment index box ("ssix") provides a mapping from levels (specified by the level assignment box) to byte ranges of the indexed subsegment. In other words, the box provides a compact index of how the data in the subsegment is sorted according to the levels in the partial subsegment. It enables clients to easily access data for partial subsegments by downloading the data ranges in the subsegment. When a subsegment index box is present, each byte in the subsegment is assigned a level. If the range is not associated with any information in the level assignment, any level not included in the level assignment can be used. There are 0 or 1 segment index boxes for each segment index box that only indexes leaf segments (i.e., only indexes subsegments without segment indexes). The subsegment index box (if any) is the next box after the associated segment index box. The subsegment index box records the subsegment indicated in the immediately preceding segment index box. Each level can be assigned to exactly one partial subsegment, i.e., the byte range of a level is continuous. The levels of partial subsegments are assigned by increasing numbers within the subsegment, i.e., a sample of a partial subsegment can depend on any sample of a previous partial subsegment in the same subsegment, but not vice versa. For example, each partial subsegment contains samples with the same temporal sublayer, and the partial subsegments appear within the subsegment in increasing temporal sublayer order. When partial subsegments are accessed in this way, the final media data box may be incomplete, that is, less data is accessed than is indicated to be presented by the length indication of the media data box. The length of the media data box may need to be adjusted, or padding may be used. The padding_flag in the level assignment box indicates whether this missing data can be replaced with zeros. If not, sample data for samples assigned to unvisited levels are not presented, and extreme care should be taken.

[0121] MPEG-DASH defines a segment container format for ISOBMFF and MPEG-2 transport streams. Other specifications may specify segment formats based on other container formats. For example, a segment format based on the Matroska container file format has been proposed and can be summarized as follows. When a Matroska file is carried as a DASH segment or similar file, the association of DASH units and Matroska units can be specified as follows. A (DASH) subsegment can be defined as one or more contiguous clusters of Matroska-encapsulated content. The initialization segment of DASH may be required to include an EBML header, a (Matroska) segment header, (Matroska) segment information, and tracks, and may optionally include other level 1 elements and padding. The segment index of DASH may include a Matroska hint element.

[0122] The Omnidirectional Media Format (OMAF), formally known as ISO / IEC 23090-2, is a standard developed by the Moving Picture Experts Group (MPEG), formally known as ISO / IEC JTC1 / SC29 / WG11. The first version of OMAF, hereinafter referred to as OMAF v1, was technically completed at the end of 2017. At the time of writing this disclosure, revision work on OMAF v2 has been initiated. Some key definitions and concepts of OMAF are described in this section as examples where embodiments may be implemented. Aspects of the invention are not limited to OMAF or extensions thereof, but are described as one possible basis on which the invention may be partially or fully implemented.

[0123] OMAF defines media formats by extending ISOBMFF, HEIF, and DASH to enable omnidirectional media applications focused on 360-degree content (e.g., video, images, audio, text). OMAF is related to the omnidirectional stream of a single 3DoF content, where the viewer is located at the center of a unit sphere with three degrees of freedom (Yaw-Pitch-Roll). The next stage of standardization (MPEG-1 Phase 1b) will enable multiple 3DoF and 3DoF+ content consumption, as well as overlay support for interaction with the user.

[0124] OMAF specifies a coordinate system consisting of a unit sphere and three coordinate axes, namely the X (back to front), Y (side, left to right), and Z (vertical, upward), which intersect at the center of the sphere.

[0125] The position of a point on the unit sphere is identified by a pair of spherical coordinates: azimuth (φ) and elevation (θ). Figure 1 The relationship between the spherical coordinate azimuth (φ) and elevation (θ) and the X, Y, and Z coordinate axes is shown. The value range of the azimuth angle is -180.0 degrees (inclusive) to 180.0 degrees (exclusive). The value range of the elevation angle is -90.0 degrees to 90.0 degrees (inclusive).

[0126] The global coordinate axes may be defined, for example, according to a coordinate system as described above, as coordinate axes that represent audio, video, and images associated with the same capture location and intended to be rendered together. The origin of the global coordinate axes is typically the same as the center point of the device or apparatus for omnidirectional audio / video capture and the position of the viewer's head in the three-dimensional space where the audio and video tracks are located.

[0127] 360-degree panoramic or omnidirectional three-degree-of-freedom (3DoF) content (i.e., images and videos) covers a full 360-degree field of view horizontally around the shooting location of the imaging device. The vertical field of view may vary and may be, for example, 180 degrees. A panoramic image covering a horizontal 360-degree field of view and a vertical 180-degree field of view may be represented by a sphere that has been mapped to a two-dimensional image plane using an equirectangular projection (ERP). In this case, the horizontal coordinate may be considered equivalent to longitude and the vertical coordinate may be considered equivalent to latitude without applying any transformation or scaling. In some cases, panoramic content with a horizontal field of view of 360 degrees but a vertical field of view of less than 180 degrees may be considered a special case of an equirectangular projection, where the polar areas of the sphere have not yet been mapped to a two-dimensional image plane. In some cases, panoramic content may have a horizontal field of view of less than 360 degrees and a vertical field of view of up to 180 degrees, while in other cases it may be characterized by an equirectangular projection format.

[0128] In the cubemap projection format, spherical video is projected onto the six faces (also called sides) of a cube. The cubemap can be generated, for example, by first rendering the spherical scene six times from a viewpoint with a view defined by a 90 degree viewing cone representing each cube face. The cube sides can be frame packed into the same frame, or each cube side can be processed separately (for example in encoding). There are many possible orders in which the cube sides are placed on the frames, and / or the cube sides can be rotated or mirrored. The frame width and height for frame packing can be selected to "tightly" fit the cube sides, for example with a 3x2 cube side grid, or unused constituent frames can be included, for example with a 4x3 cube side grid.

[0129] Equirectangular projection can be defined as the process of converting any sample position within a projected picture (in equirectangular projection format) into the angular coordinates of a coordinate system. The sample position within the projected picture can be defined relative to pictureWidth and pictureHeight, where pictureWidth and pictureHeight are the width and height of the sample equirectangular panoramic picture, respectively. In the following, the center points of the sample positions along the horizontal and vertical axes are labeled i and j, respectively. The angular coordinates (φ, θ) of the sample position in degrees are given by the following equirectangular mapping equations: φ = (0.5-i÷pictureWidth)*360, θ = (0.5-j÷pictureHeight)*180.

[0130] In general, 360-degree content can be mapped onto different types of solid geometric structures, such as polyhedrons (i.e., three-dimensional solid objects containing flat polygonal faces, straight edges and sharp corners or vertices, such as cubes or pyramids), cylinders (by projecting a spherical image onto a cylinder, as described above via equirectangular projection), cylinders (directly without first projecting onto a sphere), cones, etc., and then unfolded into a two-dimensional image plane. A two-dimensional image plane can also be considered as a geometric structure. In other words, 360-degree content can be mapped onto a first geometric structure and further unfolded into a second geometric structure. However, the transformation to the second geometric structure can be directly obtained from the original 360-degree content or from other wide field of view visual content. In general, an omnidirectional projection format can be defined as a format that represents (at most) 360-degree content on a two-dimensional image plane. Examples of omnidirectional projection formats include equirectangular projection formats and cube map projection formats.

[0131] In some cases, panoramic content with a horizontal FOV of 360 degrees but a vertical FOV of less than 180 degrees can be considered a special case of equirectangular projection, where the polar regions of the sphere have not yet been mapped to the 2D image plane. In some cases, panoramic images can have a horizontal FOV of less than 360 degrees and a vertical FOV of up to 180 degrees, while in other cases they have the characteristics of an equirectangular projection format.

[0132] Figure 2 A conversion from a spherical picture 210 to a packed picture 240 that may be used in content authoring is shown, as well as a corresponding conversion from a packed picture to a spherical picture to be rendered that may be used in an OMAF player. Figure 2 The examples shown in are described for packed pictures appearing in a projected omnidirectional video track. A similar description can be derived for image items. Figure 2 A unit sphere 210 aligned with the global coordinate axes and a unit sphere 220 aligned with the local coordinate axes are shown. Figure 2 A projected image 230 is shown on which regions for region packing are designated.

[0133] An example of OMAF video processing steps is as follows Figure 3 shown.

[0134] The projection structure (e.g., a sphere) can be rotated relative to the global coordinate axes. The rotation can be performed to achieve better compression performance, for example, based on the spatial and temporal activity of the content at certain spherical parts. Alternatively or additionally, the rotation can be performed to adjust the rendering orientation of the already encoded content. For example, if the horizon of the encoded content is not horizontal, it can then be adjusted by indicating that the projection structure is rotated relative to the global coordinate axes. The projection orientation can be indicated as a yaw angle, a pitch angle, and a roll angle that define the orientation of the projection structure or a local coordinate axis relative to the global coordinate axis. The projection orientation can be included in a box in a sample entry of an ISOBMFF track for omnidirectional video, for example.

[0135] The per-region packing information may be encoded as metadata in or along the bitstream. For example, the packing information may include a per-region mapping from a predefined or indicated source format to a packed frame format (e.g., from a projected picture to a packed picture, as described above).

[0136] Next, the region-by-region packing metadata of rectangles is described. For each region, the metadata defines a rectangle in the projected picture, a corresponding rectangle in the packed picture, and optional transformations of 90, 180, or 270 degrees of rotation and / or horizontal and / or vertical mirroring. The rectangle can be indicated, for example, by the position of the upper left corner and the lower right corner. The mapping can include resampling. Since the size of each rectangle in the projected and packed pictures can be different, the mechanism can infer the region-by-region resampling.

[0137] OMAF defines MPEG-DASH elements for associating various DASH elements. A SupplementalProperty element with the @schemeIdUri attribute equal to "urn:mpeg:mpegI:omaf:2018:assoc" is called an association descriptor. One or more association descriptors can appear at the adaptation set level, representation level, and pre-selection level. An association descriptor included within an adaptation set / representation / pre-selection element indicates that the parent element of the element's descriptor (i.e., the adaptation set / representation / pre-selection element) is associated with one or more elements in the MPD indicated by the XPath query in omaf2:association elements and association types signaled by omaf2:@associationKindList.

[0138] In OMAF DASH MPD, the view element with @schemeIdUri attribute equal to "urn:mpeg:mpegI:omaf:2018:vwpt" is called view information (VWPT) descriptor.

[0139] At most one VWPT descriptor may be present at the adaptation set level, and no VWPT descriptor shall be present at any other level. When no adaptation set in a media presentation contains a VWPT descriptor, the media presentation is inferred to contain only one view.

[0140] @value specifies the viewpoint ID of the viewpoint. ViewPointInfo is a container element whose child elements and attributes provide information about the viewpoint. The ViewPointInfo@label attribute specifies a string that provides a human-readable label for the viewpoint. The ViewPointInfo.Position attribute of this element specifies position information for the viewpoint.

[0141] In MPEG 123, VR-IF published a liaison statement that included the following requirements:

[0142] It should be possible to build an interactive VR application as a set of VR 360 streams and the navigation relationships and conditions between them. In short, the VR content includes enough information to describe all storyline paths for the interactive experience, while the interaction logic is left to the VR application.

[0143] It should be possible to build an interactive VR application where a specific section of the VR content will loop until the user has the correct interaction required to navigate further into the storyline of the interactive VR content.

[0144] It should be possible to build interactive VR applications by using overlays of media content (possibly with transparent elements) on top of VR 360 content and allowing users to interact with such overlaid media.

[0145] User interaction should be enabled as sensors attached to specific locations in the content are defined. Sensor definition is done within the VR content and can include information such as sensor location, size, and shape. User responses to triggering of such sensors should also be described in the sensor itself. This includes actions such as switching to a new VR stream, switching to new overlay media, or changing the location of overlay media.

[0146] Based on these requirements, it is necessary to define timing information when a user switches from one visual media entity to another; and a main timeline based on which the visual media entities are presented and relatively switched between them.

[0147] In this specification, the term "random access" refers to the ability of a decoder to start decoding a stream at a point other than the start point of the stream and to recover an accurate or approximate reconstructed media signal (e.g., a representation of a decoded picture). Random access operations may be characterized using random access points and recovery points. A random access point may be defined as a location in a media stream where decoding may be initiated (e.g., an access unit or a coded picture within a video bitstream). A recovery point may be defined as the first location in a media stream or within a reconstructed signal characterized in that the content of all media (such as decoded pictures) at or after the recovery point in output order is correct or approximately correct when decoding has started from the corresponding random access point. If the random access point is the same as the recovery point, the random access operation is instantaneous; otherwise, it may be progressive.

[0148] Random access points enable, for example, seek, fast forward and rewind operations in locally stored media streams and media streaming. In the context of on-demand streaming, a server can respond to a seek request by sending data starting from the random access point closest to (and in many cases, prior to) the requested destination of the seek operation, and / or a decoder can start decoding from the random access point closest to (and in many cases, prior to) the requested destination of the seek operation. Switching between encoded streams of different bit rates is a common method in unicast streaming to match the transmitted bit rate with the expected network throughput and avoid network congestion. It is possible to switch to another stream at a random access point. In addition, random access points enable tuning to broadcast or multicast. In addition, random access points can be encoded as a response to a scene cut in a source sequence or a response to an update request within a picture.

[0149] A viewport can be defined as an area of ​​an omnidirectional image or video suitable for display and viewing by a user. The current viewport (sometimes referred to as simply the viewport) can be defined as a portion of a spherical video that is currently displayed and can therefore be viewed by a user. At any point in time, a video rendered by an application on a head-mounted display (HMD) renders a portion of the 360-degree video (which is referred to as a viewport). Similarly, when viewing a spatial portion of 360-degree content on a conventional display, the spatial portion currently displayed is the viewport. A viewport is a window on a 360-degree world represented in an omnidirectional video displayed by a rendering display. The viewport is characterized by a horizontal field of view (VHFoV) and a vertical field of view (VVFoV). In the following, the horizontal field of view of the viewport will be abbreviated as HFoV, and the vertical field of view of the viewport will be abbreviated as VFoV accordingly.

[0150] A spheroid region can be defined as an area on a sphere that can be specified by four great circles or two azimuth circles and two elevation circles and an additional tiling angle that indicates a rotation along an axis originating from the sphere origin and passing through the center point of the spheroid region. A great circle can be defined as the intersection of a sphere with a plane passing through the center point of the sphere. A great circle is also called an orthodrome or Riemann circle. An azimuth circle can be defined as a circle on a sphere connecting all points with the same azimuth value. An elevation circle can be defined as a circle on a sphere connecting all points with the same elevation value.

[0151] OMAF specifies a common timed metadata syntax for spherical regions. The purpose of a timed metadata track is indicated by the track sample entry type. The sample format of all metadata tracks for a specified spherical region begins with a common section and may be followed by an extension section of sample entries specific to the metadata track. Each sample specifies a spherical region.

[0152] One of the specific sphere region timed metadata tracks specified in OMAF is called the recommended viewport timed metadata track, which indicates the viewport that should be displayed when the user has no gaze direction control or has released gaze direction control. The recommended viewport timed metadata track can be used to indicate a recommended viewport based on a "director's cut" or based on measurements of viewing statistics. A textual description of the recommended viewport can be provided in the example entry. The type of recommended viewport can be indicated in the example entry and can be in the following:

[0153] A recommended viewport based on a director's cut, such as a viewport suggested based on the creative intent of a content author or content provider.

[0154] The viewpoint or observation point is the point from which the user looks at the scene; it usually corresponds to the camera position. Slight head movements do not imply a different viewpoint.

[0155] As used herein, the term "observation point or viewpoint" refers to a quantity in three-dimensional space for virtual reality audio / video acquisition or playback. A viewpoint is a trajectory (e.g., a circle, area, or volume) around a center point of a device or apparatus for omnidirectional audio / video acquisition, and the position of the observer's head in three-dimensional space where the audio and video tracks are located. In some cases, the observer's head position is tracked and the rendering is adjusted for head movement in addition to head rotation, and the viewpoint can then be understood as the initial or reference position of the observer's head. In an implementation utilizing DASH (Dynamic Adaptive Streaming over HTTP), each observation point can be defined as a viewpoint by a viewpoint characteristic descriptor. The definition can be stored in a file format of type ISOBMFF or OMAF. In addition to DASH, delivery can also be HLS (HTTP Live Streaming), RTSP / RTP (Real-time Streaming Protocol / Real-time Transport Protocol) streaming.

[0156] As used herein, the term "viewpoint group" refers to one or more viewpoints that are spatially related or logically related. The viewpoints in a viewpoint group can be defined based on the relative positions defined for each viewpoint for a specified origin of the group. Each viewpoint group can also include a default viewpoint that reflects the default playback starting point when a user begins consuming audiovisual content in the viewpoint group without selecting a viewpoint for playback. The default viewpoint can be the same as the specified origin. In some embodiments, a viewpoint can be included in multiple viewpoint groups.

[0157] As used herein, the term "spatially related set of viewpoints" refers to viewpoints that have content with a spatial relationship between them. For example, content captured by a VR camera at different locations in the same basketball court, or a music concert captured from different locations on a stage.

[0158] As used herein, the term "logically related viewpoint group" refers to related viewpoints that have no clear spatial relationship but are logically related. The relative positions of the logically related viewpoints are described based on the creative intent. For example, two viewpoints that are members of a logically related viewpoint group may correspond to content from a performance area and a locker room. Another example may be two viewpoints in the locker rooms of two competing teams, which form a logically related viewpoint group to allow a user to walk between the two teams to watch the reactions of the players.

[0159] As used herein, the term "static viewpoint" refers to a viewpoint that remains stationary during a virtual reality audio / video capture and playback session. For example, a static viewpoint may correspond to virtual reality audio / video capture performed by a fixed camera.

[0160] As used herein, the term "dynamic viewpoint" refers to a viewpoint that does not remain static during a virtual reality audio / video capture and playback session. For example, a dynamic viewpoint may correspond to virtual reality audio / video capture performed by a mobile camera on a track or a mobile camera on a flying drone.

[0161] As used herein, the term "viewing setup" refers to a setup of one or more viewpoints and viewing orientations. In the context of a presentation where only one viewpoint is available, there is no need to explicitly indicate or summarize the viewpoint for a viewpoint setup. If a presentation has multiple viewpoints available, the viewpoints will be set based on one or more viewpoint groups, and the spatial or logical relationship between the viewpoints in each viewpoint group will be indicated in the view setup.

[0162] Overlay refers to the term for rendering visual media on 360-degree video content. Video and / or images can be overlaid on omnidirectional video and / or images. The encoded overlay video can be a separate stream or partial bitstream of the currently rendered 360-degree video / image. The omnidirectional streaming system can overlay the video / image on top of the omnidirectional video / image being rendered. The overlaid two-dimensional video / image can have a rectangular grid or a non-rectangular grid. The overlay process can cover the overlaid video / image or a portion of the video / image, or there can be a certain level of transparency / opacity or more than one level of transparency / opacity, where the overlaid video / image can be seen below the overlay video / image but with lower brightness. In other words, there can be an associated transparency level corresponding to the video / image in the foreground overlay and the video / image in the background (the video / image of the VR scene). The terms opacity and transparency can be used interchangeably.

[0163] The covered area may have one or more levels of transparency. For example, the covered area may have different portions with different transparencies. According to an embodiment, the transparency level may be defined within a range, such as from 0 to 1, such that the smaller the value, the smaller the transparency, and vice versa.

[0164] Additionally, the content provider may choose to overlay a portion of the same omnidirectional video on the user's current viewport. The content provider may wish to overlay the video based on the user's viewing conditions. For example, if the user's viewport does not match the content provider's recommended viewport, overlaying may be performed. In this case, the client player logic overlays the content provider's recommended viewport (as a preview window) on top of the user's current viewport. If the user's current viewport does not match, the recommended viewport may also be overlaid, so that the position of the overlaid video is based on the direction the user is watching. For example, if the recommended viewport is located to the left of the user's current viewport, the recommended viewport is overlaid on the left side of the display. The entire 360-degree video may also be overlaid. Yet another example is to use overlay visual information as a guidance mechanism to guide the user to the recommended viewport, such as guiding a hearing-impaired person.

[0165] There may be one or more conditions regarding when and how to display a visual overlay. Thus, a rendering device may need to receive information that the device can use to perform the overlay as indicated by the signaled information.

[0166] One or more overlays may be carried in a single visual media track or a single image item. When more than one overlay is carried in a single track or image item, or when overlays are carried with other media (e.g., a background), a region mapping from samples of the track or image item to overlay metadata may be provided, for example, in or associated with an OverlayStruct.

[0167] When several tracks or image items collectively carry one or more overlay and / or background visual media, a group of tracks and image items can be indicated in the container file. For example, the entity group of ISOBMFF can be used for this purpose.

[0168] An overlay may fall outside the user's field of view (FOV), that is, the user's viewport will not overlap with the overlay. Depending on the particular circumstances, it may be desirable to continue or pause playback of the overlay when the user is not viewing the overlay. For example, it may be desirable to pause the timeline of overlay playback until the overlay overlaps the user's viewport again. It may also be desirable to continue playing the overlay even if the overlay is outside the user's viewport. Therefore, there is a need for a mechanism to support multiple playback timelines, which in turn enables customized overlay playback / pause independent of the base content. Therefore, a method, apparatus, and computer program product are provided in accordance with example embodiments to enable multiple timeline support in playback of omnidirectional media content with an overlay, which in turn enables customized overlay playback behavior depending on whether the overlay overlaps the user's viewport.

[0169] Currently, a user can switch from one overlay visual entity to another by using any switching mechanism provided by the client player. When a user switches from one overlay to another, the following can occur:

[0170] - When the overlay switch needs to be seamless (seamless switching refers to the presentation of decoded data up to a certain time t, and the presentation of decoded data of another representation starting at time t)

[0171] - when an overlay switch requires playback to "switch to" the overlay from its starting sample;

[0172] - When an overlay switch requires "switching to" the overlay's playback from a certain time offset.

[0173] All of the above are valid when the user switches from one viewpoint to another. This embodiment addresses these aspects.

[0174] As reference Figure 2 As discussed, packed images that have been converted from spherical images can be used in content authoring, as well as a corresponding conversion from packed images to spherical images to be rendered for use in an OMAF player.

[0175] According to this embodiment, content creation may include the following:

[0176] - indicating in a bitstream, a container file and / or a manifest; a first presentation timeline associated with a first media component;

[0177] - indicating in the bitstream, container file and / or manifest; a second presentation timeline associated with the second media component;

[0178] - Indicated in the bitstream, container file and / or manifest; a switching mode to a second presentation timeline associated with a second media component

[0179] - Indicating in the bitstream, container file and / or manifest; whether the switching mode is with respect to a first presentation timeline associated with the first media component or with respect to a second presentation timeline associated with the second media component.

[0180] In another embodiment, content creation may also include the following:

[0181] - Indicate in the bitstream, container file and / or manifest; the main / global presentation timeline to use;

[0182] - In the absence of a master / global presentation timeline, the content author can indicate in the bitstream, container file and / or manifest; the creation of a player presentation timeline to be used; where

[0183] - Indicated in the bitstream, container file and / or manifest; the switching mode is with respect to a first presentation timeline associated with the first media component or with respect to a second presentation timeline associated with the second media component or with respect to a main / global presentation timeline or with respect to a player presentation timeline.

[0184] According to this embodiment, the content consumption step at the player may include the following:

[0185] - parsed from a bitstream, container file and / or manifest; a first presentation timeline associated with a first media component;

[0186] - parsed from a bitstream, container file and / or manifest; a second presentation timeline associated with a second media component;

[0187] - parsing from a bitstream, container file and / or manifest; switching mode to a second presentation timeline associated with a second media component

[0188] - Parsed from a bitstream, container file and / or manifest; the switching mode is about a first presentation timeline associated with a first media component or about a second presentation timeline associated with a second media component or about a master / global presentation timeline or about a player presentation timeline. In the absence of a master / global presentation timeline or an indication to create a player presentation timeline, the player can still create and maintain its own presentation timeline, and all switching modes will be about the player presentation timeline.

[0189] According to another embodiment, the content consumption step at the player may further include the following:

[0190] - Parsed from bitstream, container file and / or manifest; the main / global presentation timeline to use;

[0191] - In the absence of a master / global presentation timeline; parsing from bitstream, container file and / or manifest: creation of a player presentation timeline to be used; In the absence of neither a master / global presentation timeline nor an indication to create a player presentation timeline (as was the case with the previous embodiment), the player can still create and maintain its own presentation timeline, where

[0192] - Parsed from a bitstream, container file and / or manifest; the switching mode is about a first presentation timeline associated with a first media component or about a second presentation timeline associated with a second media component or about a master / global presentation timeline or about a player presentation timeline. In the absence of either a master / global presentation timeline or an indication to create a player presentation timeline (as in the case of the previous embodiment), the player can still create and maintain its own presentation timeline, and all switching modes will be about the player presentation timeline.

[0193] This embodiment proposes a method for signaling a switching mode when a user switches from a first media component that may be a first overlay to a second media component that may be a second overlay. In one embodiment, there may be more than two media components for consumption, and the switching mode between each other may be indicated in the bitstream, container file, and / or manifest.

[0194] In the following examples, the switching mode will be explained by two media components: a first media component, referred to herein as a switched-from media component, from which a user switches to a second media component; and a second media component, referred to herein as a switched-to media component.

[0195] The timeline switching mode can be one of the following:

[0196] ○Seamless switching;

[0197] ○ Switch to zero time;

[0198] ○ Switch to offset time;

[0199] ○ Switch to the nearest random access point.

[0200] Figure 4 A first example of a switching mode, namely seamless switching, is shown. Seamless switching refers to switching from presentation of decoded data of a media component until a certain time t, and switching to presentation of decoded data of a media component starting from time t. Figure 4 A “switched from” media component 405 and a “switched to” media component 400 are shown, with timelines shown with reference numerals 415 and 410, respectively. Dashed arrow 450 indicates when a user performs a switch of a media component.

[0201] Figure 5 A second example of a switching pattern is shown, namely switching to time zero. Switching to time zero refers to switching from presentation of decoded data of a media component until a certain time t, and switching to presentation of decoded data of a media component starting from time zero. Figure 5 A "switched from" media component 505 and a "switched to" media component 500 are shown, with timelines shown respectively at reference numerals 515 and 510. Dashed arrow 550 indicates when a user performs a switch of a media component.

[0202] Figure 6 A third example of a switching mode is shown, namely switching to an offset time. Switching to an offset time refers to switching from the presentation of decoded data of a media component until a certain time t, and switching to the presentation of decoded data of a media component starting from time t_offset. t_offset can be greater or less than time t. Figure 6 A “switched from” media component 605 and a “switched to” media component 600 are shown, with timelines shown with reference numerals 615 and 610, respectively. Dashed arrow 650 indicates when a user performs a switch of a media component.

[0203] A fourth example involves switching to a recent random access point (not shown in the figure), which refers to switching to the presentation of decoded data of a self-media component until a certain time t, and switching to the presentation of a media component starting from the recent random access point in the media component. "Recent" in the time dimension refers to a time point that may be later or earlier than time t.

[0204] According to an embodiment, the indication of the default switching mode can be encoded as a bitstream, container file and / or manifest to be used when switching from a media component to a media component. The default switching mode can be covered by other switching modes later in the presentation timeline. In addition, the switching mode can keep changing over time, and the change of the switching mode can be indicated in a bitstream, a container file (for example, in a timing metadata track of ISOBMFF) and / or a manifest. In an embodiment related to OMAF, the timeline switching can be an overlaid attribute, or can be included in the overlaid timing metadata track (sample entry and sample), similar to the viewpoint switching. A group of viewpoints in a group can show different switching behaviors over time. In addition, there can be a corresponding structure for switching between viewpoints. At any given point in time, there can be one or more configurations for switching between viewpoints.

[0205] According to other embodiments of the present invention, there may be additional signaling to indicate a retrieval mechanism for the player application to switch to the media component.

[0206] - Activate the toggle mode only after a certain amount of time has elapsed in the presentation timeline;

[0207] - Retrieve lower quality / resolution / frame rate content;

[0208] -Search for some content on the space;

[0209] - Retrieve an intermediate representation of the switched-to media component, the intermediate representation consisting of frames predicted from the switched-to media component.

[0210] In another embodiment of the present invention, switching to a media component corresponds to >1 candidate viewpoints for a given switching from a media component, and these candidate viewpoints can be located at different spatial locations. In this sense, this realizes a one-to-many viewpoint switching, where the time instances of >1 switching to the viewpoints are all the same (synchronous switching).

[0211] Some example embodiments are described below in conjunction with ISOBMFF. It should be understood that the example embodiments are provided as examples, and the embodiments are not limited to ISOBMFF.

[0212] In an example embodiment, the switching mode information may be carried by any other container box associated with a media component for which the switching mode has been declared. An example data structure for signaling the switching mode is shown below.

[0213]

[0214] num_entities_in_group specifies the number of entity_id values ​​that map to this entity group.

[0215] ref_id[i] is resolved to an entity, which may be a track or image item associated with an overlay or viewpoint, or generally a media component, between which the content provider enables switching and between which the user consuming the content is likely to switch.

[0216] In another example embodiment, ref_id[i] may be resolved to an item_ID indicating an ID of an overlay or viewpoint or generally a media component between which the content provider enables switching and between which the user consuming the content is likely to switch.

[0217] timeline_switching_mode specifies the timeline to use when switching between media components in this group.

[0218] toffset specifies the time offset to use when switching.

[0219] The switching mode between media components may be indicated by a value, where, for example, a value of "0" represents seamless switching, a value of "1" represents switching to the zeroth sample, a value of "2" represents switching to the offset time t_offset, and a value of "3" represents switching to the nearest random access point. It should be understood that any other value may be used instead.

[0220] According to an embodiment, TrackGroupBox or EntityToGroupBox of ISOBMFF may be used as a container for containing switching modes between different media components.Signaling Switching Modes with EntityToGroupBox One example is done for grouped overlays that are intended to be presented as a user-switchable alternative to another overlay in the same entity group.

[0221] An EntityToGroupBox with grouping_type equal to 'oval' specifies track and image items containing overlays that are intended to be presented as a user-switchable alternative to another overlay in the same entity group.

[0222]

[0223] ref_overlay_id[i] specifies the overlay_id from the track or picture item identified by the i-th entity_id that is a switchable overlay in this group. The i-th reference track or picture item shall have an overlay_id equal to the presence of ref_overlay_id[i]. If each track and picture item identified by the entity_id value of this entity group contains exactly one overlay, the ref_layer_id[i] syntax element may or may not be present. Otherwise, the ref_layer_id[i] syntax element shall be present.

[0224] The semantics of timeline_switching_mode and toffset are as described above.

[0225] Similar embodiments apply to a set of viewpoints that can be switched between each other.An example embodiment with a set of viewpoints is intended to be presented as a user-switchable alternative to another viewpoint.

[0226] An EntityToGroupBox with grouping_type equal to 'visw' specifies that tracks and image items contain viewpoints that are intended to be presented as a user-switchable alternative to another viewpoint in the same entity group.

[0227]

[0228] ref_viewpoint_id[i] specifies the viewpoint-id from the track or image item identified by the i-th entity_id that is a switchable viewpoint in this group. The i-th reference track or image item shall have a view_id equal to the presence of ref_viewpoint_id[i]. If each track and image item identified by the entity_id value of this entity group contains exactly one viewpoint, the ref_viewpoint_id[i] syntax element may or may not be present. Otherwise, the ref_layer_id[i] syntax element shall be present.

[0229] In another example embodiment, the switching mode between viewpoints can be part of ViewpointGroupStruct()

[0230]

[0231] ViewpointGroupStruct() provides viewpoint group information.

[0232] vwpt_group_id indicates the identifier of a viewpoint group. All viewpoints in a viewpoint group share a common reference coordinate system.

[0233] It can be understood that when two viewpoints have different vwpt_group_id values, their position coordinates are not comparable because the viewpoints belong to different coordinate systems.

[0234] vwpt_group_description is a null-terminated UTF-8 string indicating the description of the viewpoint group. An empty string is allowed.

[0235] It will be appreciated that the OMAF player is expected to start with the initial viewpoint timed metadata. Subsequently, if the user wishes to switch to a viewpoint group and there is no initial viewpoint information, the OMAF player is expected to switch to the viewpoint with the minimum value of the viewpoint identifier in the viewpoint group.

[0236] num_viewpoints_in_group indicates the number of viewpoints in the group that can be switched from the current viewpoint.

[0237] ref_viewpoint_id[i] specifies the pointing_id of such viewpoint in the group to which the current viewpoint can be switched and for which the corresponding switching mode is defined.

[0238] The semantics of timeline_switching_mode and toffset are as described above.

[0239] In another example embodiment, the switching mode between views may be signaled in a timed metadata track called view timeline switch timed metadata track with a sample entry "vpts".

[0240] Sample entry syntax

[0241] class vptsSampleEntry()extends MetaDataSampleEntry('vpts'){

[0242] unsigned int(32)num_viewpoint_timeline_collections;

[0243] }

[0244] Sample syntax

[0245]

[0246]

[0247] num_viewpoint_timeline_collections indicates the number of viewpoint timeline collections that signal the timeline switching mode in the sample to which this sample entry is applied.

[0248] num_viewpoints_in_this_collection[i] indicates the number of views in the i-th view timeline collection. When the user switches between any views in a given view timeline collection, the player is expected to follow the timeline_switching_mode indicated by that collection.

[0249] viewpoint_id[i][j] represents the viewpoint ID of the jth viewpoint in the i-th viewpoint timeline set.

[0250] Indicating the main timeline in ISOBMFF

[0251] In an example embodiment, the main timeline of the presentation may be signaled as part of a PresentationTimelineBox.

[0252] The ISOBMFF EditBox is a container box used to contain a mapping from a track's media timeline to a presentation timeline (shared between tracks). The EditBox can be used as a container to associate a presentation timeline with a track. An example EditBox is provided below:

[0253] Box Type: "prtl"

[0254] Container: EditBox

[0255] Mandatory: No

[0256] Quantity: zero or one

[0257] If present, the EditBox assigns the track to the specific presentation timeline identified by the timeline identifier provided in the box. When the EditBox is not present, the track is implicitly assigned to the presentation timeline with the timeline identifier equal to 0.

[0258] During playback of omnidirectional media content, tracks associated with the same presentation timeline will play synchronously. Presentation timelines with different timeline identifiers may be paced asynchronously and may have different playback states (e.g., one may be paused while another may be in regular playback mode).

[0259] In some embodiments, the example PresentationTimelineBox specifies a presentation timeline, but does not indicate how to control the presentation timeline. An example PresentationTimelineBox is provided below:

[0260]

[0261] (flags&1) equal to 1 specifies that when the track becomes active (e.g. visible), its presentation time is set to the current presentation time of the timeline identified by master_timeline_id. Specific timeline_id values ​​are reserved (for example) for indicating the timeline of the entire presentation. Thus, master_timeline_id equal to 0 specifies that the track has been synchronized to the timeline of the entire presentation.

[0262] (flags&2) equal to 2 specifies that the presentation time of this track is paused when it is inactive and resumed when it is active.

[0263] (flags&4) equal to 4 specifies that the presentation time of the track starts at 0 when the track becomes active (e.g., visible).

[0264] (flags&8) equal to 8 specifies that when the track becomes active (eg, visible), the presentation time of the track begins at the beginning of the first non-empty entry in the EditListBox.

[0265] This embodiment can be used not only for switching of overlays but also for switching of viewpoints.

[0266] timeline_id provides the timeline identifier of the presentation timeline to which this track is assigned.

[0267] Regardless of the file format used for the overlay or viewpoint, the apparatus of the example embodiments may be provided by any of a variety of computing devices, including, for example, a video encoder, a video decoder, a computer workstation, a server, etc., or by any of a variety of mobile computing devices, such as a mobile terminal, such as a smart phone, a tablet computer, a video game console, etc. Alternatively, the apparatus may be embodied by a virtual reality system, such as a virtual reality headset capable of receiving one or more data streams and rendering visual and audiovisual content that can be presented to a user.

[0268] Indicates switching modes in MPEG-DASH

[0269] Some example embodiments are described below in conjunction with MPEG-DASH.It should be understood that the example embodiments are provided as examples, and the embodiments are not limited to MPEG-DASH.

[0270] According to one embodiment, the switching mode information may be carried in an MPD with SupplementalProperty and / or EssentialProperty descriptors having an association / relationship with the media components between which the switching mode has been declared.

[0271] In an example embodiment, an association descriptor of OMAF may indicate associations between overlays that are consumed as an alternative to presentation utilizing @associationKindList indicating relationships between overlays that may be switched by the user.

[0272] According to one embodiment, a new attribute called @switchingModes is defined, which indicates the switching mode between the overrides listed by @associationKindList. The value of @switchingModes is a list of the above switching mode values.

[0273] In an example embodiment, instead of defining an attribute @switchingMode, a new element SwitchingModes is defined to be carried within a descriptor and used with an associated descriptor of OMAF, similar to what is described above. The SwitchingModes element may include a SwitchingModes@mode attribute, which is a list of switching mode values ​​mentioned above, and another attribute SwitchingModes@offset, which indicates an offset to be used if @SwitchingModes has a value of 2.

[0274] In another example embodiment, the viewpoint information descriptor may indicate switchable modes between viewpoints. A new property ViewPointInfo@switchingModes is defined, which indicates the switching modes between viewpoints. The value of ViewPointInfo@switchingModes is the same as defined above.

[0275] According to one embodiment, an element including timeline information is authored as an MPD and / or parsed from an MPD. This element may be, for example, a required descriptor element. This element may be, for example, contained in an AdaptationSet element (thus describing all representations of an adaptation set) or in a Representation element (thus describing a representation). This element includes, but is not limited to, one or more of the following:

[0276] -Presentation timeline identifier. In one embodiment, the MPD author sets this value equal to the timeline_id of the PresentationTimelineBox.

[0277] - indicates that when a representation becomes active (e.g., visible), its presentation time is set to the current presentation time of the main timeline. In one embodiment, the main timeline is aligned with the media presentation timeline inferred from the MPD. In one embodiment, the main timeline can be indicated, for example, by including its identifier into the element.

[0278] - Indicates that the presentation time of the representation is paused when it is not active and resumed when it becomes active.

[0279] - Indicates that the presentation time of the representation starts at 0 when the track becomes active (e.g., visible).

[0280] - Indicates that when a representation becomes active (e.g., visible), the presentation time of the representation starts from the earliest presentation time of its media samples. In one embodiment, the MPD author sets this value to the presentation time of the first non-empty entry in the EditListBox.

[0281] -Indicates a repeating or looping representation, i.e., indicates that when the presentation time of the representation reaches the maximum presentation time, it is reset to a value that can be presumably equal to 0 or that can be indicated in the element. In other words, the representation is repeated seamlessly. In one embodiment, the MPD author sets this indication when (flag&1) in the EditListBox is equal to 1.

[0282] According to one embodiment, the DASH client concludes whether the presentation time of the associated representation follows the media presentation timeline inferred from the MPD. The conclusion can be based on information according to any of the above embodiments. For example, elements including the described timeline information are parsed from the MPD to conclude whether the associated representation follows the media presentation timeline inferred from the MPD. When the representation follows the media presentation timeline inferred from the MPD, a (sub)segment request is usually issued. When the representation does not follow the media presentation timeline inferred from the MPD, a time range of the timeline of the representation is derived so that the time range corresponds to the next (sub)segment request to be made. For example, when the presentation time for the representation has been reset (for example due to activation of the representation or due to a loop), the time range may be different from the corresponding time range of the next (sub)segment request according to the media presentation timeline inferred from the MPD. The time range can be further converted into a segment index and / or a byte range for a (sub)segment. Then, a request, such as an HTTP GET request, can be issued based on the time range.

[0283] Indicates the MasterTimeline in MPEG-DASH

[0284] According to one embodiment, the creation of the playback timeline may be carried in the MPD at the MPD level and / or the period level and / or the adaptation set level and / or the representation set level.

[0285] In an example embodiment, the MPD @masterTimelineID element indicates the ID of the master timeline used for all periods and adaptation sets in the MPD. A new MPD with a different value for @masterTimelineID indicates that the media represented in the new MPD has a different master timeline. A new MPD with the same value for @masterTimelineID indicates that the media represented in the new MPD has the same master timeline as the previous MPD (example after an MPD refresh).

[0286] The presence of the @masterTimelineID element indicates to the player that a new timeline needs to be created by the player and that the media represented by the MPD follows the master timeline.

[0287] In another example embodiment, the Period@masterTimelineID element indicates the ID of the master timeline for a given period.

[0288] In another example embodiment, when an adaptation set containing a first view is associated with one or more adaptation sets containing a second or more views, the association descriptor shall exist as a child element of the AdaptationSet element containing the first view.

[0289] In this case, the association descriptor should include the following two items:

[0290] An XPath string in an Association element that evaluates to one or more AdaptationSet elements containing the second or more viewpoints.

[0291] Only one "vpts" value for the Association@associationKindList attribute of the association element. In this case:

[0292] o When Association@associationKindList includes a "vpts" value and the number of elements evaluated to by the XPath string in the above association element is greater than 1, the viewpoint timeline switching applies to all viewpoints.

[0293] o When Association@associationKindList includes a "vpts" value and the number of elements in the association element above to which the XPath string evaluates is equal to 1, the viewpoint timeline switching applies solely to other viewpoints.

[0294] There can be multiple such associated descriptors within an adaptation set containing a viewpoint.

[0295] As described above, when an adaptation set containing a viewpoint is associated with one or more adaptation sets containing other viewpoints, they are intended to switch between each other based on a given switching pattern.

[0296] In an example embodiment, the Association@mastertimelineID element indicates the master timeline ID of the visual media associated with the Association@associationKindList indicated by "vpts" through the association descriptor.

[0297] Figure 7 is a flow chart illustrating a method according to an embodiment. A method for encoding, comprising: generating 701 a bitstream defining a presentation, the presentation comprising omni-directional visual media content and a first visual media component and a second visual media component; indicating 702 in the bitstream a first presentation timeline associated with the first visual media component; indicating 703 in the bitstream a second presentation timeline associated with the second visual media component; indicating 704 in the bitstream a switching mode to the second presentation timeline associated with the second visual media component; and indicating 705 in the bitstream whether the switching mode is with respect to the first presentation timeline or with respect to the second presentation timeline.

[0298] An apparatus according to an embodiment includes: a module for generating a bitstream defining a presentation, the presentation including omnidirectional visual media content and a first visual media component and a second visual media component; a module for indicating in the bitstream a first presentation timeline associated with the first visual media component; a module for indicating in the bitstream a second presentation timeline associated with the second visual media component; a module for indicating in the bitstream a switching mode to the second presentation timeline associated with the second visual media component; a module for indicating in the bitstream whether the switching mode is about the first presentation timeline or about the second presentation timeline. The module includes a processor, a memory, and computer program code residing in the memory, wherein the processor may further include a processor circuit.

[0299] Figure 8 801 is a flowchart illustrating a method according to another embodiment. A method for decoding comprises: determining 802 a presentation from a bitstream, the presentation comprising omnidirectional visual media content and a first visual media component and a second visual media component; decoding 802 a first presentation timeline associated with the first visual media component from the bitstream; decoding 803 a second presentation timeline associated with the second visual media component from the bitstream; decoding 804 a switching mode to the second presentation timeline associated with the second visual media component from the bitstream; decoding 805 whether the switching mode is about the first presentation timeline or about the second presentation timeline from the bitstream.

[0300] An apparatus according to an embodiment includes: a module for determining a presentation from a bitstream, the presentation including omnidirectional visual media content and a first visual media component and a second visual media component; a module for decoding a first presentation timeline associated with the first visual media component from the bitstream; a module for decoding a second presentation timeline associated with the second visual media component from the bitstream; a module for decoding from the bitstream a switching mode to the second presentation timeline associated with the second visual media component; a module for decoding from the bitstream whether the switching mode is about the first presentation timeline or about the second presentation timeline. The module includes a processor, a memory, and computer program code residing in the memory, wherein the processor may further include a processor circuit.

[0301] exist Fig. 9 An example of a data processing system for an apparatus is shown in . Several functions may be performed with a single physical device, for example, all computing processes may be performed in a single processor if desired. The data processing system includes a main processing unit 100, a memory 102, a storage device 104, an input device 106, an output device 108, and a graphics subsystem 110, all of which are connected to each other via a data bus 112.

[0302] The main processing unit 100 is a conventional processing unit arranged to process data within a data processing system. The main processing unit 100 may include or be implemented as one or more processors or processor circuits. The memory 102, storage device 104, input device 106, and output device 108 may include conventional components known to those skilled in the art. The memory 102 and storage device 104 store data in the data processing system 100.

[0303] Computer program code resides in the memory 102 for implementing, for example, Figure 7 or Figure 8 The method of the flowchart of the embodiment of the present invention is shown in Figure 1. The input device 106 inputs data into the system, and the output device 108 receives data from the data processing system and forwards the data, for example, to a display. The data bus 112 is a conventional data bus, and although shown as a single line, it can be any combination of the following: processor bus, PCI bus, graphics bus, ISA bus. Therefore, the technician will easily recognize that the device can be any data processing device, such as a computer device, a personal computer, a server computer, a mobile phone, a smart phone or an Internet access device (such as an Internet tablet computer).

[0304] Various embodiments of the present invention may be implemented with the aid of a computer program code that resides in a memory and causes an associated device to perform the present invention. For example, a device may include circuits and electronic devices for processing, receiving, and sending data, computer program code in a memory, and a processor that causes the device to perform the features of the embodiments when the computer program code is run. In addition, a network device such as a server may include circuits and electronic devices for processing, receiving, and sending data, computer program code in a memory, and a processor that causes the network device to perform the features of the embodiments when the computer program code is run.

[0305] If desired, the different functions discussed herein may be performed in a different order and / or simultaneously with others. In addition, if desired, one or more of the above-described functions and embodiments may be optional or may be combined.

[0306] Although various aspects of the embodiment are set out in the independent claims, further aspects include other combinations of features from the described embodiments and / or the independent claims with features from the dependent claims, not just the combinations explicitly set out in the claims.

[0307] In the above, some embodiments have been described with respect to DASH or MPEG-DASH. It should be understood that the embodiments can be similarly implemented with any other similar streaming system and / or protocols similar to those used in DASH and / or any segmentation and / or manifest formats similar to those used in DASH and / or any client operations similar to DASH clients. For example, some embodiments can be implemented with the M3U manifest format. In addition, the embodiments are not limited to media descriptions for streaming, but are also applicable to other types of media applications (e.g., conferencing). For example, the IETF SDP protocol can be used as a media description to implement the embodiments.

[0308] In the above, some embodiments have been described with respect to ISOBMFF. It should be understood that the embodiments may be similarly implemented using any other file format such as Matroska.

[0309] It should also be noted herein that although the above describes example embodiments, these descriptions should not be construed in a limiting sense. Rather, various variations and modifications may be made without departing from the scope of the present disclosure as defined in the appended claims.< / maxoccurs> < / minoccurs>

Claims

1. A method for virtual reality, comprising: generating a bitstream defining a presentation, the presentation comprising omnidirectional visual media content and a first visual media component and a second visual media component; indicating in the bitstream a first presentation timeline associated with the first visual media component; indicating in the bitstream a second presentation timeline associated with the second visual media component; indicating in the bitstream a switching mode regarding the first presentation timeline associated with the first visual media component, or a switching mode regarding the second presentation timeline associated with the second visual media component, the switching mode being indicated depending on a viewpoint of a user viewing the presentation; wherein the switching mode provides an indication of switching to the first visual media component or switching to the second visual media component, the first visual media component corresponding to content captured from at least a first omnidirectional camera at a first position, and the second visual media component corresponding to content captured from at least a second omnidirectional camera at a second position; as well as A variable time offset is indicated in the bitstream, the time offset specifying a time offset to be used when switching to a view identifier.

2. The method according to claim 1, wherein: Switching from the first visual media component to the second visual media component comprises at least one of the following switching: switching from a first overlay to a second overlay, wherein both the first overlay and the second overlay are within the same viewpoint; switching from a first overlay to a second overlay, wherein the first overlay and the second overlay are both in different viewpoints; Switching from a first viewpoint to a second viewpoint; Switching from a first overlay in a first viewpoint to a second viewpoint; or Switch from the first viewpoint to a second overlay in a second viewpoint.

3. The method according to claim 1, further comprising: Indicates the activation of switching from one viewpoint to another with or without an overlay.

4. The method according to claim 3, further comprising: The switch from one viewpoint to another viewpoint is indicated within a timed metadata track of the overlay, the timed metadata track comprising a sample entry and a sample.

5. The method according to claim 1, further comprising: When an overlay is not used, activation of a switch from the first visual media component to the second visual media component is indicated independent of the type of device output display.

6. The method according to claim 1, wherein: The switching comprises a spatial switch from a first region in a first viewpoint to a second region in a second viewpoint.

7. The method according to claim 1, wherein: The switching includes a timeline switch from a first time in a first viewpoint to a second time in a second viewpoint, wherein the second time is less than the first time, the second time is equal to the first time, or the second time is greater than the first time.

8. The method according to claim 7, further comprising: The timeline switch is indicated within an overlay timed metadata track, the timeline switch comprising a sample entry and a sample.

9. A non-transitory computer-readable medium comprising program instructions stored thereon, the program instructions being configured to, when executed by at least one processor, cause the at least one processor to perform the method according to claim 1.

10. A method for virtual reality, comprising: determining a presentation from the bitstream, the presentation comprising omnidirectional visual media content and a first visual media component and a second visual media component; decoding a first presentation timeline associated with the first visual media component from the bitstream; decoding a second presentation timeline associated with the second visual media component from the bitstream; decoding from the bitstream a switching mode regarding the first presentation timeline associated with the first visual media component, or a switching mode regarding the second presentation timeline associated with the second visual media component, the switching mode being indicated depending on a viewpoint of a user viewing the presentation; wherein the switching mode provides an indication of switching to the first visual media component or switching to the second visual media component, the first visual media component corresponding to content captured from at least a first omnidirectional camera at a first position, and the second visual media component corresponding to content captured from at least a second omnidirectional camera at a second position; as well as A variable time offset is decoded from the bitstream, the time offset specifying a time offset to be used when switching to a view identifier.

11. The method according to claim 10, wherein: Switching from the first visual media component to the second visual media component comprises at least one of the following switching: switching from a first overlay to a second overlay, wherein both the first overlay and the second overlay are within the same viewpoint; switching from a first overlay to a second overlay, wherein the first overlay and the second overlay are both in different viewpoints; Switching from a first viewpoint to a second viewpoint; Switching from a first overlay in a first viewpoint to a second viewpoint; or Switch from the first viewpoint to a second overlay in a second viewpoint.

12. The method according to claim 10, further comprising: Activates switching from one viewpoint to another with or without overlays.

13. The method according to claim 12, further comprising: The switch from one view to another view is decoded from the overlay timed metadata track, the timed metadata track comprising sample entries and samples.

14. The method according to claim 10, further comprising: When an overlay is not used, switching from the first visual media component to the second visual media component is activated independently of the type of device output display.

15. The method according to claim 10, wherein: The switching comprises a spatial switch from a first region in a first viewpoint to a second region in a second viewpoint.

16. The method according to claim 10, wherein: The switching includes a timeline switch from a first time in a first viewpoint to a second time in a second viewpoint, wherein the second time is less than the first time, the second time is equal to the first time, or the second time is greater than the first time.

17. The method according to claim 16, further comprising: The timeline switch is decoded from an overlying timed metadata track, the timeline switch comprising a sample entry and a sample.

18. A non-transitory computer readable medium comprising program instructions stored thereon, the program instructions being configured to, when executed on at least one processor, cause the at least one processor to perform the method according to claim 10.

19. A device for virtual reality, comprising: at least one processor; as well as at least one non-transitory memory comprising computer program code; The at least one non-transitory memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to at least perform the following operations: generating a bitstream defining a presentation, the presentation comprising omnidirectional visual media content and a first visual media component and a second visual media component; indicating in the bitstream a first presentation timeline associated with the first visual media component; indicating in the bitstream a second presentation timeline associated with the second visual media component; indicating in the bitstream a switching mode regarding the first presentation timeline associated with the first visual media component, or a switching mode regarding the second presentation timeline associated with the second visual media component, the switching mode being indicated depending on a viewpoint of a user viewing the presentation; wherein the switching mode provides an indication of switching to the first visual media component or to the second visual media component, the first visual media component corresponding to content captured from at least a first omnidirectional camera at a first position, the second visual media component corresponding to content captured from at least a second omnidirectional camera at a second position; and A variable time offset is indicated in the bitstream, the time offset specifying a time offset to be used when switching to a view identifier.

20. A device for virtual reality, comprising: at least one processor; as well as at least one non-transitory memory comprising computer program code; The at least one non-transitory memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to at least perform the following operations: determining a presentation from the bitstream, the presentation comprising omnidirectional visual media content and a first visual media component and a second visual media component; decoding a first presentation timeline associated with the first visual media component from the bitstream; decoding a second presentation timeline associated with the second visual media component from the bitstream; decoding from the bitstream a switching mode regarding the first presentation timeline associated with the first visual media component, or a switching mode regarding the second presentation timeline associated with the second visual media component, the switching mode being indicated depending on a viewpoint of a user viewing the presentation; wherein the switching mode provides an indication of switching to the first visual media component or to the second visual media component, the first visual media component corresponding to content captured from at least a first omnidirectional camera at a first position, the second visual media component corresponding to content captured from at least a second omnidirectional camera at a second position; and A variable time offset is decoded from the bitstream, the time offset specifying a time offset to be used when switching to a view identifier.

Images