Method, apparatus and computer program product for video encoding and video decoding

Abstract

The embodiment relates to a method, which includes writing a first media entity (410) and a second media entity (420) in a container file; creating (430) a media presentation description (MPD) having a first representation and a second representation; the representation belongs to an adaptation set; the representation is associated with a media entity of the container file; when one of the representations belongs to a media entity that is a thumbnail of a viewpoint or a thumbnail of an overlay, the method includes writing (440) in the MPD file association / correspondence / grouping information of the above one representation and another representation that belongs to the media entity that is a viewpoint or an overlay. The embodiment also relates to a method for parsing and a technical device for implementing the method.

Description

Technical Field

[0001] The present solution generally relates to video encoding and video decoding. Specifically, the solution relates to image signaling in a media presentation description file. Background Art

[0002] Since the beginning of photography and filmmaking, the most common types of image and video content have been captured by cameras with relatively narrow fields of view and displayed as rectangular scenes on flat displays. Such content is referred to in this application as "flat content" or "flat images" or "flat video". Cameras are primarily directional, so they only capture a limited angular field of view (the field of view they are pointing towards).

[0003] Recently, new image and video capture devices have become available. These devices are able to capture the visual and audio content around them, i.e., they can capture the entire angular field of view, sometimes referred to as a 360-degree field of view. More precisely, they can capture a spherical field of view (i.e., 360 degrees in all spatial directions). In addition, new output technologies have been invented and produced, such as head-mounted displays. These devices allow people to see the visual content around him / her to give people a sense of "immersion" in a scene captured by a 360-degree camera. The new capture and display paradigm (in which the field of view is spherical) is often referred to as virtual reality (VR), and is considered to be a common way for people to experience media content in the future. Summary of the invention

[0004] The scope of protection sought by various embodiments of the present invention is defined by the independent claims. Embodiments and features described in this specification that do not belong to the scope of the independent claims, if any, are to be interpreted as examples useful for understanding various embodiments of the present invention.

[0005] Various aspects include a method, an apparatus and a computer readable medium comprising a computer program stored therein, which are characterized by what is stated in the independent claims. Various embodiments are disclosed in the dependent claims.

[0006] According to a first aspect, a method is provided, the method comprising writing a first media entity in a container file; writing a second media entity in a container file; wherein at least one of the first media entity or the second media entity is an image, wherein when one of the media entities is an image, it is a viewpoint or an overlay or a cover image or a thumbnail of a viewpoint, or a thumbnail of an overlay, creating a media presentation description (MPD) having a first representation and a second representation; the first representation belongs to a first adaptation set, and the second representation belongs to a second adaptation set; the representation of the first adaptation set is associated with the first media entity of the container file; and the representation of the second adaptation set is associated with the second media entity of the container file; when one of the representations belongs to a media entity that is a thumbnail of a viewpoint or a thumbnail of an overlay, correspondingly writing in the MPD file association / correspondence / grouping information of the representation with another representation belonging to the media entity that is a viewpoint or an overlay.

[0007] According to a second aspect, a method is provided, which includes parsing from a media presentation description (MPD) a first representation belonging to a first adaptation set and a second representation belonging to a second adaptation set; the first representation of the first adaptation set is associated with a first media entity of a container file; the second representation of the second adaptation set is associated with a second media entity of the container file; wherein at least one representation item in the first representation or the second representation is associated with an image; when one of the representations is associated with an image, it is a viewpoint representation or an overlay representation or a cover image representation or a representation of a thumbnail of a viewpoint; or it is a representation of a thumbnail of an overlay; parsing association / correspondence / grouping information of the representations from the MPD; selecting the first representation of the first adaptation set associated with the first media entity of the container file; and selecting the second representation of the second adaptation set associated with the second media entity of the container file.

[0008] According to a third aspect, a device is provided, which comprises at least: a component for writing a first media entity in a container file; a component for writing a second media entity in a container file; wherein at least one of the first media entity or the second media entity is an image, wherein when one of the media entities is an image, it is a viewpoint or an overlay or a cover image or a thumbnail of a viewpoint, or a thumbnail of an overlay; a component for creating a media presentation description (MPD) having a first representation and a second representation; the first representation belongs to a first adaptation set, and the second representation belongs to a second adaptation set; the representation of the first adaptation set is associated with the first media entity of the container file; the representation of the second adaptation set is associated with the second media entity of the container file; when one of the representations belongs to a media entity that is a thumbnail of a viewpoint or a thumbnail of an overlay, a component for correspondingly writing in the MPD file association / correspondence / grouping information of the representation with another representation belonging to the media entity that is a viewpoint or an overlay.

[0009] According to a fourth aspect, a device is provided, which includes at least a component for parsing a first representation belonging to a first adaptation set and a second representation belonging to a second adaptation set from a media presentation description (MPD); the first representation of the first adaptation set is associated with a first media entity of a container file; the second representation of the second adaptation set is associated with a second media entity of the container file; wherein at least one of the first representation or the second representation is associated with an image; when one of the representations is associated with an image, it is a viewpoint representation or an overlay representation or a cover image representation or a thumbnail representation of a viewpoint; or a thumbnail representation of an overlay; a component for parsing association / correspondence / grouping information of the representations from the MPD; a component for selecting a first representation of the first adaptation set associated with the first media entity of the container file; a component for selecting a second representation of the second adaptation set associated with the second media entity of the container file.

[0010] According to one embodiment, the first media entity is one of: an image file; and a project; or a media track.

[0011] According to one embodiment, the second media entity is one of: an image file; and a project; or a media track.

[0012] According to a fourth aspect, a computer program product comprising a computer program code is provided, which is configured to cause an apparatus or system, when executed on at least one processor, to: write a first media entity in a container file; write a second media entity in a container file; wherein at least one of the first media entity or the second media entity is an image, wherein when one of the media entities is an image, it is a viewpoint or an overlay or a cover image or a thumbnail of a viewpoint, or a thumbnail of an overlay, to create a media presentation description (MPD) having a first representation and a second representation; the first representation belongs to a first adaptation set, and the second representation belongs to a second adaptation set; the representation of the first adaptation set is associated with the first media entity of the container file; and the representation of the second adaptation set is associated with the second media entity of the container file; when one of the representations belongs to a media entity that is a thumbnail of a viewpoint or a thumbnail of an overlay, the association / correspondence / grouping information of the representation and another representation belonging to the media entity that is a viewpoint or an overlay is correspondingly written in the MPD file.

[0013] According to a fifth aspect, a computer program product comprising a computer program code is provided, which is configured to cause an apparatus or system, when executed on at least one processor: to parse from a media presentation description (MPD), a first representation belonging to a first adaptation set and a second representation belonging to a second adaptation set; the first representation of the first adaptation set is associated with a first media entity of a container file; the second representation of the second adaptation set is associated with a second media entity of the container file; wherein at least one of the first representation or the second representation is associated with an image; when one of the representations is associated with an image, it is a viewpoint representation or an overlay representation or a cover image representation or a representation of a thumbnail of a viewpoint; or a representation of a thumbnail of an overlay; to parse association / correspondence / grouping information of the representations from the MPD; to select a first representation of the first adaptation set associated with the first media entity of the container file; and to select a second representation of the second adaptation set associated with the second media entity of the container file.

[0014] According to one embodiment, a computer program product is embodied on a non-transitory computer readable medium.

[0015] According to a sixth aspect, a device is provided, which includes at least one processor and a memory including computer program code, and the memory and the computer program code are configured to, together with the at least one processor, cause the device to perform at least the following operations: write a first media entity in a container file; write a second media entity in a container file; wherein at least one of the first media entity or the second media entity is an image, wherein when one of the media entities is an image, it is a viewpoint or an overlay or a cover image or a thumbnail of a viewpoint, or a thumbnail of an overlay, and a media presentation description (MPD) having a first representation and a second representation is created; the first representation belongs to a first adaptation set, and the second representation belongs to a second adaptation set; the representation of the first adaptation set is associated with the first media entity of the container file; and the representation of the second adaptation set is associated with the second media entity of the container file; when one of the representations belongs to a media entity that is a thumbnail of a viewpoint or a thumbnail of an overlay, the association / correspondence / grouping information of the representation and another representation belonging to the media entity that is a viewpoint or an overlay is correspondingly written in the MPD file.

[0016] According to one embodiment, the first media entity is one of: an image file; and a project; or a media track.

[0017] According to one embodiment, the second media entity is one of: an image file; and a project; or a media track.

[0018] According to the seventh aspect, a device is provided, which includes at least one processor and a memory including computer program code, and the memory and the computer program code are configured to, together with the at least one processor, enable the device to perform at least the following operations: parse from a media presentation description (MPD), a first representation belonging to a first adaptation set and a second representation belonging to a second adaptation set; the first representation of the first adaptation set is associated with a first media entity of a container file; the second representation of the second adaptation set is associated with a second media entity of the container file; wherein at least one of the first representation or the second representation is associated with an image; when one of the representations is associated with an image, it is a viewpoint representation or an overlay representation or a cover image representation or a thumbnail representation of a viewpoint; or a thumbnail representation of an overlay; parse the association / correspondence / grouping information of the representations from the MPD; select the first representation of the first adaptation set associated with the first media entity of the container file; and select the second representation of the second adaptation set associated with the second media entity of the container file.

[0019] According to one embodiment, the first media entity is one of: an image file; and a project; or a media track.

[0020] According to one embodiment, the second media entity is one of: an image file; and a project; or a media track. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Hereinafter, various embodiments will be described in more detail with reference to the accompanying drawings, in which

[0022] Figure 1 An example of OMAF system architecture is shown;

[0023] Figure 2 An apparatus for content creation according to one embodiment is shown;

[0024] Figure 3 An apparatus for content consumption according to one embodiment is shown;

[0025] Figure 4 is a flow chart illustrating a method according to one embodiment; and

[0026] Figure 5 is a flow chart illustrating a method according to another embodiment. DETAILED DESCRIPTION

[0027] This embodiment relates to video encoding and decoding.

[0028] Available media file format standards include the International Organization for Standardization (ISO) Base Media File Format (ISO / IEC14496-12, which may be abbreviated as ISOBMFF), the Moving Picture Experts Group (MPEG)-4 file format (ISO / IEC 14496-14, also known as the MP4 format), and the NAL (Network Abstraction Layer) unit structured video file format (ISO / IEC 14496-15).

[0029] Some concepts, structures and specifications of ISOBMFF are described below as an example of a container file format, based on which some embodiments can be implemented. Aspects of the present disclosure are not limited to ISOBMFF, but are described for one possible basis on which at least some embodiments can be partially or completely implemented.

[0030] The basic building block in the ISO base media file format is called a box. Each box has a header and a payload. The box header is in bytes and indicates the type of box and the size of the box. The box type is typically identified by an unsigned 32-bit integer, interpreted as a four-character code (4CC). Boxes can enclose other boxes, and the ISO file format specifies which box types are allowed within a particular type of box. Furthermore, the presence of certain boxes in each file may be mandatory, while the presence of other boxes may be optional. Furthermore, for some box types, more than one box may be allowed to be present in a file. Therefore, the ISO base media file format can be considered to specify a hierarchy of boxes.

[0031] In a file conforming to the ISO base media file format, media data may be provided in one or more instances of MediaDataBox ("mdat"), and MovieBox ("moov") may be used to encapsulate metadata for timed media. In some cases, both "mdat" and "moov" boxes may need to be present in order for a file to be operable. A "moov" box may include one or more tracks, and each track may reside in a corresponding TrackBox ("trak"). Each track is associated with a handler, which is identified by a four-character code that specifies the track type. Video, audio, and image sequence tracks may be collectively referred to as media tracks, and they contain elementary media streams. Other track types include hint tracks and timed metadata tracks.

[0032] A track consists of samples, such as audio or video frames. For video tracks, media samples may correspond to coded pictures or access units.

[0033] A media track refers to samples (which may also be referred to as media samples) formatted according to a media compression format (and its encapsulation of the ISO base media file format). A hint track refers to hint samples, which contain codebook instructions for building packets for transmission over the indicated communication protocol. A timed metadata track may refer to samples describing the referenced media and / or hint samples.

[0034] The "trak" box includes a SampleDescriptionBox in its box hierarchy that provides details about the type of encoding used, as well as any initialization information required by that encoding. The SampleDescriptionBox contains an entry count and as many sample entries as the entry count indicates. The format of the sample entry is track type specific, but derives from the generic classes (e.g., VisualSampleEntry, AudioSampleEntry). Which type of sample entry form is used to derive a track type specific sample entry format is determined by the media handler for the track.

[0035] A track reference mechanism can be used to associate tracks with each other. A TrackReferenceBox contains box(es), each of which provides a reference from a containing track to a set of other tracks. These references are labeled by the box type of the containing box(es) (e.g., the four-character code of the box).

[0036] The ISO base media file format contains three mechanisms for timed metadata that can be associated with a particular sample: sample groups, timed metadata tracks, and sample auxiliary information. An export specification may provide similar functionality through one or more of these three mechanisms.

[0037] Sample grouping in the ISO base media file format and its derived files (such as the Advanced Video Coding (AVC) file format and the Scalable Video Coding (SVC) file format) can be defined as assigning each sample in a track as a member of a sample group based on a grouping criterion. Sample groups in a sample grouping are not limited to consecutive samples, but can also contain non-adjacent samples. Since samples in a track may have more than one sample grouping, each sample grouping can have a type field to indicate the type of grouping. Sample grouping can be represented by two linked data structures: (1) SampleToGroupBox ("sbgp" box) represents the assignment of samples to sample groups; and (2) SampleGroupDescriptionBox ("sgpd" box) contains a sample group entry for each sample group that describes the properties of the group. SampleToGroupBox and SampleGroupDescriptionBox can have multiple instances based on different grouping criteria. These can be distinguished by the type field used to indicate the type of grouping. SampleToGroupBox may include a grouping_type_parameter field, which can be used, for example, to indicate the subtype of the grouping.

[0038] In ISOMBFF, the edit list provides a mapping between the presentation timeline and the media timeline. Among other things, the edit list provides a linear offset for sample presentation in a track, provides an indication of free time, and provides specific samples to dwell on for a specific period of time. The presentation timeline can be modified accordingly to provide loops, such as looping video for various areas of a scene. An example of a box that includes an edit list is provided below, the EditListBox is provided as follows:

[0039]

[0040] In ISOBMFF, an EditListBox can be contained in an EditBox, which is contained in a TrackBox ("trak").

[0041] In this example of an edit list box, flags specifies repetition of the edit list. For example, setting a particular bit (the least significant bit, i.e., flags&1 in the ANSI-C representation, where & represents a bitwise AND operation) within the box flags to 0 specifies that the edit list is not repeated, while setting a particular bit (i.e., flags&1 in the ANSI-C representation) to 1 specifies that the edit list is repeated. Values ​​of the box flags greater than 1 can be defined as being reserved for future expansion. Therefore, when the edit list box indicates playback of zero or one samples, (flags&1) should be equal to zero. When an edit list is repeated, the media at time 0 generated by the edit list immediately follows the media with the maximum time generated by the edit list, thereby seamlessly repeating the edit list.

[0042] In ISOBMFF, track groups can group tracks based on certain characteristics, or the tracks within a group have a specific relationship. However, track grouping does not allow any image items in the group.

[0043] The syntax of TrackGroupBox in ISOBMFF is as follows:

[0044]

[0045] track_group_type indicates the grouping_type, and should be set to one of the following values, or a registered value, or a value from a derived specification or registration:

[0046] 'msrc' indicates that the track belongs to a multi-source presentation. Tracks with the same track_group_id value within a TrackGroupTypeBox of track_group_type 'msrc' are mapped as originating from the same source. For example, a recording of a video phone call may have both audio and video from two participants, and the track_group_id values ​​associated with the audio and video tracks of one participant are different from the track_group_id values ​​associated with the tracks of the other participant.

[0047] A track_group_id and track_group_type pair identifies a track group in a file. Tracks containing a specific TrackGroupTypeBox with the same track_group_id and track_group_type values ​​belong to the same track group.

[0048] Entity grouping is similar to track grouping, but supports grouping track and image items in the same group.

[0049] The syntax of EntityToGroupBox in ISOBMFF is as follows.

[0050]

[0051] group_id is a non-negative integer assigned to a specific grouping and should not be equal to any group_id value of any other EntityToGroupBox, any item_ID value of the hierarchy level (file, movie or track) containing the GroupsListBox, or any track_ID value (when the GroupsListBox is contained at the file level).

[0052] num_entities_in_group specifies the number of entity_id values ​​that are mapped to this entity group.

[0053] When an item with item_ID equal to entity_id exists in the hierarchy level (file, movie or track) containing the GroupsListBox, entity_id is resolved to item, or when a track with track_ID equal to entity_id exists and the GroupsListBox is contained at the file level, entity_id is resolved to track.

[0054] Files conforming to ISOBMFF may contain any non-timed object, called an item, meta-item, or metadata item, in a meta-box (four-character code: "meta"). Although the name of the meta-box refers to metadata, an item may contain metadata or media data in general. Meta-boxes may be located at the top level of a file, within a movie box (four-character code: "moov"), and within a track box (four-character code: "trak"), but at most one meta-box may appear at each file level, movie level, or track level. A meta-box may need to contain an "hdlr" box to indicate the structure or format of the contents of the "meta" box. A meta-box may list and characterize any number of items that may be referenced, and each item may be associated with a file name and uniquely identified with the file by an item identifier (item_id), which is an integer value. Metadata items may be stored, for example, in an "idat" box of a metadata box or in an "mdat" box or reside in a separate file. If the metadata is located outside the file, its location may be declared by a DataInformationBox (four-character code: "dinf"). In the specific case where metadata is formatted using Extensible Markup Language (XML) syntax and needs to be stored directly in the MetaBox, the metadata can be encapsulated into an XMLBox (four-character code: "xml") or a BinaryXMLBox (four-character code: "bxml"). Items can be stored as contiguous byte ranges, or can be stored in several ranges, each of which is a contiguous byte range. In other words, items can be stored in segments as ranges, for example to enable interleaving. A range is a contiguous subset of a resource's bytes. Resources can be formed by concatenating ranges.

[0055] ItemPropertiesBox can associate any item with an ordered set of item properties. Item properties can be viewed as small data records. ItemPropertiesBox consists of two parts: an ItemPropertyContainerBox that contains an implicitly indexed list of item properties, and one or more ItemPropertyAssociationBoxes that associate items with item properties.

[0056] High Efficiency Image File Format (HEIF) is a standard developed by the Moving Picture Experts Group (MPEG) for storing images and image sequences. Among other things, the standard facilitates file encapsulation of data encoded according to the High Efficiency Video Coding (HEVC) standard. HEIF includes features built on top of the ISO Base Media File Format (ISOBMFF) used.

[0057] The ISOBMFF structure and features are largely used in the design of HEIF. The basic design of HEIF includes still images stored as items and image sequences stored as tracks.

[0058] In the context of HEIF, the following boxes may be included within the root-level "meta" box and may be used as described below. In HEIF, the handler value for the Handler box of the "meta" box is "pict". Resources containing coded media data (whether within the same file, or in an external file identified by a uniform resource identifier) ​​are resolved via a data information ("dinf") box, while an item location ("iloc") box stores the location and size of each item in the referenced file. The item reference ("iref") box records the relationship between items using typed references. If there is an item in a set of items that is considered to be most important in some way than the others, that item is signaled by the primary item ("pitm") box. In addition to the boxes mentioned here, the "meta" box has the flexibility to include other boxes as needed to describe the item.

[0059] Any number of image items may be included in the same file. Given a collection of images stored using the "meta" box method, it may be important to define certain relationships between the images. Examples of such relationships include indicating a cover image for the collection, providing thumbnails for some or all images in a collection, and associating some or all images in a collection with auxiliary images such as alpha planes. A cover image in a collection of images is indicated using the "pitm" box. Thumbnails or auxiliary images are linked to the main image item using an item reference of type "thmb" or "auxl", respectively.

[0060] Users can consume videos and images as visual content. However, the consumption of videos and images has been independent of each other. The development of recent applications (such as immersive multimedia) has enabled new use cases where users consume videos and images simultaneously.

[0061] Compared to the consumption of 2D content, immersive multimedia (such as omnidirectional content consumption) is more complex for the end user. This is due to the higher degree of freedom available to the end user. This freedom also leads to more uncertainty. The situation is further complicated when rendering content layers, such as in the case of overlays.

[0062] As used herein, the term omnidirectional may refer to media content that has a larger spatial extent than the field of view of the device rendering the content. Omnidirectional content may, for example, cover substantially 360 degrees in the horizontal dimension and substantially 180 degrees in the vertical dimension, but omnidirectional may also refer to content that covers less than a 360-degree view in the horizontal direction and / or a 180-degree view in the vertical direction.

[0063] Panoramic images covering a 360 degree horizontal field of view and a 180 degree vertical field of view can be represented by a sphere that has been mapped to a two-dimensional image plane using, for example, an equirectangular projection (ERP). In the case of ERP, the horizontal coordinate can be considered equivalent to the longitude of the sphere, while the vertical coordinate can be considered equivalent to the latitude, without any transformation or scaling applied. In some cases, panoramic content with a 360 degree horizontal field of view but a vertical field of view less than 180 degrees can be considered as a special case of an equirectangular projection, where the polar regions of the sphere are not mapped onto the two-dimensional image plane. In some cases, panoramic content can have a horizontal field of view less than 360 degrees and a vertical field of view of up to 180 degrees, while otherwise having the characteristics of an equirectangular projection format.

[0064] MPEG Omnidirectional Media Format (OMAF) v1 standardizes omnidirectional streaming of single 3DoF content (where the viewer is located at the center of a unit sphere and has three degrees of freedom (yaw-pitch-roll). The next phase of standardization (MPEG-I Phase 1b) is in progress. It is expected to enable multiple 3DoF and 3DoF+ content consumption (viewpoints) and overlay support for interaction with the user.

[0065] In the cube map projection format, spherical video is projected onto the six faces (also called sides) of a cube. A cube map can be generated, for example, by first rendering a spherical scene six times from a viewpoint defined by a 90 degree frustum representing each cube face. The cube sides can be frame packed into the same frame, or each cube side can be processed separately (e.g., in encoding). There are many possible orders in which the cube sides can be positioned onto the frames, and / or the cube sides can be rotated or mirrored. The frame width and height used for frame packing can be chosen to "tightly" fit the cube sides, such as in a 3×2 cube side grid, or can include unused constituent frames, such as in a 4×3 cube side grid.

[0066] In general, 360-degree content can be mapped onto different types of solid geometric structures, such as polyhedra (i.e., three-dimensional solid objects containing planar polygonal faces, straight edges and sharp corners or vertices, such as cubes or pyramids), cylinders (by projecting a spherical image onto a cylinder, as described above using equirectangular projection), cylinders (without first projecting directly onto a sphere), cones, etc., and then unfolded to a two-dimensional image plane. The two-dimensional image plane can also be considered as a geometric structure. In other words, 360-degree content can be mapped to a first geometric structure and further unfolded to a second geometric structure. However, the transformation to the second geometric structure can be obtained directly from the original 360-degree content or from other wide-angle visual content. In general, an omnidirectional projection format can be defined as a format for representing (up to) 360-degree content on a two-dimensional image plane. Examples of omnidirectional projection formats include equirectangular projection formats and cubemap projection formats.

[0067] OMAF defines a format for enabling omnidirectional media access and delivery. Media components are distributed across different bitstreams (e.g., multiple resolutions, bitrates / quality) to provide applications with the freedom to choose between them to address various system challenges, such as network bandwidth, temporal and spatial random access for user interaction.

[0068] The currently standardized Omnidirectional Media Format (OMAF) v2 supports the use of multiple omnidirectional and overlay layers for video and images. Support for the delivery of image content is required.

[0069] Figure 1 An example of the OMAF system architecture is shown. Figure 1 As shown, omnidirectional media (A) is acquired. The omnidirectional media includes separately processed image data (B i ) and audio data (B a ).

[0070] In image stitching, rotation, projection and region packing, as input (B i)The images / videos of the source media provided are stitched to generate a spherical image on a unit sphere according to the global coordinate axes. The unit sphere is then rotated relative to the global coordinate axes. The amount of rotation from the local coordinate axes to the global coordinate axes can be specified by the rotation angle indicated in the RotationBox. The local coordinate axes of the unit sphere are the axes of the coordinate system that have been rotated. The absence of a RotationBox indicates that the local coordinate axes are the same as the global coordinate axes. The spherical image on the rotated unit sphere is then converted into a two-dimensional projection image, for example using an equirectangular projection. When spatial packing of stereoscopic content is applied, the two spherical images of the two views are converted into two component images, after which frame packing is applied to pack the two component images into one projection image. Rectangular region packing can then be applied to obtain a packed image from the projection image. The packed image (D) is then provided for video and image encoding to produce an encoded image (E i ) and / or encoded video stream (E v ).

[0071] The source media's audio is used as input (B a ) is provided to an audio codec, which provides an encoded audio (E a Then, the encoded data (E i 、E v 、E a ) is packaged into files for playback (F) and delivery (i.e., streaming) (F s ).

[0072] In the OMAF player 200, the file decapsulator processes the files (F', F's) and extracts the encoded bitstream (E' i , E' v , E' a ), and parses metadata. Then, the audio, video and / or image is decoded into decoded data (D', B' a ). The decoded picture (D') is projected onto the display according to the viewpoint and orientation sensed by the head / eye tracking device. Similarly, the decoded audio (B' a ) is rendered through the speakers / headphones.

[0073] A viewport can be defined as an omnidirectional image or video area suitable for display and viewing by a user. The current viewport (sometimes simply referred to as the viewport) can be defined as the portion of the spherical video that is currently displayed and therefore viewable by (multiple) users. At any point in time, the video rendered by the application on a head-mounted display (HMD) renders a portion of the 360-degree video, which is called the viewport. Similarly, when viewing a spatial portion of 360-degree content on a conventional display, the spatial portion currently displayed is the viewport. The viewport is a window on the 360-degree world represented in the omnidirectional video displayed via the rendering display. The viewport can be characterized by a horizontal field of view (VHFoV) and a vertical field of view (VVFoV). In the following, the horizontal field of view of the viewport will be abbreviated as HFoV, and the vertical field of view of the viewport will be abbreviated as VFoV.

[0074] A spheroid region can be defined as an area on a sphere that can be specified by four great circles or two azimuth circles and two elevation circles, and additionally by a tilt angle that indicates the rotation about an axis from the sphere origin through the center point of the spheroid region. A great circle can be defined as the intersection of the sphere with a plane through the center point of the sphere. Great circles are also called orthogonal circles or Riemann circles. An azimuth circle can be defined as a circle on the sphere connecting all points with the same azimuth value. An altitude circle can be defined as a circle on the sphere connecting all points with the same altitude value.

[0075] The Omnidirectional Media Format ("OMAF") standard (ISO / IEC 23090-2) specifies a common timed metadata syntax for spherical regions. The purpose of a timed metadata track is indicated by the track sample entry type. The sample format of all metadata tracks for a specified spherical region starts with a common section and may be followed by an extension section of sample entries specific to the metadata track. Each sample specifies a spherical region.

[0076] One of the specific sphere region timed metadata tracks specified in OMAF is called the recommended viewport timed metadata track, which indicates the viewport that should be displayed when the user cannot control the viewing orientation or has released control of the viewing orientation. The recommended viewport timed metadata track can be used to indicate a recommended viewport based on a "director's cut" or based on measurements of viewing statistics. A text description of the recommended viewport can be provided in the example entry. The type of recommended viewport can be indicated in the example entry and can be one of the following:

[0077] a. a recommended viewport for each Director's Cut, e.g., a viewport suggested based on the creative intent of the content author or content provider;

[0078] b. A recommended viewport selected based on measurement of viewing statistics;

[0079] c. Defined by the application or external specification.

[0080] The viewpoint or observation point is defined as the point from which the user looks at the scene; it usually corresponds to the camera position. Slight head movements do not imply a different viewpoint.

[0081] As used herein, the term "observation point or viewpoint" refers to a volume in three-dimensional space used for virtual reality audio / video acquisition or playback. A viewpoint is a trajectory, such as a circle, area, or volume, around the center point of a device or equipment for omnidirectional audio / video acquisition and the position of the observer's head in three-dimensional space where the audio and video tracks are located. In some cases, the observer's head position is tracked, and the rendering is adjusted for head rotation and head movement, and the viewpoint can then be understood as the initial or reference position of the observer's head. In an implementation using DASH (Dynamic Adaptive Streaming over HTTP), each observation point can be defined as a viewpoint by a viewpoint attribute descriptor. The definition can be stored in a file format of type ISOBMFF or OMAF. In addition to DASH, the transmission can also be HLS (HTTP Live Streaming), RTSP / RTP (Real Time Streaming Protocol / Real Time Transport Protocol) streams.

[0082] As used herein, the term "viewpoint group" refers to one or more viewpoints that are spatially or logically related. The viewpoints in a viewpoint group can be defined based on the relative position defined for each viewpoint relative to a specified origin of the group. Each viewpoint group can also include a default viewpoint that reflects the default playback starting point when a user begins consuming audiovisual content in the viewpoint group without selecting a viewpoint for playback. The default viewpoint can be the same as the specified origin. In some embodiments, a viewpoint can be included in multiple viewpoint groups.

[0083] As used herein, the term "spatially related set of viewpoints" refers to viewpoints that have content with a spatial relationship between them. For example, content captured by a VR camera at different locations on the same basketball court, or a concert captured from different locations on a stage.

[0084] As used herein, the term "logically related viewpoint group" refers to related viewpoints that do not have an explicit spatial relationship but are logically related. The relative positions of logically related viewpoints are described based on the creative intent. For example, two viewpoints that are members of a logically related viewpoint group may correspond to content from a performance area and a locker room. Another example may be two viewpoints from the locker rooms of two competing teams, which form a logically related viewpoint group to allow a user to traverse between the two teams to see the reactions of the players.

[0085] As used herein, the term "static viewpoint" refers to a viewpoint that remains stationary during a virtual reality audio / video capture and playback session. For example, a static viewpoint may correspond to virtual reality audio / video capture performed by a fixed camera.

[0086] As used herein, the term "dynamic viewpoint" refers to a viewpoint that does not remain static during a VR audio / video capture and playback session. For example, a dynamic viewpoint may correspond to VR audio / video capture performed by a moving camera on a track or a moving camera on a flying drone.

[0087] As used herein, the term "viewing setup" refers to a setup of one or more viewpoints and viewing orientations. In the context of a presentation where only one viewpoint is available, there is no need to explicitly indicate or end a viewpoint for a viewing setup. If a presentation has multiple viewpoints available, the viewpoints will be set based on one or more viewpoint groups, and the spatial or logical relationship between the viewpoints in each viewpoint group will be indicated in the viewing setup.

[0088] The term "overlay" refers to the visual media rendered on top of the 360-degree video content.

[0089] Videos and / or images can be overlaid on omnidirectional videos and / or images. The encoded overlay video can be a separate stream or part of the bitstream of the currently rendered 360-degree video / image. The omnidirectional streaming system can overlay the video / image on top of the omnidirectional video / image being rendered. The overlaid two-dimensional video / image can have a rectangular grid or a non-rectangular grid. The overlay process can cover the overlaid video / image or a portion of the video / image, or there can be a certain level of transparency / opacity or more than one level of transparency / opacity, where the overlaid video / image can be seen below the overlaid video / image but with lower brightness. In other words, there may be relevant transparency levels corresponding to the video / image in the foreground overlay and the video / image in the background (the video / image of the VR scene). The terms opacity and transparency can be used interchangeably.

[0090] The coverage area may have one or more levels of transparency. For example, the coverage area may have different portions with different levels of transparency. According to one embodiment, the transparency level may be defined within a range, such as from 0 to 1, such that the lower the value, the lower the transparency, and vice versa.

[0091] In addition, the content provider can choose to overlay a portion of the same omnidirectional video on the user's current viewport. The content provider may want to overlay the video based on the user's viewing conditions. For example, if the user's viewport does not match the content provider recommended viewport, then overlaying can be performed. In this case, the client player logic overlays the content provider's recommended viewport (as a preview window) on the user's current viewport. If the user's current viewport does not match, the recommended viewport can also be overlaid so that the position of the overlaid video is based on the direction the user is watching. For example, if the recommended viewport is located on the left side of the user's current viewport, the recommended viewport is overlaid on the left side of the display. The entire 360-degree video can also be overlaid. Another example is to use the overlaid visual information as a guidance mechanism to guide the user towards the recommended viewport, such as guiding a hearing-impaired person.

[0092] There may be one or more conditions regarding when and how to display the visual overlay.Thus, the rendering device may need to receive information that the rendering device can use to perform the overlay indicated by the signaled information.

[0093] One or more overlays may be carried in a single visual media track or a single image item. When more than one overlay is carried in a single track or image item, or when overlays are carried along with other media (e.g., a background), a region mapping from samples of the track or image item to overlay metadata may be provided, such as in or associated with an OverlayStruct.

[0094] When several tracks or image items together carry one or more overlays and / or background visual media, a group of tracks and image items can be indicated in the container file. For example, the entity group of ISOBMFF can be used for this purpose.

[0095] The overlay may be located outside the user's field of view (FOV), i.e., the user's viewport does not overlap with the overlay. For example, after the user rotates during playback of omnidirectional media content, the user's viewport becomes non-overlapping with the visual overlay. Depending on the circumstances, playback of the overlay may need to be continued or paused when the user is not viewing the overlay. For example, the timeline of overlay playback may need to be paused until the overlay overlaps with the user's viewport again. Playback of the overlay may need to continue even if the overlay is outside the user's viewport. Therefore, a mechanism is needed to support multiple playback timelines, thereby supporting customized overlay playback / pause independent of the base content. Therefore, a method, apparatus, and computer program product are provided in accordance with example embodiments to enable multi-timeline support when playing back omnidirectional media content with an overlay, which in turn enables customized overlay playback behavior based on whether the overlay overlaps with the user's viewport.

[0096] The Matroska file format is capable of (but not limited to) storing any video, audio, picture, or subtitle track in one file. Matroska file extensions include .mkv for video (with subtitles and audio), .mk3d for stereoscopic video, .mka for audio-only files, and .mks for subtitles-only. Matroska can be used as the base format for derivative file formats such as WebM.

[0097] Matroska uses the Extensible Binary Meta Language (EBML) as its basis. EBML specifies a binary and octet (byte) aligned format inspired by XML principles. EBML itself is a generalized description of binary markup technology. Matroska files consist of elements that make up an EBML "document". An element contains an element ID, a descriptor of the element size, and the binary data itself. Elements can be nested.

[0098] The Segment element of Matroska is a container for other top-level (level 1) elements. A Matroska file may include (but is not limited to consisting of) a Segment. The multimedia data in a Matroska file is organized in clusters (or cluster elements), each cluster typically containing a few seconds of multimedia data. Clusters include Chunk Group elements, which in turn include Chunk elements. The Cues element includes metadata that can aid in random access or seeking, and may include a file pointer or corresponding timestamp for a seek point.

[0099] A Uniform Resource Identifier (URI) can be defined as a string that identifies a resource name. This identification enables interaction with a representation of a resource over a network using a specific protocol. URIs are defined by a scheme that specifies a specific syntax and associated protocols for URIs. Uniform Resource Locators (URLs) and Uniform Resource Names (URNs) are forms of URIs. A URL can be defined as a URI that identifies a network resource and specifies a way to act on or obtain a representation of a resource to specify both its primary access mechanism and its network location. A URN can be defined as a URI that identifies a resource by name in a specific namespace. A URN can be used to identify a resource without implying its location or how it is accessed.

[0100] A URL can be defined as having the following syntax: scheme: / / [user:password@]domain:port / path?query_string#fragment_id. The scheme can, but does not have to, specify the name of a protocol (e.g., http). Domain indicates an Internet Protocol address or domain name, which can be resolved via the DNS (Domain Name System) protocol. Port number indicates the Internet Protocol port used. Path indicates the resource to which the URL points, such as a file. The query string contains data to be passed to the entity that resolves the URL. It can contain name-value pairs separated by &s, for example?first_para=123&second_para=ABC. A fragment identifier specifies a part of a resource. The fragment identification scheme can depend on the MIME type of the resource. The query string can be parsed by an entity that parses the URL, such as an HTTP server, while the fragment identifier can be parsed by a client that requests the resource specified by the URL.

[0101] The Hypertext Transfer Protocol (HTTP) has been widely used to deliver real-time multimedia content over the Internet, such as in video streaming applications. Several HTTP-based adaptive streaming solutions have been introduced, such as Smooth Streaming, Adaptive HTTP Live Streaming and Dynamic Streaming, and a standardization project has been launched. Adaptive HTTP streaming (AHS) was first standardized in version 9 of the 3rd Generation Partnership Project (3GPP) packet-switched streaming (PSS) service (3GPP TS26.234 version 9: "Transparent end-to-end packet-switched streaming service (PSS); protocols and codecs"). MPEG uses 3GPP AHS version 9 as the starting point for the MPEG DASH standard (ISO / IEC 23009-1: "Dynamic adaptive streaming over HTTP (DASH)-Part 1: Media presentation description and segment formats"). MPEG DASH and 3GP-DASH are technically close to each other and can therefore be collectively referred to as DASH. Some concepts, formats, and operations of DASH are described below as examples of video streaming systems in which embodiments can be implemented. Aspects of the present invention are not limited to DASH, but a description is given for a possible basis on which the present invention can be partially or completely implemented.

[0102] In DASH, multimedia content can be stored on an HTTP server and can be delivered using HTTP. Content can be stored on the server in two parts: a media presentation description (MPD), which describes the list of available content, its various alternatives, their URL addresses and other characteristics; and segments, which contain the actual multimedia bitstream in the form of chunks in a single or multiple files. MPD provides the client with the necessary information to establish dynamic adaptive streaming over HTTP. MPD contains information describing the media presentation, such as the HTTP Uniform Resource Locator (URL) of each segment to issue a GET segment request.

[0103] To play the content, the DASH client can obtain the MPD through, for example, HTTP, email, USB flash drive, broadcast, or other transmission methods. By parsing the MPD, the DASH client can understand the program time, media content availability, media type, resolution, minimum and maximum bandwidth, and the existence of various encoding alternatives, accessibility features and required digital rights management (DRM) of multimedia components, media component locations on the network, and other content characteristics. For example, using this information, the DASH client can select an appropriate encoding alternative and start streaming content by retrieving segments using HTTP GET requests. After appropriate buffering to allow for changes in network throughput, the client can continue to retrieve subsequent segments and monitor network bandwidth fluctuations. The client can decide how to adapt to the available bandwidth by retrieving segments of different alternatives (with lower or higher bit rates) to maintain sufficient buffers.

[0104] In the context of DASH, the following definitions may be used: A media content component or media component may be defined as a continuous component of media content having an assigned media component type, which media component type may be encoded individually into a media stream. A media content may be defined as a media content period or a continuous sequence of media content periods. A media content component type may be defined as a single type of media content, such as audio, video, or text. A media stream may be defined as an encoded version of a media content component.

[0105] In DASH, a hierarchical data model is used to construct media representations as follows. Media presentation consists of a sequence of one or more time periods, each time period contains one or more groups, each group contains one or more adaptation sets, each adaptation set contains one or more representations, and each representation consists of one or more segments. A group can be defined as a collection of adaptation sets that are not expected to be presented simultaneously. An adaptation set can be defined as a set of interchangeable encoded versions of one or more media content components. A representation is one of the alternatives for media content or a subset thereof, which typically differs due to encoding choices, such as bit rate, resolution, language, codec, etc. A segment contains media data of a certain duration, as well as metadata for decoding and presenting the included media content. A segment is identified by a URI and can typically be requested via an HTTP GET request. A segment can be defined as a data unit associated with an HTTP-URL, and optionally, can be defined as a byte range specified by an MPD.

[0106] The DASH MPD conforms to the Extensible Markup Language (XML) and is therefore specified using elements and attributes defined in XML. The MPD can be specified using the following conventions: Elements in an XML document can be identified by a capitalized first letter and can be presented in bold as Element. To indicate that an element Element1 is contained within another element Element2, it can be written as Element2.Element1. If the name of an element consists of two or more compound words, camel case can be used, such as ImportantElement. An element can appear only once, or the minimum and maximum number of occurrences can be specified by <minoccurs> ... <maxoccurs>Definition. Attributes in XML documents can be identified by lowercase first letters or by preceding them with the "@" symbol, such as @attribute, for example. To refer to a specific attribute contained in an element, @attribute can be written as Element@attribute. If the name of an attribute consists of two or more compound words, camel case can be used after the first word, such as @veryImportantAttribute. Attributes may have been assigned states in XML, such as mandatory (M), optional (O), optional with default value (OD), and conditionally mandatory (CM).

[0107] In DASH, all descriptor elements are generally constructed in the same way, in that they contain the @schemeIdUri attribute, which provides a URI for identifying the scheme, as well as optional attributes @value and optional attributes @id. The semantics of the element are specific to the scheme adopted. The URI identifying the scheme can be a URN or a URL. Some descriptors are specified in MPEG-DASH (ISO / IEC23009-1), and descriptors can be specified in other specifications in addition or alternatively. When specified in specifications other than MPEG-DASH, MPD does not provide any specific information about how to use descriptor elements. It depends on the application or specification to instantiate description elements using the appropriate scheme information in the DASH format. The application or specification that uses one of these elements defines a scheme identifier in the form of a URI and defines the value space of the element when using the scheme identifier. The scheme identifier appears in the @schemeIdUri attribute. Where a set of simple enumeration values ​​is required, a text string can be defined for each value, and the string can be included in the @value attribute. If structured data is required, any extended elements or attributes can be defined in a separate namespace. The @id value may be used to refer to a unique descriptor or to a set of descriptors. In the latter case, descriptors with the same value for the attribute @id may be required to be synonymous, i.e., it may be sufficient to process one of the descriptors with the same @id value. Two elements of type DescriptorType are equivalent if the element name, the value of the @schemeIdUri, and the value of the @value attribute are equivalent. If @schemeIdUri is a URN, equivalence may refer to lexical equivalence as defined in section 5 of RFC2141. If @schemeIdUri is a URL, equivalence may refer to character-by-character equivalence as defined in section 6.2.1 of RFC3986. If the @value attribute is not present, equivalence may be determined solely by equivalence of the @schemeIdUri. Attributes and elements in the extended namespace may not be used to determine equivalence. The @id attribute may be ignored for equivalence determinations.

[0108] MPEG-DASH specifies the descriptors EssentialProperty and SupplementalProperty. For the element EssentialProperty, the media presentation author expresses that successful processing of the descriptor is critical to the correct use of the information in the parent element containing the descriptor, unless the element shares the same @id with another EssentialProperty element. If the EssentialProperty elements share the same @id, it is sufficient to process one of the EssentialProperty elements with the same @id value. At least one EssentialProperty element is expected to be processed for each different @id value. If the scheme or value of the EssentialProperty descriptor cannot be recognized, the DASH client should ignore the parent element containing the descriptor. Multiple EssentialProperty elements with the same @id value and different @id values ​​can exist in the MPD.

[0109] For the element SupplementalProperty, the media presentation author expresses that the descriptor contains supplemental information that the DASH client can use to optimize processing. If the scheme or value of the SupplementalProperty descriptor cannot be recognized, the DASH client should ignore the descriptor. There can be multiple SupplementalProperty elements in the MPD.

[0110] MPEG-DASH specifies a Viewpoint element formatted as a property descriptor. The @schemeIdUri attribute of the Viewpoint element is used to identify the viewpoint scheme used. Adaptation sets containing non-equivalent Viewpoint element values ​​contain different media content components. The Viewpoint element can also be applied to media content types that are not video. Adaptation sets with equivalent Viewpoint element values ​​are intended to be presented together. This processing should apply equally to recognized and unrecognized @schemeIdUri values.

[0111] SRD (Spatial Relationship Description) is specified in the normative Annex H of MPEG-DASH. The following contains some excerpts from the SRD specification.

[0112] The SRD scheme allows media presentation description authors to express spatial relationships between spatial objects. Spatial objects are represented by adaptation sets or sub-representations. For example, spatial relationships can be expressed, where a video represents a spatial portion of another full-frame video (e.g., a region of interest, or tile).

[0113] SupplementalProperty and / or EssentialProperty descriptors with @schemeIdUri equal to "urn:mpeg:dash:srd:2014" are used to provide spatial relationship information associated with the containing spatial object. The SRD shall only be contained in these two MPD elements (AdaptationSet and SubRepresentation).

[0114] Sub-representation level SRD can be used to represent spatial objects in one representation, such as an HEVC slice stream. In this case, the SRD descriptor can exist at the adaptation set as well as the sub-representation level.

[0115] The @value of a SupplementalProperty or EssentialProperty element using the SRD scheme is a comma-delimited list of SRD parameter values. The SRD parameters source_id, object_x, object_y, object_width, and object_height must be present, and the SRD parameters total_width, total_height, and spatial_set_id may be present conditionally or optionally.

[0116] source_id is a non-negative integer expressed in decimal that provides an identifier for the source of the content. The source_id parameter provides a unique identifier for the source of the content within a period (Period). It implicitly defines a coordinate system associated with the source. The coordinate system uses an arbitrary origin (0; 0); the x-axis runs from left to right and the y-axis runs from top to bottom. All SRDs sharing the same source_id value have the same origin and axis orientation. Spatial relationships between spatial objects using SRDs with different source_id values ​​are undefined.

[0117] For a given source_id value, a reference space is defined that corresponds to a rectangular region that contains the entire source content, with its upper left corner located at the origin of the coordinate system. The total_width and total_height values ​​in the SRD provide the size of the reference space in arbitrary units. total_width is a non-negative integer expressed in decimal notation, representing the width of the reference space in arbitrary units. total_height is a non-negative integer expressed in decimal notation, representing the height of the reference space in arbitrary units. It is allowed for spatial objects in the MPD to not cover the entire content source, for example when the entire source content is represented by two separate videos.

[0118] object_x is a non-negative integer expressed in decimal representing the horizontal position of the upper left corner of the spatial object in arbitrary units. object_y is a non-negative integer expressed in decimal representing the vertical position of the upper left corner of the spatial object in arbitrary units. object_width is a non-negative integer expressed in decimal representing the width of the spatial object in arbitrary units. object_height is a non-negative integer expressed in decimal representing the height of the spatial object in arbitrary units. The object_x and object_y parameters (object_width and object_height, respectively) represent the 2D position (2D size, respectively) of the associated spatial object in the coordinate system associated with the source. The values ​​of the object_x, object_y, object_width, and object_height parameters are relative to the values ​​of the total_width and total_height parameters defined above. The position (object_x, object_y) and size (object_width, object_height) of SRDs sharing the same source_id value can be compared after taking into account the size of the reference space, i.e. after the object_x and object_width values ​​are divided by the total_width value and the object_y and object_height values ​​are divided by the total_height value of their corresponding descriptors. Different total_width and total_height values ​​can be used in different descriptors to provide position and size information in different units for the same reference space.

[0119] spatial_set_id is a non-negative integer expressed in decimal that provides an identifier for a set of spatial objects. When not present, the spatial object associated with this descriptor does not belong to any spatial set and no spatial set information is given. MPD authors can use the spatial_set_id parameter to indicate that certain spatial objects within a given source_id have a specific spatial relationship. For example, MPD authors can group all adaptation sets corresponding to tiles at the same resolution level. In this way, DASH clients can use the spatial_set_id parameter to quickly select spatially related spatial objects.

[0120] An initialization segment may be defined as a segment containing metadata necessary to render a media stream encapsulated in a media segment. In an ISOBMFF-based segmentation format, an initialization segment may include a movie box ("moov") which may not include metadata for any samples, i.e., any metadata for the samples is provided in a "moof" box.

[0121] A media segment contains a certain duration of media data played back at normal speed, and this duration is called the media segment duration or segment duration. The content producer or service provider can select the segment duration according to the desired characteristics of the service. For example, a relatively short segment duration can be used in a real-time service to achieve a shorter end-to-end delay. The reason is that the segment duration is usually the lower limit of the end-to-end delay perceived by the DASH client, because the segment is a discrete unit for generating media data for DASH. Content generation is usually completed in a way that makes the entire media data segment available to the server. In addition, many client implementations use segments as units of GET requests. Therefore, in a typical arrangement of a live service, a DASH client can request a segment only when the entire duration of the media segment is available and encoded and encapsulated into the segment. For on-demand services, different strategies for selecting segment durations can be used.

[0122] For example, a segment may be further split into subsegments to enable downloading the segment in multiple parts. It may be required that a subsegment contains a complete access unit. Subsegments may be indexed by a segment index box, which contains information mapping the presentation time range and byte range for each subsegment. The segment index box may also describe the subsegments and stream access points within the segment by signaling their duration and byte offset. A DASH client may use the information obtained from the segment index box(es) to make HTTP GET requests for specific subsegments using byte range HTTP requests. If relatively long segment durations are used, subsegments may be used to keep the size of the HTTP response reasonable and flexible to adapt to the bitrate. The index information for the segment may be placed in a single box at the beginning of the segment or distributed across multiple index boxes within the segment. Different propagation methods are possible, such as layering, daisy chaining, and mixing. This technique avoids adding a large box at the beginning of the segment and therefore prevents possible initial download delays.

[0123] A subrepresentation is embedded in a regular representation and is described by a SubRepresentation element. The SubRepresentation element is contained in a Representation element. The SubRepresentation element describes the properties of one or more media content components embedded in the representation. For example, it may describe the exact properties of an embedded audio component (e.g., codec, sampling rate, etc.), embedded subtitles (e.g., codec), or it may describe an embedded lower-quality video layer (e.g., a lower frame rate, etc.). Subrepresentations and representations share some common properties and elements. If the @level attribute is present in the SubRepresentation element, the following applies:

[0124] Subrepresentations provide the ability to access lower quality versions of the representations that contain them. In this case, subrepresentations allow, for example, the extraction of audio tracks in a multiplexed representation, or if provided with a lower frame rate, efficient fast forward or rewind operations;

[0125] The initialization segment and / or the media segment and / or the index segment should provide enough information so that the data can be easily accessed via an HTTP partial GET request. The details of providing such information are defined by the media format used.

[0126] When using ISOBMFF segmentation, the following applies:

[0127] The initialization segment contains the level assignment box.

[0128] A subsegment index box ("ssix") exists for each subsegment.

[0129] The attribute @level specifies the level in the subsegment index to which the described subrepresentation is associated. The information in the representation, subrepresentation, and level assignment ('leva') boxes contains information about the assignment of media data to levels. The media data should have an order such that each level provides enhancements compared to lower levels.

[0130] If the @level attribute is not present, child presentation elements are used only to provide a more detailed description of the media stream embedded in the presentation.

[0131] ISOBMFF includes a so-called level mechanism to specify subsets of a file. Levels follow a dependency hierarchy such that a sample mapped to level n may depend on any sample at level m, where m<=n, and does not depend on any sample at level p, where p>n. For example, levels may be specified based on temporal sublayers (e.g., TemporalId for HEVC). Levels may be declared in a level assignment ('leva') box contained in a movie extension ('mvex') box. Levels cannot be specified for an initial movie. When a level assignment box appears, it applies to all movie fragments after the initial movie. For the context of a level assignment box, a fraction is defined as consisting of one or more movie fragment boxes and associated media data boxes, possibly including only the initial part of the last media data box. Within a fragment, data for each level appears consecutively. Level data within a fragment appears in increasing order of level value. All data in a fragment is assigned to a level. The level assignment box provides a mapping from features (e.g., scalability layers or temporal sublayers) to levels. Features may be specified by track, subtrack within a track, or sample grouping of a track. For example, a temporal level sample grouping may be used to indicate a mapping of a picture to a temporal level, which is equivalent to a temporal sublayer in HEVC. That is, an HEVC picture of a certain TemporalId value may be mapped to a particular temporal level using a temporal level sample grouping (and the same operation may be repeated for all TemporalId values). The level allocation box may then reference the temporal level sample grouping in the indicated mapping to the level.

[0132] The subsegment index box ("ssix") provides a mapping from levels (specified by the level assignment box) to byte ranges that index subsegments. In other words, this box provides a compact index that explains how the data in the subsegment is sorted into partial subsegments according to the levels. It enables clients to easily access the data of partial subsegments by downloading the data range in the subsegment. When the subsegment index box appears, each byte in the subsegment is assigned to a level. If the range is not associated with any information in the level assignment, any level not included in the level assignment can be used. There are 0 or 1 subsegment index boxes for each segment index box, which only index leaf segments, that is, only index subsegments but no segment index. The subsegment index box (if any) is the next box after the relevant segment index box. The subsegment index box records the subsegment indicated in the immediately preceding segment index box. Each level can be assigned to exactly one partial subsegment, that is, the byte range of a level is continuous. The levels of partial subsegments are assigned by increasing number within the subsegment, i.e., a sample of a partial subsegment can depend on any sample of a previous partial subsegment in the same subsegment, but not vice versa. For example, each partial subsegment contains samples with the same temporal sublayer, and the partial subsegments appear in increasing temporal sublayer order within the subsegment. When accessing partial subsegments in this way, the final media data box may be incomplete, i.e., less data is accessed than the length indicated by the media data box indicates is present. The length of the media data box may need to be adjusted, or padding may be used. The padding_flag in the level assignment box indicates whether the missing data can be replaced with zeros. If not, the sample data assigned to the samples of the unaccessed level does not exist and should be noted.

[0133] MPEG-DASH defines a segment container format for both ISOBMFF and MPEG-2 transport streams. Other specifications may specify segment formats based on other container formats. For example, a segment format based on the Matroska container file format has been proposed, which can be summarized as follows. When a Matroska file is carried as a DASH segment or similar file, the association of DASH units and Matroska units can be specified as follows. A (DASH) subsegment can be defined as one or more contiguous clusters of Matroska encapsulated content. The initialization segment of DASH may need to include an EBML header, a segment header (Matroska), segment information (Matroska), and a track, and may optionally include other level 1 elements and padding. The segment index of DASH may include a hint element of Matroska.

[0134] OMAF defines MPEG-DASH elements for associating various DASH elements. The SupplementalProperty element with the @schemeIdUri attribute equal to "urn:mpeg:mpegI:omaf:2018:assoc" is called an association descriptor. One or more association descriptors can exist at the adaptation set level, representation level, and preselection level. An association descriptor included in an adaptation set / representation / preselection element indicates that the parent element of the element's descriptor (i.e., the adaptation set / representation / preselection element) is associated with one or more elements in the MPD indicated by the XPath query in the omaf2:Association element and the association type signaled by omaf2:@associationKindList.

[0135] In OMAF DASH MPD, the Viewpoint element with the @schemeIdUri attribute equal to "urn:mpeg:mpegI:omaf:2018:vwpt" is called the Viewpoint Information (VWPT) descriptor.

[0136] At most one VWPT descriptor may be present at the adaptation set level, and no VWPT descriptor shall be present at any other level. When no adaptation set in a media presentation contains a VWPT descriptor, the media presentation is inferred to contain only one view.

[0137] @value specifies the viewpoint ID of the viewpoint. ViewPointInfo is a Container element whose child elements and attributes provide information about the viewpoint. The ViewPointInfo@label attribute specifies a string that provides a human-readable label for the viewpoint. The ViewPointInfo.Position attribute of this element specifies the position information of the viewpoint.

[0138] Search preview and thumbnail navigation provide DASH clients with the possibility to implement thumbnails for UI cleanup. This can be achieved using a separate video adaptation set and using the trick mode feature. However, this feature can be relatively complex to implement in a player and requires dual video decoders. In a simpler approach, a sequence of 20 image tiles with multiple thumbnails per image tile can be used to provide such thumbnails.

[0139] To provide easily accessible temporally-aligned thumbnails, an AdaptationSet with the new @contentType="image" can be used in the MPD. A typical use case is to enhance the scrub bar with visual onsets. The actual assets referred to are temporally equidistant rectangular tiles of thumbnails, which are combined into a single jpeg or png image. Tiles are therefore very similar to video segments from an MPD timing perspective, but are typically much longer. For videos, different spatial resolutions can be collected into one AdapationSet. To limit the implementation effort, only SegmentTemplate with $Number$ is used to describe the thumbnail tiles and their times.

[0140] It is expected that DASH clients can handle such adaptation sets by downloading the images and assigning thumbnails to the media presentation timeline using browser-based processing.

[0141] Many parameters are the same as for videos, but new for thumbnail tiles, the rectangular grid dimensions are given as the value of an EssentialProperty, and @schemeIdUri is set to "http: / / dashif.org / guidelines / thumbnail_tile".

[0142] If an EssentialProperty descriptor with @schemeIdUriset exists for "http: / / dashif.org / guidelines / thumbnail_tile", the following properties and elements of AdaptationSet should be used to describe the tiling as follows: The value of the descriptor provides the horizontal and vertical number of tiles as unsigned integers, separated by "x". These two values ​​are referred to as htiles and vtiles in the following.

[0143] @duration The duration of one tile in the Media Presentation Timeline expressed in timescale given by the value of @timescaleattritube if present, otherwise in seconds. This value is referred to as tduration in the following.

[0144] @bandwidth represents the maximum tile size in bits divided by the duration of one tile given by the tduration value.

[0145] @width and @height represent the spatial resolution of the tile. Note that the maximum size of a JPEG image is 64k in width and height.

[0146] Each tile is assigned a number starting from 1. The tile number is called tnumber.

[0147] @startNumber may be present to indicate the number of the first tile in the period. If not present, the first number defaults to 1. The value of the starting number is called startnumber.

[0148] @presentationTimeOffset may be present to indicate the presentation time of the thumbnail sequence at the start of the period. If not present, the first number defaults to 0. The timescale is provided by the value of @timescaleattritube if present, otherwise in seconds. The value of the presentation time offset is called pto.

[0149] Based on this information, the following can be derived:

[0150] Get information about how many thumbnails are included in one tile by multiplying htiles by vtiles. This number is called ttiles.

[0151] • The first tile that can be requested in a period is called startnumber, which is used in the segmentation template.

[0152] The presentation time of each tile in this period is defined as (tnumber-1)*tduration-pto.

[0153] The duration of each thumbnail is defined as tduration / ttiles. This value is called thduration.

[0154] Thumbnails are sorted from left to right, row by row, starting from the top row. The last tile in a period may have thumbnails outside the time interval. Content providers can use any fill pixels to such tiles, such as adding a black thumbnail. The number of thumbnails in a tile is called thnumber.

[0155] The presentation time of the thumbnail in the small tile is defined as (thnumber-1)*thduration, that is, the presentation in the time period is defined as (tnumber-1)*tduration+(thnumber-1)*thduration-pto.

[0156] The vertical size of each thumbnail is defined as @height / vtiles, and the horizontal size of each thumbnail is defined as @width / htiles.

[0157] The total number of thumbnails in a session (tthumbnails for short) is the upper limit of the session duration divided by thduration.

[0158] Thumbnails are small images of content taken at regular intervals. They are an effective way to visually scrub and search based on content.

[0159] There is currently no default way to add thumbnail support to a playback app. And there is no browser support out of the box. However, since thumbnails are image data, browsers have all the capabilities to implement thumbnail navigation in your app client-side.

[0160] The most common way to generate thumbnails is to render a set of images from the main content at fixed intervals, such as every 10 seconds. Information about the locations of these images then needs to be passed to the client, which can then request and load the image for a given playback position. For more efficient loading, images are often combined into larger grids (sometimes called sprites). This way, the client only needs to make one request to load a set of thumbnails, rather than a request for each image.

[0161] Currently, neither DASH nor HLS specify a method for referencing thumbnails directly from a manifest. However, the DASH-IF guidelines [DASHIFIOP] describe an extension for referencing thumbnail images. Thumbnails are displayed as single or grid images. All parameters required to load and display thumbnails are included in the manifest. This approach is also suitable for live manifests that are regularly updated by the player. The following example shows how to reference thumbnails according to the DASH-IF guidelines:

[0162] Example 1: Thumbnail reference in DASH-Manifest

[0163]

[0164] The role descriptor is specified in the MPEG DASH specification (ISO / IEC 23009-1). The URN "urn:mpeg:dash:role:2011" is defined in ISO / IEC 23009-1 and is used to identify the role scheme defined in the following table. Note that the Role@value should be assigned to the Adaptation Set containing the media component type associated with the role.

[0165] The Role@value attribute of the scheme with the value "urn:mpeg:dash:role:2011"

[0166]

[0167]

[0168] Subsets are described by Subset elements contained within a Period element.

[0169] Subsets provide a mechanism to restrict the combinations of active adaptation sets, where an active adaptation set is an adaptation set for which a DASH client renders at least one contained representation.

[0170] A Subset defines a group of one or more Adaptation Sets. The presence of a Subset element within a Period element expresses the intent of the creator of the media presentation that the client should operate as follows: at any time, the set of active Adaptation Sets should be a subset of the Adaptation Sets in one of the specified Subsets. Any Adaptation Set not explicitly included in any Subset element is implicitly included in all specified Subsets.

[0171] this means:

[0172] - Empty subsets are not allowed.

[0173] - No subset should contain all adaptation sets.

[0174] Each adaptation set for which an @id value is provided in the @contains attribute is included in the subset.

[0175] The following table provides the semantics of the attributes and elements in the subset.

[0176] Subset Element Semantics

[0177]

[0178] MPEG-DASH primarily specifies a format for delivering video segments and does not address the delivery of images.

[0179] OMAF v2 is expected to include additional "layers" such as overlays and viewpoints to enable immersive, interactive storytelling. Such additional layers may involve images that need to be delivered to the client in a standardized way.

[0180] In their current form, neither MPEG-DASH nor OMAF defines any mechanism to support cover images. When multiple views and multiple thumbnails are defined in the MPD, a DASH player cannot resolve the correspondence / association / relationship between thumbnails and views / overlays unless it downloads all viewpoints / overlays and thumbnail files. Furthermore, the current version of OMAF does not specify any mechanism to deliver overlays and views when they are images.

[0181] This embodiment is directed to image signaling in MPEG-DASH. In this embodiment, the following terms and definitions are used:

[0182] - An image is an array of one or more pixels of different color components described by image items or samples.

[0183] – A coded picture is a coded representation of a picture.

[0184] - Derivative images are representations of images as operations on other images.

[0185] - In contrast to sample data, items are data that do not require timing processing and are described by boxes contained in MetaBox.

[0186] - The picture item is a coded picture item or a derived picture item.

[0187] A derived image item is an item whose data is a derived image.

[0188] – A coded picture item is an item whose data is a coded picture.

[0189] An image file may contain an image or an encoded image or a derived image or an image item. For example, an image file may be a file conforming to HEIF, MIAF (Multiple Image Application Format), or image / jpeg MIME type. In the presented embodiments, an image file may interchangeably refer to a file stored in a file system, or to a resource accessible by an identifier such as a URI.

[0190] A URL fragment identifier (also referred to as a URL form) may be specified for a particular content type to access a portion of a resource, such as a file, indicated by the base portion of the URL (without the fragment identifier). A URL fragment identifier may be identified, for example, by a hash ('#') character in the URL.

[0191] Although the term "image file" is used in this specification, it is to be understood that embodiments are additionally or alternatively applicable to images or image items within an image file. In some embodiments, URL fragments may be used to address individual images or image items within an image file. The URL fragment scheme for ISOBMFF and derivative file formats (e.g., HEIF and MIAF) is specified in Annex C of ISOBMFF. An image item may be accessed by replacing the URL fragment item_ID with<item_ID> Add to the URI for addressing.<item_id> Is a value (representable by a string representing an unsigned integer in base 10) that identifies the item in the MetaBox at the document level with the given item_id. URL fragments are included in the URI using the delimiters "#" or "*", the latter indicating that the fragment should be addressed or parsed.

[0192] In one embodiment, a URL query string may be used to address a single image or image item within an image file.

[0193] Indicating image files in MPEG-DASH

[0194] In the following, some embodiments for indicating media entities in MPEG-DASH will be described. It should be understood that the example embodiments are provided as examples, and therefore the embodiments are not limited to MPEG-DASH. The media entity may be an image file, an item (e.g., a point cloud occupancy map), or a media track (e.g., an audio track or a video track). The embodiments relate to content creation and content consumption performed in corresponding devices. The device for content creation may be an encoding and file packaging device, such as a content server or a content capture device. The device for content consumption may be a decoding and / or rendering device, such as a virtual reality headset, a head-mounted device, or any other device capable of consuming immersive content.

[0195] Figure 2 An example of a device for content creation, namely, an apparatus according to an embodiment, is shown in FIG. The apparatus 90 includes a main processing unit 91 , a memory 92 , a user interface 94 , and a communication interface 93 . Figure 2 The device according to the embodiment shown also includes a camera module 95. Alternatively, the device can be configured to receive image and / or video data from an external camera device via a communication network. The memory 92 stores data including computer program code in the device 90. The computer program code is configured to implement the method according to various embodiments. The camera module 95 or the communication interface 93 receives the data to be processed by the processor 91 in the form of an image or video stream. The communication interface 93 forwards the processed data (i.e., image file) to, for example, a display of another device (such as a virtual reality headset). When the device 90 is a video source including a camera module 95, user input can be received from a user interface.

[0196] Figure 3 An example of a device for content consumption, i.e., an apparatus according to another embodiment, is shown in . The apparatus in this example is a virtual reality headset, such as a head mounted display (HMD) 300 for stereoscopic viewing. The head mounted display 300 includes two screen portions or two screens 320, 330 for displaying left eye images and right eye images. The displays 320, 330 are close to the eyes, so lenses are used to make the images easy to view and to spread the images to cover as much of the eye's field of view as possible. The device is attached to the user's head so that it remains in place even if the user turns his head. The device may have an orientation detection module 310 for determining head movement and head direction. The head mounted display provides the user with a three-dimensional (3D) perception of the recorded / streamed image files.

[0197] According to one embodiment, the image file to be signaled may be similar to the video media type in MPEG-DASH. In one embodiment, the image file should be signaled as an adaptation set with @mimeType="image / containerFormat" and @contentType="image". @mimeType may take a value specified by the Internet Assigned Numbers Authority.

[0198] According to one embodiment, an Adaptation Set may contain multiple representations as alternatives, such as images of different resolutions and / or codecs but the same image content. Alternatively, image files with different content component attributes but the same image content may exist in multiple Adaptation Sets. Adaptation Sets are grouped together with the same non-zero @group attribute value. Such grouping provides seamless switching between image files across AdaptationSets (for example, when the player supports only one of the image codecs between the Adaptation Sets).

[0199] Segment Information

[0200] According to one embodiment, when the top-level media type of the MIME type is indicated as "image" in @mimeType, there is no media segment information for representation in the MPD, and the MPD may only provide information for the initialization segment. The initialization segment contains an image file.

[0201] According to one embodiment, an image file is considered (eg, by a DASH client or player) as a video representation containing only a single frame, whose duration is equal to the duration of the period containing the representation.

[0202] According to one embodiment, when an image file is only a part of a single media segment provided in a representation and the media segment URL is included in the BaseURL element, the SegmentBase element is used to describe the segment information of the image file.

[0203] According to one embodiment, if the representation containing the image file has more than one media segment, either the attribute @duration or the element SegmentTimeline shall be present. The attribute @duration and the element SegmentTimeline shall not be present at the same time.

[0204] Initialize segment

[0205] According to one embodiment, when an image file is represented by a single media segment, there is no initialization segment for the representation containing the image file. According to one embodiment, the following elements and attributes are not included in the representation containing the image file:

[0206] SegmentBase.Initialization

[0207] SegmentList.Initialization

[0208] The SegmentTemplate.Initialization element, or

[0209] ·SegmentTemplate@initialization attribute.

[0210] According to one embodiment, if neither the Initialization element nor the SegmentTemplate@initialization attribute is present for a representation containing an image file, the media segments within the representation will be self-initializing.

[0211] According to one embodiment, an MPD may consist of image files (e.g., overlays) that are part of Adaptation Set 1 and video tracks (e.g., background media or viewpoints) that are part of Adaptation Set 2. According to one embodiment, if representations from Adaptation Set 1 and Adaptation Set 2 are to be viewed together as part of a viewing session, the representations in Adaptation Set 1 should have a different initialization segment than the initialization segment of the representations in Adaptation Set 2.

[0212] ImageEntity Descriptor

[0213] According to one embodiment, the ImageEntity descriptor may indicate the presence of an image file in an adaptation set / representation. An EssentialProperty or SupplementalPropertyImageEntity element with the @schemeIdUri attribute equal to "urn:mpeg:mpeg:isobmff:2019:imen" is called an ImageEntity descriptor.

[0214] According to one embodiment, there may be at most one ImageEntity descriptor at the Adaptation Set level and / or Representation Set level. In one embodiment, there may be no ImageEntity descriptor at the Period and MPD levels.

[0215] In an example embodiment, there shall be no @value attribute of the ImageEntity descriptor.The ImageEntity descriptor shall include the elements and attributes specified in the following table.

[0216]

[0217] In the example, the XML schema for the image item looks like this:

[0218]

[0219] Image files as overlays and viewpoints

[0220] In the following embodiments, embodiments using image files as overlays and viewpoints will be discussed.

[0221] According to one embodiment, a representation that belongs to an adaptation set with @mimeType="image / containerFormat" and @contentType="image" and has an Overlay Information (OVLY) descriptor is called an overlay representation (it is an image), and the OVLY descriptor is an EssentialProperty or SupplementalProperty element with the @schemeIdUri attribute equal to "urn:mpeg:mpegI:omaf:2018:ovly".

[0222] Alternatively, a representation that belongs to an adaptation set that has an ImageEntity descriptor, which is an EssentialProperty or SupplementalProperty ImageEntity element with the @schemeUri attribute equal to "urn:mpeg:mpegI:isobmff:2019:imen", and an Overlay Information (OVLY) descriptor, which is an EssentialProperty or SupplementalProperty element with the @schemeIdUri attribute equal to "urn.mpeg:mpegI:omaf:2018:ovly", is called a representation of an overlay (which is an image).

[0223] According to one embodiment, a representation that belongs to an adaptation set with @mimeType="image / containerFormat" and @contentType="image" and has a viewpoint information (VWPT) descriptor is called a representation of an overlay layer (it is an image), and the VWPT descriptor is an EssentialProperty or SupplementalProperty element with the @schemeIdUri attribute equal to "urn:mpeg:mpegI:omaf:2018:vwpt".

[0224] Alternatively, a representation that belongs to an adaptation set that has an ImageEntity descriptor, which is an EssentialProperty or SupplementalProperty ImageEntity element with the @schemeUri attribute equal to "urn:mpeg:mpegI:isobmff:2019:imen", and a Viewpoint Information (VWPT) descriptor, which is an EssentialProperty or SupplementalProperty element with the @schemeIdUri attribute equal to "urn:mpeg:mpegI:omaf:2018:vwpt", is called a representation of the overlay layer (which is an image).

[0225] Cover image

[0226] According to one embodiment, the cover image is an image that is displayed at the beginning of a viewing session before a user initiates playback of timed media.

[0227] In an example embodiment, the URN "urn:mpeg:mpegI:role:2019" is defined to identify a role scheme for a cover image as defined in the following table. Note that the Role@value should be assigned to the Adaptation Set containing the media component type associated with this cover image role:

[0228] The Role@value attribute of the scheme with the value "urn:mpeg:mpegI:role:2019"

[0229] Role@value describe coverimage The cover image for the current viewing session

[0230] Instead, the ImageEntity descriptor contains an @ImageItemType attribute that indicates the type of image item represented by the descriptor.

[0231] According to one embodiment, ImageEntity@ImageItemType equal to "coverimage" specifies that the adaptation set / representation set contains a coverimage image item.

[0232] The ImageEntity descriptor with the @ImageItemType attribute is shown in the following table.

[0233]

[0234] In the example, the XML schema for the image item looks like this:

[0235]

[0236] In an alternative embodiment, to support cover images, an AdaptationSet with a new @contentType="image" may be used in the MPD.

[0237] According to one embodiment, it is expected that an MPEG-DASH client can handle an adaptation set of cover images by downloading the images and assigning the cover images to the media presentation timeline using a browser-based process.

[0238] According to one embodiment, the EssentialProperty descriptor with @schemeIdUri set to 'urn:mpeg:mpegI:isobmff:2019:cvim' indicates that the adaptation set / representation set represents the cover image.

[0239] Grouping of cover images

[0240] In an example embodiment, if there is only one cover image in the MPD without any association or grouping information, the cover image belongs to all representations in the MPD.

[0241] In an example embodiment, a cover image may be grouped / associated to a view group if an EssentialProperty descriptor with a @schemeIdUri attribute equals "urn:mpeg:mpegI:isobmff:2019:vpci". The value of the descriptor provides the groupId of the view group to which the cover image belongs.

[0242] Alternatively, EntityGroup@group_type equal to "vpci" specifies a representation of the corresponding track and image files containing the cover image and the view group to which the cover image belongs.

[0243] According to one embodiment, at least one cover image exists in an EntityGroup where @group_type equals 'vpci'.

[0244] The EntityToGroup descriptor with the EntityGroup@group_type attribute is specified in the following table:

[0245] EntityGroup@group_type M EntityGroup@group_type='vpci'

[0246] In an example, the XML schema for an EntityGroup looks like this:

[0247]

[0248] In some embodiments, entity grouping can be carried in the MPD with an association descriptor of OMAF. When an adaptation set containing a cover image is associated with one or more adaptation sets containing viewpoints, the association descriptor should appear as a child element under each AdaptationSet element containing a cover image. In this case, the association descriptor should include the following two items:

[0249] An XPath string in an Association element that evaluates to one or more AdaptationSet elements containing viewpoint groups.

[0250] There is only one "vpci" value for the Association@associationKindList attribute of the Association element. In this case:

[0251] When Association@associationKindList includes a "vpci" value and the number of elements counted by the XPath string in the above Association element is greater than 1, the cover image applies commonly to all viewpoint groups.

[0252] When Association@associationKindList includes a "vpci" value and the number of elements computed by the XPath string in the Association element above is equal to 1, the cover image will be applied to each viewpoint group individually.

[0253] There may be multiple such associated descriptors in an adaptation set containing a cover image.

[0254] Group thumbnails for multiple viewpoints

[0255] In one embodiment, the adaptation set of thumbnails belonging to view k is grouped with the adaptation set of view k in a subset element.

[0256] According to one embodiment, if the adaptation set of thumbnails belonging to view k and the adaptation set of view k have the provided @id value, the @contains attribute of the subset element contains the @id values ​​of the adaptation set of thumbnails belonging to view k and the adaptation set of view k.

[0257] In one embodiment, the thumbnail adaptation set belonging to a viewpoint is grouped with the viewpoint adaptation set using the EntityToGroup descriptor.

[0258] In one embodiment, EntityGroup@group_type equal to 'thvp' specifies a representation containing thumbnails of viewpoints belonging to the same entity group.

[0259] In one embodiment, there is at most one thumbnail adaptation set and one viewpoint adaptation set in an EntityGroup with @group_type equal to 'thvp'.

[0260] The EntityToGroup descriptor with the EntityGroup@group_type attribute is specified in the following table:

[0261] EntityGroup@group_type M EntityGroup@group_type='thvp'

[0262] In an example, the XML schema for an EntityGroup looks like this:

[0263]

[0264] In some embodiments, entity grouping can be carried in the MPD with an association descriptor of OMAF. When an adaptation set containing thumbnails is associated with one or more adaptation sets containing viewpoints, the association descriptor should appear as a child element under each AdaptationSet element containing thumbnails. In this case, the association descriptor should include the following two items:

[0265] An XPath string within an Association element that evaluates to one or more AdaptationSet elements containing the viewpoint.

[0266] There is only one "thvp" value for the Association@associationKindList attribute of the Association element. In this case:

[0267] When Association@associationKindList includes a "thvp" value and the XPath string in the Association element above counts a number of element(s) greater than 1, the thumbnail is applied commonly to all viewpoints.

[0268] When Association@associationKindList includes a "thvp" value and the number of elements computed by the XPath string in the Association element above is equal to 1, the thumbnail will be applied to each viewpoint individually.

[0269] There may be multiple such associated descriptors in an adaptation set containing thumbnails.

[0270] Figure 4 is a flow chart showing a method according to an embodiment. The embodiment is for content creation, wherein the method at least includes

[0271] Writing 410 a first media entity in a container file;

[0272] Writing 420 a second media entity in the container file;

[0273] creating 430 a media presentation description (MPD) having a first representation and a second representation; the first representation belongs to a first adaptation set and the second representation belongs to a second adaptation set; the representation of the first adaptation set is associated with a first media entity of a container file; the representation of the second adaptation set is associated with a second media entity of the container file,

[0274] When one of the representations belongs to a media entity as a thumbnail of a viewpoint or a thumbnail of an overlay, association / correspondence / grouping information of the representation and another representation belonging to the media entity as a viewpoint or an overlay is correspondingly written 440 in the MPD file.

[0275] At least one of the first media entity or the second media entity is an image, wherein when one of the media entities is an image it is a viewpoint or an overlay or a cover image or a thumbnail of a viewpoint or a thumbnail of an overlay.

[0276] According to an embodiment, an apparatus comprises: a component for writing a first media entity in a container file; a component for writing a second media entity in a container file; wherein at least one of the first media entity or the second media entity is an image, wherein when one of the media entities is an image, it is a viewpoint or an overlay or a cover image or a thumbnail of a viewpoint, or a thumbnail of an overlay; a component for creating a media presentation description (MPD) having a first representation and a second representation; the first representation belongs to a first adaptation set, and the second representation belongs to a second adaptation set; the representation of the first adaptation set is associated with the first media entity of the container file; the representation of the second adaptation set is associated with the second media entity of the container file, when one of the representations belongs to a media entity that is a thumbnail of a viewpoint or a thumbnail of an overlay, a component for correspondingly writing in the MPD file the association / correspondence / grouping information of the representation and another representation belonging to the media entity that is a viewpoint or an overlay. The component comprises at least one processor and a memory comprising computer program code, wherein the processor may also comprise a processor circuit system. The memory and the computer program code are configured to, together with at least one processor, cause the apparatus to perform methods according to various embodiments.

[0277] Figure 5 is a flow chart showing a method according to an embodiment. This embodiment is used for content consumption, wherein the method at least includes

[0278] Parsing from a media presentation description (MPD), a first representation belonging to a first adaptation set and a second representation belonging to a second adaptation set;

[0279] Parse the association / correspondence / grouping information of representations from MPD;

[0280] A first representation of a first adaptation set associated with a first media entity of the container file is selected; and a second representation of a second adaptation set associated with a second media entity of the container file is selected.

[0281] a first representation of a first adaptation set is associated with a first media entity of a container file, a second representation of a second adaptation set is associated with a second media entity of a container file, wherein at least one of the first representation or the second representation is associated with an image. When one of the representations is associated with an image, it is a viewpoint representation or an overlay representation or a cover image representation or a thumbnail representation of a viewpoint; or a thumbnail representation of an overlay;

[0282] According to an embodiment, an apparatus includes: a component for parsing a first representation belonging to a first adaptation set and a second representation belonging to a second adaptation set from a media presentation description (MPD); the first representation of the first adaptation set is associated with a first media entity of a container file; the second representation of the second adaptation set is associated with a second media entity of a container file; wherein at least one of the first representation or the second representation is associated with an image; when one of the representations is associated with an image, it is a viewpoint representation or an overlay representation or a cover image representation or a thumbnail representation of a viewpoint; or a thumbnail representation of an overlay; a component for parsing association / correspondence / grouping information of the representation from the MPD; and a component for selecting a first representation of the first adaptation set associated with the first media entity of the container file and selecting a second representation of the second adaptation set associated with the second media entity of the container file. The component includes at least one processor and a memory including computer program code, wherein the processor may also include a processor circuit system. The memory and the computer program code are configured to, together with at least one processor, cause the apparatus to perform methods according to various embodiments.

[0283] Various embodiments may be implemented with the aid of a computer program code that resides in a memory and causes an associated device to perform the method. For example, a device may include a circuit system and electronic devices for processing, receiving and transmitting data, a computer program code in a memory, and a processor that causes the device to perform features of the embodiments when the computer program code is run. In addition, a network device such as a server may include a circuit system and electronic devices for processing, receiving and transmitting data, a computer program code in a memory, and a processor that causes the network device to perform features of the embodiments when the computer program code is run. The computer program code includes one or more operating characteristics. The above operating characteristics are defined by the above computer through configuration based on the type of the above processor, wherein the system can be connected to the above processor via a bus, wherein the programmable operating characteristics of the system include at least such as Figure 4 or Figure 5 The features specified in the flowchart.

[0284] A computer program product according to an embodiment may be embodied on a non-transitory computer readable medium. According to another embodiment, the computer program product may be downloaded in the form of data packets via a network.

[0285] If desired, the different functions discussed herein may be performed in a different order and / or simultaneously with other functions. In addition, if desired, one or more of the above functions and embodiments may be optional or may be combined.

[0286] Although various aspects of the embodiment are set out in the independent claims, further aspects comprise other combinations of features from the described embodiments and / or the dependent claims with features of the independent claims and not just the combinations explicitly set out in the claims.

[0287] It is also noted herein that while the above describes example embodiments, these descriptions should not be viewed in a limiting sense. Rather, several variations and modifications may be made without departing from the scope of the present disclosure as defined in the appended claims.< / maxoccurs> < / minoccurs>

Claims

1. A method for processing a video, comprising: Writing a first media entity in a container file; writing a second media entity in the container file; wherein at least one of the first media entity or the second media entity is an image; wherein the image comprises one of the following: a viewpoint image or an overlay image or a cover image or a thumbnail of the viewpoint image; or a thumbnail of the overlay image; creating a media presentation description (MPD) having a first representation and a second representation; the first representation belonging to a first adaptation set and the second representation belonging to a second adaptation set; wherein the representation of the first adaptation set is associated with the first media entity of the container file; and the representation of the second adaptation set is associated with the second media entity of the container file; When one of the representations belongs to the first media entity which is the thumbnail of the viewpoint image or the thumbnail of the overlay image, the method includes: writing correspondence information between the representation and another representation belonging to the second media entity which is the viewpoint image or the overlay image in the MPD file accordingly.

2. The method of claim 1, wherein the first media entity is stored as one of: an image file; a project; or a media track.

3. The method of claim 1 or 2, wherein the second media entity is stored as one of: an image file; a project; or a media track.

4. A method for processing a video, comprising: Parsing a first representation belonging to a first adaptation set and a second representation belonging to a second adaptation set from a media presentation description (MPD); wherein the first representation of the first adaptation set is associated with a first media entity of a container file; and wherein the second representation of the second adaptation set is associated with a second media entity of a container file; wherein at least one of the first representation or the second representation is associated with an image; wherein the image is one of: a viewpoint image representation or an overlay image representation or a cover image representation or a representation of a thumbnail of a viewpoint image; or a representation of a thumbnail of an overlay image; parsing correspondence information of the representation from the MPD; The first representation of the first adaptation set associated with the first media entity of the container file is selected; and the second representation of the second adaptation set associated with the second media entity of the container file is selected.

5. The method of claim 4, wherein the first media entity is one of: an image file; a project; or a media track.

6. The method of claim 4 or 5, wherein the second media entity is one of: an image file; a project; or a media track.

7. A device for processing video, comprising at least: means for writing a first media entity in a container file; means for writing a second media entity in a container file; At least one of the first media entity or the second media entity is an image; the image comprises one of the following: a viewpoint image or an overlay image or a cover image or a thumbnail of the viewpoint image; or a thumbnail of the overlay image; means for creating a media presentation description (MPD) having a first representation and a second representation; the first representation belonging to a first adaptation set and the second representation belonging to a second adaptation set; wherein the representation of the first adaptation set is associated with the first media entity of the container file; and the representation of the second adaptation set is associated with the second media entity of the container file; When one of the representations belongs to the first media entity which is the thumbnail of the viewpoint image or the thumbnail of the overlay image, the device includes: a component for writing correspondence information between the representation and another representation belonging to the second media entity which is the viewpoint image or the overlay image in the MPD file accordingly.

8. The apparatus of claim 7, wherein the first media entity is stored as one of: an image file; a project; or a media track.

9. An apparatus according to claim 7 or 8, wherein the second media entity is stored as one of: an image file; a project; or a media track.

10. A device for processing video, comprising at least: means for parsing from a media presentation description (MPD) a first representation belonging to a first adaptation set and a second representation belonging to a second adaptation set; wherein the first representation of the first adaptation set is associated with a first media entity of a container file; and wherein the second representation of the second adaptation set is associated with a second media entity of a container file; wherein at least one of the first representation or the second representation is associated with an image; wherein the image comprises one of: a viewpoint image representation or an overlay image representation or a cover image representation or a representation of a thumbnail of a viewpoint image; or a representation of a thumbnail of an overlay image; means for parsing correspondence information for said representation from said MPD; Means for selecting the first representation of the first adaptation set associated with the first media entity of the container file, and selecting the second representation of the second adaptation set associated with the second media entity of the container file.

11. The apparatus of claim 10, wherein the first media entity is stored as one of: an image file; a project; or a media track.

12. An apparatus according to claim 10 or 11, wherein the second media entity is stored as one of: an image file; a project; or a media track.

13. A computer program product, comprising computer program code, wherein the computer program code is configured to, when executed on at least one processor, cause an apparatus or system to implement any one of methods 1 to 6.

Images