Viewpoint metadata for omnidirectional video

Abstract

To provide novel methods for signaling information regarding different viewpoints in a multi-viewpoint omnidirectional media presentation.SOLUTION: In some embodiments, a container file (which may use an ISO base media file format) is generated containing several tracks. The tracks are grouped using a track-group identifier, where each group identifier is associated with a different viewpoint. In some embodiments, a manifest (such as an MPEG-DASH MPD) is generated, where the manifest includes viewpoint identifiers that identify the viewpoint associated with each stream. In some embodiments, metadata included in a container file and / or in a manifest provides information on a position of each viewpoint, intervals during which each viewpoint is available, transition effects for transitions between viewpoints, and / or recommended projection formats for corresponding field-of-view ranges.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] Regarding viewpoint metadata for omnidirectional video. [Background technology]

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a nonprovisional patent application of U.S. Provisional Patent Application No. 62 / 653,363 (filed April 5, 2018) and U.S. Provisional Patent Application No. 62 / 675,524 (filed May 23, 2018), both of which are entitled "Viewpoint Metadata for Omnidirectional Video" and are incorporated herein by reference in their entireties, and claims the benefit under 35 U.S.C. § 119(e).

[0003] Spherical video, or 360° video, is a fast-growing new format emerging in the media industry, enabled by the growing availability of VR devices, which can provide a greater sense of presence to the viewer. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] ISO / IEC14496-12, "Coding of Audio-Visual Objects, Part 12:ISO Base Media File Format", 2015 [Non-patent document 2] ISO / IEC23009-1, "Dynamic adaptive streaming over HTTP (DASH), Part 1: Media Presentation Description and Segment Formats", May 2014 [Non-patent document 3] ISO / IEC JTC1 / SC29 / WG11 N17399 “FDIS 23090-2 Omnidirectional Media Format”, February 2018 [Non-patent document 4] ISO / IEC JTC1 / SC29 / WG11 N17331 “MPEG-I Phase 1b Requirements”, February 2018 [Non-Patent Document 5] "Extended VSRS for 360-degree video", MPEG121, Gwangju, Korea, January 2018, m41990 [Non-patent document 6] C. Fehn, "Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV," SPIE Stereoscopic Displays and Virtual Reality Systems XI, vol. 5291, May 2004, pages 93-104. [Non-Patent Document 7] Paul Rosenthal, Lars Linsen, "Image-space point cloud rendering", Proceedings of Computer Graphics International, pages 136-143, 2008. Summary of the Invention [Problem to be solved by the invention]

[0005] Compared to traditional rectilinear video (2D or 3D), 360° video presents a new set of difficult technical challenges for video processing and delivery. Enabling a comfortable and immersive user experience requires high video quality and very low latency, but large video size can be an obstacle to delivering 360° video with high quality. [Means for solving the problem]

[0006] ISO-based media file format The ISO / IEC 14496 MPEG-4 standard contains several sections that define file formats for the storage of time-based media. These sections are all based on and derived from the ISO Base Media File Format (ISO BMFF), described in ISO / IEC 14496-12, "Coding of Audio-Visual Objects, Part 12: ISO Base Media File Format," 2015. ISO BMFF is a structural, media-independent definition. It primarily contains structure and media data information for the timed presentation of media data, such as audio and video. There is also support for untimed data, such as metadata, at different levels within the file structure. The logical structure of the file is that of a video, containing a set of time-parallel tracks. The file's temporal structure is such that tracks contain sequences of samples in time, and these sequences are mapped to the overall video timeline. ISO BMFF is based on the concept of a box-structured file. A box structure file consists of a series of boxes (also called atoms) of a certain size and type. The type is a 32-bit value, usually chosen to be four printable characters, also known as a 4-character code (4CC). Un-timed data can be contained in metadata boxes at the file level, or added to video boxes or to one of the streams of timed data called tracks within a video.

[0007] Dynamic Streaming over HTTP (DASH) MPEG Dynamic Adaptive Streaming over HTTP (MPEG-DASH) is a delivery format that dynamically adapts to changing network conditions. MPEG-DASH is described in ISO / IEC 23009-1, "Dynamic adaptive streaming over HTTP (DASH), Part 1: Media Presentation Description and Segment Formats," May 2014. Dynamic HTTP streaming requires various bitrate alternatives of multimedia content to be available at the server. In addition, multimedia content may be composed of several media components (e.g., audio, video, text, etc.), each of which can have different characteristics. In MPEG-DASH, these characteristics are described by a Media Presentation Description (MPD).

[0008] Figure 2 shows the MPD hierarchical data model. The MPD describes a sequence of Periods, where a consistent set of encoded versions of the components of the media content remains unchanged during a Period. Each Period has a start time and a duration, and consists of one or more Adaptation Sets.

[0009] A conformance set represents a set of encoded versions of one or several media content components that have common characteristics, such as language, media type, picture aspect ratio, role, accessibility, and rating characteristics. For example, a conformance set may contain different bitrates of the video component of the same multimedia content. Another conformance set may contain different bitrates of the audio component of the same multimedia content (e.g., low-quality stereo and high-quality surround sound). Each conformance set typically contains multiple representations.

[0010] A representation describes a deliverable encoded version of one or several media components that differ from other representations by bitrate, resolution, number of channels, or other characteristics. Each representation consists of one or more segments. Attributes of the Representation element, such as @id, @bandwidth, @qualityRanking, and @dependencyId, are used to specify the properties of the associated representation. A Representation can also contain sub-representations, which are part of the representation, to describe and extract partial information from the representation. Sub-representations can provide the ability to access lower-quality versions of the representations they are contained in.

[0011] A segment is the largest unit of data that can be retrieved in a single HTTP request. Each segment has a URL, an addressable location on the server, that can be downloaded using HTTP GET or HTTP GET with a byte range.

[0012] To use this data model, a DASH client parses the MPD XML document and selects a set of adaptation sets appropriate for its environment based on the information provided in each of the AdaptationSet elements. Within each adaptation set, the client selects one representation, typically based on the value of the @bandwidth attribute, but also taking into account the client's decoding and rendering capabilities. The client downloads the initialization segment of the selected representation and then accesses the content by requesting the entire segment or a byte range of the segment. After the presentation starts, the client continues to consume the media content by successively requesting media segments or portions of media segments and playing the content according to the media presentation timeline. The client can switch representations taking into account updated information from its environment. The client is expected to play the content continuously over a period of time. When the client is consuming the media contained in a segment towards the end of the media announced in the representation, the media presentation ends, a new period begins, or the MPD is re-fetched.

[0013] Descriptors in DASH MPEG-DASH uses descriptors to provide application-specific information about media content. Descriptor elements are all structured similarly; they include an @schemeIdUri attribute to provide a URI to identify the scheme, an optional @value attribute, and an optional @id attribute. The semantics of the elements are specific to the scheme used. The URI identifying the scheme can be a URN or a URL. The MPD does not provide any concrete information on how to use these elements. It is up to applications using the DASH format to instantiate description elements with the appropriate scheme information. A DASH application using one of these elements first defines a scheme identifier in the form of a URI, and then defines the value space for the element when that scheme identifier is used. When structured data is used, any extension elements or attributes may be defined in a separate namespace. Descriptors can appear at several levels within the MPD. The presence of an element at the MPD level means that the element is a child of an MPD element. The presence of an element at the AdaptationSet level indicates that the element is a child of an AdaptationSet element. The presence of an element at the representation level indicates that the element is a child element of the Representation element.

[0014] Omnidirectional Media Format The Omnidirectional Media Format (OMAF) is a systems standard developed by MPEG as Part 2 of MPEG-I, a set of standards for the encoding, representation, storage, and delivery of immersive media. OMAF defines a media format that enables omnidirectional media applications and focuses primarily on 360° video, images, audio, and associated timed-metadata tracks. The final draft of the OMAF International Standard (FDIS) was released in early 2018 and is documented in ISO / IEC JTC1 / SC29 / WG11 N17399, "FDIS 23090-2 Omnidirectional Media Format," February 2018.

[0015] As part of MPEG-I Phase 1b, an extension of OMAF to support several new features, including 3DoF plus motion parallax, and support for multiple viewpoints, was planned for 2019. The requirements for Phase 1b were released in February 2018 and are described in ISO / IEC JTC1 / SC29 / WG11 N17331 "MPEG-I Phase 1b Requirements", February 2018. OMAF and the MPEG-I Phase 1b requirements address the following concepts: Field-of-view (FoV) is the extent of the observable world on a captured / recorded content or physical display device.

[0016] Viewpoint is the point from which the user views the scene, which usually corresponds to the camera position. Slight head movements do not necessarily imply a different viewpoint. A sample is all of the data associated with a single point in time. A track is a timed sequence of related samples in an ISO base media file. In the case of media data, a track corresponds to a sequence of images or sampled audio. Boxes are object-oriented building blocks in ISO-based media files defined by a unique type identifier and a length.

[0017] In some embodiments, systems and methods are provided for signaling position information for one or more viewpoints in an omnidirectional video presentation. In some embodiments, the method includes receiving a manifest (e.g., an MPEG-DASH MPD) for an omnidirectional video presentation, the video presentation having at least one omnidirectional video associated with a viewpoint, determining, based on the manifest, whether a timed-metadata track for viewpoint positions is provided for the viewpoint, and, in response to determining that the timed-metadata track is provided, determining the viewpoint position based on information in the timed-metadata track.

[0018] In some embodiments, determining whether a time-specific metadata track for the viewpoint position is provided includes determining whether a flag in the manifest indicates that the viewpoint position is dynamic.

[0019] In some embodiments, the manifest includes coordinates indicating the first viewpoint position.

[0020] In some embodiments, the timed metadata track is identified in the manifest, and the method further includes importing the timed metadata track.

[0021] In some embodiments, the timed metadata track contains the viewpoint position in Cartesian coordinates, while in other embodiments, the timed metadata track contains the viewpoint position in longitude and latitude coordinates.

[0022] In some embodiments, the method further includes displaying a user interface to a user, the user interface allowing the user to select a omnidirectional video based on a viewpoint position of the omnidirectional video, and the omnidirectional video is displayed to the user in response to the user selection of the omnidirectional video.

[0023] In some embodiments, the omnidirectional video presentation includes at least a first omnidirectional video and a second omnidirectional video. In such embodiments, displaying the user interface can include displaying the first omnidirectional video to a user and displaying user interface elements or other indications of the second omnidirectional video at a position in the first omnidirectional video that corresponds to a position of the viewpoint of the second omnidirectional video.

[0024]

[0009] Methods for signaling information about various viewpoints in a multi-view omnidirectional media presentation are described herein. In some embodiments, a container file (which may use the ISO Base Media File Format) containing several tracks is generated. The tracks are grouped using track group identifiers, with each track group identifier associated with a different viewpoint. In some embodiments, a manifest (such as an MPEG-DASH MPD) is generated, where the manifest includes a viewpoint identifier that identifies the viewpoint associated with each stream. In some embodiments, metadata included in the container file and / or the manifest provides information about one or more of the following: the location of each viewpoint, the validity range of each viewpoint, the interval over which each viewpoint is available, transition effects for transitions between viewpoints, and recommended projection formats for different viewing ranges.

[0025] In some embodiments, a method comprises generating a container file (e.g., an ISO Base Media File Format file). At least first and second 360-degree video data are received, the first video data representing a view from a first perspective and the second 360-degree video data representing a view from a second perspective. A container file is generated for at least the first video data and the second video data. In the container file, the first video data is organized into a first set of tracks and the second video data is organized into a second set of tracks. Each track in the first set of tracks includes a first track group identifier associated with the first perspective and each track in the second set of tracks includes a second track group identifier associated with the second perspective.

[0026] In some such embodiments, each of the tracks in the first set of tracks includes a respective instance of a viewpoint group type box that includes a first track group identifier, and each of the tracks in the second set of tracks includes a respective instance of a viewpoint group type box that includes a second track group identifier.

[0027] In some embodiments, the container file is organized into a hierarchical box structure and includes a viewpoint list box that identifies at least a first viewpoint information box and a second viewpoint information box. The first viewpoint information box includes at least (i) a first track group identifier and (ii) an indication of a time period during which video from the first viewpoint is available. The second viewpoint information box includes at least (i) a second track group identifier and (ii) an indication of a time period during which video from the second viewpoint is available. The time interval indication can be a list of instances of each viewpoint available interval box.

[0028] In some embodiments, the container file is organized into a hierarchical box structure, and the container file includes a viewpoint list box that identifies at least a first viewpoint information box and a second viewpoint information box. The first viewpoint information box includes at least (i) a first track group identifier and (ii) an indication of the location of the first viewpoint. The second viewpoint information box includes at least (i) a second track group identifier and (ii) an indication of the location of the second viewpoint. The location indication may include Cartesian coordinates or latitude and longitude coordinates, among other options.

[0029] In some embodiments, the container file is organized into a hierarchical box structure, and the container file includes a viewpoint list box that identifies at least a first viewpoint information box and a second viewpoint information box. The first viewpoint information box includes at least (i) a first track group identifier and (ii) an indication of the coverage area of ​​the first viewpoint. The second viewpoint information box includes at least (i) a second track group identifier and (ii) an indication of the coverage area of ​​the second viewpoint.

[0030] In some embodiments, the container file is organized into a hierarchical box structure, and the container file includes a transition effect list box that identifies at least one transition effect box, and each transition effect box includes (i) a source viewpoint identifier, (ii) a destination viewpoint identifier, and (iii) a transition type identifier. The transition type identifier can identify a basic transition, a viewpoint path transition, or an auxiliary information viewpoint transition, among other options. In the case of a viewpoint path transition, a path viewpoint transition box may be provided that includes a list of viewpoint identifiers. In the case of an auxiliary information viewpoint transition, an auxiliary information viewpoint transition box may be provided that includes a track identifier.

[0031] In some embodiments, the container file is organized into a hierarchical box structure that includes a meta box, and the meta box identifies at least one recommended projection list box. Each recommended projection list box can include information identifying (i) a projection type and (ii) a field of view range corresponding to the projection type. The information identifying the field of view range can include (i) a minimum horizontal field of view angle, (ii) a maximum horizontal field of view angle, (iii) a minimum vertical field of view angle, and (iv) a maximum vertical field of view angle.

[0032] In some embodiments, a method is provided for generating a manifest, such as an MPEG-DASH MPD. At least first 360-degree video data representing a view from a first perspective and second 360-degree video data representing a view from a second perspective are received. A manifest is generated. At least one stream in a first set of streams is identified in the manifest, with each stream in the first set representing at least a portion of the first video data. At least one stream in a second set of streams is also identified in the manifest, with each stream in the second set representing at least a portion of the second video data. Each stream in the first set is associated in the manifest with a first perspective identifier, and each stream in the second set is associated in the manifest with a second perspective identifier.

[0033] In some embodiments, each stream in the first set is associated in the manifest with a respective matching set that has a first viewpoint identifier as an attribute, and each stream in the second set is associated in the manifest with a respective matching set that has a second viewpoint identifier as an attribute.

[0034] In some embodiments, each stream in the first set is associated in the manifest with a respective matching set having a first viewpoint identifier in a first descriptor, and each stream in the second set is associated in the manifest with a respective matching set having a second viewpoint identifier in a second descriptor.

[0035] In some embodiments, the manifest further includes an attribute indicating a coverage area for each of the viewpoints. In some embodiments, the manifest further includes an attribute indicating a location for each of the viewpoints. The location attribute may include Cartesian coordinates or latitude and longitude coordinates.

[0036] In some embodiments, the manifest further includes, for each viewpoint, information indicating at least one time period during which video for each viewpoint is available.

[0037] In some embodiments of the method for generating a manifest, the first video data and the second video data are received in a container file, where the first video data is organized into a first set of tracks and the second video data is organized into a second set of tracks, each of the tracks in the first set of tracks including a first track group identifier associated with a first viewpoint and each of the tracks in the second set of tracks including a second track group identifier associated with a second viewpoint. The viewpoint identifiers used in the manifest can be equal to the respective track group identifiers in the container file.

[0038] Some embodiments may be implemented by a client device, such as a device with a head-mounted display or other display device for 360-degree video. In some such methods, a manifest identifying multiple 360-degree video streams is received, where the manifest includes information identifying a viewpoint position for each respective stream. A first video stream identified in the manifest is acquired and displayed. A user interface element indicating a viewpoint position for a second video stream identified in the manifest is overlaid on the display of the first video stream. In response to a selection of the user interface element, the second video stream is acquired and displayed.

[0039] In some such embodiments, the manifest further includes information identifying a scope of coverage for at least one of the identified streams, and the client further displays an indication of the scope of coverage.

[0040] In some embodiments, the manifest further includes information identifying an available period of the second video stream, and the user interface element is displayed only during the available period.

[0041] In some embodiments, the manifest further includes information identifying a transition type for a transition from the first video stream to the second video stream. In response to a selection of the user interface element, the client presents a transition having the identified transition type, and after presenting the transition, the second video stream is displayed.

[0042] In some embodiments, the manifest further includes information identifying the location of at least one virtual viewpoint. In response to a selection of the virtual viewpoint, the client synthesizes a view from the virtual viewpoint and displays the synthesized view. One or more synthesized views can be used in a transition.

[0043] A method for selecting a projection format is further described. In some embodiments, a client receives a manifest identifying a plurality of 360-degree video streams. The manifest includes information identifying each projection format for each of the video streams. The manifest further includes information identifying each range of field of view sizes for each of the projection formats. The client determines a field of view size for display. The client then selects at least one of the video streams such that the determined field of view size falls within the identified range of field of view sizes for the projection format of the selected video stream. The client acquires at least one of the selected video streams and displays the acquired video stream using the determined field of view size.

[0044] Also included in this disclosure is a system comprising a processor and a non-transitory computer-readable medium storing instructions operable, when executed on the processor, to perform any of the methods described herein. Also included in this disclosure is a non-transitory computer-readable storage medium storing one or more container files, or manifests, generated using the methods disclosed herein. [Effects of the Invention]

[0045] A novel method and system for signaling information about different viewpoints in a multi-view omnidirectional media presentation is provided. [Brief explanation of the drawings]

[0046] [Figure 1A] 1 is a diagram of an exemplary communication system in which one or more embodiments may be implemented. [Figure 1B] 1B is a system diagram illustrating an exemplary wireless transmit / receive unit (WTRU) that can be used within the communication system of FIG. 1A according to an embodiment. [Figure 2] FIG. 1 is a diagram illustrating a hierarchical data model of an MPD. [Figure 3]FIG. 1 illustrates a user interface (UI) showing available viewpoints in an example presentation. [Figure 4] FIG. 10 illustrates another UI design example showing multiple viewpoints by overlaying available viewpoint icons on the HMD display. [Figure 5] FIG. 1 illustrates an example of viewpoints with different coverage areas. [Figure 6] FIG. 10 illustrates the change in viewpoint availability over time. [Figure 7] 1A and 1B are diagrams illustrating examples of an actual viewpoint and an intermediate virtual viewpoint. [Figure 8] FIG. 10 illustrates pixel mapping from a reference viewpoint to a target virtual viewpoint. DETAILED DESCRIPTION OF THE INVENTION

[0047] Exemplary Network for Implementing the Embodiments 1A illustrates an exemplary communication system 100 in which one or more disclosed embodiments may be implemented. The communication system 100 may be a multiple access system providing content, such as voice, data, video, messaging, and broadcast, to multiple wireless users. The communication system 100 enables the multiple wireless users to access such content by sharing system resources, including wireless bandwidth. For example, the communication system 100 may use one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tailed unique word DFT spread OFDM (ZT UW DTS-s OFDM), unique word OFDM (UW-OFDM), resource block-filtered OFDM, filter bank multicarrier (FBMC), and the like.

[0048] 1A, communications system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, RAN 104, CN 106, public switched telephone network (PSTN) 108, Internet 110, and other networks 112, although it will be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. By way of example, the WTRUs 102a, 102b, 102c, 102d, any of which may be referred to as a “station” and / or “STA,” are configured to transmit and / or receive wireless signals and may include user equipment (UE), mobile stations, fixed or mobile subscriber units, subscription-based units, pagers, cellular phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, wireless sensors, hotspot or Mi-Fi devices, Internet of Things (IoT) devices, watches or other wearables, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in industrial and / or automated processing chain situations), consumer electronic devices, devices operating in commercial and / or industrial wireless networks, etc. Any of the WTRUs 102a, 102b, 102c, and 102d may be referred to interchangeably as UEs.

[0049] The communications system 100 may also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communications networks, such as the CN 106 / 115, the Internet 110, and / or other networks 112. By way of example, the base stations 114a, 114b may be a Base Transceiver Station (BTS), a Node B, an eNodeB, a Home Node B, a Home eNodeB, a gNB, an NR Node B, a site controller, an access point (AP), a wireless router, and the like. While the base stations 114a, 114b are each shown as a single element, it will be understood that the base stations 114a, 114b may include any number of interconnected base stations and / or network elements.

[0050] The base station 114a may be part of the RAN 104 / 113, which may also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. The base station 114a and / or base station 114b may be configured to transmit and / or receive wireless signals on one or more carrier frequencies, which may be referred to as a cell (not shown). These frequencies may be licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide coverage for wireless service to a particular geographic area, which may be relatively fixed over time or may change. A cell may be further divided into cell sectors. For example, the cell associated with the base station 114a may be divided into three sectors. Thus, in one embodiment, the base station 114a may include three transceivers, i.e., one for each sector of the cell. In an embodiment, the base station 114a may use multiple-input multiple-output (MIMO) technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming can be used to transmit and / or receive signals in desired spatial directions.

[0051] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0052] More specifically, as noted above, the communication system 100 may be a multiple access system and may use one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. For example, the base station 114a in the RAN 104 / 113 and the WTRUs 102a, 102b, 102c may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 115 / 116 / 117 using Wideband CDMA (WCDMA). WCDMA may include communication protocols such as High Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High Speed ​​UL Packet Access (HSUPA).

[0053] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 116 using Long Term Evolution (LTE), and / or LTE Advanced (LTE-A), and / or LTE Advanced Pro (LTE-A Pro).

[0054] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as New Radio (NR) radio access that may establish an air interface 116 using NR.

[0055] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, the base station 114a and the WTRUs 102a, 102b, 102c may implement both LTE radio access and NR radio access, e.g., using a dual connectivity (DC) principle. Thus, the radio interface utilized by the WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions sent to and from multiple types of base stations (e.g., eNBs and gNBs).

[0056] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement a wireless technology such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi)), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile Communications (GSM), GSM Evolution Enhanced Data Rates (EDGE), GSM EDGE (GERAN), and the like.

[0057] 1A may be, for example, a wireless router, a Home NodeB, a Home eNodeB, or an access point, and may utilize any suitable RAT to facilitate wireless connectivity in a localized area, such as a workplace, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., used by drones), a roadway, and similar locations. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In an embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish a picocell or femtocell. 1A, the base station 114b may have a direct connection to the Internet 110. Therefore, the base station 114b may not need to access the Internet 110 via the CN 106 / 115.

[0058] The RAN 104 / 113 can communicate with the CN 106 / 115, which may be any type of network configured to provide voice, data, application, and / or Voice over Internet Protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data may have various quality of service (QoS) requirements, such as different throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, and the like. The CN 106 / 115 may provide call control, billing services, mobile location-based services, prepaid calling, Internet connectivity, video distribution, etc., and / or perform high-level security functions such as user authentication. Although not shown in FIG. 1A , it will be understood that the RAN 104 / 113 and / or the CN 106 / 115 can communicate directly or indirectly with other RANs that use the same RAT as the RAN 104 / 113 or a different RAT. For example, in addition to being connected to a RAN 104 / 113 that can utilize NR radio technology, the CN 106 / 115 can also communicate with another RAN (not shown) that uses GSM, UMTS, CDMA2000, WiMAX, E-UTRA, or WiFi radio technology.

[0059] The CN 106 / 115 may also serve as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or other networks 112. The PSTN 108 may include a circuit-switched telephone network providing plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols, such as Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and / or Internet Protocol (IP) in the TCP / IP Internet protocol suite. The network 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, the network 112 may include another CN connected to one or more RANs, which may use the same RAT as the RAN 104 / 113 or a different RAT.

[0060] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communications system 100 may include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks over different wireless links.) For example, the WTRU 102c shown in FIG. 1A may be configured to communicate with a base station 114a that can use cellular-based wireless technology and with a base station 114b that can use IEEE 802 wireless technology.

[0061] 1B is a system diagram illustrating an example WTRU 102. As shown in FIG. 1B, the WTRU 102 may include, among other things, a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and / or other peripherals 138. It will be understood that the WTRU 102 may include any sub-combination of the foregoing elements and still be consistent with an embodiment.

[0062] The processor 118 may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DCP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), field programmable gate array (FPGA) circuitry, any other type of integrated circuit (IC), a state machine, and the like. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. While FIG. 1B depicts the processor 118 and the transceiver 120 as separate components, it will be understood that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0063] The transmit / receive element 122 may be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) over the air interface 116. For example, in one embodiment, the transmit / receive element 122 may be an antenna configured to transmit and / or receive RF signals. In an embodiment, the transmit / receive element 122 may be an emitter / detector configured to transmit and / or receive IR, UV, or visible light signals, for example. In yet another embodiment, the transmit / receive element 122 may be configured to transmit and / or receive both RF and light signals. It will be understood that the transmit / receive element 122 may be configured to transmit and / or receive any combination of wireless signals.

[0064] 1B as a single element, the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may use MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0065] The transceiver 120 may be configured to modulate signals transmitted by the transmit / receive element 122 and demodulate signals received by the transmit / receive element 122. As mentioned above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers to enable the WTRU 102 to communicate over multiple RATs, such as, for example, NR and IEEE 802.11.

[0066] The processor 118 of the WTRU 102 may be coupled to and may receive user input data from a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. Additionally, the processor 118 may access information from and store data in any type of suitable memory, such as non-removable memory 130 and / or removable memory 132. The non-removable memory 130 may include random access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 may access information from, and store data in, memory that is not physically located on the WTRU 102, such as on a server or home computer (not shown).

[0067] The processor 118 may receive power from the power source 134 and may be configured to distribute and / or control power to other components in the WTRU 102. The power source 134 may be any suitable device for providing power to the WTRU 102. For example, the power source 134 may include one or more dry batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel-metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, and the like.

[0068] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to, or instead of, information from the GPS chipset 136, the WTRU 102 may receive location information from base stations (e.g., base stations 114a, 114b) over the air interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. It will be appreciated that the WTRU 102 may obtain location information by way of any suitable location-determination method while remaining consistent with an embodiment.

[0069] The processor 118 may further be coupled to other peripherals 138, which may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos and / or videos), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth module, a frequency modulation (FM) radio unit, a digital music player, a media player, a video game player module, an internet browser, a virtual reality and / or augmented reality (VR / AR) device, an activity tracker, and the like. The peripherals 138 may include one or more sensors, which may be one or more of a gyroscope, an accelerometer, a Hall effect sensor, a magnetometer, a direction sensor, a proximity sensor, a temperature sensor, a time sensor, a geolocation sensor, an altimeter, a light sensor, a touch sensor, a barometer, a gesture sensor, a biometric sensor, and / or a humidity sensor.

[0070] The WTRU 102 may include a full-duplex radio, in which case the transmission and reception of some or all of the signals (e.g., associated with a particular subframe for both the UL (e.g., for transmission) and the downlink (e.g., for reception)) are coincident and / or occur simultaneously. The full-duplex radio may include an interference management unit to reduce and / or substantially eliminate self-interference by signal processing by hardware (e.g., a choke) or by a processor (e.g., by a separate processor (not shown) or by the processor 118). In an embodiment, the WTRU 102 may include a half-duplex radio, in which case the transmission and reception of some or all of the signals (e.g., associated with a particular subframe for the UL (e.g., for transmission) or the downlink (e.g., for reception)) are coincident and / or occur simultaneously.

[0071] Although the WTRU is shown in Figures 1A-1B as a wireless terminal, it is contemplated that in some representative embodiments such a terminal may use a wired communication interface with the communication network (e.g., temporarily or permanently).

[0072] In an exemplary embodiment, the other network 112 may be a WLAN.

[0073] A WLAN in infrastructure basic service set (BSS) mode may have an access point (AP) for the BSS and one or more stations (STAs) associated with the AP. The AP may have access to or interface with a distribution system (DS) or another type of wired / wireless network that carries traffic to and / or from the BSS. Traffic originating from outside the BSS to a STA may arrive through the AP and be delivered to the STA. Traffic originating from a STA to a destination outside the BSS may be sent to the AP and delivered to the destination. Traffic between STAs within a BSS may be sent, for example, through the AP, in which case the source STA may send traffic to the AP, and the AP may deliver the traffic to the destination STA. Traffic between STAs within a BSS may be considered and / or referred to as peer-to-peer traffic. Peer-to-peer traffic may be sent between (e.g., directly between) the source and destination STAs using direct link setup (DLS). In some representative embodiments, the DLS may use 802.11e DLS or 802.11z Tunneled DLS (TDLS). A WLAN using an Independent BSS (IBSS) mode may have no APs, and the STAs within or using the IBSS (e.g., all of the STAs) may communicate directly with each other. The IBSS mode of communication is sometimes referred to herein as an "ad hoc" mode of communication.

[0074] When using the 802.11ac infrastructure mode of operation, or a similar mode of operation, an AP can transmit beacons on a fixed channel, such as a primary channel. The primary channel can be a fixed width (e.g., a 20 MHz wide bandwidth) or a width that is dynamically set via signaling. The primary channel can be the operating channel of the BSS and can be used by STAs to establish a connection with the AP. In some representative embodiments, for example, in an 802.11 system, carrier sense multiple access with collision avoidance (CSMA / CA) can be implemented. With CSMA / CA, STAs (e.g., every STA), including the AP, can sense the primary channel. If a particular STA senses / detects and / or determines that the primary channel is busy, the particular STA may back off. One STA (e.g., only one station) can transmit at any given time in a given BSS.

[0075] A high-throughput (HT) STA may use a 40 MHz wide channel for communication, for example, by combining a primary 20 MHz channel with adjacent or non-adjacent 20 MHz channels to form the 40 MHz wide channel.

[0076] A very high throughput (VHT) STA can support 20 MHz, 40 MHz, 80 MHz, and / or 160 MHz wide channels. 40 MHz and / or 80 MHz channels can be formed by combining adjacent 20 MHz channels. A 160 MHz channel can be formed by combining eight adjacent 20 MHz channels or two non-adjacent 80 MHz channels, which can be called an 80+80 configuration. For the 80+80 configuration, after channel encoding, the data can be passed through a segment parser that can split it into two streams. Inverse fast Fourier transform (IFFT) processing and time-domain processing can be performed separately for each stream. The streams can be mapped to two 80 MHz channels, and the data can be transmitted by the transmitting STA. At the receiver of the receiving STA, the operations described above for the 80+80 configuration can be reversed, and the combined data can be sent to the media access control (MAC).

[0077] Sub-1 GHz operating modes are supported by 802.11af and 802.11ah. Channel operating bandwidths and carriers are reduced in 802.11af and 802.11ah compared to those used in 802.11n and 802.11ac. 802.11af supports 5 MHz, 10 MHz, and 20 MHz bandwidths in the TV White Space (TVWS) spectrum, and 802.11ah supports 1 MHz, 2 MHz, 4 MHz, 8 MHz, and 16 MHz bandwidths using non-TVWS spectrum. According to representative embodiments, 802.11ah can support meter-type control / machine-type communications, such as MTC devices in macro coverage areas. MTC devices can have limited functionality, for example, including support for some and / or limited bandwidths (e.g., only support for them). MTC devices can include batteries with above-threshold battery life (e.g., to maintain very long battery life).

[0078] WLAN systems that can support multiple channels and channel bandwidths, such as 802.11n, 802.11ac, 802.11af, and 802.11ah, include a channel that can be designated as a primary channel. The primary channel can have a bandwidth equal to the largest common operating bandwidth supported by all STAs in the BSS. The bandwidth of the primary channel can be set and / or limited by the STA that supports the smallest bandwidth operating mode among all STAs operating in the BSS. In an 802.11ah example, the primary channel can be 1 MHz wide for STAs (e.g., MTC-type devices) that support (e.g., only support) 1 MHz mode, even if the AP and other STAs in the BSS support 2 MHz, 4 MHz, 8 MHz, 16 MHz, and / or other channel bandwidth operating modes. Carrier sensing and / or network allocation vector (NAV) setting can depend on the state of the primary channel. For example, if the primary channel is busy due to a STA (that only supports 1 MHz mode of operation) transmitting to the AP, the entire available frequency band may be considered busy even though most of the frequency band remains idle and available.

[0079] In the United States, the available frequency bands that can be used by 802.11ah are 902MHz to 928MHz. In South Korea, the available frequency bands are 917.5MHz to 923.5MHz. In Japan, the available frequency bands are 916.5MHz to 927.5MHz. The total bandwidth available for 802.11ah is 6MHz to 26MHz, depending on country regulations.

[0080] 1A-1B and the corresponding description thereof, one or more, or all, of the functions described herein with respect to one or more of the WTRUs 102a-d, base stations 114a-b, eNodeBs 160a-c, MME 162, SGW 164, PGW 166, gNBs 180a-c, AMFs 182a-b, UPFs 184a-b, SMFs 183a-b, DNs 185a-b, and / or any other devices described herein may be performed by one or more emulation devices (not shown). The emulation devices may be one or more devices configured to emulate one or more, or all, of the functions described herein. For example, the emulation devices may be used to test other devices and / or to simulate network and / or WTRU functionality.

[0081] The emulation device may be designed to perform one or more tests of other devices in a laboratory environment and / or an operator network environment. For example, one or more emulation devices may perform one or more or all functions but be fully or partially implemented and / or deployed as part of a wired and / or wireless communication network to test other devices in the communication network. One or more emulation devices may perform one or more or all functions but be temporarily implemented and / or deployed as part of a wired and / or wireless communication network. The emulation device may be directly coupled to another device to perform testing and / or may perform testing using wireless communication over the air.

[0082] The one or more emulation devices may perform one or more, including all, functions but are not implemented / deployed as part of a wired and / or wireless communication network. For example, the emulation devices may be utilized in a test lab and / or in a test scenario in an undeployed (e.g., test) wired and / or wireless communication network to perform testing of one or more components. The one or more emulation devices may be test equipment. Direct RF coupling and / or wireless communication via RF circuitry (which may include, e.g., one or more antennas) may be used by the emulation devices to transmit and / or receive data.

[0083] Detailed Description Problems Addressed in Some Embodiments Omnidirectional video provides a 360-degree experience, allowing viewers to view video in all directions around a central viewing position. However, viewers have generally been limited to a single viewpoint and are unable to navigate the scene by changing their viewpoint. For large-scale events such as the Olympic Games Opening Ceremony, an NFL or NBA tournament, or a carnival parade, a single 360-degree camera is insufficient to capture the entire scene. Capturing the scene from multiple viewpoints and allowing users to switch between different viewpoints while watching the video can provide an enhanced experience. Figure 3 illustrates a user interface that can be presented to a user in some embodiments to indicate available viewpoints. In this example, the user interface displays a top-down view of the venue and provides an indication of the locations of available viewpoints. In this case, viewpoint 302 is the active viewpoint (the viewpoint from which the user is currently experiencing the presentation) and is highlighted. Other viewpoints, such as viewpoints 304, 306, 308, 310, 312, 314, and 316, may be displayed to indicate their availability, but are not currently selected by the user.

[0084] During playback, a user interface, such as that shown in Figure 3, is superimposed on the rendered frame, e.g., in one of four corners, and the user can select different viewpoints using a user input device, such as a touchscreen or HMD controller. A viewpoint switch is then triggered, and the user's view is transitioned so that frames from the target viewpoint are rendered on the display. In some embodiments, a transition effect (e.g., fade-out / fade-in) accompanies the transition between viewpoints.

[0085] FIG. 4 illustrates another user design example in which available viewpoint locations are indicated using icons shown as an overlay on content 400 displayed on a head-mounted display. The location of each viewpoint icon in the user's view corresponds to the spatial location of an available viewpoint. In the example of FIG. 4, icons 406 and 414 may be displayed to correspond to viewpoints 306 and 314, respectively, in FIG. 3. The viewpoint icons may be rendered with correct depth effects that allow the user to perceive each viewpoint location in three-dimensional space within the scene. For example, icon 416 (corresponding to viewpoint location 316) may be displayed with larger dimensions than icons 406 and 414 to indicate that the viewpoint corresponding to icon 416 is closer to the current viewpoint. The user may select a viewpoint icon to switch the user's view of the rendered scene to the associated viewpoint.

[0086] In an exemplary embodiment, to enable support for multiple viewpoints, information about available viewpoints is signaled to a player (which may be, for example, an omnidirectional media player with a DASH client running on a user device such as an HMD). This information may include aspects such as the number of available viewpoints, the position and range of each viewpoint, and when video data is available for the viewpoint. Furthermore, because most omnidirectional media presentations are experienced via head-mounted displays, sudden changes in viewpoint may feel unnatural to a viewer immersed in a virtual environment. Therefore, it is preferable to support viewpoint transition effects that would provide smooth transitions when a user changes their viewpoint. These transitions can also be used by content creators to guide the user experience.

[0087] Viewpoint Media Component Grouping In some embodiments, media samples for omnidirectional media content with multiple viewpoints are stored in several tracks within a container file. A video player playing or streaming the content operates to identify which tracks belong to which viewpoint. To enable this, a mapping is made between the media tracks in the file and the viewpoints to which they belong. In some embodiments, this mapping is signaled at the media container (file format) level. In some embodiments, this mapping is signaled at the transport protocol level (DASH).

[0088] Media container level signaling (file formats) In ISO / IEC 14496-12 (ISO BMFF), a TrackGroupBox is defined to allow grouping several tracks in a container file that share certain characteristics or have a specific relationship. A TrackGroupBox contains zero or more boxes, and the specific characteristic or relationship is indicated by the box type of the contained boxes. The contained boxes contain an identifier that can be used to conclude that tracks belong to the same track group. Tracks that contain the same type of contained boxes within a TrackGroupBox and have the same identifier value within these contained boxes belong to the same track group. aligned(8) class TrackGroupBox extends Box('trgr') { } A track group type is defined by extending TrackGroupTypeBox, which contains a track_group_id identifier and a track_group_type that stores a four-character code that identifies the group type. The track_group_id and track_group_type pair identifies a track group within a file.

[0089] To group together several media tracks belonging to a single viewpoint, in some embodiments a new group type (ViewpointGroupTypeBox) is defined as follows: aligned(8) class ViewpointGroupTypeBox extends TrackGroupTypeBox('vpgr') { / / additional viewpoint data can be defined here }

[0090] In some embodiments, media has a ViewpointGroupTypeBox within a TrackGroupBox, and tracks belonging to the same viewpoint have the same value of track_group_id in each ViewpointGroupTypeBox. A 3DoF+ omnidirectional media player can therefore identify available viewpoints by parsing each track in the container and examining the number of unique track_group_id values ​​in the ViewpointGroupTypeBox for each track.

[0091] Transport-Level Signaling (DASH) The OMAF standard defines delivery-related interfaces for DASH. In some embodiments, information related to different viewpoints is signaled in the Media Presentation Descriptor. In DASH, each media component is represented by an AdaptationSet element in the MPD. In some embodiments, AdaptationSet elements that belong to the same viewpoint are grouped by defining additional attributes for the AdaptationSet element or by adding a descriptor to the AdaptationSet in which a viewpoint identifier is provided.

[0092] Several descriptors are defined in the MPEG-DASH standard. These include the SupplementalProperty descriptor, which can be used by media presentation authors to indicate that the descriptor contains auxiliary information that can be used by DASH clients for optimized processing. The semantics of the signaled information is specific to the scheme used, which is identified by the @schemeIdUri attribute. In this disclosure, several new XML elements and attributes are described to signal viewpoint-related information. The new elements can be defined in the same namespace as defined in the latest version of the OMAF standard (urn:mpeg:mpegI:omaf:2017) or in a separate new namespace (urn:mpeg:mpegI:omaf:2019) to distinguish between OMAF v1 and OMAF v2 features. For illustrative purposes, the namespace (urn:mpeg:mpegI:omaf:2017) will be used in the rest of this document.

[0093] In order to identify and describe the viewpoint to which a media component belongs, an embodiment is described in which an @schemeIdUri attribute is added to the SupplementalProperty element, equal to "urn:mpeg:mpegI:omaf:2017:ovp". Such a descriptor is referred to herein as an OMAF Viewpoint (OVP) descriptor. In some embodiments, there may be at most one OVP descriptor at the adaptation set level. An OVP descriptor may have an @viewpoint_id attribute with a value representing a unique viewpoint identifier. Example semantics for @viewpoint_id are given in Table 1. AdaptationSet elements with the same @viewpoint_id value may be recognized by a player as belonging to the same viewpoint.

[0094] [Table 1]

[0095] Signaling viewpoint information To allow a player to identify attributes belonging to different viewpoints (e.g., spatial relationships between viewpoints, viewpoint availability, etc.), in some methods described herein, additional metadata describing the viewpoints is signaled in the container file (and in the MPD file, in the case of streaming). Examples of viewpoint attributes signaled in some embodiments include viewpoint position, viewpoint validity range, viewpoint type, and viewpoint availability. The viewpoint position specifies the viewpoint location in the 3D space of the captured scene. The viewpoint validity range is the distance from the viewpoint within which objects can be rendered at a certain quality level. The certain quality level can be, for example, a minimum quality level, a quality level above a known quality threshold, a guaranteed quality level, or a quality level accepted or deemed acceptable by the provider of the omnidirectional media content. For example, objects within the validity range are of sufficient size in the rendered image to provide a resolution that provides good quality and ensures an acceptable viewing experience for the user. The viewpoint validity range depends on the characteristics of the capture device (e.g., camera sensor resolution, field of view, etc.). The coverage area may be determined at least in part by the camera lens density, which refers to the number of lenses integrated into a 360-degree video camera.

[0096] FIG. 5 shows an example of the coverage areas of various cameras. In this example, viewpoints 502, 504, 506, 508, 510, 512, 514, and 516 are shown with dotted circles 503, 505, 507, 509, 511, 513, 515, and 517 indicating the coverage area of ​​each viewpoint. The omnidirectional cameras located at viewpoints 502 and 510 may include more lenses to cover a wider area; therefore, the coverage areas of viewpoints 502 and 510 may cover penalty areas 520 and 522, as shown in FIG. 5. In this example, cameras along the sides of the field may have fewer lenses; therefore, the coverage areas of these viewpoints (504, 506, 508, 512, 514, and 516) may be smaller than that of viewpoints 502 and 510. Generally, omnidirectional cameras with more lenses, more component cameras, or higher quality component cameras (e.g., component cameras with higher quality optics, higher resolution, and / or higher frame rates) can be associated with higher coverage areas.

[0097] In another embodiment, the viewpoint coverage may be determined at least in part by parameters of the camera lens, such as focal length, aperture, depth of field, and focus distance. The coverage may define a minimum range and a maximum range, between which no stitching error occurs.

[0098] Viewpoints can be classified as real or virtual. A real viewpoint is one at which a real capture device is positioned to capture a scene from that viewpoint. A virtual viewpoint refers to a viewpoint at which the rendering of a viewport at that position requires further processing, such as view synthesis, and may utilize auxiliary information and / or video data from one or more other (e.g., real) viewpoints. Viewpoint availability specifies at what times during the presentation media data is available for the viewpoint.

[0099] User interactions with the viewport scene, such as zooming in or out, may be supported within the valid range. A virtual viewpoint can only be identified within the valid range of one or more cameras. The valid range can also be used as a criterion for generating transition paths. For example, a transition from viewpoint A to viewpoint B may involve multiple transition viewpoints if the valid ranges of the viewpoints cover the transition path.

[0100] Media container level signaling of viewpoint information In ISO BMFF, viewpoint-related information for static viewpoints can be signaled at the file level in a "meta" box (MetaBox). The "meta" box holds static metadata and contains only one mandatory "hdlr" box (HandlerBox), which declares the structure or format of the MetaBox. In some embodiments, for OMAF v2 metadata, the four-letter code "omv2" is used for the handler_type value in the "hdlr" box. To identify the viewpoints available in a file, some embodiments use a box called OMAFViewpointListBox, which contains a list of OMAFViewpointInfoBox instances. Each OMAFViewpointInfoBox holds information about a certain viewpoint. An example syntax for OMAFViewpointListBox is as follows: Box Type: 'ovpl' Container: MetaBox Mandatory: No Quantity: Zero or one aligned(8) class OMAFViewpointListBox extends Box('ovpl') { unsigned int(16) num_viewpoints; OMAFViewpointInfoBox viewpoints[]; }

[0101] An example of the semantics of an OMAFViewpointListBox is: num_viewpoints indicates the number of viewpoints in the media file. viewpoints is a list of OMAFViewpointInfoBox instances.

[0102] An example of the syntax of an OMAFViewpointInfoBox is given below. Box Type: 'ovpi' Container: OMAFViewpointListBox Mandatory: No Quantity: Zero or more aligned(8) class OMAFViewpointInfoBox extends Box('ovpi') { unsigned int(16) viewpoint_id; bit(1) effective_range_flag; bit(1) virtual_viewpoint_flag; bit(1) dynamic_position_flag; bit(5) reserved; if (effective_range_flag == 1) { unsigned int(32) effective_range; } unsigned int(32) num_availability_intervals; OMAFViewpointPositionGlobalBox(); / / optional OMAFViewpointPositionCartesianBox(); / / optional OMAFViewpointAvailabilityIntervalBox availability_intervals[]; Box other_boxes[];}

[0103] An example of the semantics of an OMAFViewpointInfoBox is: viewpoint_id is a unique identifier for the viewpoint.

[0104] The virtual_viewpoint_flag indicates whether the viewpoint is a virtual viewpoint (there is no capture device placed at the viewpoint position) or a captured viewpoint. The information needed to create the virtual viewpoint is signaled in the OMAFVirtualViewpointConfigBox.

[0105] dynamic_position_flag indicates whether the position is static or dynamic. If this flag is set, the timed-metadata track is used to provide the viewpoint position. Otherwise, the position is indicated by the OMAFViewpointPositionGlobalBox and / or OMAFViewpointPositionCartesianBox in this OMAFViewpointInfoBox.

[0106] The effective_range is the radius that defines a volumetric sphere centered at the viewpoint within which the viewpoint provides rendering at a certain quality (e.g., a minimum level of quality, quality above a known quality threshold, a guaranteed quality level, or a quality level acknowledged or deemed acceptable by the provider of the omnidirectional media content).

[0107] num_availability_intervals indicates the number of time intervals this viewpoint is available for.

[0108] availability_intervals is a list of OMAFViewpointAvailabilityIntervalBox instances.

[0109] In some embodiments, when the viewpoint position in space changes over time, the position information is signaled using a timed metadata track. A timed metadata track is a track in the media container (ISO BMFF) file, and the samples represent the dynamic metadata information. For dynamic viewpoint position information, some embodiments use a timed metadata track with sample entry type "vpps". A sample entry for this track can look like this: aligned(8) class OMAFDynamicViewpointSampleEntry extends MetadataSampleEntry('vpps') { unsigned int(16) viewpoint_id; unsigned int(3) coordinate_system_type; bit(5) reserved; } An example of the semantics of OMAFDynamicViewpointSampleEntry is as follows:

[0110] viewpoint_id is the identifier of the viewpoint whose position the samples in this timed metadata track define.

[0111] The coordinate_system_type indicates the coordinate system used to define the position of the viewpoint.

[0112] In some embodiments, the sample for the viewpoint position metadata track has the following structure: aligned(8) class OMAFViewpointPositionSample { if (coordinate_system_type == 1) { ViewpointPositionGlobalStruct(); }else if (coordinate_system_type == 2) { ViewpointPositionCartesianStruct(); } }

[0113] The sample format can rely on the coordinate system types defined in the sample entry of the timed-metadata track. ViewpointPositionGlobalStruct and ViewpointPositionCartesianStruct are described in further detail below.

[0114] Transport protocol level signaling of viewpoint information To identify and describe the set of viewpoints available in a media presentation, some embodiments include a SupplementaryProperty descriptor at the Period level. This descriptor may have an @schemeIdUri equal to "urn:mpeg:mpegI:omaf:2017:ovl" and is referred to herein as an OMAF Viewpoint List (OVL) descriptor. In some embodiments, there may be at most one OVL descriptor at the Period level. An OVL descriptor may contain at least one ovp element. An ovp element has an @id attribute with a value representing a unique viewpoint identifier and may contain sub-elements with information about the viewpoints.

[0115] Table 2 lists example elements and attributes used to signal viewpoint information in MPD files directed to DASH clients. Further details are given below.

[0116] [Table 2-1]

[0117] [Table 2-2]

[0118] In Table 2 and other tables in this disclosure, elements are in bold and attributes are not in bold and are preceded by an @. "M" indicates that the attribute is mandatory in the particular embodiment indicated in the table, "O" indicates that the attribute is optional in the particular embodiment indicated in the table, "OD" indicates that the attribute is optional with a default value in the particular embodiment indicated in the table, and "CM" indicates that the attribute is conditionally mandatory in the particular embodiment indicated in the table. <minoccurs> .. <maxoccurs>(N=Not Restricted).

[0119] The data types for the various elements and attributes are those defined in the XML Schema. The XML Schema for ovp is provided below in the section "XML Schema for DASH Signaling."

[0120] Viewpoint The "real" viewpoints correspond to 360° video cameras positioned at various locations capturing the scene from different vantage points. In some embodiments, the viewpoints can represent views from virtual positions. The virtual positions can represent points not associated with physical camera locations. The virtual positions can represent points at which synthetic content can be rendered, or at which content captured by one or more cameras at other (real) viewpoints can be transformed, processed, or combined to synthesize a virtual view. To provide the player with useful information about the camera settings used to capture the scene and their layout, in some embodiments, the spatial relationship between viewpoints is signaled by providing the position of each viewpoint. Location information can be represented in various ways in various embodiments. In some embodiments, global geolocation coordinates similar to those used by GPS systems can be used to identify camera / viewpoint locations. Alternatively, a Cartesian coordinate system can be used for positioning.

[0121] Media container level signaling of viewpoint position Described here are two example boxes that, when present in an OMAFViewpointInfoBox, can be used to identify the position of the viewpoint: OMAFViewpointPositionGlobalBox and OMAFViewpointPositionCartesianBox. In some embodiments, these boxes are optional. An example syntax for the proposed position boxes is given below. Additional boxes can also be introduced to provide position information based on other coordinate systems. Box Type: 'vpgl' Container: OMAFViewpointInfoBox Mandatory: No Quantity: Zero or one aligned(8) class ViewpointPositionGlobalStruct() { signed int(32) longitude; signed int(32) latitude; signed int(32) altitude; } aligned(8) class OMAFViewpointPositionGlobalBox extends Box('vpgl') { ViewpointPositionGlobalStruct(); }

[0122] In some embodiments, double precision or floating point types are used for longitude, latitude, and / or altitude values. Box Type: 'vpcr' Container: OMAFViewpointInfoBox Mandatory: No Quantity: Zero or one aligned(8) class ViewpointPositionCartesianStruct() { signed int(32) x; signed int(32) y; signed int(32) z; } aligned(8) class OMAFViewpointPositionCartesianBox extends Box('vpcr') { ViewpointPositionCartesianStruct(); }

[0123] Transport protocol level signaling of viewpoint positions To signal the position of the viewpoint, in some embodiments, an ovp:position element can be added to the ovp element. This element can contain an ovp:position:global element and / or an ovp:position:cartesian element. In some embodiments, at most one of each of these elements is present within the ovp:position element. The attributes of the ovp:position:global element provide the position of the viewpoint with respect to global geolocation coordinates in degrees. In some embodiments, the ovp:position:global element has three attributes: @longitude, @latitude, and @altitude. In some embodiments, the @altitude attribute is optional and may not be present. The attributes of the ovp:position:cartesian attribute provide the position of the viewpoint with respect to Cartesian coordinates. In some embodiments, three attributes are defined for the ovp:position:cartesian element: @x, @y, and @z, where only @z is optional.

[0124] Perspective availability In some cases, a viewpoint may not be available for the entire duration of a media presentation. Therefore, in some embodiments, the availability of a viewpoint is signaled before the media samples for that viewpoint are processed. This allows a player to process samples for a track belonging to a particular viewpoint only when the viewpoint is available.

[0125] The change in viewpoint availability over time is illustrated in FIG. 6. At time t1, only viewpoints 601, 602, 603, and 604 are available. Later, during the presentation at time t2, a penalty shot is awarded to one of the teams, with most of the players positioned near the goal on the right side. At that time point, two additional viewpoints 605 and 606 become available to the user until time t3. The time interval between t2 and t3 is the available interval for viewpoints 605 and 606. Using viewpoint availability information (e.g., received from the server), the player or streaming client operates to indicate to the user the availability of the additional viewpoints at time t2 during playback, for example, using the UI shown in FIG. 3 or FIG. 4. When the available interval begins, the player can present the user with the option to switch to any or all of the viewpoints available (e.g., newly available) during the available interval. As shown in FIG. 6, the user may be given the option to switch to viewpoint 605 or 606 starting at time t2. At the end of the available interval, the player may remove the option to switch to a viewpoint that is no longer available after the available interval ends. In some embodiments, when the available interval ends (e.g., at time t3 shown in FIG. 6), if the user is still in one of these viewpoints, the user may return to the viewpoint the user was in before switching to the viewpoint that is no longer available (e.g., viewpoint 605 or 606 shown in FIG. 6). In some embodiments, the available interval of a viewpoint may also be signaled for virtual viewpoints. However, the availability of these viewpoints depends on the availability of other reference viewpoints as well as any auxiliary information used to support the rendering of the virtual viewpoint.

[0126] Media container level signaling of viewpoint availability In some embodiments, a box (OMAFViewpointAvailaibilityIntervalBox) is introduced to signal the available interval. Zero or multiple instances of this box can be present in an OMAFViewpointInfoBox. When no OMAFViewpointAvailaibilityIntervalBox instance is present for a viewpoint, this indicates that the viewpoint is available for the entire duration of the presentation. Box Type: 'vpai' Container: OMAFViewpointInfoBox Mandatory: No Quantity: Zero or more aligned(8) class OMAFViewpointAvailabilityIntervalBox extends Box('vpai') { bit(1) open_interval_flag; bit(7) reserved; unsigned int(64) start_time; / / mandatory unsigned int(64) end_time; }

[0127] An example of the semantics for an OMAFViewpointAvailabilityIntervalBox is: open_inverval_flag is a flag indicating whether the available interval is an open interval (value 1) where the viewpoint is available from start_time to the end of the presentation, or a closed interval (value 0). If the flag is set (value 1), the end_time field in this box is not present.

[0128] start_time is the presentation time at which the viewpoint is available (corresponding to the configuration time for the first sample in the interval).

[0129] end_time is the presentation time after which the viewpoint is no longer available (corresponding to the construction time of the last sample in the interval).

[0130] Transport protocol level signaling of viewpoint availability In some embodiments, to signal the availability of a viewpoint in an MPD file, one or more ovp:availability elements may be added to an instance of the ovp element. This element denotes the availability period and has two attributes, @start and @end, indicating the presentation time at which the viewpoint is available and the presentation time of the last sample of the available interval, respectively.

[0131] Virtual Viewpoint In some embodiments, a virtual viewpoint is generated using an omnidirectional virtual view synthesis process. In some embodiments, this process utilizes one or more input (reference) viewpoints, their associated depth maps, and additional metadata describing the transformation vector between the input viewpoint position and the virtual viewpoint position. In some such embodiments, each pixel of the input omnidirectional viewpoint is mapped to a position in the virtual viewpoint sphere by mapping pixels in the reference viewpoint's equirectangular frame to points in 3D space and then projecting them to the target virtual viewpoint. One such view synthesis process is described in great detail in "Extended VSRS for 360-degree video," MPEG121, Gwangju, Korea, January 2018, m41990, and is shown in FIG. 8. In the example of FIG. 8, point 802 has a position described by angular coordinates (φ, θ) and depth z relative to input viewpoint 804. In generating a virtual viewpoint 806 that is displaced by a vector (Tx, Ty, Tz) from the input viewpoint 804, the angular coordinates (φ′, θ′) for point 802 are found with respect to the virtual viewpoint 806. The displacement vector (Tx, Ty, Tz) may be determined based on the viewpoint position signaled in the container file, manifest, timed metadata track, or otherwise.

[0132] In various embodiments, various techniques can be used to generate the virtual viewpoint. The synthesized virtual viewpoint frames from various reference viewpoints can then be merged together using a blending process to generate a final equidistant cylindrical frame at the virtual viewpoint. Holes that appear in the final frame due to occlusion of the reference viewpoint can be handled using inpainting and filling steps.

[0133] A virtual viewpoint is a viewpoint that is not captured. A viewport can render at the virtual viewpoint using video data from other viewpoints and / or other auxiliary information. In some embodiments, the information used to render a scene from the virtual viewpoint is signaled in an OMAFVirtualViewpointConfigBox, which is present in the OMAFViewpointInfoBox when the virtual_viewpoint flag is set. In some embodiments, the OMAFVirtualViewpointConfigBox may be defined as follows: Box Type: vvpc' Container: OMAFViewpointInfoBox Mandatory: No Quantity: Zero or more aligned(8) class OMAFVirtualViewpointConfigBox extends Box('vvpc') { unsigned int(5) synthesis_method; unsigned int(3) num_reference_viewpoints; unsigned int(16) reference_viewpoints[]; / / optional boxes but no fields }

[0134] An example of the semantics for the OMAFVirtualViewpointConfigBox fields is given below.

[0135] synthesis_method indicates which synthesis method is used to generate the virtual viewpoint. The value of synthesis_method can be an index into a list of view synthesis methods, for example, depth-image based rendering, image-warping based synthesis, etc.

[0136] num_reference_viewpoints indicates the number of viewpoints used as references in synthesizing virtual viewpoints.

[0137] reference_viewpoints is a list of viewpoint ids that will be used as references when compositing viewports for this viewpoint.

[0138] In another embodiment, the identifiers of the tracks containing the information necessary for the compositing process are signaled directly in the virtual viewpoint configuration box, which can be implemented as follows: aligned(8) class OMAFVirtualViewpointConfigBox extends Box('vvpc') { unsigned int(5) synthesis_method; unsigned int(3) num_reference_tracks; unsigned int(16) reference_track_ids[]; / / optional boxes but no fields }

[0139] An example of the semantics of the OMAFVirtualViewpointConfigBox fields for this embodiment is as follows:

[0140] synthesis_method indicates which synthesis method is used to generate the virtual viewpoint. The value of synthesis_method is an index into a list of view synthesis methods, for example, depth-image based rendering, image-warping based synthesis, etc.

[0141] num_reference_tracks indicates the number of tracks in the container file that are used as references in the synthesis of virtual viewpoints.

[0142] reference_track_ids is a list of track identifiers for tracks used in compositing the viewport for this viewpoint.

[0143] Perspective Group Signaling In a large-scale event such as the FIFA World Cup, several events may take place in parallel at different venues or locations. For example, several competitions may take place in different stadiums, perhaps in different cities. In some embodiments, viewpoints may be grouped based on the geolocation of the events / venues. In some embodiments, a ViewpointGroupStruct structure is used to store information about groups of viewpoints within a media container file. An example syntax for this structure is as follows: aligned(8) class ViewpointGroupStruct() { unsigned int(8) viewpoint_group_id; signed int(32) longitude; signed int(32) latitude; unsigned int(8) num_viewpoints; unsigned int(16) viewpoint_ids[]; string viewpoint_group_name; }

[0144] An example of the semantics of the ViewpointGroupStruct fields is as follows:

[0145] viewpoint_group_id is a unique id that identifies the viewpoint group.

[0146] longitude is the longitude coordinate of the geolocation of the event / venue where the viewpoint is located.

[0147] latitude is the geolocation latitude coordinate of the event / venue where the viewpoint is located.

[0148] num_viewpoints is the number of viewpoints in the viewpoint group.

[0149] viewpoint_ids is an array with the ids of the viewpoints that are part of the viewpoint group.

[0150] viewpoint_group_name is a string with a name that describes the group.

[0151] To signal the viewpoint groups available in a media container file, an OMAFViewpointGroupsBox can be added to a MetaBox in an ISO BMFF container file. An example of the syntax of an OMAFViewpointGroupsBox is given below: Box Type: 'ovpg' Container: MetaBox Mandatory: No Quantity: Zero or one aligned(8) class OMAFViewpointGroupsBox extends Box('ovpg') { unsigned int(8) num_viewpoint_groups; ViewpointGroupStruct viewpoint_groups[]; }

[0152] An example of the semantics for the fields in this box is: num_viewpoint_groups is the number of viewpoint groups.

[0153] viewpoint_groups is an array of ViewpointGroupStruct instances, providing information about each viewpoint group.

[0154] For transport protocol level signaling (e.g., DASH), an ovg element is defined to signal the view group available in a media presentation and may be signaled in the OVL descriptor mentioned above. An OVL descriptor may contain one or more ovg elements. An ovg element has an @id attribute with a unique view group identifier and a value representing other attributes that describe the group. Table 3 lists the attributes of an example ovg element.

[0155] [Table 3]

[0156] Signaling the viewpoint shift effect The following examples of transition types are disclosed herein: basic transitions, viewpoint path transitions, and auxiliary information transitions. Basic transitions are predefined transitions that can be used when switching from one viewpoint to another. An example of such a transition is a fade-to-black transition, where the rendered view gradually fades out to black and then fades in frames from the new viewpoint. Viewpoint path transitions allow content creators to specify a path that the player can follow across other viewpoints when switching to a target viewpoint. Auxiliary information transitions are transitions that rely on auxiliary information provided by the content creator in a separate track. For example, the auxiliary track may contain depth information that can be used to render intermediate virtual views as the viewport moves from a first viewpoint to a target viewpoint.

[0157] In some embodiments, the transition can be based on the rendering of intermediate virtual views. This can be done using a view synthesis process, such as depth-image-based rendering (DIBR), described in, for example, C. Fehn, "Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV," SPIE Stereoscopic Displays and Virtual Reality Systems XI, vol. 5291, May 2004, pp. 93-104. DIBR uses depth information to project pixels in a 2D plane to their positions in 3D space and reproject them to another plane. Because no capture device exists at these intermediate viewpoints (e.g., no 360-degree camera), they are referred to herein as virtual viewpoints. The number of intermediate virtual viewpoints rendered between the source and destination viewpoints determines the smoothness of the transition and depends on the player / device capabilities and the availability of auxiliary information for these intermediate viewpoints.

[0158] FIG. 7 illustrates an embodiment using virtual viewpoints. In the example of FIG. 7, viewpoints 702, 704, 706, 708, 710, 712, 714, and 716 are only viewpoints using capture devices, while the remaining intermediate viewpoints (703, 705, 707, 709, 711, 713, 715, and 717) are virtual viewpoints. Other types of auxiliary information include a point cloud stream, additional reference frames from nearby viewpoints (to improve the quality of the virtual view), and occlusion information (to support the hole-filling step in the view synthesis process and improve the quality of the virtual view obtained at the intermediate viewpoints). In some embodiments, the point cloud stream is used to enable rendering of a virtual view at any viewpoint position between the source and destination viewpoints. In some embodiments, the point cloud is rendered using the technique described in Paul Rosenthal and Lars Linsen, "Image-space point cloud rendering," Proceedings of Computer Graphics International, pp. 136-143, 2008.

[0159] Media container-level signaling of viewpoint transition effects Some embodiments operate to signal transition effects between pairs of viewpoints in a container file as a list of boxes in a new OMAFViewpointTransitionEffectListBox, which can be placed in a MetaBox at the file level. In some embodiments, at most one instance of this box exists in a MetaBox. The boxes in the OMAFViewpointTransitionEffectListBox are instances of OMAFViewpointTransitionBox. Example syntax for the two boxes is given below: Box Type: 'vptl' Container: MetaBox Mandatory: No Quantity: Zero or one aligned(8) class OMAFViewpointTransitionEffectListBox extends Box('vptl') { OMAFViewpointTransitionBox transitions[]; } Box Type: 'vpte' Container: OMAFViewpointTransitionEffectListBox Mandatory: No Quantity: One or more aligned(8) class OMAFViewpointTransitionEffectBox extends Box('vpte') { unsigned int(16) src_viewpoint_id; / / mandatory unsigned int(16) dst_viewpoint_id; / / mandatory unsigned int(8) transition_type; / / mandatory / / additional box to specify the parameters of the transition }

[0160] An example of the semantics for the fields of an OMAFViewpointTransitionBox is: src_viewpoint_id is the id of the source viewpoint.

[0161] dst_viewpoint_id is the id of the destination viewpoint.

[0162] transition_type is an integer that identifies the type of transition. A value of 0 indicates a basic transition. A value of 1 indicates a viewpoint path transition. A value of 2 indicates an auxiliary information transition. The remaining values ​​are reserved for future transitions.

[0163] In some embodiments, additional boxes may be present in an OMAFViewpointTransitionBox that relate to and provide further information about specific types of transitions. Additional boxes may be defined for each of the previously defined transition types. OMAFBasicViewpointTransitionBox is present when the transition_type field of OMAFViewpointTransitionBox is equal to 0. This box contains only one field, basic_transition_type, whose value indicates a specific transition from a set of predefined basic transitions. OMAFPathViewpointTransitionBox is present when the transition_type field of OMAFViewpointTransitionBox is equal to 1. This box contains a list of viewpoint identifiers that the player can follow when the user requests a transition to a target viewpoint. In some embodiments, a field may also be provided to indicate the speed of the transition along the path. OMAFAuxiliaryInfoViewpointTransitionBox is present when the transition_type field of OMAFViewpointTransitionBox is equal to 2. This box contains two fields: a type field that specifies the nature of the transition (e.g., generating a virtual viewpoint), and an aux_track_id that provides a reference to one of the tracks in the file that contains the timed auxiliary information used to implement the transition effect. An example syntax for the three aforementioned boxes is given below. aligned(8) class OMAFBasicViewpointTransitionBox extends Box('vptb') { unsigned int(8) basic_transition_type; } aligned(8) class OMAFPathViewpointTransitionBox extends Box('vptp') { unsigned int(16) intermediate_viewpoints[]; } aligned(8) class OMAFAuxiliaryInfoViewpointTransitionBox extends Box('vpta') { unsigned int(8) type; unsigned int(32) aux_track_id; }

[0164] Transport protocol level signaling of viewpoint transition effects (e.g., DASH) Viewpoint transition effect information signaled at the container level may also be signaled at the transport protocol level in the manifest file. If the container file includes viewpoint transition effect information, this information preferably matches the information signaled in the manifest file. In some embodiments, viewpoint transition effects are signaled within OVL descriptors, such as those described above. Transition effects between pairs of viewpoints may be signaled by an ovp:transition element. In one example, this element has three attributes: @src, @dst, and @type. These attributes specify the id of the source viewpoint, the id of the destination viewpoint, and the type of transition effect, respectively. For some types of transition effects, the ovp:transition element may contain child elements that provide further information used by clients to render these transitions.

[0165] Table 4 lists example elements and attributes that may be used to signal viewpoint transition effects in an MPD file.

[0166] [Table 4]

[0167] Signaling the recommended projection format for the FoV Within different FoV ranges, different projection formats may be advantageous. For example, a rectilinear projection format may work well at a 90° field of view, but an undesirable stretching effect may be visible when using rectilinear projection at a larger field of view, such as 130°. Conversely, a projection format such as an "asteroid" stereoscopic projection, or fisheye projection format, may not work well at a 90° FoV, but may present a suitable rendering experience at higher FoVs.

[0168] In some embodiments, an OMAFRecommendedProjectionListBox is provided as further metadata information in the "meta" box to signal recommended projection formats for a range of device field of view (FoV) values. This box contains one or more OMAFRecommendedProjectionBox instances. The OMAFRecommendedProjectionBox defines the horizontal and vertical FoV ranges and provides recommended projection types for the specified FoV ranges. A player or streaming client receiving this signaling can determine the size of the field of view of the device on which the player or streaming client is running (e.g., it can look up the device's FoV capabilities from a local database or obtain this characteristic via an API call to the HMD's operating system). The player or streaming client can compare this determined field of view size with the FoV ranges defined in the OMAFRecommendedProjectionBoxes to determine which of the recommended projection types corresponds to the device's field of view. The player or streaming client can then request content in the determined recommended projection format. Example syntax for these boxes is provided below. Box Type: 'orpl' Container: MetaBox Mandatory: No Quantity: Zero or one aligned(8) class OMAFRecommendedProjectionListBox extends Box('orpl') { OMAFRecommendedProjectionBox recommendations[]; } Box Type: 'orpr' Container: OMAFRecommendedProjectionListBox Mandatory: No Quantity: One or more aligned(8) class OMAFRecommendedProjectionBox extends Box('orpr') { bit(3) reserved = 0; unsigned int(5) projection_type; unsigned int(32) min_hor_fov; unsigned int(32) min_ver_fov; unsigned int(32) max_hor_fov; unsigned int(32) max_ver_fov; }

[0169] An example of the semantics of the field OMAFRecommendedProjectionBox is: projection_type indicates the type of mapping of the projected picture to a spherical coordinate system, as specified by the OMAF standard. The value of projection_type can be an index into a list of rendering projections, including rectilinear, asteroid, azimuthal equidistant, fisheye, etc.

[0170] min_hor_fov and min_ver_fov provide the minimum horizontal and vertical display field of view, in units of 2-16 degrees. min_hor_fov can range from 0 to 360x216, inclusive. min_ver_fov can range from 0 to 180x216, inclusive.

[0171] max_hor_fov and max_ver_fov provide the maximum horizontal and vertical display field of view, in units of 2-16 degrees. max_hor_fov can range from 0 to 360x216, inclusive. max_ver_fov can range from 0 to 180x216, inclusive.

[0172] If a projection format is recommended for a particular FoV, min_hor_fov is equal to max_hor_fov and min_ver_fov is equal to max_ver_fov.

[0173] In another embodiment, a content creator or content provider can provide information identifying recommended viewports for devices with different FoV configurations with appropriate projection recommendations, and different devices with different FoVs can use the recommended projection format to render the 360 ​​video content according to the recommended viewports.

[0174] OMAF describes the recommended viewport information box (RcvpInfoBox) as follows: class RcvpInfoBox extends FullBox('rvif',0,0) { unsigned int(8) viewport_type; string viewport_description; } viewport_type specifies the preferred viewport type, as listed in Table 5.

[0175] [Table 5]

[0176] In some embodiments, a further type of recommended viewport (which may be assigned, for example, to type 2) is used based on the FOV of the rendering device. In some embodiments, the viewport_description in the RcvpInfoBox may be used to indicate the recommended rendering projection and corresponding rendering FOV range. In some embodiments, an optional box is added to the RcvpInfoBox based on the viewport_type to indicate further parameters to be used for the corresponding recommended type. For example, an OMAFRecommendedProjectionBox may be signaled when a viewport type is associated with an FOV. class RcvpInfoBox extends FullBox('rvif',0,0) { unsigned int(8) viewport_type; string viewport_description; Box[] other_boxes; / / optional }

[0177] In another embodiment, the recommended viewports can conform to multiple recommendation types or subtypes to provide users with flexible selection. For example, viewing statistics can be further divided into statistics by measurement period (e.g., weekly, monthly), geography (country, city), or age (youth, adult). Table 6 shows a hierarchical recommendation structure that can be used in some embodiments.

[0178] [Table 6]

[0179] In some embodiments, a recursive RcvpInfoBox structure is used to support a hierarchical recommendation structure. The other_boxes field proposed in the RcvpInfoBox structure can contain an RcvpInfoBox to specify a subtype, such as: class RcvpInfoBox extends FullBox('rvif',0,0) { unsigned int(8) viewport_type; string viewport_description; RcvpInfoBox(); / / optional; }

[0180] A single Director's Cut recommended viewport can propose multiple tracks, each of which can support one or more recommended rendering projections for FOV ranges. An example structure of an RcvpInfoBox is shown below. A viewport_type value of 0 in the first RcvpInfoBox may indicate that such a recommended viewport is from the Director's Cut, and a value of viewport_type (e.g., 1) in the second RcvpInfoBox may indicate that the track associated with this Director's Cut recommended viewport is recommended for devices with a particular rendering FOV. One or more instances of OMAFRecommendedProjectionBox may be signaled to provide recommended projections for corresponding FOV ranges. RcvpInfoBox{ viewport_type = 0; / / recommended director's cut RcvpInfoBox { Viewport_type=1; / / recommended for device FOV OMAFRecommendedProjectionBox(); / / projection method 1 OMAFRecommendedProjectionBox(); / / projection method 2 viewport_description; } viewport_description; }

[0181] In a DASH MPD, SupplementalProperty and / or EssentialProperty descriptors with @schemeIdUri equal to "urn:mpeg:dash:crd" can be used to provide a Content Recommendation Description (CRD). The @value of a SupplementalProperty or EssentialProperty element using the CRD scheme can be implemented as a comma-separated list of values ​​for the CRD parameters, as shown in Table 7.

[0182] [Table 7]

[0183] XML Schema for DASH Signaling An example of an XML schema for DASH signaling that can be used in some embodiments is as follows: <?xml version="1.0" encoding="UTF-8"?> <xs:schema xmlns:xs="http: / / www.w3.org / 2001 / XMLSchema" targetnamespace="urn:mpeg:mpegI:omaf:2017" xmlns:omaf="urn:mpeg:mpegI:omaf:2017" elementformdefault="qualified"> <xs:element name="ovp" type="omaf:viewpointType" / > <xs:element name="ovg" type="omaf:viewpointGroupType" / > <xs:complextype name="viewpointType"> <xs:attribute name="id" type="xs:string" use="required" / > <xs:attribute name="effective_range" type="xs:unsignedInt" use="optional" / > <xs:attribute name="virtual" type="xs:boolean" use="optional" default="false" / > <xs:attribute name="synthesisMethod" type="xs:unsignedByte" use="optional" / > <xs:attribute name="refViewpointIds" type="xs:boolean" use="optional" / > <xs:attribute name="dynamicPosition" type="xs:boolean" use="optional" default="false" / > <xs:element name="position" type="omaf:viewpointPositionType" minOccurs="0" maxOccurs="1" / > <xs:element name="availability" type="omaf:viewpointAvailabilityType" maxOccurs="unbounded" / > <xs:element name="transition" type="omaf:vpTransitionType" minOccurs="0" maxOccurs="unbounded" / > < / xs:complextype> <xs:complextype name="viewpointPositionType"> <xs:element name="global" type="omaf:viewpointGlobalPositionType" maxOccurs="1" / > <xs:element name="cartesian" type="omaf:viewpointCartesianPositionType" maxOccurs="1" / > < / xs:complextype> <xs:complextype name="viewpointGlobalPositionType" use="optional" maxoccurs="1"> <xs:attribute name="longitude" type="xs:double" use="required" / > <xs:attribute name="latitude" type="xs:double" use="required" / > <xs:attribute name="altitude" type="xs:double" use="optional" default="0" / > < / xs:complextype> <xs:complextype name="viewpointCartesianPositionType" use="optional" maxoccurs="1"> <xs:attribute name="x" type="xs:int" use="required" / > <xs:attribute name="y" type="xs:int" use="required" / > <xs:attribute name="z" type="xs:int" use="optional" default="0" / > < / xs:complextype> <xs:complextype name="viewpointAvailabilityType" use="optional" maxoccurs="unbounded"> <xs:attribute name="start" type="xs:unsignedLong" use="required" / > <xs:attribute name="end" type="xs:unsignedLong" use="optional" / > < / xs:complextype> <xs:complextype name="vpTransitionType" use="optional" maxoccurs="unbounded"> <xs:attribute name="src" type="xs:string" use="required" / > <xs:attribute name="dst" type="xs:string" use="required" / > <xs:attribute name="type" type="xs:unsignedByte" use="required" / > <xs:element name="omaf:vpBasicTransitionType" use="optional" maxOccurs="1" / > <xs:element name="omaf:vpPathTransitionType" use="optional" maxOccurs="1" / > <xs:element name="omaf:vpAuxTransitionType" use="optional" maxOccurs="1" / > < / xs:complextype> <xs:complextype name="vpBasicTransitionType"> <xs:attribute name="type" type="unsignedByte" use="required" / > < / xs:complextype> <xs:complexttype name="vpPathTransitionType"> <xs:attribute name="viewpoints" type="xs:string" use="required" / > < / xs:complexttype> <xs:complextype name="vpAuxTransitionType"> <xs:attribute name="auxIdList" type="xs:string" use="required" / > < / xs:complextype> <xs:complextype name="viewpointGroupType"> <xs:attribute name="id" type="xs:string" use="required" / > <xs:attribute name="name" type="xs:string" use="optional" / > <xs:attribute name="longitude" type="xs:double" use="required" / > <xs:attribute name="latitude" type="xs:double" use="required" / > <xs:attribute name="viewpointIds" type="xs:string" use="required" / > < / xs:complextype> < / xs:schema>

[0184] Further embodiments In some embodiments, the method includes receiving at least first 360-degree video data representing a view from a first perspective and second 360-degree video data representing a view from a second perspective, and generating a container file (e.g., an ISO Base Media File Format file) for at least the first and second video data, wherein the first video data is organized into a first set of tracks and the second video data is organized into a second set of tracks, each track in the first set of tracks including a first track group identifier associated with the first perspective and each track in the second set of tracks including a second track group identifier associated with the second perspective.

[0185] In some such embodiments, each of the tracks in the first set of tracks includes a respective instance of a viewpoint group type box that includes a first track group identifier, and each of the tracks in the second set of tracks includes a respective instance of a viewpoint group type box that includes a second track group identifier.

[0186] In some embodiments where the container file is organized into a hierarchical box structure and the container file includes a viewpoint list box that identifies at least a first viewpoint information box and a second viewpoint information box, the first viewpoint information box includes at least (i) a first track group identifier and (ii) an indication of a time interval for which video is available from the first viewpoint, and the second viewpoint information box includes at least (i) a second track group identifier and (ii) an indication of a time interval for which video is available from the second viewpoint. The time interval indication can be a list of instances of the available interval box for each viewpoint.

[0187] In some embodiments where the container file is organized into a hierarchical box structure and the container file includes a viewpoint list box that identifies at least a first viewpoint information box and a second viewpoint information box, the first viewpoint information box includes at least (i) a first track group identifier and (ii) an indication of the location of the first viewpoint, and the second viewpoint information box includes at least (i) a second track group identifier and (ii) an indication of the location of the second viewpoint. The location indication may include Cartesian coordinates or latitude and longitude coordinates.

[0188] In some embodiments in which the container file is organized into a hierarchical box structure and the container file includes a viewpoint list box that identifies at least a first viewpoint information box and a second viewpoint information box, the first viewpoint information box includes at least (i) a first track group identifier and (ii) an indication of the coverage area of ​​the first viewpoint, and the second viewpoint information box includes at least (i) a second track group identifier and (ii) an indication of the coverage area of ​​the second viewpoint.

[0189] In some embodiments where the container file is organized into a hierarchical box structure and the container file includes a transition effect list box that identifies at least one transition effect box, each transition effect box includes a source viewpoint identifier, a destination viewpoint identifier, and a transition type identifier. The transition type identifier can identify a basic transition or a viewpoint path transition. If the transition type identifier identifies a path viewpoint transition box, the path viewpoint transition box can include a list of viewpoint identifiers. If the transition type identifier identifies an auxiliary information viewpoint transition box, the auxiliary information viewpoint transition box can include a track identifier.

[0190] In some embodiments in which the container file is organized into a hierarchical box structure that includes a meta box, the meta box identifies at least one recommended projection list box, and each recommended projection list box includes information identifying (i) a projection type and (ii) a corresponding field of view range. The information identifying the corresponding field of view range may include a minimum horizontal field of view angle, a maximum horizontal field of view angle, a minimum vertical field of view angle, and a maximum vertical field of view angle.

[0191] Some embodiments include a non-transitory computer storage medium that stores a container file generated according to any of the methods described herein.

[0192] In some embodiments, a method includes receiving at least first 360-degree video data representing a view from a first perspective and second 360-degree video data representing a view from a second perspective, and generating a manifest, such as an MPEG-DASH MPD, in which at least one stream in a first set of streams is identified in the manifest, each stream in the first set representing at least a portion of the first video data, and at least one stream in a second set of streams is identified, each stream in the second set representing at least a portion of the second video data, each stream in the first set being associated in the manifest with a first perspective identifier, and each stream in the second set being associated in the manifest with a second perspective identifier.

[0193] In some such embodiments, each of the streams in the first set is associated in the manifest with a respective matching set having a first viewpoint identifier as an attribute, and each of the streams in the second set is associated in the manifest with a respective matching set having a second viewpoint identifier as an attribute, which may be an @viewpoint_id attribute.

[0194] In some embodiments, each of the streams in the first set is associated in the manifest with a respective conformance set having a first viewpoint identifier in a first descriptor, and each of the streams in the second set is associated in the manifest with a respective conformance set having a second viewpoint identifier in a second descriptor. The first and second descriptors may be SupplementalProperty descriptors.

[0195] In some embodiments, the manifest includes attributes indicating a coverage area for each of the viewpoints. In some embodiments, the manifest includes attributes indicating a location for each of the viewpoints. The location attributes may include Cartesian coordinates or latitude and longitude coordinates. In some embodiments, the manifest includes information indicating at least one time period for which video is available for each viewpoint.

[0196] In some embodiments, the first video data and the second video data are received in a container file (such as an ISO Base Media File Format file), where the first video data is organized into a first set of tracks and the second video data is organized into a second set of tracks, each of the tracks in the first set of tracks including a first track group identifier associated with a first viewpoint and each of the tracks in the second set of tracks including a second track group identifier associated with a second viewpoint. The viewpoint identifiers used in the manifest are equal to the respective track group identifiers in the container file.

[0197] In some embodiments, a method includes receiving a manifest identifying a plurality of 360-degree video streams, the manifest including, for each identified stream, information identifying a viewpoint position for each stream, acquiring and displaying a first video stream identified in the manifest, and overlaying a user interface element on the display of the first video stream that indicates a viewpoint position for a second video stream identified in the manifest. In some embodiments, the method includes acquiring and displaying the second video stream in response to a selection of the user interface element.

[0198] In some embodiments where the manifest further includes information identifying a validity range for at least one of the identified streams, the method further includes displaying an indication of the validity range. In some embodiments where the manifest further includes information identifying a usable period of the second video stream, the user interface element is displayed only during the usable period.

[0199] In some embodiments, the manifest includes information identifying a transition type for a transition from the first video stream to the second video stream. In response to a selection of a user interface element, the method includes presenting a transition having the identified transition type and obtaining and displaying the second video stream, wherein the second video stream is displayed after the presentation of the transition.

[0200] In some embodiments in which the manifest further includes information identifying the location of at least one virtual viewpoint, the method further includes synthesizing a view from the virtual viewpoint in response to a selection of the virtual viewpoint, and displaying the synthesized view.

[0201] In some embodiments, the method includes receiving a manifest (MPEG-DASH MPD) identifying a plurality of 360-degree video streams, the manifest including information identifying each projection format of each of the video streams, the manifest further including information identifying each range of field of view sizes for each of the projection formats; determining a field of view size for display; selecting at least one of the video streams such that the determined field of view size is within the identified range of field of view sizes for the projection format of the selected video stream; and acquiring at least one of the selected video streams and displaying the acquired video stream using the determined field of view size.

[0202] A further embodiment includes a system comprising a processor and a non-transitory computer-readable medium storing instructions operable, when executed on the processor, to perform any of the methods described herein.

[0203] It should be noted that the various hardware elements of one or more of the described embodiments are referred to as “modules,” which perform (i.e., implement, execute, and the like) various functions described herein with respect to each module. As used herein, a module includes hardware deemed appropriate by one of ordinary skill in the art for a given implementation (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices, etc.). It should be noted that each described module may also include executable instructions for performing one or more functions described as being performed by the respective module, and that these instructions may take or include the form of hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and / or the like, and may be stored in any suitable non-transitory computer-readable medium or media, commonly referred to as RAM, ROM, etc.

[0204] Although functions and elements are described above in particular combinations, those skilled in the art will understand that each function or element can be used alone or in any combination with the other functions and elements. In addition, the methods described herein can be implemented in a computer program, software, or firmware embodied in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random-access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs). A processor in conjunction with software can be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.< / maxoccurs> < / minoccurs>

Claims

1. receiving information from a server identifying a view and a respective attribute set; receiving, from the server, information identifying one or more omnidirectional videos captured from each of the identified viewpoints; rendering a video of the one or more omnidirectional videos based on at least one attribute of a set of viewpoint attributes for each of the identified viewpoints; A method for providing

2. The method of claim 1 , wherein the set of attribute sets includes a viewpoint position.

3. The method of claim 1 , wherein the set of attribute sets includes viewpoint coverage.

4. The method of claim 1 , wherein the set of attribute sets includes a viewpoint type.

5. The method of claim 1 , wherein the set of attributes includes viewpoint availability.

6. 1. An apparatus comprising: at least one processor; When implemented by the at least one processor, the apparatus receiving, from a server, information identifying a viewpoint and a respective attribute set; receiving, from the server, information identifying one or more omnidirectional videos captured from each of the identified viewpoints; Rendering a video of the one or more omnidirectional videos based on at least one attribute of a set of viewpoint attributes for each of the identified viewpoints. Memory that stores instructions and A device comprising:

7. The apparatus of claim 6 , wherein the set of attributes includes a viewpoint position.

8. The apparatus of claim 6 , wherein the set of attribute sets includes viewpoint coverage.

9. The apparatus of claim 6 , wherein the set of attribute sets includes a viewpoint type.

10. The apparatus of claim 6 , wherein the set of attributes includes viewpoint availability.

11. by at least one processor, receiving information from a server identifying a view and a respective attribute set; receiving, from the server, information identifying one or more omnidirectional videos captured from each of the identified viewpoints; rendering a video of the one or more omnidirectional videos based on at least one attribute of a set of viewpoint attributes for each of the identified viewpoints; comprising instructions for carrying out a method comprising: Non-transitory computer-readable medium.

Images