Anchoring scene descriptions to the user environment for streaming immersive media content.

Abstract

An exemplary device for presenting media data includes a memory configured to store media data defining one or more virtual objects in a virtual scene; and one or more processors implemented in a circuit and configured to receive a bitstream scene description including data describing one or more virtual objects in the virtual scene and scene anchors, the scene anchors representing correspondences between the virtual scene and a real-world presentation environment; determine the correspondences between the virtual scene and the real-world presentation environment using the scene anchors; and present the one or more virtual objects at locations in the real-world presentation environment in accordance with the determined correspondences.

Description

Priority Claim

[0001]

[0001] This application claims the priority of U.S. Patent Application No. 17 / 659,760, filed on April 19, 2022, and U.S. Provisional Application No. 63 / 177,261, filed on April 20, 2021, the entire contents of each of which are incorporated herein by reference. U.S. Patent Application No. 17 / 659,760, filed on April 19, 2022, claims the benefit of U.S. Provisional Application No. 63 / 177,261, filed on April 20, 2021.

Technical Field

[0002]

[0002] This disclosure relates to the storage and transport of media data, particularly for immersive media content.

Background Art

[0003]

[0003] Digital video functionality can be incorporated into a wide range of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular phones or satellite radiotelephones, video teleconferencing devices, etc. Digital video devices implement video compression techniques, such as those defined by standards such as MPEG-2, MPEG-4, ITU-T H.263 or ITU-T H.264 / MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.265 (also known as High Efficiency Video Coding (HEVC)), and extensions to such standards, to transmit and receive digital video information more efficiently.

[0004]

[0004] Video compression techniques perform spatial and / or temporal predictions to reduce or eliminate redundancy inherent in video sequences. In block-based video coding, a video frame or slice may be divided into macroblocks. Each macroblock may be further divided. Macroblocks in an intracoded (I) frame or slice are encoded using spatial predictions with respect to adjacent macroblocks. Macroblocks in an intercoded (P or B) frame or slice may use spatial predictions with respect to adjacent macroblocks in the same frame or slice, or temporal predictions with respect to other reference frames.

[0005]

[0005] After the video data has been encoded, the video data may be packetized for transmission or storage. The video data may be assembled into a video file that conforms to any of the various standards, such as AVC, an International Organization for Standardization (ISO) based media file format, and its extensions. [Overview of the project]

[0006]

[0006] Generally speaking, this disclosure describes techniques for streaming and presenting immersive media content, such as augmented reality (AR), virtual reality (VR), or mixed reality (MR) content. Such content may generally be referred to as extended reality (XR) content. Generally speaking, XR content may include a virtual scene containing one or more virtual objects. The virtual scene may be presented to the user as three-dimensional (3D) navigable media content. For example, the user may be able to navigate the virtual scene using a controller and / or real-world movements. To enable proper movement within a real-world presentation environment and to ensure that virtual objects are presented properly, this disclosure describes techniques for anchoring the virtual scene to a real-world presentation environment. For example, one or more real-world anchor points may be identified and used to anchor the virtual scene to a real-world presentation environment. These anchor points may be defined in a scene description, which may include a scene graph.

[0007]

[0007] In one example, a method for presenting media data includes receiving a scene description of a bitstream, the scene description including data describing scene anchors and one or more virtual objects in a virtual scene, the scene anchors representing a correspondence between a virtual scene and a real-world presentation environment, determining the correspondence between a virtual scene and a real-world presentation environment using the scene anchors, and presenting one or more virtual objects at locations in the real-world presentation environment according to the determined correspondence.

[0008]

[0008] In another example, a device for presenting media data includes a memory configured to store media data defining one or more virtual objects in a virtual scene, and one or more processors implemented in the circuit, the one or more processors receiving a scene description in bitstream, the scene description including data describing scene anchors and one or more virtual objects in the virtual scene, the scene anchors representing a correspondence between the virtual scene and a real-world presentation environment; determining the correspondence between the virtual scene and the real-world presentation environment using the scene anchors; and presenting one or more virtual objects at locations in the real-world presentation environment according to the determined correspondence.

[0009]

[0009] In another example, a computer-readable storage medium, when executed, causes a processor to receive a bitstream of a scene description, the scene description containing data describing scene anchors and one or more virtual objects in a virtual scene, where the scene anchors represent a correspondence between a virtual scene and a real-world presentation environment; it stores instructions to use the scene anchors to determine the correspondence between the virtual scene and the real-world presentation environment; and to present one or more virtual objects at locations in the real-world presentation environment according to the determined correspondence.

[0010]

[0010] In another example, a device for presenting media data includes means for receiving a scene description of a bitstream, the scene description including data describing scene anchors and one or more virtual objects in a virtual scene, where the scene anchors represent a correspondence between the virtual scene and a real-world presentation environment; means for determining the correspondence between the virtual scene and the real-world presentation environment using the scene anchors; and means for presenting one or more virtual objects at locations in the real-world presentation environment according to the determined correspondence.

[0011]

[0011] Details of one or more examples are described in the accompanying drawings and the following description. Other features, purposes, and advantages will become apparent from the description and drawings, as well as from the claims. [Brief explanation of the drawing]

[0012] [Figure 1]

[0012] A block diagram illustrating an exemplary system that implements techniques for streaming media data over a network. [Figure 2]

[0013] A block diagram showing in more detail an exemplary set of components of the retrieval unit in Figure 1. [Figure 3]

[0014] A conceptual diagram illustrating elements of exemplary multimedia content. [Figure 4]

[0015] A block diagram showing elements of an exemplary video file that could correspond to segments of representation. [Figure 5]

[0016] A graph illustrating the scene graph structure of an exemplary Graphics Language Transmission Format (glTF). [Figure 6]

[0017] A block diagram illustrating an exemplary MPEG-I architecture, including key interfaces between various components. [Figure 7]

[0018] A conceptual diagram illustrating an exemplary circular buffer. [Figure 8]

[0019] A block diagram illustrating an exemplary MPEG-I scene description architecture. [Figure 9]

[0020] A conceptual diagram showing an exemplary anchor XR space indicated by the scene description. [Figure 10]

[0021] A block diagram showing another exemplary client device 300 capable of performing the techniques of this disclosure. [Figure 11]

[0022] A flowchart illustrating an exemplary method using the techniques of this disclosure. [Modes for carrying out the invention]

[0013]

[0023] In general, this disclosure describes techniques for streaming immersive media content for Extended Reality (XR) content, such as augmented reality (AR), mixed reality (MR), or virtual reality (VR) content. More specifically, this disclosure describes techniques for signaling a correspondence between a virtual environment and a real-world presentation environment in which the virtual environment is presented. In this way, a client device can orient and / or scale the virtual environment to fit the real-world presentation environment, for example, to ensure that the boundaries of the virtual environment do not extend beyond the boundaries of the real-world presentation environment. Thus, a client device can avoid presenting the virtual environment in a way that could cause the boundaries of the virtual environment to extend beyond the boundaries of the real-world presentation environment, which can help avoid collisions between the user and real-world objects such as walls, chairs, and tables. Similarly, these techniques can ensure that a larger real-world presentation environment is available to the user when a larger real-world room is available.

[0014]

[0024] These techniques can be beneficial in the context of streamed immersive media content in that a server device that sends immersive media content needs to be equipped with details representing a plurality of different real-world presentation environments for each recipient of the immersive media content. Instead, these techniques enable each respective client device to determine an appropriate real-world anchor point corresponding to an anchor point when defined in a scene description. In this way, while still allowing each individual presentation of the immersive media content to conform to the corresponding real-world presentation environment in which the immersive media content is presented, excessive processing by the server can be avoided.

[0015]

[0025] The techniques of the present disclosure can be applied to video files compliant with video data encapsulated according to any of the ISO base media file format, scalable video coding (SVC) file format, advanced video coding (AVC) file format, 3rd Generation Partnership Project (3GPP (registered trademark)) file format, and / or multi-view video coding (MVC) file format, or other similar video file formats.

[0016]

[0026] In HTTP streaming, frequently used operations include HEAD, GET, and Partial GET. The HEAD operation retrieves the header of a file associated with a given Uniform Resource Locator (URL) or Uniform Resource Name (URN) without retrieving the payload associated with the URL or URN. The GET operation retrieves the entire file associated with a given URL or URN. The Partial GET operation receives a range of bytes as an input parameter and retrieves several consecutive bytes from the file, the number of bytes corresponding to the received byte range. Thus, a Partial GET operation can obtain one or more individual movie fragments, so a movie fragment for HTTP streaming can be given. Within a movie fragment, there may be several track fragments from different tracks. In HTTP streaming, a media presentation can be a structured collection of data accessible to the client. The client may request and download media data information in order to present the streaming service to the user.

[0017]

[0027] In an example of streaming 3GPP data using HTTP streaming, there may be multiple representations for the video and / or audio data of the multimedia content. As described below, different representations may correspond to different coding characteristics (e.g., different profiles or levels of a video coding standard), different coding standards or extensions of coding standards (such as multi-view and / or scalable extensions), or different bitrates. The manifest of such representations may be defined in a Media Presentation Description (MPD) data structure. A media presentation may correspond to a structured set of data accessible by an HTTP streaming client device. The HTTP streaming client device may request and download media data information to present the streaming service to the user of the client device. The media presentation may be described in an MPD data structure that may include updates to the MPD.

[0018]

[0028] A media presentation may include a sequence of one or more periods. Each period may continue until the start of the next period or, in the case of the last period, until the end of the media presentation. Each period may include one or more representations of the same media content. A representation may be one of several alternative encoded versions of audio, video, timed text, or other such data. Representations may differ by encoding type (e.g., for video data, by bitrate, resolution, and / or codec; for audio data, by bitrate, language, and / or codec). The term representation may be used to refer to a section of encoded audio or video data corresponding to a particular period of multimedia content and encoded in a particular way.

[0019]

[0029] Representations for a particular period can be assigned to a group indicated by an attribute in the MPD that shows the adaptive set to which the representation belongs. Representations within the same adaptive set are generally considered substitutable for each other in that a client device can dynamically and seamlessly switch between these representations, for example, to perform bandwidth adaptation. For example, each representation of video data for a particular period can be assigned to the same adaptive set so that one of the representations can be selected to decode to present media data, such as video or audio data, of multimedia content for the corresponding period. Media content within a period may, in some examples, be represented by either one representation from group 0, if any, or a combination of many or one representations from each non-zero group. Timing data for each representation of a period may be expressed relative to the start time of the period.

[0020]

[0030] A representation may contain one or more segments. Each representation may contain an initialization segment, or each segment of a representation may be self-initializing. If present, the initialization segment may contain initialization information for accessing the representation. Generally, initialization segments do not contain media data. Segments may be uniquely referred to by identifiers such as a Uniform Resource Locator (URL), Uniform Resource Name (URN), or Uniform Resource Identifier (URI). The MPD may assign an identifier to each segment. In some examples, the MPD may also provide a range of bytes in the form of a range attribute that may correspond to data for a segment in a file accessible by a URL, URN, or URI.

[0021]

[0031] Different representations may be selected for substantially simultaneous retrieval of different types of media data. For example, a client device may select audio representations, video representations, and timed text representations from which to retrieve segments. In some examples, a client device may select a specific set of adaptations for performing bandwidth adaptation; that is, a client device may select an adaptation set containing video representations, an adaptation set containing audio representations, and / or an adaptation set containing timed text. Alternatively, a client device may select an adaptation set for one type of media (e.g., video) and directly select representations of other types of media (e.g., audio and / or timed text).

[0022]

[0032] Figure 1 is a block diagram illustrating an exemplary system 10 that implements a technique for streaming media data over a network. In this example, system 10 includes a content creation device 20, a server device 60, and a client device 40. The client device 40 and the server device 60 are connected communicatively by a network 74, which may include the Internet. In some examples, the content creation device 20 and the server device 60 may also be connected by network 74 or another network, or they may be connected communicatively directly. In some examples, the content creation device 20 and the server device 60 may be the same device.

[0023]

[0033] In the example shown in Figure 1, the content creation device 20 comprises an audio source 22 and a video source 24. The audio source 22 may comprise, for example, a microphone that generates electrical signals representing captured audio data to be encoded by an audio encoder 26. Alternatively, the audio source 22 may comprise a storage medium for storing previously recorded audio data, an audio data generator such as a computerized synthesizer, or any other source of audio data. The video source 24 may comprise a video camera that generates video data to be encoded by a video encoder 28, a storage medium encoded with previously recorded video data, a video data generation unit such as a computer graphics source, or any other source of video data. The content creation device 20 may not necessarily be communicatively coupled to the server device 60 in all examples, but it may store multimedia content on separate media that can be read by the server device 60.

[0024]

[0034] Raw audio and video data may comprise analog or digital data. Analog data may be digitized before being encoded by the audio encoder 26 and / or video encoder 28. Audio source 22 may acquire audio data from a call participant while the participant is speaking, and simultaneously, video source 24 may acquire video data of the call participant. In other examples, audio source 22 may comprise a computer-readable storage medium containing stored audio data, and video source 24 may comprise a computer-readable storage medium containing stored video data. Thus, the techniques described herein may be applied to live, streaming, real-time audio and video data, or archived, pre-recorded audio and video data.

[0025]

[0035] An audio frame corresponding to a video frame is generally an audio frame that contains audio data captured (or generated) by an audio source 22 simultaneously with video data captured (or generated) by a video source 24 contained within the video frame. For example, while a call participant generates audio data by speaking, audio source 22 captures the audio data, and simultaneously, i.e., while audio source 22 is capturing the audio data, video source 24 captures the video data of the call participant. Thus, an audio frame may temporally correspond to one or more specific video frames. Therefore, an audio frame corresponding to a video frame generally corresponds to a situation in which audio data and video data are captured simultaneously, and a situation in which an audio frame and a video frame each contain simultaneously captured audio data and video data, respectively.

[0026]

[0036] In some examples, the audio encoder 26 can encode a timestamp in each encoded audio frame, representing the time the audio data of the encoded audio frame was recorded, and similarly, the video encoder 28 can encode a timestamp in each encoded video frame, representing the time the video data of the encoded video frame was recorded. In such examples, the audio frames corresponding to the video frames may include an audio frame with a certain timestamp and a video frame with the same timestamp. The content creation device 20 may include an internal clock from which the audio encoder 26 and / or the video encoder 28 can generate timestamps, or which the audio source 22 and video source 24 can use to associate audio data and video data with timestamps, respectively.

[0027]

[0037] In some examples, an audio source 22 can send data to an audio encoder 26 corresponding to the time the audio data was recorded, and a video source 24 can send data to a video encoder 28 corresponding to the time the video data was recorded. In some examples, the audio encoder 26 can encode a sequence identifier in the encoded audio data to indicate the relative time order of the encoded audio data without necessarily indicating the absolute time the audio data was recorded, and similarly, the video encoder 28 can use a sequence identifier to indicate the relative time order of the encoded video data. Similarly, in some examples, the sequence identifier can be mapped to a timestamp or, in some cases, correlated with a timestamp.

[0028]

[0038] The audio encoder 26 generally generates a stream of encoded audio data, while the video encoder 28 generates a stream of encoded video data. Each individual stream of data (whether audio or video) is sometimes called an elementary stream. An elementary stream is a single, digitally encoded (and possibly compressed) component of a representation. For example, the encoded video or audio portion of a representation can be an elementary stream. An elementary stream can be converted to a packetized elementary stream (PES) before being encapsulated within a video file. Within the same representation, a stream ID may be used to distinguish PES packets belonging to one elementary stream from others. The basic data unit of an elementary stream is a packetized elementary stream (PES) packet. Therefore, encoded video data generally corresponds to an elementary video stream. Similarly, audio data corresponds to one or more of each elementary stream.

[0029]

[0039] Many video coding standards, such as the ITU-T H.264 / AVC and the upcoming High Efficiency Video Coding (HEVC) standard, define syntax, semantics, and a decoding process for error-free bitstreams, all of which conform to a specific profile or level. While video coding standards typically do not specify an encoder, the encoder is obligated to ensure that the resulting bitstream conforms to the decoder's standard. In the context of video coding standards, a “profile” corresponds to a subset of algorithms, features, or tools, and the constraints applied to them. For example, a “profile” defined by the H.264 standard is a subset of the entire bitstream syntax specified by the H.264 standard. A “level” corresponds to, for example, the picture resolution and the limits on decoder resource consumption, such as decoder memory and computation related to the block processing rate. Profiles can be signaled by profile_idc (profile indicator) values, while levels can be signaled by level_idc (level indicator) values.

[0030]

[0040] The H.264 standard acknowledges that significant variations in encoder and decoder performance may still be required depending on the values ​​taken by syntax elements in the bitstream, such as a specified size of the decoded picture, within the boundaries imposed by the syntax of a given profile. The H.264 standard further acknowledges that in many applications, it is neither practical nor economical to implement a decoder capable of handling all hypothetical uses of the syntax within a particular profile. Therefore, the H.264 standard defines a "level" as a specified set of constraints imposed on the values ​​of syntax elements in the bitstream. These constraints may be simple restrictions on values. Alternatively, these constraints may take the form of constraints on combinations of value operations (e.g., picture width × picture height × number of pictures decoded per second). The H.264 standard further specifies that individual implementations may support different levels for each supported profile.

[0031]

[0041] A profile-compliant decoder typically supports all features defined in the profile. For example, as a coding feature, B-picture coding is not supported in the H.264 / AVC baseline profile, but is supported in other H.264 / AVC profiles. A level-compliant decoder should be able to decode any bitstream that does not require resources beyond the limits defined in the level. Profile and level definitions can serve an explainability purpose. For example, during a video transmission, pairs of profile and level definitions may be negotiated and agreed upon for the entire transmission session. More specifically, in H.264 / AVC, a level may define limits on the number of macroblocks that need to be processed, the decoded picture buffer (DPB) size, the coded picture buffer (CPB) size, the vertical motion vector range, the maximum number of motion vectors per two consecutive blocks such as macroblocks (MBs), and whether a B-block can have subblock divisions smaller than 8x8 pixels. In this way, a decoder can determine whether it is capable of properly decoding a bitstream.

[0032]

[0042] In the example shown in Figure 1, the encapsulation unit 30 of the content creation device 20 receives an elementary stream containing encoded video data from the video encoder 28 and an elementary stream containing encoded audio data from the audio encoder 26. In some examples, the video encoder 28 and the audio encoder 26 may each include a packetizer for forming PES packets from the encoded data. In other examples, the video encoder 28 and the audio encoder 26 may each interface with their respective packetizers for forming PES packets from the encoded data. In yet another example, the encapsulation unit 30 may include a packetizer for forming PES packets from the encoded audio data and the encoded video data.

[0033]

[0043] The video encoder 28 can encode video data of multimedia content in various ways to produce different representations of multimedia content using various characteristics such as pixel resolution, frame rate, compliance with various coding standards, compliance with various profiles and / or levels of profiles for various coding standards, representations having one or more views (for example, for two-dimensional or three-dimensional playback), or other such characteristics at various bitrates. The representations used in this disclosure may comprise one of audio data, video data, text data (for example, for closed captions), or other such data. The representations may include elementary streams, such as audio elementary streams or video elementary streams. Each PES packet may include a stream_id that identifies the elementary stream to which the PES packet belongs. The encapsulation unit 30 is responsible for assembling the elementary streams into video files (for example, segments) of the various representations.

[0034]

[0044] The encapsulation unit 30 receives PES packets of the elementary stream of the representation from the audio encoder 26 and the video encoder 28, and forms corresponding network abstraction layer (NAL) units from the PES packets. The coded video segment can be organized into NAL units that provide a “network-friendly” video representation to address applications such as videophone, storage, broadcast, or streaming. NAL units can be classified into video coding layer (VCL) NAL units and non-VCL NAL units. VCL units may include a core compression engine and may include block, macroblock, and / or slice-level data. Other NAL units may be non-VCL NAL units. In some examples, a coded picture in one time instance, typically presented as a primary coded picture, may be contained within an access unit that may contain one or more NAL units.

[0035]

[0045] Non-VCL NAL units may include, in particular, parameter set NAL units and SEI NAL units. Parameter sets may include sequence-level header information (in a sequence parameter set (SPS)) and rarely changing picture-level header information (in a picture parameter set (PPS)). With parameter sets (e.g., PPS and SPS), rarely changing information does not need to be repeated per sequence or per picture, thus improving coding efficiency. Furthermore, the use of parameter sets allows for out-of-band transmission of critical header information, avoiding the need for redundant transmission for error tolerance. In an example of out-of-band transmission, a parameter set NAL unit, such as an SEI NAL unit, may be transmitted on a different channel than other NAL units.

[0036]

[0046] Supplemental Enhancement Information (SEI) is not required to decode coded picture samples from VCL NAL units, but may contain information that can assist processes related to decoding, display, error tolerance, and other purposes. SEI messages may be included in non-VCL NAL units. SEI messages are a normative part of some standards specifications and are therefore not always required for standard-compliant decoder implementations. SEI messages can be sequence-level SEI messages or picture-level SEI messages. SEI messages may contain some sequence-level information, such as the scalability information SEI message in the SVC example and the view scalability information SEI message in MVC. These exemplary SEI messages may carry, for example, information about the extraction of operating points and the characteristics of those operating points. In addition, the encapsulation unit 30 may form a manifest file, such as a Media Presentation Descriptor (MPD) that describes the characteristics of the representation. The encapsulation unit 30 may format the MPD according to Extensible Markup Language (XML).

[0037]

[0047] The encapsulation unit 30 may provide data for one or more representations of multimedia content to the output interface 32, along with a manifest file (e.g., MPD). The output interface 32 may include a Universal Serial Bus (USB) interface, a network interface or interface for writing to a storage medium such as a CD or DVD writer or burner, an interface to a magnetic or flash storage medium, or other interfaces for storing or transmitting media data. The encapsulation unit 30 can provide data for each representation of the multimedia content to the output interface 32, which can then send that data to the server device 60 via network transmission or storage medium. In the example in Figure 1, the server device 60 includes a storage medium 62 for storing various multimedia content 64, each multimedia content 64 including its own manifest file 66 and one or more representations 68A-68N (representation 68). In some examples, the output interface 32 may also send data directly to the network 74.

[0038]

[0048] In some examples, representation 68 can be separated into adaptive sets. That is, various subsets of representation 68 may include each common set of characteristics, such as codecs, profiles and levels, resolution, number of views, segment file formats, text type information that can identify the language or other characteristics of the text and / or audio data to be displayed using the representation and / or decoded and presented, for example, by a speaker, camera angle information that can describe the camera angle of the scene or the real-world camera perspective for the representation in the adaptive set, and rating information that describes the content suitability for a particular audience.

[0039]

[0049] The manifest file 66 may contain data indicating a subset of representations 68 corresponding to a particular adaptation set, and common characteristics of the adaptation set. The manifest file 66 may also contain data representing individual characteristics of individual representations of the adaptation set, such as bitrate. In this way, the adaptation set can enable simplified network bandwidth adaptation. Representations within the adaptation set may be indicated using child elements of the adaptation set element in the manifest file 66.

[0040]

[0050] The server device 60 includes a request processing unit 70 and a network interface 72. In some examples, the server device 60 may include multiple network interfaces. Furthermore, some or all of the functions of the server device 60 may be implemented on other devices in the content distribution network, such as routers, bridges, proxy devices, switches, or other devices. In some examples, intermediate devices in the content distribution network may include components that cache data for multimedia content 64 and substantially conform to the components of the server device 60. Generally, the network interface 72 is configured to send and receive data over the network 74.

[0041]

[0051] The request processing unit 70 is configured to receive network requests for data on the storage medium 62 from client devices, such as the client device 40. For example, the request processing unit 70 may implement the Hypertext Transfer Protocol (HTTP) version 1.1, as described in RFC 2616, "Hypertext Transfer Protocol - HTTP / 1.1", Network Working Group, IETF, June 1999, by R. Fielding et al. That is, the request processing unit 70 may be configured to receive HTTP GET requests or partial GET requests and to provide data for the multimedia content 64 in response to these requests. A request may specify one segment of the representation 68, for example, using the segment's URL. In some examples, a request may also specify one or more byte ranges of a segment, thus comprising a partial GET request. The request processing unit 70 may be further configured to service HTTP HEAD requests for header data for one segment of the representation 68. In any case, the request processing unit 70 may be configured to process requests for providing the requested data to the requesting device, such as the client device 40.

[0042]

[0052] As an addition or alternative, the request processing unit 70 may be configured to deliver media data via a broadcast or multicast protocol such as eMBMS. The content creation device 20 may create DASH segments and / or subsegments in substantially the same manner as described, but the server device 60 may deliver these segments or subsegments using eMBMS or another broadcast or multicast network transport protocol. For example, the request processing unit 70 may be configured to receive multicast group join requests from client devices 40. That is, the server device 60 can advertise the Internet Protocol (IP) address associated with the multicast group to client devices associated with specific media content (e.g., a broadcast of a live event), including client device 40. Client device 40 can then submit a request to join the multicast group. This request may be propagated across the network 74, for example, to routers that make up the network 74, so that the routers direct traffic destined for the IP address associated with the multicast group to joining client devices such as client device 40.

[0043]

[0053] As shown in the example in Figure 1, the multimedia content 64 includes a manifest file 66 which may correspond to a Media Presentation Description (MPD). The manifest file 66 may include descriptions of different alternative representations 68 (for example, video services with different qualities), which may include, for example, codec information, profile values, level values, bitrate, and other description characteristics of the representation 68. The client device 40 may retrieve the MPD of the media presentation to determine how to access segments of the representation 68.

[0044]

[0054] More specifically, the retrieval unit 52 may retrieve configuration data (not shown) of the client device 40 to determine the decoding capability of the video decoder 48 and the rendering capability of the video output 44. The configuration data may also include any or all of the following: language preferences selected by the user of the client device 40, one or more camera perspectives corresponding to depth preferences set by the user of the client device 40, and / or rating preferences selected by the user of the client device 40. The retrieval unit 52 may comprise, for example, a web browser or media client configured to submit HTTP GET requests and partial GET requests. The retrieval unit 52 may correspond to software instructions executed by one or more processors or processing units (not shown) of the client device 40. In some examples, all or part of the functions described with respect to the retrieval unit 52 may be implemented in hardware, or in a combination of hardware, software, and / or firmware, and hardware may be provided that is essential for executing software or firmware instructions.

[0045]

[0055] The extraction unit 52 may compare the decoding and rendering capabilities of the client device 40 with the characteristics of the representation 68 indicated by the information in the manifest file 66. The extraction unit 52 may first extract at least a portion of the manifest file 66 to determine the characteristics of the representation 68. For example, the extraction unit 52 may request a portion of the manifest file 66 that describes the characteristics of one or more adaptive sets. The extraction unit 52 may select a subset of representations 68 (e.g., adaptive sets) that have characteristics that can be satisfied by the coding and rendering capabilities of the client device 40. The extraction unit 52 may then determine the bitrate of the representations in the adaptive sets, determine the amount of network bandwidth currently available, and extract a segment from one of the representations that has a bitrate that can be satisfied by the network bandwidth.

[0046]

[0056] Generally, higher bitrate representations can result in higher quality video playback, but when available network bandwidth decreases, lower bitrate representations can provide sufficient quality video playback. Therefore, when available network bandwidth is relatively high, the extraction unit 52 can extract data from a relatively high bitrate representation, but when available network bandwidth is low, the extraction unit 52 can extract data from a relatively low bitrate representation. In this way, the client device 40 can stream multimedia data over the network 74 while also adapting to the changing network bandwidth availability of the network 74.

[0047]

[0057] As an addition or alternative, the retrieval unit 52 may be configured to receive data according to a broadcast or multicast network protocol such as eMBMS or IP multicast. In such an example, the retrieval unit 52 may submit a request to join a multicast network group associated with a particular media content. After joining the multicast group, the retrieval unit 52 may receive data from the multicast group without further requests issued to the server device 60 or content creation device 20. When the data from the multicast group is no longer needed, the retrieval unit 52 may submit a request to leave the multicast group, for example, to stop playback or to change the channel to a different multicast group.

[0048]

[0058] The network interface 54 can receive data from a selected representation segment and provide it to the extraction unit 52, which in turn can provide the segment to the decapsulation unit 50. The decapsulation unit 50 decapsulates the video file elements into a constituent PES stream, depackets the PES stream to extract the encoded data, and can send the encoded data to either the audio decoder 46 or the video decoder 48, depending on whether the encoded data is part of an audio stream or a video stream, as indicated by the PES packet header of the stream. The audio decoder 46 decodes the encoded audio data and sends the decoded audio data to the audio output 42, and the video decoder 48 decodes the encoded video data and sends the decoded video data, which may contain multiple views of the stream, to the video output 44.

[0049]

[0059] The video encoder 28, video decoder 48, audio encoder 26, audio decoder 46, encapsulation unit 30, extraction unit 52, and decapsulation unit 50 may each be implemented as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic circuits, software, hardware, firmware, or any combination thereof, where applicable. Each of the video encoder 28 and video decoder 48 may be included in one or more encoders or decoders, and any of them may be integrated as part of a composite video encoder / decoder (codec). Similarly, each of the audio encoder 26 and audio decoder 46 may be included in one or more encoders or decoders, and any of them may be integrated as part of a composite codec. The device including the video encoder 28, video decoder 48, audio encoder 26, audio decoder 46, encapsulation unit 30, extraction unit 52, and / or decapsulation unit 50 may include an integrated circuit, a microprocessor, and / or a wireless communication device such as a cellular phone.

[0050]

[0060] The client device 40, the server device 60, and / or the content creation device 20 may be configured to operate in accordance with the techniques of this disclosure. As an example, this disclosure describes these techniques with respect to the client device 40 and the server device 60. However, it should be understood that the content creation device 20 may be configured to perform these techniques instead of (or in addition to) the server device 60.

[0051]

[0061] The encapsulation unit 30 may form a NAL unit comprising a header that identifies the program to which the NAL unit belongs, and a payload, for example, audio data, video data, or data describing the transport or program stream to which the NAL unit corresponds. For example, in H.264 / AVC, the NAL unit includes a 1-byte header and a payload of varying size. A NAL unit that includes video data in its payload may have video data at various granularity levels. For example, the NAL unit may have blocks of video data, multiple blocks, slices of video data, or entire pictures of video data. The encapsulation unit 30 may receive encoded video data from the video encoder 28 in the form of PES packets of elementary streams. The encapsulation unit 30 may associate each elementary stream with a corresponding program.

[0052]

[0062] The encapsulation unit 30 can also assemble access units from multiple NAL units. Generally, an access unit may comprise one or more NAL units for representing frames of video data, and, if audio data corresponding to those frames is available, such audio data. Generally, an access unit includes all NAL units across one output time instance, for example, all audio and video data across one time instance. For example, if each view has a frame rate of 20 frames per second (fps), each time instance may correspond to a time interval of 0.05 seconds. During this time interval, the unique frames of all views of the same access unit (same time instance) can be rendered simultaneously. In one example, an access unit may comprise a coded picture in one time instance, which may be presented as a primary coded picture.

[0053]

[0063] Therefore, an access unit may contain all audio and video frames of a common time instance, for example, all views corresponding to time X. This disclosure also refers to the encoded pictures of a particular view as “view components.” That is, a view component may contain the encoded pictures (or frames) of a particular view at a particular time. Therefore, an access unit can be defined as one that contains all view components of a common time instance. The decoding order of an access unit does not necessarily have to be the same as the output order or display order.

[0054]

[0064] A media presentation may include a media presentation description (MPD) which may contain descriptions of different alternative representations (e.g., video services with different qualities), and this description may include, for example, codec information, profile values, and level values. An MPD is an example of a manifest file, such as manifest file 66. To determine how to access movie fragments of various presentations, a client device 40 may retrieve the MPD of the media presentation. Movie fragments may be placed in movie fragment boxes (moof boxes) of a video file.

[0055]

[0065] A manifest file 66 (which may, for example, include an MPD) may advertise the availability of segments in representation 68. That is, the MPD may include information indicating the wall clock time when one of the first segments in representation 68 becomes available, and information indicating the duration of the segments in representation 68. In this way, the retrieval unit 52 of the client device 40 may determine when each segment is available based on the start time and the duration of the segments preceding a particular segment.

[0056]

[0066] After the encapsulation unit 30 assembles the NAL unit and / or access unit into a video file based on the received data, the encapsulation unit 30 passes the video file to the output interface 32 for output. In some examples, the encapsulation unit 30 may store the video file locally or send the video file to a remote server via the output interface 32, rather than sending the video file directly to the client device 40. The output interface 32 may comprise, for example, a transmitter, transceiver, a device for writing data to a computer-readable medium such as an optical drive or magnetic media drive (e.g., a floppy disk drive), a Universal Serial Bus (USB) port, a network interface, or other output interface. The output interface 32 outputs the video file to a computer-readable medium such as a transmission signal, magnetic media, optical media, memory, a flash drive, or other computer-readable medium.

[0057]

[0067] The network interface 54 can receive NAL units or access units via the network 74 and provide the NAL units or access units to the decapsulation unit 50 via the extraction unit 52. The decapsulation unit 50 decapsulates the elements of the video file into a constituent PES stream, depackets the PES stream to extract the encoded data, and can send the encoded data to either the audio decoder 46 or the video decoder 48, depending on whether the encoded data is part of an audio stream or a video stream, as indicated by the PES packet header of the stream, for example. The audio decoder 46 decodes the encoded audio data and sends the decoded audio data to the audio output 42, and the video decoder 48 decodes the encoded video data and sends the decoded video data, which may contain multiple views of the stream, to the video output 44.

[0058]

[0068] As will be described in more detail below, the client device 40 may be configured to perform various techniques of this disclosure, either alone or in any combination. Generally, the retrieval unit 52 may be configured to retrieve media data (e.g., scene data) as described above, and a bitstream containing a scene description. The scene description may include anchor point data representing the correspondence between the virtual scene represented by the media data and the real-world presentation environment. The client device 40 may also be configured to use the anchor point data to anchor the virtual scene to the real-world presentation environment and to transform the virtual scene as needed, for example, by rotation, translation, and / or scaling.

[0059]

[0069] Figure 2 is a block diagram showing in more detail an exemplary set of components of the retrieval unit 52 of Figure 1. In this example, the retrieval unit 52 includes an eMBMS middleware unit 100, a DASH client 110, and a media application 112.

[0060]

[0070] In this example, the eMBMS middleware unit 100 further includes an eMBMS receiving unit 106, a cache 104, and a proxy server unit 102. In this example, the eMBMS receiving unit 106 is configured to receive data via eMBMS in accordance with File Delivery over Unidirectional Transport (FLUTE), as described by T. Paira et al., "FLUTE - File Delivery over Unidirectional Transport," Network Working Group, RFC6726, November 2012, available, for example, at tools.ietf.org / html / rfc6726. That is, the eMBMS receiving unit 106 can receive files via broadcast from, for example, a server device 60 that can act as a Broadcast / Multicast Service Center (BM-SC).

[0061]

[0071] When the eMBMS middleware unit 100 receives data relating to a file, the eMBMS middleware unit may store the received data in the cache 104. The cache 104 may comprise a computer-readable storage medium such as flash memory, a hard disk, RAM, or any other suitable storage medium.

[0062]

[0072] The proxy server unit 102 can act as a server for the DASH client 110. For example, the proxy server unit 102 can provide the DASH client 110 with an MPD file or other manifest file. The proxy server unit 102 can advertise the availability time for segments in the MPD file and hyperlinks from which the segments can be retrieved. These hyperlinks may include a local host address prefix corresponding to the client device 40 (for example, 127.0.0.1 for IPv4). In this way, the DASH client 110 can request a segment from the proxy server unit 102 using an HTTP GET or partial GET request. For example, for a segment available from the link http: / / 127.0.0.1 / rep1 / seg3, the DASH client 110 can construct an HTTP GET request that includes a request for http: / / 127.0.0.1 / rep1 / seg3 and submit that request to the proxy server unit 102. The proxy server unit 102 may retrieve the requested data from the cache 104 and, in response to such a request, provide that data to the DASH client 110.

[0063]

[0073] Figure 3 is a conceptual diagram showing the elements of an exemplary multimedia content 120. The multimedia content 120 may correspond to multimedia content 64 (Figure 1) or another multimedia content stored in the storage medium 62. In the example in Figure 3, the multimedia content 120 includes a media presentation description (MPD) 122 and multiple representations 124A to 124N (representation 124). Representation 124A includes arbitrary header data 126 and segments 128A to 128N (segment 128), and representation 124N includes arbitrary header data 130 and segments 132A to 132N (segment 132). The character N is used for convenience to specify the last movie fragment in each of the representations 124. In some examples, there may be different numbers of movie fragments between representations 124.

[0064]

[0074] MPD122 may have a different data structure from representation 124. MPD122 may correspond to manifest file 66 in Figure 1. Similarly, representation 124 may correspond to representation 68 in Figure 1. Generally, MPD122 may contain data that collectively describes the characteristics of representation 124, such as coding and rendering characteristics, adaptive sets, the profile to which MPD122 corresponds, text type information, camera angle information, rating information, trick mode information (e.g., information indicating representations that include time subsequences), and / or information for extracting remote periods (e.g., for inserting targeted advertisements into media content during playback).

[0065]

[0075] Header data 126, if present, may describe characteristics of segment 128, such as the time location of a random access point (RAP, also known as a stream access point (SAP)), and the random access points of segment 128 may include the random access point, the byte offset to the random access point within segment 128, the uniform resource locator (URL) of segment 128, or other aspects of segment 128. Header data 130, if present, may describe similar characteristics of segment 132. Additional or alternative, such characteristics may be entirely contained within MPD 122.

[0066]

[0076] Segments 128 and 132 include one or more coded video samples, each of which may include a frame or slice of video data. Each coded video sample in segment 128 may have similar characteristics, such as height, width, and bandwidth requirements. Such characteristics may be described by data in MPD122, but such data is not shown in the example in Figure 3. MPD122 may include characteristics described by the 3GPP specification in addition to any or all of the signaled information described herein.

[0067]

[0077] Each of segments 128 and 132 can be associated with a unique uniform resource locator (URL). Thus, each of segments 128 and 132 can be retrieved independently using a streaming network protocol such as DASH. In this way, a destination device, such as client device 40, can use an HTTP GET request to retrieve segment 128 or 132. In some examples, client device 40 can use an HTTP partial GET request to retrieve a specific byte range of segment 128 or 132.

[0068]

[0078] Figure 4 is a block diagram showing elements of an exemplary video file 150 that may correspond to a segment of representation, such as one of segments 128 and 132 in Figure 3. Each of segments 128 and 132 may contain data that substantially matches the sequence of data shown in the example in Figure 4. The video file 150 is sometimes said to encapsulate segments. As explained above, video files by the ISO-based media file format and its extensions store data in a set of objects called "boxes". In the example in Figure 4, the video file 150 includes a file type (FTYP) box 152, a movie (MOOV) box 154, a segment index (sidx) box 162, a movie fragment (MOOF) box 164, and a movie fragment random access (MFRA) box 166. While Figure 4 represents an example of a video file, it should be understood that other media files may contain other types of media data (e.g., audio data, timed text data, etc.) structured similarly to the data in the video file 150, according to the ISO-based media file format and its extensions.

[0069]

[0079] The File Type (FTYP) box 152 generally describes the file type of the video file 150. The File Type box 152 may contain data identifying a specification that describes the best use of the video file 150. Alternatively, the File Type box 152 may be placed before the MOOV box 154, the Movie Fragment box 164, and / or the MFRA box 166.

[0070]

[0080] In some examples, a segment such as video file 150 may include an MPD update box (not shown) before the FTYP box 152. The MPD update box may contain information indicating that the MPD corresponding to the representation containing video file 150 should be updated, along with information for updating the MPD. For example, the MPD update box may provide a URI or URL for the resource used to update the MPD. In another example, the MPD update box may contain data for updating the MPD. In some examples, the MPD update box may come immediately after the segment type (STYP) box (not shown) for video file 150, where the STYP box may define the segment type for video file 150.

[0071]

[0081] In the example in Figure 4, the MOOV box 154 includes a movie header (MVHD) box 156, a track (TRAK) box 158, and one or more movie extension (MVEX) boxes 160. Generally, the MVHD box 156 may describe the general characteristics of the video file 150. For example, the MVHD box 156 may include data describing when the video file 150 was first created, when the video file 150 was last modified, the timeline of the video file 150, the duration of playback of the video file 150, or other data that generally describes the video file 150.

[0072]

[0082] The TRAK box 158 may contain data about the track of the video file 150. The TRAK box 158 may contain a Track Header (TKHD) box that describes the characteristics of the track corresponding to the TRAK box 158. In some examples, the TRAK box 158 may contain an encoded video picture, while in other examples, the encoded video picture of the track may be contained in a movie fragment 164 that can be referenced by the data in the TRAK box 158 and / or the sidx box 162.

[0073]

[0083] In some examples, video file 150 may contain two or more tracks. Therefore, MOOV box 154 may contain several TRAK boxes equal to the number of tracks in video file 150. TRAK box 158 may describe the characteristics of the corresponding track in video file 150. For example, TRAK box 158 may describe temporal and / or spatial information about the corresponding track. When the encapsulation unit 30 (Figure 1) includes a parameter set track in a video file such as video file 150, TRAK boxes similar to TRAK box 158 in MOOV box 154 may describe the characteristics of the parameter set track. The encapsulation unit 30 may signal the presence of a sequence-level SEI message in the parameter set track within the TRAK box describing the parameter set track.

[0074]

[0084] The MVEX box 160 may, for example, describe the properties of the corresponding movie fragment 164 so as to signal that the video file 150 contains the movie fragment 164 in addition to the video data contained in the MOOV box 154, if any. In the context of streaming video data, coded video pictures may be contained in the movie fragment 164 rather than in the MOOV box 154. Therefore, all coded video samples may be contained in the movie fragment 164 rather than in the MOOV box 154.

[0075]

[0085] A MOOV box 154 may contain several MVEX boxes 160 equal to the number of movie fragments 164 in the video file 150. Each MVEX box 160 may describe one corresponding characteristic of one of the movie fragments 164. For example, each MVEX box may contain a movie extension header box (MEHD) box that describes the duration of one corresponding movie fragment 164.

[0076]

[0086] As described above, the encapsulation unit 30 may store a sequence dataset in a video sample that does not contain the actual coded video data. A video sample may generally correspond to an access unit, which is a representation of a coded picture in a particular time instance. In the context of AVC, a coded picture includes one or more VCL NAL units containing information to constitute all the pixels of the access unit, and other related non-VCL NAL units such as SEI messages. Thus, the encapsulation unit 30 may include a sequence dataset in one of the movie fragments 164, which may contain a sequence-level SEI message. The encapsulation unit 30 may further signal the presence of the sequence dataset and / or the sequence-level SEI message as being present in one of the movie fragments 164 within one of the MVEX boxes 160 corresponding to one of the movie fragments 164.

[0077]

[0087] The SIDX box 162 is an optional element of the video file 150. That is, a video file conforming to the 3GPP file format, or any other such file format, does not necessarily contain a SIDX box 162. According to the example of the 3GPP file format, a SIDX box may be used to identify a subsegment of a segment (for example, a segment contained within the video file 150). The 3GPP file format defines a subsegment as "a self-contained set of one or more consecutive movie fragment boxes, each having a corresponding media data box and a media data box containing data referenced by a movie fragment box, which must follow that movie fragment box and precede the next movie fragment box containing information about the same track." The 3GPP file format also indicates that a SIDX box "contains a sequence of references to subsegments of the (sub)segment documented by the box. The referenced subsegments are consecutive in presentation time. Similarly, bytes referenced by a segment index box are always consecutive within a segment. The referenced size gives a count of the number of bytes in the referenced material."

[0078]

[0088] The SIDX box 162 generally provides information representing one or more subsegments of a segment contained within the video file 150. For example, such information may include the playback time at which the subsegment begins and / or ends, the byte offset of the subsegment, whether the subsegment contains a stream access point (SAP) (e.g., starts with one), the type of SAP (e.g., whether the SAP is an instantaneous decoder refresh (IDR) picture, a clean random access (CRA) picture, or a broken link access (BLA) picture), the position of the SAP within the subsegment (relative to playback time and / or byte offset), and so on.

[0079]

[0089] A movie fragment 164 may contain one or more coded video pictures. In some examples, a movie fragment 164 may contain one or more picture groups (GOPs), each of which may contain several coded video pictures, such as frames or pictures. Furthermore, as described above, a movie fragment 164 may, in some examples, contain a sequence dataset. Each movie fragment 164 may contain a movie fragment header box (MFHD, not shown in Figure 4). The MFHD box may describe characteristics of the corresponding movie fragment, such as the sequence number of the movie fragment. The movie fragments 164 may be included in the order of their sequence numbers in the video file 150.

[0080]

[0090] The MFRA box 166 may describe random access points within movie fragments 164 of the video file 150. This can help perform trick modes, such as performing seeks to specific time locations (i.e., playback time) within segments encapsulated by the video file 150. The MFRA box 166 is generally optional and, in some examples, does not need to be included in the video file. Similarly, a client device, such as the client device 40, does not necessarily need to refer to the MFRA box 166 in order to correctly decode and display the video data of the video file 150. The MFRA box 166 may include several Track Fragment Random Access (TFRA) boxes (not shown), equal to the number of tracks in the video file 150, or, in some examples, equal to the number of media tracks (e.g., non-hint tracks) in the video file 150.

[0081]

[0091] In some examples, a movie fragment 164 may contain one or more stream access points (SAPs), such as IDR pictures. Similarly, an MFRA box 166 may provide location instructions for the SAPs within the video file 150. Thus, a time subsequence of the video file 150 may be formed from the SAPs of the video file 150. The time subsequence may also contain other pictures, such as P frames and / or B frames, dependent on the SAPs. Frames and / or slices of the time subsequence may be structured within a segment so that frames / slice of the time subsequence that depend on other frames / slice of the subsequence can be properly decoded. For example, in a data hierarchy, data used for predicting other data may also be included in the time subsequence.

[0082]

[0092] Figure 5 is a graph showing an exemplary Graphics Language Transmission Format (glTF) scene graph structure. One of the techniques for enabling immersive three-dimensional (3D) user experiences is scene description. Scene description can be used to describe the composition of a 3D scene, where the scene description includes data that references and positions various two-dimensional (2D) and 3D assets within the scene. The information provided in the scene description can then be used by a presentation engine to properly render the 3D scene using techniques such as physically based rendering (PBR) to produce a realistic scene.

[0083]

[0093] A scene description typically includes a scene graph, which is a directed acyclic graph, usually a plain tree structure, representing an object-based hierarchy of the scene's geometry. The leaf nodes of the graph represent geometric primitives such as images, textures, or media data buffers. Each node in the graph holds pointers to its children, which can be, among other things, other nodes, geometric elements, transformation matrices, accessors to media data buffers, camera information for rendering, and so on.

[0084]

[0094] Spatial transformations are represented as nodes in a graph, and are expressed by transformation matrices. A typical use of a transformation node is to describe the rotation, translation, or scaling of an object in its child nodes. The scene graph also supports animation nodes, which allow animation properties to change over time, and thus describe dynamic content.

[0085]

[0095] The scene graph architecture has the advantage of reducing processing complexity, for example, when traversing the graph for rendering. An exemplary operation simplified by the graph representation is culling, where a branch of the graph is dropped from processing if the parent node's space is considered not visible or relevant to the rendering of the current view frustum (detail culling).

[0086]

[0096] While there are many proprietary solutions for scene description (typically at the heart of game engines, VFX design tools, or AR / VR authoring tools), some solutions have also been standardized. Specifically, in 2001, the Virtual Reality Modeling Language (VRML), using XML syntax, was the first scene description solution to be standardized for web use. Later, in 2005, OpenSceneGraph, an open-source project using OpenGL, was released and has been used as a component in several computer games and rendering platforms such as Delta3D or FlightGear. In 2010, X3D, standardized by ISO / IEC, became the successor to VRML. X3D introduced binary and JSON formats for scene graph description and featured new capabilities such as multi-texture rendering, shading, real-time environment lighting, and culling. In 2015, the Khronos group, which also manages OpenGL, released the Graphics Library Transmission Format (glTF). glTF is a scene description format based on the JSON format, intended to be efficient and interoperable as a common description format for 3D content tools and services.

[0087]

[0097] A typical glTF scene graph, such as the one shown in Figure 5, is a tree of nodes that describe content properties, access to content data, possible dynamic animations, and parameters for rendering. In the example in Figure 5, the scene graph structure includes a scene node 170 with child nodes 172. Node 172 has child camera nodes 174, mesh nodes 176, and skin nodes 178. The mesh node has child material nodes 180 and accessor nodes 182. Animation node 184 is also a parent to nodes 172 and accessor nodes 182, and skin node 178 has an edge to accessor node 182. Accessor node 182 has a child buffer view node 188, which in turn has a child buffer node 194. Material node 180 has a child texture node 186. Texture node 186 has child sampler nodes 190 and image nodes 192.

[0088]

[0098] Typically, camera node 174 contains data describing the view on which rendering is to be performed. 3D objects may also be described in mesh subnodes, such as mesh node 176, whose child nodes provide information on how to access media data through buffers and texture information (e.g., accessor node 182, buffer view node 188, buffer node 194, material node 180, texture node 186, sampler node 190, and image node 192). Animation nodes, such as animation node 184, may contain data describing the dynamic changes of 3D objects over time.

[0089]

[0099] glTF 2.0 provides a solid and efficient baseline for interchangeable and interoperable scene descriptions. However, glTF 2.0 has traditionally focused on static scenes and assets, which makes it unsuitable for addressing the demands and needs of dynamic and rich 3D scenes in immersive environments.

[0090]

[0100] The specific gaps in glTF 2.0 include the following:

[0091] • There is no support for video and timed media such as moving meshes and point clouds.

[0092] • There is no audio support.

[0093] • Limited support for interaction with scenes and assets within scenes.

[0094] For example, there is no support for local and real-time media, which are crucial for AR experiences.

[0095]

[0101] After carefully evaluating various alternatives, including older scene graph solutions developed for home use and the potential to define something new from scratch, MPEG has decided to leverage the extensions provided by glTF2.0 to develop its solution for immersive multimedia.

[0096]

[0102] Figure 6 is a block diagram illustrating an exemplary MPEG-I architecture including key interfaces between various components. In this example, the MPEG-I architecture includes a scene description and update 200, a media access function (MAF) 206, a format plugin 216, a protocol plugin 218, a media access application programming interface (API) 204, a MAF plugin 202, a presentation engine 220, a network 210, local storage 208, decoders 212A-212D (decoder 212), a texture buffer 214A, a color buffer 214B, an index buffer 214C, and a metadata buffer 214D (buffer 214). The scene description and update 200 may correspond to the scene graph in Figure 5.

[0097]

[0103] MPEG developed the architecture for MPEG-I to guide the work on scene descriptions, which act as an entry point for the consumption of immersive media presentations. The design primarily focuses on buffers (e.g., texture buffer 214A, color buffer 214B, index buffer 214C, and metadata buffer 214D) as means for data exchange throughout the media access and rendering pipeline. It also defines a media access API 204 to request media referenced by the scene description and update 200, which are made accessible through buffer 214. This design aligns well with the principles of glTF 2.0 and integrates well with the video decoding interface.

[0098]

[0104] The glTF extension specifies defining timed accessors (MPEG_Accessors_Timed_Extensions) to provide access to timed media and metadata within a scene. Accessors within glTF define the type and layout of data stored in a buffer, as seen through a buffer view. When timed data is read from a buffer, the data in the buffer is expected to change dynamically over time. Buffer elements are extended to add support for circular buffers used with timed data. According to the MPEG extension, scenes containing timed media and / or metadata shall use the Timed Accessors extension to access the data. Timed accessors are extensions of regular accessors to indicate that the underlying data buffer is dynamic.

[0099]

[0105] Figure 7 is a conceptual diagram illustrating an exemplary circular buffer. To support timed data access, the buffer element is extended by MPEG to provide circular buffer functionality. The extension is named "MPEG_Circular_Buffer" and can be included as part of the "Buffer" structure. A buffer that provides access to timed data may contain the "MPEG_Circular_Buffer" extension.

[0100]

[0106] In the example in Figure 7, the circular buffer 240 contains empty data 234A, 234B (which may correspond to parts of the circular buffer 240 that do not contain usable media data) and decoded media frames, each containing headers 230A-230D and payloads 232A-232D. Although four frames are shown in this example, generally, the circular buffer 240 can store any number of frames, depending on the amount of memory allocated to the circular buffer 240 and the size of the frames.

[0101]

[0107] The client device 40 in Figure 1 can maintain a read pointer 236 and a write pointer 238. Generally, the read pointer 236 identifies the starting point of the next media frame to be read (in this case, the frame containing the header 230A and payload 232A), while the write pointer 238 represents the portion of the circular buffer 240 on which the subsequent frame can be written. As described above, the client device 40 can manage the circular buffer 240 in such a way that the read pointer 236 does not pass the write pointer 238, and the write pointer 238 does not overtake the read pointer 236, in order to avoid buffer overflows and underflows.

[0102]

[0108] In this example, the client device 40 can allocate memory in the circular buffer 240 based on several frames to be stored, the dimensions of each frame, and the per-texel sample format of the frame. Multiple frames may be used to swap the buffer and ensure smooth access to the buffer. The accessors of the client device 40 can maintain both a read pointer 236 and a write pointer 238, as shown. A frame is read at the read pointer 236 for rendering. A new input frame from the media decoder is inserted at the write pointer 238. Previous data in that frame may be overwritten, and the frame buffer may be resized accordingly. The client device 40 can store values ​​for the read pointer 236 and the write pointer 238 as offsets in the buffer.

[0103]

[0109] The renderer on client device 40 can ensure that timestamp(write_pointer) > timestamp(read_pointer). When overwriting a frame in the buffer with a new decoded video frame, the renderer can ensure that the read_pointer is moved to the frame with the oldest timestamp. This may result in frame drops, but it can also ensure that simultaneous access to the same frame in the buffer does not occur.

[0104]

[0110] The MPEG media extension identified by MPEG_media provides an array of MPEG media items to be used in the scene.

[0105]

[0111] The MPEG video texture extension, identified by "MPEG_video_texture," provides the possibility of linking glTF texture objects to media and their respective tracks listed by the MPEG_media object. The MPEG video texture extension also provides references to timed accessors, using timed accessor objects, to which the decoded timed texture is made available.

[0106]

[0112] Figure 8 is a block diagram illustrating an exemplary MPEG-I scene description architecture. The MPEG-I scene description solution is designed to describe scenes that mix media from different input modalities. Interaction with these different modalities is ensured through a dedicated interface to the presentation engine that consumes the scene description. Interaction with the viewer's local real world is performed through the i-1 and ii interfaces. These interfaces can leverage existing APIs such as OpenXR.

[0107]

[0113] In this example, the MPEG-I scene description architecture includes a local camera 250, a local microphone 252, a controller device 254, a sensor 256, a scene graph 262, a scene graph update 264, 2D / 3D media data 258, and a presentation engine 260. The local camera 250 may be used to capture images of local real-world objects that can be used as anchor points for the virtual scene. The local microphone 252 may be used to capture user voice. The controller device 254 may be a game controller device including one or more buttons, a trackpad, etc., for receiving user input. The sensor 256 may include a gyroscope, motion sensor, light sensor, or other types of sensors for detecting user posture, virtual camera orientation, user location, etc. The 2D / 3D media data 258 may include data for rendering the virtual scene, one or more virtual objects including, for example, walls, floors, ceilings, chairs, tables, or any other virtual objects, and the locations of objects within the virtual scene. Scene graph 262 and scene graph update 264 may correspond to the scene description and update 200 in Figure 6 and the scene graph in Figure 5.

[0108]

[0114] OpenXR is an application programming interface (API) for developing XR applications that address a wide range of XR devices. XR refers to a blend of real-world and virtual-world environments generated by computers through human interaction. XR includes technologies such as virtual reality (VR), augmented reality (AR), and mixed reality (MR). OpenXR is the interface between an application and the XR runtime. The runtime handles functions such as frame configuration, user-initiated actions, and tracking information.

[0109]

[0115] OpenXR is designed to be a layered API, meaning that users or applications can insert API layers between their application and the runtime implementation. These API layers intercept OpenXR functions from the layer above and then provide additional functionality by performing different actions than would be performed without that layer. In the simplest cases, a layer simply calls the next layer with the same arguments, but more complex layers can implement API functionality that does not exist in the layer or the runtime below it. This mechanism is essentially an architecturalized "function shimming" or "intercepting" feature designed in OpenXR, meaning it replaces the more informal way of "hooking" API calls.

[0110]

[0116] An application can determine which API layers are available to it by calling the xrEnumerateApiLayerProperties function to obtain a list of available API layers. The application can then select the desired API layers from this list and provide them to the xrCreateInstance function when creating an instance.

[0111]

[0117] The API layer can implement OpenXR functions that may or may not be supported by the underlying runtime. To expose these new functions, the API layer must expose them in the form of OpenXR extensions. It must not expose new OpenXR functions without the associated extension.

[0112]

[0118] An OpenXR instance is an object that allows an OpenXR application to communicate with the OpenXR runtime. The application achieves this communication by calling xrCreateInstance and receiving a handle to the resulting XrInstance object.

[0113]

[0119] The XrInstance object stores and tracks the state of an OpenXR-related application without storing any such state in the application's global address space. This allows applications to create multiple instances of the object and securely encapsulate their OpenXR state, as the object is opaque to the application. While the OpenXR runtime can limit the number of concurrent XrInstance objects that can be created and used, they must support the creation and use of at least one XrInstance object per process.

[0114]

[0120] A space is represented by an XrSpace handle, which is created by the application and then used in API calls. Whenever an application calls a function that returns coordinates, it provides an XrSpace to specify the reference frame on which those coordinates are represented. Similarly, when providing coordinates to a function, the application specifies which XrSpace the runtime should use to interpret those coordinates.

[0115]

[0121] OpenXR defines a set of well-known reference spaces that applications use to automatically perform spatial inference. These reference spaces are VIEW, LOCAL, and STAGE. Each reference space has well-defined meanings that establish where its origin lies and how its axes are oriented.

[0116]

[0122] A runtime tracking system improves its understanding of the world over time, and the runtime can track space independently. For example, even if the LOCAL space and STAGE space map their origins to static locations in the world, respectively, a runtime with an inside-out tracking system can continuously introduce fine-tuning to the origins of each space in order to maintain each origin in its designated position.

[0117]

[0123] Beyond the well-known reference space, runtime exposes other independently tracked spaces, such as the pose action space, which tracks the motion controller's pose over time.

[0118]

[0124] As described above, the scene description, including scene graph 262 and scene graph update 264, serves as the entry point in the dataset for consuming immersive media data. The scene description describes all virtual objects in the scene and their spatial and temporal relationships. Currently, the scene description (and no other elements) describes the relationships between the virtual objects in the scene and real-world objects. However, in order to support AR and MR applications, this disclosure recognizes the need to describe such relationships between virtual objects and real-world objects.

[0119]

[0125] More specifically, this disclosure describes techniques for anchoring scenes in real-world space. In one example, the anchor used may be mapped to OpenXR space.

[0120]

[0126] Figure 9 is a conceptual diagram illustrating an exemplary anchored XR space indicated by a scene description. According to the techniques of this disclosure, a scene description node may include a reference to that space, indicating that the scene is anchored to an XR space. The anchored XR space may be a base space, such as the local, view, or stage. Alternatively, the anchored XR space may be an application-defined space that is set up at runtime based on the application or user input. The scene description indicates the type of anchored XR space. If the anchored space is an application-defined space, the scene description may provide several constraints on the XR space that the application must adhere to.

[0121]

[0127] A scene description may indicate a transformation of the scene coordinate system to the anchor XR space. For example, rotation, translation, and scaling of the origin of the scene description coordinate space may be performed to align it in XR space. If no transformation is performed, the coordinate system of the scene description is aligned with that of the anchor space.

[0122]

[0128] XR runtime systems like OpenXR allow querying of boundary spaces in XR space. A scene description can request that the presentation engine align the scene's extent, i.e., the scene's bounding box, with the bounding box of the anchored XR space. The scene description can indicate whether this alignment involves scaling to match the two spaces. Scaling may be in both dimensions, or scaling may preserve the spatial extent relationship of the scene. If scaling is not applied, the presentation engine aligns the long edges of the scene bounding box with the long edges of the XR space, and then centers the scene bounding box so that it aligns with the center of the XR space bounding box.

[0123]

[0129] In the example shown in Figure 9, the anchor XR space is a "stage" type, corresponding to the floor of the viewer's living room.

[0124]

[0130] According to the techniques of this disclosure, an MPEG_scene_anchor extension can be added to a scene node as a glTF2.0 extension. An AR scene may include an MPEG_scene_anchor extension to describe the anchoring of the scene to the real-world XR space. This extension may be declared in an extensionsUsed element and may be included in the extensionsRequired element of a glTF2.0 scene.

[0125]

[0131] For example, if the presentation engine run by the client device 40 in Figure 1 does not support scene anchor extension, the presentation engine may revert to rendering the entire virtual scene.

[0126]

[0132] Table 1 below shows an exemplary set of elements in the MPEG_Scene_Anchor extension and their properties.

[0127] [Table 1-1]

[0128] [Table 1-2]

[0129]

[0133] The following dataset represents an exemplary instance of the scene anchor extension described above.

[0130]

number

[0131]

[0134] Figure 10 is a block diagram showing another exemplary client device 300 capable of performing the techniques of the present disclosure. Client device 300 may correspond to client device 40 in Figure 1. In this example, client device 300 includes a network interface 302, an extraction unit 304, a user interface device 306, a camera 308, a sensor 310, a microphone 312, a display 314, a decoder 316, scene data 320, and a presentation engine 330. Any or all of the various units and devices may be implemented in software, hardware, or firmware. When implemented in software or firmware, it should be understood that essential hardware such as memory for storing instructions and one or more processors for executing instructions may also be provided. The hardware may include one or more processing circuits such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated circuits or discrete logic circuits.

[0132]

[0135] The network interface 302 is generally configured to send and receive data over a network. For example, the network interface 302 may include a network interface card (NIC) configured to send and receive Ethernet® data. Alternatively, the network interface 302 may be a wireless network interface configured to send and receive network data according to one of various protocols, such as one of the various IEEE 802.11 network protocols. The network interface 302 may correspond to the network interface 54 in Figure 1.

[0133]

[0136] The retrieval unit 304 may correspond to the retrieval unit 52 in Figures 1 and 2. The retrieval unit 304 may be configured to retrieve media data via the network interface 302 according to various retrieval protocols. For example, the retrieval unit 304 may be configured to retrieve media data via unicast according to DASH. As another example, the retrieval unit 304 may be configured to receive media data via broadcast or multicast, for example, according to eMBMS. As yet another example, the retrieval unit 304 may be configured to retrieve media data according to the Real-Time Transport Protocol (RTP). Other protocols may also be used for retrieving media data.

[0134]

[0137] Decoder 316 may correspond to the audio decoder 46 and video decoder 48 in Figure 1. Additionally or alternatively, decoder 316 may correspond to decoder 212 in Figure 6. Generally, decoder 316 decodes various types of media data, such as 2D video, 3D video, audio, and metadata. Decoder 316 can output the decoded media data to scene data 320.

[0135]

[0138] Scene data 320 represents one or more memories (storage devices) for storing media data 322. In addition, according to the techniques of this disclosure, scene data 320 also stores a scene graph 324. Scene graph 324 may correspond to the scene graph in Figure 5, the scene description and update 200 in Figure 6, and / or scene graph 262 and scene graph update 264 in Figure 8. Scene graph 324 may be updated in accordance with various inputs, such as when the user moves their head (such as detected by camera 308 and / or sensor 310), when the user interacts with a virtual object (e.g., touches it) (such as detected through collisions in response to movements detected by camera 308, sensor 310, and / or user interface device 306), or through animations defined in scene graph 324.

[0136]

[0139] Scene graph 324 also includes at least one scene anchor by the techniques of the present disclosure. The scene anchor may be a reference space in a real-world presentation environment, e.g., local, view, or stage. According to the OpenXR specification, view space tracks a view origin used to generate view transformations for the primary viewer (user) with up / down, left / right, and forward / backward movement. Local space establishes a gravity-aligned world lock origin to exclude pitch and roll with up / down, left / right, and forward / backward movement. Stage space is an empty, runtime-defined flat rectangular space on which one may walk around with an origin at the center of the rectangle on the floor, with up / down, left / right, and forward / backward movement aligned with the axes of the stage.

[0137]

[0140] Therefore, view space can be used for virtual objects that are persistently presented on the display regardless of the orientation and direction of the user viewport, such as reticles and head-up displays (HUDs). Local space can be used for virtual objects that are not placed relative to the physical floor of the real-world presentation environment. Stage space can be used for other virtual objects that the user can interact with and that can be placed relative to the physical floor (stage) of the real-world presentation environment.

[0138]

[0141] The user interface device 306 may include a controller or other device for the user to provide input to the client device 300. Such a controller may include one or more buttons, a trackpad, an analog control stick, an analog trigger, and so on. Using these various elements of the user input device 306, the user can provide input that represents movement in the virtual world, interaction with virtual objects (for example, opening a door or storage box), opening a menu, and so on.

[0139]

[0142] Camera 308 represents a camera used to capture image or video data of a real-world presentation environment. In some examples, camera 308 may detect light emitted from a device coupled to a client device 300, such as an infrared lightbox, which can be used to determine the user's location and orientation within the real-world presentation environment. Additionally or alternatively, camera 308 may detect real-world objects such as floors, ceilings, walls, chairs, tables, etc., and the client device 300 may warn the user of potential collisions with real-world objects. Similarly, sensor 310 may be a gyroscope and / or other sensor for detecting information about the user's posture, location, and / or orientation within the real-world presentation environment. Microphone 312 may be a microphone for capturing the user's voice, for example, for a virtual presentation or game, for example, when multiple people are interacting with a virtual environment and the voice is sent to other users.

[0140]

[0143] Display 314 represents a display for providing visual output to the user. Display 314 may include two screens, for example, a left-eye screen and a right-eye screen, to provide separate outputs to the user's eyes. Alternatively, display 314 may include a single display configured to rapidly alternate between left-eye presentations and right-eye presentations, and the user's eyes may alternately shutter in a manner synchronized with the display.

[0141]

[0144] In this example, the presentation unit 330 includes an anchor point detection unit 332 and a conversion unit 334. The presentation unit 330 can receive data from the scene graph 324 to determine the anchor point type, for example, local, view, or stage. The presentation unit 330 can also receive image input from the camera 308 and sensor data from the sensor 310. The anchor point detection unit 332 can determine the type of anchor point to be used from the data in the scene graph 324 and identify the corresponding real-world anchor point using the image data from the camera 308. The presentation unit 330 can use the anchor points identified in the real-world presentation environment to align the virtual scene with the real-world presentation environment. For example, a real-world anchor point could be a point on the floor, a surface (e.g., a table), etc. In some examples, visual markers on real-world objects, such as quick response (QR) codes, may be used to represent real-world anchor points.

[0142]

[0145] In some examples, the presentation unit 330 can receive data from an application (not shown) running on a client device 300 that represents an application-defined space corresponding to a real-world presentation environment. For example, a user can use a user interface device 306 to define the application-defined space. The presentation unit 330 may receive data from the application that represents the application-defined space. The application-defined space may include real-world anchor points.

[0143]

[0146] The transformation unit 334 can transform the virtual scene to match the real-world presentation environment. For example, the transformation unit 334 can determine the orientation of the virtual scene and the orientation of the real-world presentation environment, and then transform the virtual scene so that the orientations of the virtual scene and the real-world presentation environment are aligned. That is, the scene graph 324 can include data representing a transformation from the scene coordinate system to the real-world coordinate system of the real-world presentation environment. Such a transformation may include rotation and / or translation. As an addition or alternative, the transformation unit 334 may scale the virtual scene to fit within the real-world presentation environment according to a scaling transformation.

[0144]

[0147] After aligning virtual scene anchor points with real-world anchor points and performing any necessary transformations, the presentation unit 330 can present media data 322 via the display 314. For example, the media data 322 may include data defining virtual objects, textures, colors, and positions relative to the virtual objects. The presentation unit 330 can present virtual objects at locations corresponding to the orientation of the user's virtual camera (determined by data received from camera 308 and sensor 310). The presentation unit 330 can update the presentation according to user movements detected from the user interface device 306, camera 308, and / or sensor 310, and / or based on updates to the scene graph 324.

[0145]

[0148] For example, a user can interact with a virtual object, for instance, by colliding with it or by pressing a button to interact with it. The presentation unit 330 can determine animations to render on the virtual object in response to such actions, such as moving, rotating, or deforming the virtual object. Such animations may be defined, for example, in animation node 184 of the exemplary scene graph in Figure 5. Thus, in subsequent output frames, the presentation unit 330 can update the location and orientation of the virtual object relative to the user's view, determined by data received from camera 308 and sensor 310. Such updates may further include modifications to the presentation of the virtual object based on the location of the light source, as well as texture and color information about the virtual object.

[0146]

[0149] In some examples, the scene graph 324 may contain data for the MPEG_scene_anchors described above with respect to Table 1. As shown in Table 1, an MPEG_scene_anchor may include a poseFeedback element and a sceneUnderstandingStream element. When the user's pose changes as detected by the camera 308 and sensor 310, the client device 300 can send pose information updates via the uplink stream indicated by the poseFeedback element using the network interface 302. If a locally captured view captured by 308 is to be uploaded, the client device 300 can send the locally captured view via the uplink stream indicated by the sceneUnderstandingStream element using the network interface 302.

[0147]

[0150] Thus, the client device 300 represents an example of a device for presenting media, comprising a memory configured to store media data defining one or more virtual objects in a virtual scene, and one or more processors implemented in the circuit, the one or more processors receiving a scene description in a bitstream containing data describing one or more virtual objects and scene anchors in the virtual scene, the scene anchors representing a correspondence between the virtual scene and a real-world presentation environment; the device is configured to use the scene anchors to determine the correspondence between the virtual scene and the real-world presentation environment; and to present one or more virtual objects at locations in the real-world presentation environment according to the determined correspondence.

[0148]

[0151] Figure 11 is a flowchart illustrating an exemplary method using the technique of this disclosure. The method of Figure 11 may be performed, for example, by the client device 40 in Figure 1, a device or set of devices conforming to the MPEG-I architecture in Figure 6 or Figure 8, or the client device 300 in Figure 10. For illustrative and explanatory purposes, the method of Figure 11 will be described in relation to the client device 300 in Figure 10. However, it should be understood that other devices or sets of devices may be configured to perform this method or a similar method.

[0149]

[0152] First, the client device 300 can receive a bitstream containing a scene description (350). For example, the client device 300 can retrieve the bitstream according to DASH, RTP, eMBMS, or other unicast, broadcast, or multicast network protocols. That is, receiving a scene description can include any method for obtaining a scene description, such as wireless or wired transmission and reception over a network from a computer-readable medium such as a transmission medium or storage medium such as a CD-ROM, DVD, Blu-ray® disc, or a storage device such as a hard drive or flash drive. The bitstream may further include media data about one or more virtual objects such as walls, ceilings, floors, doors, tables, chairs, etc. The scene description may include a scene graph that is periodically updated by scene graph updates. For example, the scene description may correspond to the exemplary scene graph in Figure 5, the scene description and update 200 in Figure 6, the scene graph 262 and scene graph update 264 in Figure 8, or the scene graph 324 in Figure 10.

[0150]

[0153] The presentation unit 330 of the client device 300 can determine anchor points from the scene description (352). According to the technique of the present disclosure, the scene description may include data about anchor points, such as MPEG_scene_anchors, as described above with respect to Table 1. The data about anchor points may include, for example, data indicating whether the reference space type (e.g., a real-world presentation environment) should be a view, local, stage, or application type of reference space. The data about anchor points may further indicate whether the scene data should be transformed (e.g., rotated, translated, and / or scaled) to match a real-world presentation environment. Anchor points may further include various actions that the user may perform, such as motion received via a controller or real-world repositioning of a device worn by the user. Anchor points may also include object identifiers shared between the presentation application and the server. Such objects may be used as anchor points. In general, the scene description may include data that associates scene anchor points with real-world anchor points, such as specific locations on the floor (e.g., midpoints on the floor).

[0151]

[0154] The presentation unit 330 can also receive data representing a real-world presentation environment (354). In some examples, this data may be provided by an application that receives input data from a user. For example, a user can use a user interface device 306 to define a real-world presentation environment, for example, by physically or virtually tracking the real-world presentation environment using a user interface device 302. In some examples, the presentation unit 330 can automatically detect the real-world presentation environment using a camera 308. In some examples, a client device 300 can receive image and / or video data captured by the camera 308, upload this data via a sceneUnderstandingStream as indicated by the scene description, and then receive data indicating the location of real-world anchor points in the real-world presentation environment via a network interface 302.

[0152]

[0155] Next, the presentation unit 330 can receive data about the virtual scene (356). For example, the bitstream may include data about one or more virtual objects, including data defining the objects themselves, the location of the objects in the virtual scene, the texture of the objects, and the color of the objects. The presentation unit 330 can anchor the virtual scene to the real-world presentation environment at determined real-world anchor points (358). The presentation unit 330 can also transform the virtual scene to align with the real-world presentation environment, for example, by using rotation, translation, and / or scaling (360). The presentation unit 330 can then present the virtual scene, including presenting the virtual objects at the appropriate positions within the virtual scene (362).

[0153]

[0156] The user can move a virtual camera (representing the user's eyes) through the virtual scene using the user interface device 306 or using physical movement that can be detected by the camera 308 and / or sensor 310. In response to such movement, the presentation unit 330 can update the presentation to take the new virtual camera location into account. For example, the position of a "view" anchored object may remain unchanged, while the position of a "local" or "stage" anchored object may change based on user movement or virtual object animation.

[0154]

[0157] Thus, the method in Figure 11 represents an example of a method that includes receiving a bitstream scene description, the scene description containing data describing scene anchors and one or more virtual objects in a virtual scene, the scene anchors representing a correspondence between a virtual scene and a real-world presentation environment; determining the correspondence between a virtual scene and a real-world presentation environment using the scene anchors; and presenting one or more virtual objects at locations in the real-world presentation environment according to the determined correspondence.

[0155]

[0158] Various examples of the techniques used in this disclosure are summarized in the following clauses.

[0156]

[0159] Clause 1: A method for presenting media data, comprising: receiving a scene description of a bitstream containing data describing one or more virtual objects and scene anchors in a virtual scene; the scene anchors representing a correspondence between a virtual scene and a real-world presentation environment; determining the correspondence between the virtual scene and the real-world presentation environment; and presenting one or more virtual objects at locations in the real-world presentation environment according to the determined correspondence.

[0157]

[0160] Clause 2: A scene anchor contains data defining a type about a real-world presentation environment, the type being one or more of local, view, or stage, as described in Clause 1.

[0158]

[0161] Clause 3: The method according to Clause 1, wherein a scene anchor includes data defining a type about a real-world presentation environment, the type being an application-defined space, and the method further comprises receiving data defining an application-defined space.

[0159]

[0162] Clause 4: A scene description is a method in any of Clauses 1 to 3, which includes data representing the transformation from the scene coordinate system to the coordinate system of the real-world presentation environment.

[0160]

[0163] Clause 5: The method according to claim 4, wherein the transformation includes one or more of rotation, translation, or scaling.

[0161]

[0164] Clause 6: The method of any one of Clauses 1 to 5, wherein presenting one or more virtual objects comprises aligning a virtual scene with a real-world presentation environment.

[0162]

[0165] Clause 7: A scene anchor contains data defining a type of real-world presentation environment, the type being one or more of local, view, or stage, as described in Clause 1.

[0163]

[0166] Clause 8; The method according to Clause 1, wherein a scene anchor includes data defining a type about a real-world presentation environment, the type being an application-defined space, and the method further comprises receiving data defining an application-defined space.

[0164]

[0167] Clause 9: A scene description as described in Clause 1, which includes data representing the transformation from the scene coordinate system to the coordinate system of the real-world presentation environment.

[0165]

[0168] Clause 10: The method according to claim 9, wherein the transformation includes one or more of rotation, translation, or scaling.

[0166]

[0169] Clause 11: The method according to Clause 1, comprising presenting one or more virtual objects to align a virtual scene with a real-world presentation environment.

[0167]

[0170] Clause 12: A device for presenting media data, comprising a device or means for performing any method described in Clauses 1 to 11.

[0168]

[0171] Clause 13; one or more means comprising one or more processors implemented in the circuit, as described in Clause 12.

[0169]

[0172] Clause 14: The device described in Clause 12, further comprising memory configured to store bitstream data.

[0170]

[0173] Clause 15: A device is a device as described in Clause 12, comprising at least one of an integrated circuit, a microprocessor, and a wireless communication device.

[0171]

[0174] Clause 16: A device for extracting media data, comprising means for receiving a scene description of a bitstream containing data describing one or more virtual objects and scene anchors in a virtual scene, the scene anchors representing a correspondence between a virtual scene and a real-world presentation environment, means for determining the correspondence between the virtual scene and the real-world presentation environment, and means for presenting one or more virtual objects at locations in the real-world presentation environment according to the determined correspondence.

[0172]

[0175] Clause 17: A method for presenting media data, comprising: receiving a scene description of a bitstream, the scene description comprising data describing scene anchors and one or more virtual objects in a virtual scene, the scene anchors representing a correspondence between a virtual scene and a real-world presentation environment, determining the correspondence between the virtual scene and the real-world presentation environment using the scene anchors, and presenting one or more virtual objects at locations in the real-world presentation environment according to the determined correspondence.

[0173]

[0176] Clause 18: A scene anchor contains data defining a type about a real-world presentation environment, the type being one or more of local, view, or stage, as described in Clause 17.

[0174]

[0177] Clause 19: The method of Clause 17, wherein a scene anchor includes data defining a type about a real-world presentation environment, the type being an application-defined space, and the method further comprises receiving data defining an application-defined space.

[0175]

[0178] Clause 20: A scene description as described in Clause 17, which includes data representing the transformation from the scene coordinate system to the coordinate system of the real-world presentation environment.

[0176]

[0179] Clause 21: A transformation is any method described in Clause 20, which includes one or more of the following: rotation, translation, or scaling.

[0177]

[0180] The method according to Clause 20, wherein presenting one or more virtual objects comprises determining a signaled location for one or more virtual objects in a virtual scene and converting the signaled location to a real-world location according to a transformation in order to generate a location in a real-world presentation environment.

[0178]

[0181] Clause 23: The method of Clause 17, comprising presenting one or more virtual objects, which involves aligning a virtual scene to a real-world presentation environment according to a scene anchor.

[0179]

[0182] Clause 24: The method of Clause 17, wherein the scene description further includes data defining an action representing an input received from a user, the method further comprises determining a modification to at least one location of one or more virtual objects or virtual cameras in response to the action, and presenting one or more virtual objects further comprises presenting one or more virtual objects in accordance with the modification to at least one location of one or more virtual objects or virtual cameras.

[0180]

[0183] Clause 25: The method of Clause 17, wherein the scene description further includes data defining object identifiers for one or more virtual objects.

[0181]

[0184] Clause 26: A device for presenting media data comprising a memory configured to store media data defining one or more virtual objects in a virtual scene, and one or more processors implemented in the circuit, the one or more processors A device configured to receive a bitstream scene description, the scene description containing data describing scene anchors and one or more virtual objects in a virtual scene, the scene anchors representing a correspondence between a virtual scene and a real-world presentation environment; to determine the correspondence between a virtual scene and a real-world presentation environment using the scene anchors; and to present one or more virtual objects at locations in the real-world presentation environment according to the determined correspondence.

[0182]

[0185] Clause 27: A scene anchor contains data defining the type of a real-world presentation environment, the type being one or more of local, view, or stage, as described in Clause 26.

[0183]

[0186] Clause 28: A scene anchor contains data defining a type about a real-world presentation environment, the type being an application-defined space, and the method further comprises receiving data defining an application-defined space, as described in Clause 26.

[0184]

[0187] Clause 29: A scene description is a device as described in Clause 26, which includes data representing the transformation from the scene coordinate system to the coordinate system of the real-world presentation environment.

[0185]

[0188] Clause 30: The transformation includes one or more of the following: rotation, translation, or scaling, as described in Clause 29.

[0186]

[0189] Clause 31: The device as described in Clause 29, wherein one or more processors are configured to present one or more virtual objects, determine signaled locations for one or more virtual objects in a virtual scene, and translate the signaled locations to real-world locations according to a transformation in order to generate locations in a real-world presentation environment.

[0187]

[0190] Clause 32: The device described in Clause 26, wherein one or more processors are configured to align a virtual scene to a real-world presentation environment according to a scene anchor in order to present one or more virtual objects.

[0188]

[0191] Clause 33: The device as described in Clause 26, wherein the scene description further includes data defining an action representing an input received from a user, one or more processors are further configured to determine a modification to at least one location of one or more virtual objects or virtual cameras in response to the action, and one or more processors are configured to present one or more virtual objects in accordance with the modification to at least one location of one or more virtual objects or virtual cameras.

[0189]

[0192] Clause 34: A scene description of a device as described in Clause 26, further including data defining object identifiers for one or more virtual objects.

[0190]

[0193] Clause 35: The device is the device described in Clause 26, comprising at least one of an integrated circuit, a microprocessor, and a wireless communication device.

[0191]

[0194] Clause 36: A computer-readable storage medium storing instructions that, when executed, cause the processor to receive a bitstream scene description, the scene description comprising data describing scene anchors and one or more virtual objects in a virtual scene, the scene anchors representing correspondences between the virtual scene and a real-world presentation environment; to determine the correspondences between the virtual scene and a real-world presentation environment using the scene anchors; and to present one or more virtual objects at locations in the real-world presentation environment according to the determined correspondences.

[0192]

[0195] Clause 37: A scene anchor is a computer-readable storage medium as described in Clause 36, containing data that defines a type about a real-world presentation environment, the type being one or more of local, view, or stage.

[0193]

[0196] Clause 38: A computer-readable storage medium as described in Clause 36, containing data that defines a type of real-world presentation environment, the type being an application-defined space, and the method further comprising receiving data that defines the application-defined space.

[0194]

[0197] Clause 39; A scene description is a computer-readable storage medium as described in Clause 36, containing data representing the transformation from the scene coordinate system to the coordinate system of the real-world presentation environment.

[0195]

[0198] Clause 40: The transformation includes one or more of rotation, translation, or scaling of the computer-readable storage medium as described in Clause 39.

[0196]

[0199] Clause 41: A computer-readable storage medium as described in Clause 39, comprising instructions causing a processor to present one or more virtual objects, which in turn causes the processor to determine signaled locations for one or more virtual objects in a virtual scene, and to translate the signaled locations according to a transformation to generate locations in a real-world presentation environment.

[0197]

[0200] Clause 42; A computer-readable storage medium as described in Clause 36, comprising instructions causing a processor to present one or more virtual objects, including instructions causing the processor to align a virtual scene to a real-world presentation environment according to a scene anchor.

[0198]

[0201] Clause 43: A computer-readable storage medium as described in Clause 36, wherein the scene description further includes data defining an action representing an input received from a user, and further includes instructions causing the processor to determine a modification to at least one location of one or more virtual objects or virtual cameras in response to the action, and instructions causing the processor to present one or more virtual objects include instructions causing the processor to present one or more virtual objects in accordance with the modification to at least one location of one or more virtual objects or virtual cameras.

[0199]

[0202] Clause 44: A scene description further includes data defining object identifiers for one or more virtual objects, as described in Clause 36, on a computer-readable storage medium.

[0200]

[0203] Clause 45: A device for presenting media data, comprising means for receiving a scene description of a bitstream, the scene description comprising data describing scene anchors and one or more virtual objects in a virtual scene, the scene anchors representing a correspondence between a virtual scene and a real-world presentation environment, means for determining the correspondence between a virtual scene and a real-world presentation environment using the scene anchors, and means for presenting one or more virtual objects at locations in the real-world presentation environment according to the determined correspondence.

[0201]

[0204] Clause 46: A scene anchor contains data defining the type of a real-world presentation environment, the type being one or more of local, view, or stage, as described in Clause 45.

[0202]

[0205] Clause 47: A scene anchor contains data defining a type about a real-world presentation environment, the type being an application-defined space, and the method further comprises receiving data defining an application-defined space, as described in Clause 45.

[0203]

[0206] Clause 48: A scene description is a device as described in Clause 45, which includes data representing the transformation from the scene coordinate system to the coordinate system of the real-world presentation environment.

[0204]

[0207] Clause 49: The transformation includes one or more of the following: rotation, translation, or scaling, as described in Clause 48.

[0205]

[0208] Clause 50: The device according to Clause 48, comprising means for presenting one or more virtual objects, means for determining signaled locations for one or more virtual objects in a virtual scene, and means for converting the signaled locations according to a transformation to real-world locations in order to generate locations in a real-world presentation environment.

[0206]

[0209] Clause 51: The device described in Clause 45, comprising means for presenting one or more virtual objects, including means for aligning a virtual scene to a real-world presentation environment according to a scene anchor.

[0207]

[0210] Clause 52: The device according to Clause 45, wherein the scene description further includes data defining an action representing an input received from a user, and means for determining a modification to at least one location of one or more virtual objects or virtual cameras in response to the action, and means for presenting one or more virtual objects further includes means for presenting one or more virtual objects in accordance with the modification to at least one location of one or more virtual objects or virtual cameras.

[0208]

[0211] Clause 53: A scene description further includes data defining object identifiers for one or more virtual objects, as described in Clause 45.

[0209]

[0212] Clause 54: A method for presenting media data, comprising: receiving a scene description of a bitstream, the scene description including data describing scene anchors and one or more virtual objects in a virtual scene, the scene anchors representing a correspondence between a virtual scene and a real-world presentation environment, determining the correspondence between the virtual scene and the real-world presentation environment using the scene anchors, and presenting one or more virtual objects at locations in the real-world presentation environment according to the determined correspondence.

[0210]

[0213] Clause 55: A scene anchor contains data defining a type of real-world presentation environment, the type being one or more of local, view, or stage, as described in Clause 54.

[0211]

[0214] Clause 56: The method of Clause 54 or 55, wherein a scene anchor includes data defining a type about a real-world presentation environment, the type being an application-defined space, and the method further comprises receiving data defining an application-defined space.

[0212]

[0215] Clause 57: A scene description is a method of any of Clauses 54 to 56, which includes data representing a transformation from the scene coordinate system to the coordinate system of the real-world presentation environment.

[0213]

[0216] Clause 58: The transformation is one or more of the following: rotation, translation, or scaling, as described in Clause 57.

[0214]

[0217] The method of Clause 59: Presenting one or more virtual objects comprises determining a signaled location for one or more virtual objects in a virtual scene and converting the signaled locations to real-world locations according to a transformation in order to generate locations in a real-world presentation environment.

[0215]

[0218] Clause 60: The method of any one of Clauses 54 to 59, wherein presenting one or more virtual objects comprises aligning a virtual scene to a real-world presentation environment according to a scene anchor.

[0216]

[0219] Clause 61: The method according to any one of Clauses 54 to 60, wherein the scene description further includes data defining an action representing an input received from a user, the method further comprises determining a modification to at least one location of one or more virtual objects or virtual cameras in response to the action, and presenting one or more virtual objects comprises presenting one or more virtual objects in accordance with the modification to at least one location of one or more virtual objects or virtual cameras.

[0217]

[0220] Clause 62: A scene description is provided in any of the manner described in Clauses 54 to 61, further comprising data defining object identifiers for one or more virtual objects.

[0218]

[0221] Clause 63: A device for presenting media data, comprising: a memory configured to store media data defining one or more virtual objects in a virtual scene; and one or more processors implemented in circuitry, which receive a bitstream scene description, the scene description comprising data describing scene anchors and one or more virtual objects in a virtual scene, the scene anchors representing a correspondence between a virtual scene and a real-world presentation environment, and which are configured to use the scene anchors to determine the correspondence between a virtual scene and a real-world presentation environment, and to present one or more virtual objects at locations in the real-world presentation environment according to the determined correspondence.

[0219]

[0222] Clause 64: A scene anchor contains data defining a type of real-world presentation environment, the type being one or more of local, view, or stage, as described in Clause 63.

[0220]

[0223] Clause 65: A scene anchor is a device as described in Clause 63 or 64, which includes data defining a type about a real-world presentation environment, the type being an application-defined space, and the method further comprises receiving data defining an application-defined space.

[0221]

[0224] Clause 66: A scene description is a device as described in any of Clauses 63 to 65, which includes data representing a transformation from the scene coordinate system to the coordinate system of the real-world presentation environment.

[0222]

[0225] Clause 67: The transformation includes one or more of the following: rotation, translation, or scaling, as described in Clause 66.

[0223]

[0226] Clause 68: The device according to Clause 66 or 67, wherein one or more processors are configured to present one or more virtual objects, determine signaled locations for one or more virtual objects in a virtual scene, and translate the signaled locations to real-world locations according to a transformation in order to generate locations in a real-world presentation environment.

[0224]

[0227] Clause 69: A device according to any one of Clauses 63 to 68, wherein one or more processors are configured to align a virtual scene to a real-world presentation environment according to a scene anchor in order to present one or more virtual objects.

[0225]

[0228] Clause 70: The device according to any one of Clauses 63 to 69, wherein the scene description further includes data defining an action representing an input received from a user, one or more processors are further configured to determine a modification to at least one location of one or more virtual objects or virtual cameras in response to the action, and one or more processors are configured to present one or more virtual objects in accordance with the modification to at least one location of one or more virtual objects or virtual cameras.

[0226]

[0229] Clause 71: A scene description of a device as described in any of Clauses 63 to 70 further includes data defining object identifiers for one or more virtual objects.

[0227]

[0230] Clause 72: A device is a device according to any one of Clauses 63 to 71, comprising at least one of an integrated circuit, a microprocessor, and a wireless communication device.

[0228]

[0231] Clause 73: A computer-readable storage medium storing instructions that, when executed, cause the processor to receive a bitstream scene description, the scene description containing data describing scene anchors and one or more virtual objects in a virtual scene, the scene anchors representing correspondences between a virtual scene and a real-world presentation environment, the scene anchors being used to determine correspondences between a virtual scene and a real-world presentation environment, and the virtual objects being presented at locations in the real-world presentation environment according to the determined correspondences.

[0229]

[0232] Clause 74: A scene anchor is a computer-readable storage medium as described in Clause 73, containing data that defines a type of real-world presentation environment, the type being one or more of local, view, or stage.

[0230]

[0233] Clause 75: A computer-readable storage medium as described in Clause 73 or 74, containing data that defines a type about a real-world presentation environment, the type being an application-defined space, and the method further comprising receiving data that defines the application-defined space.

[0231]

[0234] Clause 76: A scene description is a computer-readable storage medium as described in any of Clauses 73 to 75, containing data representing a transformation from the scene coordinate system to the coordinate system of the real-world presentation environment.

[0232]

[0235] Clause 77: The transformation includes one or more of rotation, translation, or scaling of the computer-readable storage medium as described in Clause 76.

[0233]

[0236] Clause 78: A computer-readable storage medium according to Clause 76 or 77, wherein instructions causing a processor to present one or more virtual objects include instructions causing the processor to determine signaled locations for one or more virtual objects in a virtual scene and to translate the signaled locations according to a transformation to generate locations in a real-world presentation environment.

[0234]

[0237] Clause 79: A computer-readable storage medium according to any one of Clauses 73 to 78, comprising instructions causing a processor to present one or more virtual objects, which in turn causes the processor to align a virtual scene to a real-world presentation environment according to a scene anchor.

[0235]

[0238] A computer-readable storage medium according to any one of Clauses 73 to 79, wherein the scene description further includes data defining an action representing an input received from a user, and further includes instructions causing the processor to determine a modification to at least one location of one or more virtual objects or virtual cameras in response to the action, and the instructions causing the processor to present one or more virtual objects further include instructions causing the processor to present one or more virtual objects in accordance with the modification to at least one location of one or more virtual objects or virtual cameras.

[0236]

[0239] Clause 81: A scene description further includes data defining object identifiers for one or more virtual objects, as described in any of Clauses 73 to 80, on a computer-readable storage medium.

[0237]

[0240] Clause 82: A device for presenting media data, comprising means for receiving a scene description of a bitstream, the scene description comprising data describing scene anchors and one or more virtual objects in a virtual scene, the scene anchors representing a correspondence between a virtual scene and a real-world presentation environment, means for determining the correspondence between a virtual scene and a real-world presentation environment using the scene anchors, and means for presenting one or more virtual objects at locations in the real-world presentation environment according to the determined correspondence.

[0238]

[0241] Clause 83: A scene anchor contains data defining a type of real-world presentation environment, the type being one or more of local, view, or stage, as described in Clause 82.

[0239]

[0242] Clause 84: A scene anchor is a device as described in Clause 82 or 83, which includes data defining a type about a real-world presentation environment, the type being an application-defined space, and the method further comprises receiving data defining an application-defined space.

[0240]

[0243] Clause 85: A scene description is a device as described in any of Clauses 82 to 84, which includes data representing a transformation from the scene coordinate system to the coordinate system of the real-world presentation environment.

[0241]

[0244] Clause 86: The transformation includes one or more of the following: rotation, translation, or scaling, as described in Clause 85.

[0242]

[0245] The device according to claim 85 or 86, wherein the means for presenting one or more virtual objects comprises means for determining a signaled location for one or more virtual objects in a virtual scene, and means for converting the signaled locations according to a transformation to real-world locations in order to generate locations in a real-world presentation environment.

[0243]

[0246] Clause 88: A device according to any one of Clauses 82 to 87, comprising means for presenting one or more virtual objects, for aligning a virtual scene to a real-world presentation environment according to a scene anchor.

[0244]

[0247] Clause 89: The device according to any one of Clauses 82 to 88, wherein the scene description further includes data defining an action representing an input received from a user, and means for determining a modification to at least one location of one or more virtual objects or virtual cameras in response to the action, and means for presenting one or more virtual objects further includes means for presenting one or more virtual objects in accordance with the modification to at least one location of one or more virtual objects or virtual cameras.

[0245]

[0248] Clause 90: A scene description is a device as described in any of Clauses 82 to 89, further including data defining object identifiers for one or more virtual objects.

[0246]

[0249] In one or more examples, the described functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on or transmitted through a computer-readable medium and executed by a hardware-based processing unit. The computer-readable medium may include computer-readable storage media corresponding to tangible media such as data storage media, or communication media including any medium that facilitates the transfer of computer programs from one location to another, for example, according to a communication protocol. Thus, the computer-readable medium may generally correspond to (1) non-transient tangible computer-readable storage media, or (2) communication media such as signals or carrier waves. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described herein. A computer program product may include computer-readable media.

[0247]

[0250] As an example, and not an limitation, such computer-readable storage media may include RAM, ROM, EEPROM®, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and can be accessed by a computer. Furthermore, any connection is appropriately called a computer-readable medium. For example, if instructions are transmitted from a website, server or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of a medium. However, it should be understood that computer-readable storage media and data storage media refer to non-temporary, tangible storage media, rather than connections, carriers, signals, or other temporary media. As used herein, the terms "disk" and "disc" include Compact Disc (CD), LaserDisc® (disc), Optical Disc (disc), Digital Multipurpose Disc (disc) (DVD), Floppy Disk (disk), and Blu-ray® Disc (disc), where a Disk typically reproduces data magnetically, and a Disc (disc) reproduces data optically using a laser. Any combination of the above should also be included within the scope of computer-readable media.

[0248]

[0251] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated circuits or discrete logic circuits. Therefore, the term “processor” as used herein may refer to any of the above-described structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some embodiments, the functions described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a composite codec. Also, the techniques can be adequately implemented with one or more circuits or logic elements.

[0249]

[0252] The techniques of this disclosure can be implemented in a wide variety of devices or apparatus, including wireless handsets, integrated circuits (ICs), or sets of ICs (e.g., chipsets). While this disclosure describes various components, modules, or units to highlight the functional aspects of devices configured to perform the disclosed techniques, these components, modules, or units do not necessarily need to be implemented by different hardware units. Rather, as described above, the various units, along with suitable software and / or firmware, can be combined in a codec hardware unit, including one or more processors described above, or provided by a set of interoperable hardware units.

[0250]

[0253] We have described various examples. These and other examples fall within the scope of the following claims. The invention described in the original claims of this application is listed below. [C1] A method for presenting media data, The process involves receiving a bitstream scene description, the scene description including data describing scene anchors and one or more virtual objects within a virtual scene, the scene anchors representing a correspondence between the virtual scene and a real-world presentation environment, Using the aforementioned scene anchor, the correspondence between the virtual scene and the real-world presentation environment is determined, In accordance with the determined response, present one or more virtual objects at a location within the real-world presentation environment, A method for providing this. [C2] The method according to C1, wherein the scene anchor includes data defining a type for the real-world presentation environment, the type being one or more of local, view, or stage. [C3] The scene anchor includes data defining the type of the real-world presentation environment, the type being an application-defined space, The method according to C1, further comprising receiving data that defines the application definition space. [C4] The method according to C1, wherein the scene description includes data representing a transformation from the scene coordinate system to the coordinate system of the real-world presentation environment. [C5] The method according to C4, wherein the transformation includes one or more of rotation, translation, or scaling. [C6] Presenting one or more virtual objects means that Determining the signaled location of one or more virtual objects within the virtual scene, To generate the aforementioned locations within the real-world presentation environment, the signaled locations are converted to real-world locations according to the conversion, A method for C4 comprising: [C7] The method of C1, wherein presenting one or more virtual objects comprises aligning the virtual scene to the real-world presentation environment according to the scene anchor. [C8] The scene description further includes data that defines actions representing input received from the user, The method further comprises determining a modification to at least one of the one or more virtual objects or virtual cameras at the location in response to the action, The method of C1, wherein presenting the one or more virtual objects comprises presenting the one or more virtual objects in accordance with the modification of the location of at least one of the one or more virtual objects or the virtual camera. [C9] The method according to C1, wherein the scene description further includes data defining object identifiers for one or more virtual objects. [C10] A device for presenting media data, Memory configured to store media data that defines one or more virtual objects within a virtual scene, The circuit comprises one or more processors implemented within the circuit, wherein the one or more processors The process involves receiving a bitstream scene description, the scene description including data describing a scene anchor and one or more virtual objects within the virtual scene, the scene anchor representing a correspondence between the virtual scene and a real-world presentation environment. Using the aforementioned scene anchor, the correspondence between the virtual scene and the real-world presentation environment is determined, In accordance with the determined response, present one or more virtual objects at a location within the real-world presentation environment, A device configured to perform the following actions. [C11] The scene anchor includes data defining a type for the real-world presentation environment, the type being one or more of local, view, or stage, as described in C10. [C12] The scene anchor includes data defining a type for the real-world presentation environment, the type being an application-defined space, The device according to C10, further comprising receiving data defining the application definition space. [C13] The device according to C10, wherein the scene description includes data representing a transformation from the scene coordinate system to the coordinate system of the real-world presentation environment. [C14] The device according to C13, wherein the transformation includes one or more of rotation, translation, or scaling. [C15] In order to present the one or more virtual objects, the one or more processors Determining the signaled location of one or more virtual objects within the virtual scene, To generate the aforementioned locations within the real-world presentation environment, the signaled locations are converted to real-world locations according to the conversion, A device as described in C13, configured to perform the following actions. [C16] The device according to C10, wherein the one or more processors are configured to align the virtual scene to the real-world presentation environment according to the scene anchor in order to present the one or more virtual objects. [C17] The scene description further includes data that defines actions representing input received from the user, The one or more processors are further configured to determine a modification to at least one of the one or more virtual objects or virtual cameras at the location in response to the action. The device according to C10, wherein the one or more processors are configured to present the one or more virtual objects in accordance with the modification of the location of at least one of the one or more virtual objects or the virtual camera. [C18] The device according to C10, wherein the scene description further includes data defining object identifiers for one or more virtual objects. [C19] Integrated circuits, Microprocessor, and Wireless communication devices A device according to C10, comprising at least one of the following. [C20] When executed, the processor, The process involves receiving a bitstream scene description, the scene description including data describing scene anchors and one or more virtual objects within a virtual scene, the scene anchors representing a correspondence between the virtual scene and a real-world presentation environment, Using the aforementioned scene anchor, the correspondence between the virtual scene and the real-world presentation environment is determined, In accordance with the determined response, present one or more virtual objects at a location within the real-world presentation environment, A computer-readable storage medium that stores commands to perform an action. [C21] The computer-readable storage medium described in C20, which includes data defining a type for the real-world presentation environment, wherein the type is one or more of local, view, or stage. [C22] The scene anchor includes data defining a type for the real-world presentation environment, the type being an application-defined space, The computer-readable storage medium according to C20, further comprising receiving data defining the application-defined space. [C23] The computer-readable storage medium described in C20, wherein the scene description includes data representing a transformation from the scene coordinate system to the coordinate system of the real-world presentation environment. [C24] The computer-readable storage medium according to C23, wherein the transformation includes one or more of rotation, translation, or scaling. [C25] The instruction that causes the processor to present the one or more virtual objects means that the processor, Determining the signaled location of one or more virtual objects within the virtual scene, To generate the aforementioned locations within the real-world presentation environment, the signaled locations are converted to real-world locations according to the conversion, A computer-readable storage medium as described in C23, equipped with instructions to perform the following actions. [C26] The computer-readable storage medium according to C20, wherein the instruction causing the processor to present the one or more virtual objects includes an instruction causing the processor to align the virtual scene to the real-world presentation environment according to the scene anchor. [C27] The scene description further includes data that defines actions representing input received from the user, The processor further provides instructions that cause it to determine a modification to at least one of the one or more virtual objects or virtual cameras at the location in response to the action, The computer-readable storage medium according to C20, wherein the instruction causing the processor to present the one or more virtual objects comprises an instruction causing the processor to present the one or more virtual objects in accordance with the modification of the one or more virtual objects or the location of at least one of the virtual cameras. [C28] The computer-readable storage medium according to C20, wherein the scene description further includes data defining object identifiers for one or more virtual objects. [C29] A device for presenting media data, Means for receiving a bitstream scene description, the scene description including data describing scene anchors and one or more virtual objects in a virtual scene, the scene anchors representing a correspondence between the virtual scene and a real-world presentation environment, Means for determining the correspondence between the virtual scene and the real-world presentation environment using the aforementioned scene anchor, Means for presenting one or more virtual objects in a location within the real-world presentation environment, in accordance with the determined correspondence, A device equipped with the following features. [C30] The scene anchor includes data defining a type for the real-world presentation environment, the type being one or more of local, view, or stage, as described in C29. [C31] The scene anchor includes data defining a type for the real-world presentation environment, the type being an application-defined space, The device according to C29, further comprising receiving data defining the application definition space. [C32] The device according to C29, wherein the scene description includes data representing a transformation from the scene coordinate system to the coordinate system of the real-world presentation environment. [C33] The device according to C32, wherein the transformation includes one or more of rotation, translation, or scaling. [C34] The means for presenting one or more virtual objects is: Means for determining the signaled location of one or more virtual objects within the virtual scene, In order to generate the location within the real-world presentation environment, means for converting the signaled location to a real-world location according to the conversion, A device as described in C32, comprising: [C35] The device according to C29, wherein the means for presenting the one or more virtual objects comprises means for aligning the virtual scene to the real-world presentation environment according to the scene anchor. [C36] The scene description further includes data that defines actions representing input received from the user, The system further comprises means for determining a modification to the location of at least one of the one or more virtual objects or virtual cameras in response to the aforementioned action, The device according to C29, wherein the means for presenting the one or more virtual objects comprises means for presenting the one or more virtual objects in accordance with the modification of the location of at least one of the one or more virtual objects or the virtual camera. [C37] The device according to C29, wherein the scene description further includes data defining object identifiers for one or more virtual objects.

Claims

1. A method for presenting media data, Receiving a scene description of a bitstream, the scene description includes data describing scene anchors and one or more virtual objects in a virtual scene, the scene anchors representing a correspondence between the virtual scene and a real-world presentation environment, the representation comprising i) defining a type for the real-world presentation environment, and ii) indicating whether there is an alignment between the real-world presentation environment and the virtual scene. Using the scene anchors, the correspondence between the virtual scene and the real-world presentation environment is determined, and the determination comprises: i) determining the anchor point for the virtual scene in the real-world presentation environment based on the type; and ii) determining whether the virtual scene and the real-world presentation environment are aligned based on the instructions. In accordance with the determined response, present one or more virtual objects at a location within the real-world presentation environment, A method for providing this.

2. The method according to claim 1, wherein the type is one or more of local, view, stage, or application-defined space, and the method optionally further comprises receiving data that defines the application-defined space.

3. The method according to claim 1, wherein the scene description includes data representing a transformation from the scene coordinate system to the coordinate system of the real-world presentation environment, and optionally the transformation includes one or more of rotation, translation, or scaling.

4. Presenting one or more virtual objects means Determining the signaled location of one or more virtual objects within the virtual scene, To generate the aforementioned locations within the real-world presentation environment, the signaled locations are converted to real-world locations according to the conversion, The method according to claim 3, comprising:

5. The method according to claim 1, wherein presenting one or more virtual objects comprises aligning the virtual scene to the real-world presentation environment according to the scene anchor.

6. The aforementioned scene description further includes data that defines actions representing input received from the user, The method further comprises determining a modification to at least one of the one or more virtual objects or virtual cameras at the location in response to the action, The method according to claim 1, wherein presenting the one or more virtual objects comprises presenting the one or more virtual objects in accordance with the modification of the location of at least one of the one or more virtual objects or the virtual camera.

7. The method according to claim 1, wherein the scene description further includes data defining object identifiers for one or more virtual objects.

8. A device for presenting media data, Memory configured to store media data that defines one or more virtual objects within a virtual scene, The circuit comprises one or more processors implemented within the circuit, and the one or more processors are Receiving a scene description of a bitstream, the scene description including data describing scene anchors and one or more virtual objects in a virtual scene, the scene anchors representing a correspondence between the virtual scene and a real-world presentation environment, the representation comprising i) defining a type for the real-world presentation environment, and ii) indicating whether there is an alignment between the real-world presentation environment and the virtual scene. Using the scene anchors, the correspondence between the virtual scene and the real-world presentation environment is determined, and the determination comprises: i) determining the anchor point for the virtual scene in the real-world presentation environment based on the type; and ii) determining whether the virtual scene and the real-world presentation environment are aligned based on the instructions. In accordance with the determined response, present one or more virtual objects at a location within the real-world presentation environment, A device configured to perform the following actions.

9. The device according to claim 8, wherein the scene anchor includes data defining a type for the real-world presentation environment, the type being one or more of local, view, or stage, or the scene anchor includes data defining a type for the real-world presentation environment, the type being an application-defined space, and the device further comprises receiving data defining the application-defined space.

10. The device according to claim 8, wherein the scene description includes data representing a transformation from the scene coordinate system to the coordinate system of the real-world presentation environment, and optionally the transformation includes one or more of rotation, translation, or scaling.

11. In order to present the one or more virtual objects, the one or more processors Determining the signaled location of one or more virtual objects within the virtual scene, To generate the aforementioned locations within the real-world presentation environment, the signaled locations are converted to real-world locations according to the conversion, The device according to claim 10, configured to perform the following:

12. The device according to claim 8, wherein, in order to present the one or more virtual objects, the one or more processors are configured to align the virtual scene to the real-world presentation environment according to the scene anchor.

13. The aforementioned scene description further includes data that defines actions representing input received from the user, The one or more processors are further configured to determine a modification to at least one of the one or more virtual objects or virtual cameras at the location in response to the action. The device according to claim 8, wherein the one or more processors are configured to present the one or more virtual objects in accordance with the modification of the location of at least one of the one or more virtual objects or the virtual camera, and / or the scene description further includes data defining object identifiers for the one or more virtual objects.

14. Integrated circuits, Microprocessor, and Wireless communication devices The device according to claim 8, comprising at least one of the following.

15. A computer-readable storage medium storing instructions that, when executed, cause a processor to perform the method according to any one of claims 1 to 7.

Images