Use case description to access media

Abstract

Techniques for processing media content are provided. For example, a process may include using a scene description to obtain media information and buffer information for a media object. The media information may identify the expected format for storing the media object in one or more output buffers of a media pipeline. The buffer information may include output buffer attributes. The process may include using the media information and buffer information to initialize a media pipeline for the media object. The process may include obtaining a media object with a delivery format and generating one or more processed media objects. Each processed media object may be converted from the delivery format to the expected format for storage in an output buffer. The process may include outputting the processed media object (a) to (one or more) output buffers using the initialized media pipeline.

Description

Technical Field

[0001] This application relates to systems and methods for processing media. For example, aspects of this disclosure pertain to a Media Access Function (MAF) for retrieving, processing, and delivering media objects. Background Technology

[0002] Many devices and systems allow media data to be processed and output for consumption. Media data can include video data and / or audio data. For example, digital video data can include large amounts of data to meet the needs of consumers and video providers. For instance, consumers of video data expect the highest quality video with high fidelity, resolution, frame rate, etc. As a result, the large amount of video data required to meet these needs places a burden on the communication networks and devices that process and store video data.

[0003] Various video codec technologies can be used to compress video data. Video codecs are performed according to one or more video codec standards. For example, video codec standards include ITU-T H.261, ISO / IEC MPEG-1 Visual, ITU-T H.262 or ISO / IEC MPEG-2 Visual, ITU-T H.263, ISO / IEC MPEG-4 Visual, ITU-T H.264 or ISO / IEC MPEG-4 AVC, including its Scalable Video Codec (SVC) and Multi-View Video Codec (MVC) extensions, and High-Efficiency Video Codec (HEVC), also known as ITU-T H.265 and ISO / IEC 23008-2, including its Scalable Codec extension (i.e., Scalable High-Efficiency Video Codec, SHVC) and Multi-View Extension (i.e., Multi-View High-Efficiency Video Codec, MV-HEVC). Video codecs typically utilize prediction methods that take advantage of redundancy present in video images or sequences (e.g., inter-frame prediction, intra-frame prediction, etc.). A key goal of video encoding and decoding technology is to compress video data into a form that uses a lower bit rate, while avoiding or minimizing the degradation of video quality.

[0004] After video data is encoded, it can be packetized for transmission or storage. Video data can be assembled into video files conforming to any of various standards, such as the International Organization for Standardization (ISO) Basic Media File Format and its extensions, such as ITU-T H.264 / AVC. This packetized video data can be transmitted in various ways, such as via network streaming over computer networks. Summary of the Invention

[0005] In some examples, systems and techniques for decoupling fetching and rendering operations during media access and delivery are described. According to at least one illustrative example, a method for processing media content is provided. The method includes: acquiring media information and buffer information of media objects determined from a scene description of at least one media scene, wherein the media information identifies a desired format for storing the media objects in one or more output buffers of a media pipeline, and wherein the buffer information includes one or more attributes of the one or more output buffers; initializing a media pipeline for the media objects based on the media information and buffer information; acquiring media objects of a delivery format; generating one or more processed media objects, at least partially by processing the acquired media objects, wherein each of the one or more processed media objects is converted from the delivery format to the desired format for storing each processed media object in an output buffer of the one or more output buffers; and outputting the one or more processed media objects to the one or more output buffers using the initialized media pipeline.

[0006] In another example, an apparatus for processing media content is provided, the apparatus including a memory and one or more processors (e.g., implemented in a circuit) coupled to the memory. The one or more processors are configured and capable of: acquiring media information and buffer information of a media object determined from a scene description of at least one media scene, wherein the media information identifies a desired format for storing the media object in one or more output buffers of a media pipeline, and wherein the buffer information includes one or more attributes of the one or more output buffers; initializing a media pipeline for the media object based on the media information and the buffer information; acquiring a media object in a delivery format; generating one or more processed media objects at least partially by processing the acquired media objects, wherein each of the one or more processed media objects is converted from the delivery format to the desired format to store each processed media object in an output buffer of the one or more output buffers; and outputting the one or more processed media objects to the one or more output buffers using the initialized media pipeline.

[0007] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided, which, when executed by one or more processors, cause the one or more processors to: acquire media information and buffer information of a media object determined from a scene description of at least one media scene, wherein the media information identifies an expected format for storing the media object in one or more output buffers of a media pipeline, and wherein the buffer information includes one or more attributes of the one or more output buffers; initialize a media pipeline for the media object based on the media information and the buffer information; acquire a media object of a delivery format; generate one or more processed media objects at least in part by processing the acquired media objects, wherein each of the one or more processed media objects is converted from the delivery format to the expected format to store each processed media object in an output buffer of one or more output buffers; and output the one or more processed media objects to one or more output buffers using the initialized media pipeline.

[0008] In another example, an apparatus is provided, comprising: components for acquiring media information and buffer information of a media object determined from a scene description of at least one media scene, wherein the media information identifies an expected format for storing the media object in one or more output buffers of a media pipeline, and wherein the buffer information includes one or more attributes of the one or more output buffers; components for initializing a media pipeline for the media object based on the media information and the buffer information; components for acquiring a media object of a delivery format; components for generating one or more processed media objects at least partially by processing the acquired media object, wherein each of the one or more processed media objects is converted from the delivery format to the expected format to store each processed media object in an output buffer of one or more output buffers; and components for outputting the one or more processed media objects to one or more output buffers using the initialized media pipeline.

[0009] In some respects, media information and buffer information are provided as parameters for Media Access Functions (MAF) application programming interface (API) calls made to media objects that describe the scene.

[0010] In some respects, media information includes one or more source locations used to obtain media objects.

[0011] In some respects, media information identifiers are used to replay one or more rendering parameters of a media object from one or more output buffers.

[0012] In some respects, one or more calls to the MAF API are used to initialize the media pipeline.

[0013] In some aspects, the methods, apparatuses, and computer-readable media described above also include controlling each of one or more output buffers by at least in part by passing buffer information to a buffer application programming interface (API) through one or more calls to a buffer API.

[0014] In some aspects, initializing the media pipeline includes allocating a new buffer in response to determining that a buffer identifier does not belong to an existing buffer, the allocation being based on one or more buffer attributes associated with the buffer identifier.

[0015] In some respects, buffer information includes a list of buffers, each entry in which a buffer identifier is included and is associated with one or more buffer attributes.

[0016] In some respects, one or more output buffers include at least one cyclic buffer.

[0017] In some respects, the scene description includes a Graphical Language Sending Format (glTF) scene description.

[0018] In some respects, at least one media scene includes one or more immersive media objects, which include at least one of three-dimensional (3D) mesh objects, 3D point cloud objects, virtual reality (VR) objects, augmented reality (AR) objects, and 6 degrees of freedom (DoF) objects.

[0019] In some respects, a media object is obtained in a delivery format from one or more source locations identified in the media information.

[0020] In some respects, at least one of a Uniform Resource Locator (URL), a Uniform Resource Identifier (URI), and a local file path is used to identify one or more source locations in media information.

[0021] In some respects, one or more source locations identified in the media information include the default media source location and one or more alternative media source locations that are different from the default media source location.

[0022] In some respects, the selected source location is determined based on one or more selection factors identified in the media information, including one or more of playback preferences, playback capabilities, and network characteristics.

[0023] In some respects, media objects are obtained from the source location selected based on view information determined from one or more MAF API calls.

[0024] In some respects, the view information determined from one or more MAF API calls includes the position of media objects relative to the viewer's location.

[0025] In some aspects, the methods, apparatus, and computer-readable media described above further include: configuring source positioning based on view information; wherein the configuration includes adjusting one or more of the following: adjusting the level of detail of the acquired media object, adjusting the bit rate of the acquired media object, and adjusting the visible portion of the acquired media object.

[0026] In some aspects, the device may be a mobile device (e.g., a mobile phone or so-called "smartphone," or other mobile device), a personal computer, a laptop computer, an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a smart wearable device, a server computer, a camera (e.g., a digital camera, an Internet Protocol (IP) camera, etc.), a multi-camera system, or other device, or may be a combination of the above. In some aspects, the device includes one or more cameras for capturing one or more images. In some aspects, the device includes a display for displaying one or more images, announcements, and / or other displayable data. In some aspects, the aforementioned device may include one or more sensors.

[0027] This summary is not intended to identify key or essential features of the subject matter of the claims, nor is it intended to be used in isolation to determine the scope of the claims. The subject matter should be understood by referring to the entire specification of this patent, any or all of the drawings, and the appropriate portions of each claim.

[0028] The foregoing and other features and embodiments will become more apparent from the following description, claims and drawings. Attached Figure Description

[0029] The illustrative embodiments of the present invention are described in detail below with reference to the accompanying drawings:

[0030] Figure 1 It is a diagram illustrating the structure of an example file conforming to the ISO Basic Media File Format (ISOBMFF) based on some examples;

[0031] Figure 2 This is a diagram illustrating examples of ISO base media files (formatted according to ISOBMFF) containing data and metadata for video rendering, based on some examples.

[0032] Figure 3 This is a diagram illustrating examples of media systems, including a rendering engine and Media Access Function (MAF), based on several examples.

[0033] Figure 4This is a diagram illustrating another example of a media system including a rendering engine and MAF with plugins, based on some examples;

[0034] Figure 5 It is a diagram illustrating different examples of media pipelines based on some examples;

[0035] Figure 6 This is another diagram illustrating different examples of media pipelines based on some examples;

[0036] Figure 7 This is a diagram illustrating another example of a system including a rendering engine and MAF, based on some examples;

[0037] Figure 8 This is a flowchart illustrating an example of the process of capturing Moving Picture Experts Group (MPEG) media objects using media pipes and buffers, based on some examples.

[0038] Figure 9 This is a flowchart illustrating an example of the process of fetching media objects using media pipes and buffers, based on some examples;

[0039] Figure 10 It is a block diagram illustrating examples of video encoding devices based on some examples; and

[0040] Figure 11 This is a block diagram illustrating an example of a video decoding device based on some examples.

[0041] Figure 12 This is a diagram illustrating an example system architecture used to implement some of the aspects described in this article. Detailed Implementation

[0042] Certain aspects and embodiments of this disclosure are provided below. Some of these aspects and embodiments can be applied independently, and some can be applied in combination, as will be apparent to those skilled in the art. In the following description, specific details are set forth for purposes of explanation in order to provide a thorough understanding of embodiments of this application. However, it will be apparent that various embodiments can be practiced without these specific details. The drawings and description are not intended to be limiting.

[0043] The following description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the subsequent description of exemplary embodiments will provide those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of this application as set forth in the appended claims.

[0044] As systems used to present audiovisual and / or multimedia content have grown in both popularity and complexity, the processing workload involved in delivering such content to end users in a seamless and enjoyable manner has also increased considerably. This increased processing workload can be attributed to the significant expansion in the scale and range of multimedia content types delivered to users. Examples of multimedia content include two-dimensional (2D) media content, three-dimensional (3D) media content (also known as immersive media content or 360-degree media), and so on.

[0045] 2D media content can include a number of discrete media element types (e.g., video tracks, audio tracks, metadata, or caption tracks). Immersive or 3D media content can include more complex information than 2D media content. For example, in addition to including one or more media element types mentioned above for 2D media content, immersive media content can include multiple additional media element types. For example, additional media element types can be delivered to the rendering engine along with the aforementioned 2D media element types, typically simultaneously. Additional media element types associated with immersive media content can include, but are not limited to, 2D textures, 3D meshes, 3D point clouds, depth maps, transparency / specular reflection data, occlusion data, and / or other element types. Furthermore, some immersive media content can also be interactive, in which case more media element types may be required, such as: 3DoF data for immersive scenes, where users can change their gaze direction (e.g., looking around a 3D virtual scene in a virtual reality environment); and 6DoF data for immersive scenes, where users can change their gaze direction and their position within the scene (e.g., walking around in a 3D virtual scene).

[0046] Accordingly, immersive media content may involve a significantly greater computational workload compared to 2D media content, not only in the final processing of rendering immersive media elements at the rendering engine, but also in the backend retrieval, preprocessing, and delivery of immersive media elements to the rendering engine before rendering. Additionally, in the context of presenting immersive media content, a robust architecture for buffering or otherwise delivering the necessary media elements to the rendering engine can be important, as immersive media content typically does not follow a linear time scheme, unlike traditional 2D media content. For example, an open-world 3D environment that users can freely roam and explore may present content based on a calculated user location rather than a timestamp.

[0047] While both 2D and immersive media content can be presented within the same general framework, where various media elements are retrieved and delivered to a rendering engine (e.g., under the Moving Picture Experts Group (MPEG) standard), the optimal delivery and presentation of immersive media elements may require additional considerations given the factors mentioned above. Furthermore, as the complexity of immersive media content continues to increase rapidly, scalable and efficient buffering solutions become increasingly desirable. Moreover, with the growing adoption of immersive media by institutions and consumers, a lightweight yet robust buffering solution capable of seamless operation even on low-end consumer computing hardware is expected.

[0048] This document describes systems, apparatuses, processes (also referred to as methods), and computer-readable media (collectively, “Systems and Technologies”) that provide an improved media access architecture. The media access architecture can be used to deliver media content and media elements for presentation. Media content and media elements can include, but are not limited to, one or more of the media content types and / or media element types described above. The media access architecture can include “Media Access Functions” or “MAFs,” which will be described in more detail below. As described in more detail below, the media access architecture provides flexible solutions for the retrieval, preprocessing, buffering, and delivery of various media types. Examples of various media types include, but are not limited to, the 2D media elements and 3D / immersive media elements described above. As used herein, media elements can include video elements, audio elements, and / or other media elements.

[0049] For illustrative and exemplary purposes, the currently disclosed media access architecture is described with reference to the Moving Picture Experts Group (MPEG) standards, although it should be noted that this is not intended to be construed as restrictive, and the media access architecture can be used with a variety of other types of non-MPEG media (e.g., VVC media, HEVC media, AVC media, Open Media Consortium (AOMedia) Video 1 (AV1) media, etc.) without departing from the scope of this disclosure. The MPEG standards include proposed improvements to existing scene description formats to support immersive media and have a specific instance of glTF 2.0 (“Graphics Language Sending Format”). The media access architecture described herein can provide specific extensions that address identified gaps within existing MPEG and glTF 2.0-based frameworks(one or more) while maintaining compatibility and / or interoperability.

[0050] In some examples, the systems and techniques described herein provide improved delivery of media content and elements (immersive or otherwise) by decoupling media fetching from media rendering. Media access functionality (MAF) (e.g., the media access functionality of the aforementioned media access architecture, as described herein) can be provided between various media sources delivering media content to a player device or system and the rendering engine of the player device or system. The media access architecture (including the MAF) can be part of a player device or system, integrated as a hardware component, a logical component, or some combination of both. The rendering engine can be used to process and render the content of the scene. This scene content can be delivered to the rendering engine via the MAF or otherwise made available to the rendering engine. The MAF can perform media fetching and related tasks, such as preprocessing, decoding, format conversion, etc., while the rendering engine can perform rendering functions, resulting in the decoupling of the rendering engine's rendering from the media content fetched from the media source via the MAF. In some examples of this decoupling dynamic, the MAF can asynchronously retrieve and prepare media content for rendering upon request from the rendering engine. At a later time, the rendering engine can then access the media content retrieved and prepared via the MAF and render the scene content.

[0051] In some cases, MAF can be configured to deliver any given media element in a file format requested or desired by any given rendering engine, in which case MAF is not limited to any specific rendering engine(s). In some cases, MAF is not limited to any specific media source. For example, MAF can access or retrieve media content from any media source that MAF can address or reach (e.g., locally or via wired or wireless network). In some cases, MAF is not limited to any specific input file format. For example, MAF can process media content in any input file format that MAF can read, or can retrieve appropriate plugins for the input file format. In some examples, MAF can be a plugin usable by one or more rendering engines, which can be integrated into (e.g., integrated into software, etc.) one or more rendering engines, a combination of both, and / or otherwise implemented.

[0052] During runtime, the rendering engine can parse and process the scene description of a given media content item (referred to as a media content item). Based on parsing and processing the scene description, the rendering engine can determine information about the specific media elements it will render. For example, each media content item is associated with a scene description, which specifies multiple media elements for that media content item. Based on the scene description (and any subsequent scene description updates), the rendering engine can communicate with the MAF, allowing the MAF to initiate a retrieval of any media elements (from one or more media sources) required for rendering by the rendering engine. This communication from the rendering engine to the MAF can be performed using the MAF's application programming interface (API), also referred to herein as the "MAF API". For example, the rendering engine can call the MAF API, which allows the MAF to initiate the retrieval of one or more media elements from one or more media sources. In some examples, the MAF API call from the rendering engine to the MAF API may include parameters and other information specifying how the MAF will retrieve one or more specific media elements. In an illustrative example, a MAF API call may include: 1) the source location or identifier of the requested media element; 2) an indication of which parts of the media element are required and at what resolution / level of detail; 3) timing information for when the requested media element must be available; 4) output information, including details of the requested format for the media element and how the media element will be delivered to the rendering engine; any combination thereof; and / or any other suitable information.

[0053] In response to MAF API calls, MAF can initialize, allocate, or initialize one or more media pipelines to perform media fetching and / or decoding tasks. For example, a media pipeline can be initialized or allocated to perform both media fetching and decoding tasks, while in another example, a media pipeline can be initialized or allocated so that it is dedicated to performing either a media fetching task or a decoding task. In some cases, MAF (or the rendering engine) can initialize a media pipeline when it determines that a media pipeline has not yet been created. If a media pipeline has already been created, in some examples, MAF can determine, for example, based on one or more parameters specified in the MAF API call, that an existing media pipeline needs to be reconfigured; in this case, MAF can allocate or reallocate the existing media pipeline as needed.

[0054] Media and / or metadata are captured and processed through one or more media pipelines. For example, a media pipeline may take one or more media elements or other tracks of media content items (including metadata) as input and perform one or more processing, decoding, and / or encoding operations. A media pipeline may include decoding and encoding engines (e.g., video decoding engines, audio encoding engines, etc.), processing engines (e.g., performing tasks including but not limited to streaming, demultiplexing, decoding, decryption, encoding, format conversion, etc.), and one or more buffers (e.g., loop buffers or other types of buffers). One or more media pipelines may terminate at a final output buffer or a set of final output buffers. In some examples, (one or more) final output buffers may be included in a media pipeline.

[0055] The final output buffer is used to exchange data with the rendering engine. For example, the MAF can use the media pipeline to retrieve the requested media element from the source location identified in the API call. The MAF can decode the requested media element from its delivery format (e.g., the format that the media source location provides to the MAF of the requested media element) into the format identified in the API call (e.g., the format used by the rendering engine for rendering). If necessary, the MAF can perform one or more preprocessing operations. The MAF can then output the media in the format identified in the API call to one or more final output buffers.

[0056] In some examples, buffer management (e.g., buffer instantiation and allocation) can be performed by MAF, the rendering engine, or some combination thereof using the buffer API. In some cases, MAF may bear primary responsibility for buffer management. Because each media pipeline can include one or more buffers, buffer instantiation can be performed concurrently with and / or in conjunction with media pipeline instantiation. It is also possible to initialize one or more buffers in an existing media pipeline (e.g., new buffers can be added to an existing media pipeline).

[0057] As previously mentioned, in some cases, it may be necessary to reconfigure or allocate existing media pipelines (which may contain one or more buffers) based on parameters and / or media information contained in the MAF API calls. Therefore, MAF can also utilize the buffer API to reconfigure, allocate, or reallocate existing buffers, for example, to configure existing buffers in relation to any newly received MAF API calls. In one illustrative example, media pipelines and / or buffers may already exist because similar or identical media elements are currently being processed via different API calls, or were recently processed in response to a previous API call. In another illustrative example, certain frequently used and / or complex media pipelines and / or buffers may remain active instead of being discarded or released after the task is completed (this can result in improved efficiency in terms of time and / or computational resources).

[0058] Using the systems and techniques described herein, MAF decouples the rendering engine from increasingly complex tasks associated with media retrieval and processing. These tasks may include connecting to multiple different source locations, handling delivery formats that differ from the rendering engine's expected formats, performing hardware and / or other resource management and allocation, and performing decoding, encoding, and preprocessing tasks. Advantageously, the MAF API (and in some cases the Buffer API) is used to abstract the challenges and operations of these tasks, extracting them into API calls that provide a consistent language for any given rendering engine to access different media elements and sources.

[0059] Video codecs implement video compression techniques to efficiently encode and decode video data. Video compression techniques may include applying different prediction modes, including spatial prediction (e.g., intra-frame prediction or intra-prediction), temporal prediction (e.g., inter-frame prediction or inter-prediction), inter-layer prediction (across different layers of video data), and / or other prediction techniques to reduce or eliminate redundancy inherent in the video sequence. A video encoder can segment each frame of the original video sequence into rectangular regions called video blocks or codec units (described in more detail below). These video blocks can be encoded using specific prediction modes.

[0060] Different types of media formats can be used to encapsulate video, such as encoded video. An example of a media format includes the ISO Basic Media File Format (ISOBMFF, as specified in ISO / IEC 14496-12, the entire contents of which are incorporated herein by reference for all purposes). Other media file formats derived from ISOBMFF (ISO / IEC 14496-12) include the MPEG-4 file format (ISO / IEC 14496-14), the 3GPP file format (3GPP TS 26.244), and the AVC file format (ISO / IEC 14496-15). For example, a video bitstream encoded as described above can be written to or packaged into one or more files using ISOBMFF, file formats derived from ISOBMFF, some other file formats, and / or combinations of file formats including ISOBMFF. One or more ISOBMFF files can be played back using a video player device, sent by an encoding device (or file generation device) and then displayed by a player device, stored, and / or used in any other suitable manner.

[0061] ISOBMFF is used as the foundation for many codec container formats (such as AVC file format) and many multimedia container formats (such as MPEG-4 file format, 3GPP file format (3GP), DVB file format, etc.). Continuous media (e.g., audio and video), still media (e.g., images), and metadata can be stored in files conforming to ISOBMFF. Files constructed according to ISOBMFF can be used for a variety of purposes, including local media file playback, progressive download of remote files, segmentation for Dynamic Adaptive Streaming (DASH) over HTTP, containerization instructions for content to be streamed, recording of received live media streams, and other suitable purposes. For example, although ISOBMFF was originally designed for storage, it has proven to be very valuable for media streaming (e.g., for progressive download or DASH). Movie clips defined in ISOBMFF can be used for streaming purposes.

[0062] ISOBMFF is designed to contain timing-controlled media information in flexible and scalable formats that facilitate media exchange, management, editing, and presentation. Media presentation can be "local" to the system containing the presentation, or it can be performed via a network or other streaming mechanism (e.g., DASH or other suitable streaming mechanisms). A "presentation" as defined by the ISOBMFF specification can include media files associated with a sequence of images, typically associated with sequential capture by a video capture device, or associated for some other reason. In some examples, a presentation may also be referred to as a movie, video presentation, or rendering. In some examples, a presentation can include audio. A single presentation can be contained in one or more files, one of which contains metadata for the entire presentation. Metadata includes information such as timing and frame control data, descriptors, pointers, parameters, and other information describing the presentation. Metadata does not include the video and / or audio data itself. Files other than the file containing the metadata do not need to be formatted according to ISOBMFF and only need to be formatted such that they can be referenced by the metadata.

[0063] The ISO Basic Media File (IBML) file structure is object-oriented, and the structure of individual objects within the file can be directly inferred from their type. Objects in an IBMFF file are referred to as "boxes" by the ISOBMFF specification. An IBMFF file is structured as a series of boxes, which can contain other boxes. A box is a fundamental syntactic structure in ISOBMFF, including the four-character encoded box type, the box's byte count, and the payload. A box may include a header that provides its size and type. The size describes the entire size of the box, including the header, fields, and all boxes contained within it. Boxes with types not recognized by the player device are typically ignored and skipped.

[0064] An ISOBMFF file can contain different types of boxes. For example, a movie box (“moov”) contains metadata for the continuous media streams present in the file, where each media stream is represented as a track in the file. The track's metadata is encapsulated in a track box (“trak”), while the track's media content is either encapsulated in a media data box (“mdat”) or directly in a separate file. Different types of tracks are possible. For example, ISOBMFF specifies the following types of tracks: media tracks, which contain primary media streams; cue tracks, which include media delivery indications or representations of received packet streams; and timing metadata tracks, which contain time-synchronized metadata.

[0065] The media content of a track comprises a sequence of samples known as media samples, such as audio or video access units or frames. These media samples differ from the image samples described above, where image samples are individual color components of pixels. As used herein, the term "media sample" refers to media data (audio or video) associated with a single time (e.g., a single point in time, a time range, or other time). The metadata for each track includes a list of sample description entries, where each entry provides the codec or container format used in the track and the initialization data required to process that format. Each sample is associated with one of the sample description entries for the track.

[0066] ISOBMFF enables the use of sample-specific metadata using various mechanisms. Specific boxes within the sample table box (“stbl”) have been standardized to respond to common needs. For example, the synchronization sample box (“stss”) can be used to list randomly accessed samples for a track. The sample grouping mechanism maps samples to groups sharing the same trait based on a four-character grouping type, specified as a sample group description entry in a file. Several grouping types have been specified in ISOBMFF.

[0067] Figure 1 This is a diagram illustrating an example of a file 100 having a file structure conforming to the ISO Basic Media File Format. The ISO Basic Media File 100 may also be referred to as a media format file. Media presentations may, but are not always, contained within a single file; in this case, the media presentation is self-contained within the file. File 100 includes a movie container 102 (or “movie box”). The movie container 102 may contain metadata about the media, which may include, for example, one or more video tracks and one or more audio tracks. For example, video track 110 may contain information about various layers of the video, which may be stored in one or more media information containers 114. For example, media information container 114 may include a sample table that provides information about video samples of the video. In various embodiments, video data chunks 122 and audio data chunks 124 are contained within media data container 104. In some embodiments, video data chunks 122 and audio data chunks 124 may be contained within one or more other files (in addition to file 100).

[0068] In various implementations, the presentation (e.g., motion sequences) can be contained in several files. All timing and frame control (e.g., position and size) information can be in the ISO base media file, and auxiliary files can be in virtually any format.

[0069] ISO files have logical, temporal, and physical structures. These different structures are not required to be coupled. The file's logical structure is the logical structure of the film, which in turn contains a set of time-parallel tracks (e.g., video track 110). The file's temporal structure consists of tracks containing temporally sampled sequences, and these sequences are mapped to the overall film's timeline via an optional edit list.

[0070] The physical structure of the file separates the data required for logical, temporal, and structural deconstruction from the media data sample itself. This structural information is centralized in movie boxes (e.g., movie container 102), which may be extended temporally through movie clip boxes. Movie boxes record the logical and temporal relationships of the samples and also contain pointers to sample locations. These pointers can point to the same file or another file, which can be referenced by, for example, a Uniform Resource Locator (URL).

[0071] Each media stream is contained in a track dedicated to that media type. For example, in Figure 1 In the example illustrated, movie container 102 includes video track 110 and audio track 116. Movie container 102 may also include a cue track 118, which may include send instructions from video track 110 and / or audio track 116, or other information that may represent other tracks in movie container 102 or other movie containers (not shown) of file 100. Each track can be further parameterized by sample entries. For example, in the illustrated example, video track 110 includes media information container 114, which includes a table of samples (referred to as a “sample table”). Sample entries contain a “name” of the exact media type (e.g., the type of decoder required to decode the stream) and any parameter settings required by that decoder. The name may be in the form of a four-character code (e.g., moov, trak, or other suitable name code). Various media types have defined sample entry formats. Sample entries may also include pointers to video data blocks (e.g., video data block 122) in box 120 of media data container 104. Box 120 includes interwoven, time-ordered video samples (organized into video data blocks, such as video data block 122), audio frames (e.g., in audio data block 124), and cue instructions (e.g., in cue instruction block 126).

[0072] Support for metadata can take different forms. In one example, timing metadata can be stored in an appropriate track and synchronized with the media data described by the metadata as expected. In a second example, there is non-timing metadata that is typically supported for attachment to a movie or a separate track. Structure support is generic and, as with media data, allows metadata resources to be stored elsewhere in the file or in another file.

[0073] In some cases, a single track in a video file can contain multiple layers. A video track can also include a track header (e.g., track header 112), which may contain information about the contents of the video track (e.g., video track 110). For example, the track header may include a track content information (also referred to as a "tcon") box. The tcon box can list all the layers and sublayers in the video track. The video file may also include an operation point information box (also referred to as an "oinf" box). The oinf box records information about the operation point, such as the layers and sublayers that make up the operation point, dependencies between operation points (if any), the operation point's profile, level and layer parameters, and other such operation point-related information. In some cases, an operation point may also be called a run point.

[0074] Figure 2 This diagram illustrates another example of an ISO Basic Media File 200 formatted according to ISOBMFF. The ISO Basic Media File 200 can also be referred to as a media format file. The ISO Basic Media File 200 contains data and metadata used for video rendering. At the highest level of file 200, there are file type boxes 210, movie boxes 220, and one or more segments 230a, 230b, 230c to 230n. Other boxes that can be included at this level but are not shown in this example include free space boxes, metadata boxes, and media data boxes, etc.

[0075] File type box 210 is identified by the box type "ftyp". File type box 210 is typically placed as early as possible within the ISO basic media file 200. File type box 210 identifies the ISOBMFF specification most suitable for parsing the file. In this case, "most suitable" means that the ISO basic media file 200 may have been formatted according to a specific ISOBMFF specification, but is likely compatible with other iterations of that specification. This most suitable specification is called the brand name. Player devices can use the brand name to determine whether the device can decode and display the file's content. File type box 210 may also include a version number, which can be used to indicate the version of the ISOBMFF specification. File type box 210 may also include a list of compatible brand names, including a list of other brand names that the file is compatible with. ISO basic media files can be compatible with multiple brand names.

[0076] When an ISO basic media file includes a file type box (such as ISO basic media file 200), there is only one file type box. In some cases, the ISO basic media file may omit the file type box for compatibility with older player devices. When an ISO basic media file does not include a file type box, the player device may use a default primary version name (e.g., mp41), a secondary version name (e.g., "0"), and a compatible version name (e.g., mp41, isom, iso2, avc1, etc.).

[0077] The ISO Basic Media File 200 also includes a movie box 220, which contains metadata for rendering. The movie box 220 is identified by the box type "moov". ISO / IEC 14496-12 specifies that a rendering can only include one movie box 220, whether included in one file or multiple files. Often, the movie box is placed near the beginning of the ISO Basic Media File (e.g., as indicated by the placement of the movie box 220 in the ISO Basic Media File 200). The movie box 220 includes a movie header box 222 and may include one or more track boxes 224, as well as other boxes.

[0078] The movie header box 222, identified by the box type "mvhd", may include information that is independent of the media and related to the overall presentation. For example, the movie header box 222 may include information such as creation time, correction time, time scale, and / or presentation duration. The movie header box 222 may also include an identifier that identifies the next track in the presentation. For example, in the illustrated example, the identifier may point to the track box 224 contained by the movie box 220.

[0079] Track box 224, identified by box type "trak", can contain information for the track used in presentation. Presentation can include one or more tracks, each independent of other tracks in the presentation. Each track can include temporal and spatial information specific to the content within the track, and each track can be associated with a media box. The data in the track can be media data, in which case the track is a media track, or the data can be packetization information for a streaming protocol, in which case the track is a cue track. Media data includes, for example, video and audio data. Figure 2 In the example shown, the example track box 224 includes a track header box 224a and a media box 224b. Track boxes may include other boxes, such as track reference boxes, track group boxes, edit boxes, user data boxes, meta boxes, etc.

[0080] The track header box 224a, identified by the box type "tkhd", can specify the characteristics of the track contained within the track box 224. For example, the track header box 224a can include the track's creation time, correction time, duration, track identifier, layer identifier, group identifier, volume, width, and / or height. For media tracks, the track header box 224a can also identify whether the track is enabled, whether the track should be played as part of a presentation, whether the track can be used for a preview presentation, and other uses of the track. Track presentation is typically assumed to occur at the beginning of the presentation. The track box 224 can include an edit list box (not shown) that can include an explicit timeline map. Among other things, the timeline map can specify the track's offset time, where the offset indicates the track's start time after the beginning of the presentation.

[0081] In the illustrated example, track box 224 also includes a media box 224b identified by the box type "mdia". Media box 224b can contain objects and information about the media data in the track. For example, media box 224b can contain a handler reference box, which can identify the media type of the track and the process by which the media in the track is rendered. As another example, media box 224b can contain a media information box, which can specify the characteristics of the media in the track. The media information box can also include a sample table, as described above regarding... Figure 1 As described, each sample describes a chunk of media data (e.g., video or audio data), including, for example, the location of the data for that sample. The sample data is stored in a media data box, which will be discussed further below. Like most other boxes, media box 224b may also include a media header box.

[0082] like Figure 2As shown, the example ISO base media file 200 also includes multiple segments 230a, 230b, 230c through 230n to be presented. Segments may also be referred to as movie segments. Segments (e.g., in some cases may include Common Media Application Format (CMAF) chunks) can be presented in a temporally extended manner. In some examples, a segment may provide information that may already be included in a movie box (“moov”). A movie segment (or CMAF chunk) may include at least one movie segment box (identified by the box type “moof”), followed by a media data box (identified by the box type “mdat”). For example, segment 230a may include a movie segment (moof) box 232 and a media data (mdat) box 238, and may be presented in an extended manner by including additional information that would otherwise be stored in movie box 220. Clips 230a, 230b, 230c through 230n are not ISOBMFF boxes, but rather describe movie clip boxes and media data boxes referenced by movie clip boxes (e.g., movie clip box 232 and media data box 238 are referenced by movie clip box 232). Movie clip box 232 and media data box 238 are the highest-level boxes, but are grouped here to indicate the relationship between movie clip box 232 and media data box 238. Using movie clip boxes (e.g., movie clip box 232), the rendering can be built incrementally.

[0083] In some examples, movie clip box 232 may include movie clip header box 234 and track clip box 236, as well as other boxes not shown here. Movie clip header box 234, identified by box type "mfhd", may include a serial number. The player device can use the serial number to verify that clip 230a contains the next piece of data for rendering. In some cases, the content of a file or the file used for rendering may be provided to the player device out of order. For example, network packets may frequently arrive in a different order than the original order in which they were sent. In these cases, the serial number can assist the player device in determining the correct order of the clips.

[0084] Movie clip box 232 may also include one or more track clip boxes 236, identified by the box type "traf". Movie clip box 232 may include a set of track clips, zero or more for each track. Track clips may contain zero or more track runs, each track run describing a continuous sample run of a track. In addition to adding samples to a track, track clips can also be used to add blank time to a track.

[0085] Media data box 238, identified by the box type "mdat", contains media data. Within a video track, media data box 238 may contain video frames, access units, NAL units, or other forms of video data. Media data boxes may alternatively or additionally include audio data. A presentation may include zero or more media data boxes contained in one or more separate files. The media data is described by metadata. In the illustrated example, the media data in media data box 238 may be described by metadata in track segment box 236, which is included in movie segment box 232. In other examples, the media data in media data boxes may be described by metadata in movie box 220. Metadata can reference specific media data by absolute offsets within file 200, allowing media data headers and / or free space within media data box 238 to be skipped.

[0086] Other segments 230b, 230c to 230n in the ISO basic media file 200 may contain boxes similar to those illustrated for the first segment 230a, and / or may contain other boxes.

[0087] As mentioned above, ISOBMFF supports not only local playback of media but also streaming media data over a network. One or more files comprising a movie presentation may include additional tracks called cue tracks, which contain instructions that assist the streaming server in forming and sending one or more files as packets. These instructions may include, for example, data the server wants to send (e.g., header information) or references to media data segments. Segments may include a portion of an ISO BMFF file, including movie boxes and associated media data and other boxes (if present). Segments may also include a portion of an ISO BMFF file, including one or more movie clip boxes and associated media data and other boxes (if present). Files may include separate cue tracks for different streaming protocols. Cue tracks can also be added to files without reformatting the file.

[0088] One method for streaming media data is dynamic adaptive streaming via Hypertext Transfer Protocol (HTTP) or DASH (defined in ISO / IEC 23009-1:2014). DASH, also known as MPEG-DASH, is an adaptive bitrate streaming technology that uses a traditional HTTP web server to achieve high-quality streaming of media content. DASH operates by breaking media content into a sequence of small HTTP-based file segments, where each segment contains content at a short time interval. Using DASH, the server can serve media content at different bitrates. The client device playing the media can choose from alternative bitrates as it downloads the next segment, adapting to changing network conditions. DASH uses the Internet's HTTP web server infrastructure to deliver content over the World Wide Web. DASH is independent of the codecs used to encode and decode media content and therefore works with codecs such as H.264 and HEVC.

[0089] Additionally, scene description data can be leveraged to provide support for immersive media presentation and experiences, drawing upon, but not limited to, MPEG media and media objects. In some examples, scene description data can be based on glTF 2.0 (or other standards and formats), and specific sets of extensions can be provided to address identified gaps. glTF is a file format (e.g., for 3D scenes) and can be modeled using the JavaScript Object Notation (JSON) standard. For example, glTF is an application programming interface (API)-neutral runtime asset delivery format.

[0090] In some examples, 3D scenes can be described by scene description data. Scene description data can be used to process and render 3D scenes. Scene description data can similarly be used to describe (and subsequently process and render) immersive media scenes, presentations, and experiences. Scene description data can be included in files, documents, syntax, or included with media data, or as standalone information (without any media data). For example, a scene description document (or file) can include scene description data. In some examples, scene description data (e.g., scene description data in a scene description document) can describe 3D, immersive, and / or other types of scenes with a hierarchical node structure (e.g., scene description data includes descriptions of node hierarchies, materials, cameras, and descriptive information about the meshes, animations, and other constructs of media objects used to process and render the 3D scene).

[0091] The node structure of the aforementioned scene description data can also be considered as a directed acyclic graph (e.g., a tree structure) that uses an object-based hierarchical structure to represent the geometry of the scene (e.g., 3D, immersive, or otherwise). For example, leaf nodes of a directed acyclic graph can represent geometric primitives such as polygons. Each node in the graph can hold pointers to its child nodes. Child nodes can include, but are not limited to, individual nodes, one or more groups of other nodes, geometric elements, transformation matrices, etc. Spatial transformations can be attached to certain nodes of the graph (e.g., by attaching transformation matrices corresponding to the desired spatial transformation). Using such a scene description data structure, scene description data can be used to provide geometric and spatial information for processing and rendering 3D scenes.

[0092] In some examples, the scene description data provides a binary data representation of the aforementioned directed acyclic graph structure (also known as the "scene graph"). Using the representation contained within the scene description data, the scene graph logic can be utilized by the rendering engine responsible for rendering the 3D scene described by the scene description data. The rendering engine can reduce its processing complexity by traversing the scene graph during rendering. For example, if it is determined that the space of a parent node is not visible or is irrelevant to the rendering of the current viewing frustum (e.g., the region of the displayed modeled space), a culling operation can be performed, where branches of the scene graph are discarded from processing without having to be fully traversed.

[0093] The glTF 2.0 scene description provides a compact and low-level representation of the scene graph and simplifies processing by providing a flat, hierarchical JSON representation of the scene graph. The glTF 2.0 scene description is designed to eliminate redundancy and provide efficient indexing of the different objects contained within the scene graph. Furthermore, the scene description can include information for accessing or retrieving various media objects, files, and other media used to render the 3D scene.

[0094] As described above, this article describes systems and technologies that provide an improved media access architecture. Figure 3This is a diagram illustrating an example of a media system 300. As shown, the media system 300 includes a rendering engine 340 and a Media Access Function (MAF) 320. The rendering engine 340 processes and renders assets, which may include, for example, 3D scenes, immersive scenes, 2D scenes, and / or other types of scenes that include media content. In some examples, the rendering engine 340 performs processing and rendering tasks for the media scene described by a scene description document 348 that includes scene description data. The MAF 320 can retrieve and prepare media and / or media objects for rendering. For example, the MAF 320 may receive one or more requests from the rendering engine 340 and may retrieve and prepare media and / or media objects for rendering in response to receiving one or more requests. The media objects retrieved by the MAF 320 may include media objects containing the media scene described by the scene description document 348. As illustrated, the rendering engine 340 can interact and communicate with the MAF 320 via the MAF application programming interface (API) 325 located between the rendering engine 340 and the MAF 320. The media system 300 also includes a buffer API 335, which provides an interface to the buffer management 330 for the rendering engine 340 and MAF 320.

[0095] As will be described in more detail below, MAF API 325 and Buffer API 335 can be used to generate one or more media pipelines 372, 374 that decouple the rendering task of rendering engine 340 from the media fetching task of MAF 320. In an illustrative example, rendering engine 340 parses scene description document 348, identifies media information of media objects required to render a specific media scene, and requests each media object from MAF 320 by passing the media information to MAF API 325 in one or more calls (e.g., one or more API calls). MAF 320 can then retrieve each media object from the source identified in the MAF API call (e.g., a network source such as network source 312, a local source such as local source 314, and / or other sources). For example, MAF 320 can send one or more media requests to a network source, and the network source can respond to one or more requests with one or more media objects. In another example, MAF 320 can retrieve one or more media objects from local source 314.

[0096] Once one or more media objects are received, MAF 320 can use one or more media pipes (e.g., media pipes 372 and 374) to process and perform format conversions (where appropriate) on the one or more media objects. Media pipes 372 and 374 can be initialized, instantiated, or allocated by MAF 320. For example, MAF 320 can determine one or more parameters of media pipes 372 and 374 based on MAFAPI calls and / or calls to buffer API 335. Media pipes 372 and 374 terminate at multiple buffers 332a, 332b to 332n. Media pipes 372 and 374 deliver the processed media objects to each of buffers 332a, 332b to 332n in the expected format requested by rendering engine 340 in the MAFAPI calls. Buffers 332a, 332b to 332n can be any suitable type of buffer, such as a cyclic buffer, a static buffer, or some combination thereof (for example, buffer 332a can be a buffer of a different type than buffer 332b, or it can be a buffer of the same type).

[0097] In some examples, data exchange (e.g., media objects and / or metadata) between MAF 320 and rendering engine 340 does not occur directly. For instance, data exchange between MAF 320 and rendering engine 340 can instead be performed via buffers 332a-n. Each buffer 332a-n may contain header information describing the content stored within each buffer 332a-n (e.g., the processed media objects(s) it contains) and the timing associated with each buffer 332a-n. Figure 3 As illustrated, the functionality of buffer management 330 is exposed to MAF 320 and rendering engine 340 via buffer API 335.

[0098] By utilizing media pipelines 372, 374 and buffers 332a-n, the media rendering task performed by the rendering engine 340 can be decoupled from the media fetching task performed by the MAF 320. Although media objects can be stored in various locations, file formats, containers, etc., the rendering engine 340 can abstract these complexities into standardized MAF API calls. Furthermore, the rendering engine 340 can consistently locate the requested media object stored in buffers 332a-n in the requested format and at the requested time.

[0099] Scene description document 348 may include media information used to parameterize the aforementioned MAF API calls (e.g., those made by rendering engine 340 to MAF API 325). Scene description document 348 may also include buffer information used to parameterize buffer API calls (e.g., those made by MAF 320, rendering engine 340, or both to buffer API 335). In some examples, buffer information is included in the MAF API call. For example, based on a received MAF API call that includes buffer information, MAF 320 may use the buffer information to invoke buffer API 335 itself.

[0100] In an illustrative example, rendering engine 340 receives and parses scene description document 348 (and any subsequent scene description updates) and identifies external media or other media objects that need to be rendered. For each media object, rendering engine 340 may determine or generate media information based on information or data obtained from parsing scene description document 348. When rendering engine 340 requests each media object, it may include the media information in an API call to MAF API 325. In some cases, media information may include timing information to identify the rendering time of each media object. The rendering time may be the time when the media object needs to be rendered in the rendered scene, or it may be an earlier time when the media object needs to be delivered to one or more buffers 332a-n (e.g., before being rendered).

[0101] In some examples, the media information obtained from parsing the scenario description document 348 may include one or more media source locations (e.g., media source locations may be provided as Uniform Resource Locators (URLs), Uniform Resource Identifiers (URIs), etc.) instructing the MAF 320 where the requested media object can be found. In some cases, one or more media source locations may include a primary or default media source location. In some examples, a primary or default media source location may be provided along with one or more alternative media source locations, such that if the MAF 320 fails to obtain the requested media object from either the primary media source location or any of the previously attempted alternative media source locations, the MAF 320 can fall back to an alternative media source location. In some examples, the MAF 320 may perform a selection among different media source locations. For example, the MAF 320 may select one or more source locations from different media source locations based on user preferences and / or network characteristics such as bandwidth and latency (which indicate whether a particular media source location is the most preferred among different media source locations).

[0102] In some examples, the media information in the MAF API call may specify certain portions of the media object to be retrieved by the MAF 320, and / or may specify the level of detail that the rendering engine 340 expects the media object to be delivered. In one illustrative example, the media object may be provided as a point cloud representing a model of a person. In such an example, the MAF API call may specify that the MAF 320 retrieves only the portion of the point cloud corresponding to the face of the person. In another illustrative example, the media object may include a high-resolution texture, and the MAF API call may request a version of the texture with a resolution lower than the native high-resolution texture.

[0103] In some examples, the media information in the MAF API call can specify how the rendering engine 340 expects the MAF 320 to deliver the processed media object to one or more of the buffers 332a-n. For example, the media information may include the requested or expected format of the processed media object, one or more buffer identifiers for delivering the processed media object, instructions for one or more processing steps that the MAF 320 should apply to generate the processed media object, etc. The format of the media object obtained from the media source location may differ from the expected format requested by the rendering engine 340. For example, the media source location may provide the media object as a compressed video stream in YUV420, and the rendering engine 340 may request the media object to be provided in the buffer as interleaved RGBA color data. In some examples, the media source location may provide the media object as a compressed mesh stream with Draco, and the rendering engine 340 may request the media object to be provided in a vertex index list buffer, an XYZ position buffer (X corresponds to the horizontal direction, Y to the vertical direction, and Z to the depth direction), and a texture coordinate buffer. In some cases, media source location can provide media objects as compressed point clouds with MPEG V-PCC, and rendering engine 340 can request media objects as interleaved XYZRGBA buffers.

[0104] The rendering engine 340 can call the MAF API 325 individually for each requested media object (and in some cases, the corresponding media and buffer information for that requested media object), and / or can call the MAF API 325 for multiple media objects requested at once. Based on the media information and any other parameters included in the MAF API call, the MAF 320 can initialize one or more of the media fetching and decoding pipelines 372, 374 as necessary. In some examples, the MAF 320 can use timing information provided by the rendering engine 340 in the MAF API call to initialize or allocate (one or more) media pipelines 372, 374 at an appropriate time (e.g., shortly before the media pipelines are needed).

[0105] Based on MAF API calls and media information, MAF 320 can ensure that each requested media object is available in the appropriate buffers 332a-n at the appropriate time for the rendering engine 340 to access. In some cases, before delivering the requested media object, MAF 320 can use the media information to decode, process, and / or reformat each media element to match the format requested by the rendering engine 340, as previously described.

[0106] In some examples, the scene description document 348 contains all media and buffer information that can be passed in calls to the MAF API 325 and Buffer API 335. In some cases, the rendering engine 340 may obtain or determine a portion of the media information from sources other than the scene description document 348, which may include lookup tables, internal or external databases, stored user preferences or settings, etc.

[0107] In some examples, media and / or buffer information may be explicitly or directly represented in the scene description document 348 (e.g., as stored values, variables, relations, etc.). As previously described, the rendering engine 340 may parse the scene description document 348 to obtain media and buffer information. In some examples, the rendering engine 340 may parse the scene description document 348 and store the determined media and / or buffer information before making a later API call to the MAF API 325, Buffer API 335, or both. For example, the rendering engine 340 may parse the scene description document 348 upon initial receipt and may subsequently use the determined media and / or buffer information when an API call is required.

[0108] In some cases, the rendering engine 340 can perform the initial parsing of the scene description document 348 and can send multiple (e.g., all in some cases) MAF API and buffer API calls. In this case, the media information and / or buffer information may include timing parameters specifying the time each media object must be provided in one of the buffers 332a-n, and the MAF 320 can use these timing parameters to perform its own scheduling(s)(s).

[0109] In some implementations, rendering engine 340, MAF 320, or both rendering engine 340 and MAF 320 may be included in a media playback device configured to render and / or display a media scene (e.g., integrated into a mobile device, personal computer, laptop computer, extended reality (XR) device (e.g., virtual reality (VR) device, augmented reality (AR) device, or mixed reality (MR) device), set-top box, television, game console, smart wearable device, or other media playback device, etc.). In some cases, one or more of rendering engine 340 and MAF 320 may be provided remotely (e.g., via the Internet or wired or wireless networks). For example, rendering engine 340 may be located locally on the media playback device, while MAF 320 may be located remotely. In such an example, rendering engine 340 may send an MAF API 325 call to MAF 320 via the Internet, and MAF 320 may deliver (e.g., stream) a requested media object to rendering engine 340 via the Internet. Similarly, one or more buffers 322a-n may be located locally (e.g., in a media playback device) and / or remotely from the rendering engine 340 and / or the MAF 320. Continuing in the context of the above example, a remote MAF may deliver processed media objects to a local buffer (e.g., in local storage or memory) associated with the rendering engine 340, and / or a remote MAF may deliver processed media objects to a remote buffer (e.g., in the cloud or on a networked server), which may then deliver (e.g., stream) the processed media objects to the rendering engine 340.

[0110] The use of media pipelines 372 and 374 can provide flexibility and interoperability for the selection and positioning of MAF 320 and rendering engine 340. For example, prior compatibility between different rendering engines and MAF may have been limited by design constraints or choices (e.g., a rendering engine designed to accept media objects or buffers of certain file formats; an MAF designed to output media objects with different sets of file formats or different buffer configurations). The media pipelines described herein can be utilized to interconnect otherwise incompatible instances of MAF 320 and rendering engine 340. About Figure 5 and Figure 6 It provides a more detailed discussion of the media pipeline and how it transforms media objects from the delivery format provided by the media source into the expected format of the rendering engine.

[0111] Back Figure 3 Note that although two media pipes 372 and 374 are shown, in some cases, more media pipes may be provided, or only one media pipe may be provided. For example, in some cases, one or more media pipes may be initialized for each media object requested in the rendering engine 340's call to the MAF API 325. In such examples, there may be at least a 1:1 correspondence between the number of media objects requested by the rendering engine 340 and the number of media pipes 372 and 374 created to handle these requests. In some examples, the number of media pipes may vary at any given time, because in some examples, inactive or idle media pipes may be removed (e.g., to free up system resources). In some cases, the removal of inactive pipes may be performed immediately after the media pipes have finished outputting the processed media objects to buffers 332a-n, after a specified or predetermined amount of idle time, and / or as required by resource management strategies. In some examples, the correspondence between the number of media pipes and the number of requested media objects may be less than 1:1. For example, instead of creating media pipelines to handle each individual request for a media object, it's possible to create media pipelines to handle different request types. In an illustrative example, a given media pipeline can handle all media object requests that have the same delivery format, intermediate processing steps, and expected format in the output buffer. In some cases, initializing, allocating, or otherwise managing media pipelines can combine both of these approaches. In an illustrative example, a dedicated (or more permanent) media pipeline can be used to handle common or recurring media object requests, and dynamically created (or more temporary) media pipelines can be used for the remainder of media object requests.

[0112] although Figure 3The diagram illustrates buffers 332a-n, but in some cases, more or fewer buffers may be used. For example, in a 1:1 correspondence, the number of buffers may equal the number of media pipes, where each media pipe is associated with a given output buffer. In some examples, a single media pipe may output processed media elements to multiple different buffers, and / or a single buffer may receive processed media elements from multiple different media pipes, in which case a 1:1 correspondence may not exist. The relationship between a given pipe and its associated output buffer(s) can be determined from MAF API calls. For example, the rendering engine may parse the scene description document 348 and determine the buffer information for each media object. In response to some or all of the MAF API calls that include this buffer information, MAF 320 may initialize and / or allocate the necessary buffers for each media object. For example, MAF API calls may instruct MAF 320 on the location, identity, and / or certain parameters of the buffer in which the rendering engine 340 expects to find the processed media object. Regarding buffer information, note that buffer information may be determined simultaneously with media information or determined separately from media information.

[0113] Figure 4 This is another example diagram depicting a media system 400, including examples of different types of output buffers 432a-e that can be used for data exchange between the MAF 420 and the rendering engine 440. In some cases, Figure 4 The media system 400 shown in the diagram can be related to the above-mentioned media system 400. Figure 3 The media system 300 described is the same as or similar to the media system 400, and / or one or more components of the media system 300 may be the same as or similar to one or more corresponding components of the media system 300. As illustrated, output buffers 432a-e may include, but are not limited to, texture buffer 432a, color buffer 432b, audio buffer 432c, index buffer 432d, and metadata buffer 432e. One or more buffers 432a-e may be cyclic buffers or static buffers.

[0114] As illustrated, each of buffers 432a-e corresponds to a different type of media or media object requested or utilized by the rendering engine 440 when rendering the media scene. Each buffer is also shown as receiving the output of the corresponding decoder 482. Although Figure 4 Media channels (such as) not described Figure 3The media pipelines 372 and 374 are used, but as described above, the decoder 482 may be included in or communicate with one or more media pipelines to process media objects received at one or more media pipelines. In some examples, the MAF 420 may analyze each MAFAPI request to determine the media type of each requested media object. Based on the determined media type, the MAF 420 may route each requested media object to one or more appropriate decoders 482 for processing the determined media type. Subsequently, each decoder to which one or more media objects are routed may output the processed media object to one or more appropriate buffers 432a-e for the determined media type. For example, a texture media object may be output to a texture buffer 432a, while a color media object may be output to a color buffer 432b. In some examples, the delivery of media objects to output buffers 432a-e of a specific type can be specified by the rendering engine 440 (e.g., the rendering engine 440 constructs MAF API calls that specify the same output buffer for each requested media object of a given type, such as audio buffer 432c for audio media objects).

[0115] In the illustrative example, after receiving an MAF API request from rendering engine 440, MAF 420 retrieves one or more media objects specified by the media information in the MAF API request. At least some of the requested media objects can be retrieved from local storage 414, where a specific path or location within local storage 414 is determined from the media information. At least some of the requested media objects can be retrieved from a remote source 412, which... Figure 4 The image is shown as cloud storage. In some examples, when retrieving one or more media objects from a remote source 412, the MAF 420 may receive a URL and / or URI from the media information, which indicates the source from which one or more media objects can be retrieved.

[0116] In some examples, the MAF 420 may use one or more protocol plugins 423 to communicate with a remote source 412 identified in media information requested from the MAF API (e.g., using a protocol provided by protocol plugin 423). One or more protocol plugins may be integrated with the MAF 420 and may be downloaded or acquired, or a combination of both, in response to determining that the MAF 420 requires protocol plugin 423 in order to reach the remote source and acquire the requested media object.

[0117] In some cases, the MAF 420 may use one or more format plugins 427 to convert between different delivery formats initially provided by local storage 414, remote source 412, or other sources, and different expected formats requested by the rendering engine 440 (e.g., included in media information in MAF API calls). Similar to protocol plugin 423, one or more format plugins 427 may be integrated with the MAF 420 and may be downloaded or acquired in response to determining that the MAF 420 requires format plugin 427 to process or convert the requested media object, or a combination of both.

[0118] In some examples, MAF 420 itself can be implemented using a plugin of rendering engine 440 (e.g., as illustrated in MAF plugin 443). In some examples, MAF 420 can be provided as a separate component of media system 400 (in which case MAF 420 is not implemented using a plugin). MAF plugin 443 can provide rendering engine 440 with the ability to utilize MAF API 325 and request media objects from MAF 325. In some examples, MAF plugin 443 can transform or convert media information (e.g., generated when rendering engine 440 parses scene description 448) into appropriately formatted or constructed MAF API calls. In some examples, MAF plugin 443 can intercept media object requests generated by rendering engine 440 through a first structure and can transform (e.g., automatically transform) requests from the first structure into the structure of MAF API calls (e.g., MAF plugin 443 can be used to transform existing requests into MAF API calls), thereby providing compatibility with rendering engines that were not previously decoupled from media fetching.

[0119] Figure 5This is a block diagram of different examples and configurations of media pipeline sets 572-578, where in some examples these media pipelines may be the same as or similar to one or more media pipelines discussed above and elsewhere. As shown, one or more media tracks 522 may include media data, and in some cases include other data (e.g., metadata, etc.). For example, media track 522 may include a complete media object (such as a 3D asset). In some cases, media track 522 may include one or more components of a media object (e.g., media components such as color components (red, green, blue, chroma, luminance, etc.), depth, vertex position, polygon, texture, etc.). The media data and / or other data are output to one or more decoders, which may perform one or more decoding operations 582 to decode the media data and / or other data. The decoded data may be output for processing by one or more processing operations 584. The processed data may then be output to one or more output buffers 532. In some examples, one output buffer (e.g., one of multiple output buffers 532) may receive processed data from multiple media pipelines. In some cases, the output buffer receives processed data from one or more media pipes, but the output buffer itself is not included in one or more media pipes. In some examples, the scene description obtained by the rendering engine 540 can describe the properties and / or parameters of all five media pipes 572-578. The following is about... Figure 10 and Figure 11 Examples of decoding operations 582 (one or more) are described below. Processing operations 584 may convert formats and / or create new static and / or dynamic buffers. For example, processing operations 584 (one or more) may create an index buffer based on metadata received from a media object (included in media track 522) or media track 522 itself.

[0120] In some cases, each media pipeline may perform one or more processing steps on the input media track 522. The processing operation(s) 584 may include, but are not limited to, streaming, demultiplexing, decoding, decryption, format conversion, any combination thereof, and / or other processing functions. For example, format conversion may be performed to match the expected format for outputting the processed media track to the buffer 532. As previously described, the buffer 532 may be used to exchange data with the rendering engine 540.

[0121] Although not shown, media access functions (e.g., such as MAF 320) can perform the setup and management (e.g., initialization) of media pipelines 572-578. Media pipelines 572-578 can be constructed based on information determined from the scene description (e.g., media information obtained by the rendering engine 540 through parsing the scene description and provided to the MAF via MAF API calls, as previously described). In some cases, the scene description can provide support for pipelines or media pipelines (e.g., the processing model of the MPEG-1 scene description can provide support for the pipeline concept).

[0122] In one example, pipe 572 is illustrated as receiving a single track (track #1) as input, applying decoding and processing operations, and then providing the processed media object in buffer #1 to rendering engine 540. If the input track contains only media components (e.g., not a complete media object), in some cases, pipe 572 may provide the processed media components in buffer #1 to the rendering engine. In some examples, pipe 572 may contain multiple instances of elements and components that appear only once in pipe 572.

[0123] In another example, pipe 574 receives two tracks (track #2 and track #3) as input (each track may include one or more media objects and / or media components). In such an example, the two tracks are decoded separately (e.g., because they may be acquired in different formats or encodings). Processing operations are applied to track #2 before it is output to buffer #2. Track #3 is also output to buffer #2, but no processing operations are applied. In some examples, the media pipe can be configured to provide multiple tracks to the same output buffer because the requested media object is stored in multiple tracks. For example, when the requested media object includes multiple different media components, the requested media object may be stored in multiple tracks. In some examples, the media pipe may provide multiple tracks to the same output buffer based on a request from rendering engine 540 (e.g., rendering engine 540 expects the requested media object to be multiple tracks, and rendering engine 540 may then combine them into a single requested media object). As a specific example, the first track (e.g., track #2) may include color information, and the second track (e.g., track #3) may include 3D position or depth information. Each distinct track type can be output to a separate decoder within the media pipeline and processed by that decoder. The media pipeline can then combine or mix the tracks into a single output buffer, as shown in pipe 574. In addition to separate decoding, each track can also undergo separate processing (or some tracks can undergo no processing at all). For example, the 3D position / depth information in a track (e.g., track #3) can be processed to convert to an XYZ coordinate format (where X corresponds to the horizontal direction, Y to the vertical direction, and Z to the depth direction), while the color information from a track (e.g., track #2) can be passed to the buffer without processing.

[0124] In the example of pipe 576, a single input track is received (e.g., track #4) and, after decoding and processing are performed, is provided to two separate output buffers (e.g., buffers #3 and #4). Specifically, the single input track is provided to one buffer in a decoded and processed form, and to the other buffer in a decoded but unprocessed form. A configuration such as pipe 576 can be used in examples where a single media track contains multiple (e.g., two in this case) media components, such as one or more color components (e.g., red, green, blue, chroma, luminance, etc.), depth information (e.g., for a 3D scene), vertex positions (e.g., for a 3D scene), polygons (e.g., for a 3D scene), textures, etc. For example, track #4 may include a single stream storing both color information (as a first media component) and depth information (as a second media component). If the rendering engine 540 expects to provide color and depth information in separate output buffers, the media pipeline 576 can configure the decoder #4 to separate the two distinct media components after decoding, configure the processing elements to process only one of the media components (e.g., color or depth information) as needed, and provide the media components to their own output buffers (e.g., buffer #3 and buffer #4). The method illustrated in pipeline 576 is the opposite of that illustrated in pipeline 574, and the discussion therein envisions an example of acquiring color and depth information in two separate tracks and then mixing them by the pipeline into a single output buffer.

[0125] In the example of pipe 578, two input tracks are received (tracks #5 and #6), subjected to the same decoding and processing operations, and provided to the same output buffer (buffer #5). In some examples, the two outputs (e.g., a processed media object and / or a processed media component) may be stored separately in the output buffer. In some examples, the two outputs may be mixed and stored together in the output buffer (e.g., as described above with respect to pipe 574; if both input tracks are media components, the processed media components may be combined into a media object).

[0126] Figure 6These are block diagrams of two example media pipelines 672 and 674, illustrating how the internal processing functions and buffer outputs of different example pipeline configurations can be constructed (e.g., initialized by MAF or otherwise configured) based on the specific structure of the media objects or tracks received as input and / or based on information determined from the scene description (e.g., media information obtained by the rendering engine based on parsing the scene description and included in the MAF via MAF API calls, as previously described). For example, the input to the first media pipeline 672 is shown as a single track 622. In some examples, track 622 may include multiple identical media objects, each of which is provided to the second media pipeline 674 in separate tracks 624a-e. For instance, the two media pipelines 672 and 674 may receive, process, and output the same set of media objects, but in different ways.

[0127] Because the second media pipe 674 receives a set of media objects in discrete tracks 624a-e, each of the discrete tracks can be directly provided to the corresponding decoder (e.g., HEVC decoders 682a-c and patch decoder 682d) and processing operations 684a, 684b, and 684c. However, the first media pipe 672 receives a set of media objects packaged or containerized into a single track 622. Therefore, the first media pipe 672 is illustrated as initially passing a single track 622 to a demultiplexer 688 (illustrated as demultiplexer 688). The demultiplexer 688 can separate the single track 622 into individual media objects that it comprises. Although not directly shown, the output of the demultiplexer 688 may include the same five media objects provided to the second media pipe 672 in the form of individual tracks 624a-e. The first media pipe 672 can then decode and / or process the individual media objects in the same manner as the second media pipe 674.

[0128] The second media conduit 674 is shown as outputting media objects contained in separate tracks 624a-d to corresponding separate output buffers 634a-d. In this example, the second media conduit 674 outputs the media objects in a structure reflecting the initial delivery structure of the media objects. For example, as Figure 6As shown, at least four media objects are received in tracks 624a-d, and each of the four output buffers 634a-d receives a specific processed media object. In one example, buffer 634a receives a geometry media object from geometry track 624a, buffer 634b receives a textured media object from texture track 624b, buffer 634c receives a occupied media object from occupied track 624c, and buffer 634d receives a patched media object from patch track 624d. Note that metadata from static metadata track 624e is used by processing steps 684a and 684b and is therefore not output to the buffers.

[0129] Although the first media pipeline 672 uses a demultiplexer 688 to generate individual representations of media objects for internal pipeline processing operations, it ultimately uses a 3D reconstruction operation 686 to integrate these individual media object representations into a single output of buffer 632. The output structure of the first media pipeline 672 reflects the initial delivery structure that initially provided the media objects. For example, a single track 622 is provided, and a single output buffer 632 receives the processed media objects.

[0130] In some examples, the initial delivery structure of media objects (or tracks comprising one or more media objects) received at the media pipeline may differ from the output structure of the media pipeline. For instance, a media pipeline may receive multiple inputs of media objects and output a single processed media object to a buffer. In some cases, a media pipeline may receive multiple inputs of media objects and output more than one but fewer processed media objects than the number of input media objects. Figure 6 In the context of this, in some examples, the second media pipe 674 may receive multiple tracks 624a-e as input and provide output to a single output buffer 632 (e.g., by utilizing a 3D reconstruction operation 686 prior to the pipe output stage).

[0131] In some examples, a media pipeline can receive a single input, which includes multiple media objects. From this single input, the media pipeline can then output multiple processed media objects to multiple output buffers. Figure 6 In some cases, the first media pipe 672 may receive a single track 622 and provide four output buffers 634a-d as shown for the second media pipe 674. In some cases, the media pipe may provide one output track to the output buffer. In some cases, the media pipe may provide multiple output tracks to the output buffer.

[0132] For example, Figure 6The diagram depicts processed media objects (e.g., media objects generated from geometry tracks 624a and texture tracks 624b received as input at the second media conduit 674) provided to separate output buffers 634a, b. However, in one illustrative example, both processed media objects may be provided to the same output buffer (e.g., one of the two output buffers 634a, b). In another illustrative example, a 3D reconstruction operation 686 of the first media conduit 674 may output multiple processed media objects to output buffer 632. Multiple processed media objects may be obtained prior to the 3D reconstruction operation 686 (e.g., the outputs of processing operations 684a-c(s) may be provided to output buffer 632 instead of the 3D reconstruction operation 686). In some cases, multiple processed media objects may be provided to a single output buffer 632 in a 1:1 correspondence or some other correspondence ratio, or to multiple output buffers.

[0133] like Figure 5 and Figure 6 As illustrated in the examples, media pipelines can be configured with different designs, architectures, input / output structures, etc. In some cases, media pipeline configuration and settings can be provided by the MAF API, as previously discussed. Figure 3 and Figure 4 The MAF API 325 is described. For example, one or more media pipeline settings or configuration parameters can be specified in MAF API calls generated by the rendering engine. In some cases, as previously described, one or more media pipeline settings or configuration parameters can be determined based on media information (and / or buffer information) obtained from the rendering engine that parses the scene description or scene graph.

[0134] In some examples, the MAF API (e.g., MAF API 325) may provide or offer one or more of the methods listed in Table 1 below:

[0135]

[0136] Table 1

[0137] In some examples, the rendering engine can provide media information as a parameter to the initialize() function. The media information may include information related to the retrieval of the requested media object, the decoding of the requested media object, one or more processing operations to be applied to the requested media object, the expected delivery format of the processed media object, and the expected or identified output buffer for receiving the processed media object output from the media pipeline.

[0138] In some cases, the rendering engine can use the `initialize()` function to initialize the media pipeline. In some examples, the rendering engine can use the `initialize()` function to trigger the creation and / or initialization of the media pipeline, where the rendering engine itself does not perform the creation or initialization of the media pipeline. In some cases, when the rendering engine does not create or initialize the media pipeline, the media pipeline can be created or initialized by MAF (e.g., based on or in response to the `initialize()` function used by the rendering engine). For example, as previously described, the rendering engine can use the MAF API and MAF API functions to interact with the MAF interface and send commands and / or requests to MAF.

[0139] Example Interface Definition Language (IDL) descriptions for media pipeline interfaces are provided in Table 2 below (example data type definitions are provided in Table 3):

[0140]

[0141] Table 2

[0142] Table 3 below provides examples of IDL data type definitions:

[0143]

[0144]

[0145]

[0146] Table 3

[0147] In some examples (e.g., Table 2), the “Pipeline” parameter can be used to provide a representation of the media pipeline. The “Pipeline” parameter can store the current pipeline state, “PipelineState,” and can maintain a tracking output buffer for the media pipeline. In some cases, PipelineState can be IDLE, READY, ACTIVE, or ERROR. The IDLE state indicates that the media pipeline has not yet been initialized. The READY state indicates that the media pipeline has been initialized and is ready to start fetching media. The ACTIVE state indicates that the media pipeline is actively fetching media. The ERROR state indicates that the media pipeline has encountered an error that stops media access.

[0148] In some examples (e.g., Tables 2 and 3), the “MediaInfo” parameter may include location information corresponding to the media object to be provided to the media pipeline. In some cases, MediaInfo may include information provided by the scene description. In some examples, MediaInfo may include information provided by the rendering engine (e.g., information obtained from parsing the scene description document by the rendering engine). In some examples, MediaInfo may include information provided by the MPEG_media extension. A name may be assigned to MediaInfo. In some cases, MediaInfo may provide one or more alternative locations for the MAF or media pipeline to access the media object (e.g., one or more alternative locations may be provided in an alternative array). An alternative location may contain one or more of a MIME type, track set, and URI, which can be used to access the requested media object from the alternative location. The track set may include an indication of the track to be accessed from the alternative location. In some examples, if MediaInfo does not include an indication of the track to be accessed, the media pipeline may access all tracks or components of the alternative location.

[0149] In some examples (e.g., Tables 2 and 3), the "TimeInfo" parameter can indicate the point in time when the media and / or media object will be accessed. TimeInfo can also indicate when media access begins. In some examples, the semantics of one or more TimeInfo fields can be those of one or more fields provided in the MPEG_media extension.

[0150] In some examples, the "BufferInfo" parameter can provide information about the expected format of the media pipeline's output buffer (e.g., the requested or expected format of the rendering engine making API calls to the MAF API). In some cases, BufferInfo may include information about one or more buffers and may describe the format of samples and frames stored in a given buffer. One or more tracks from MediaInfo can be fed into the same buffer. When multiple MediaInfo tracks are provided to the same buffer, the link between the track providing the media object to the buffer and the buffer that will store the processed media object (e.g., the output of the media pipeline from the buffer) can be indicated by the bufferId property. Buffers can be allocated by MAF and / or the rendering engine. Buffer handlers can be used to read or pass references to allocated buffers. If MAF allocates a buffer, read-only access may be allowed in some examples. In some cases, MAF may allocate a buffer if the media pipeline is initialized without a valid buffer handler. In some examples, buffer format information may include componentType, sampleType, offset, and / or stride information. In some cases, buffer format information can correspond to the componentType, Type, offset, and stride information in the corresponding accessor and bufferView. attributeType can correspond to primitive properties.

[0151] In some examples, the "ViewInfo" parameter can represent the current position of one or more objects rendered by the rendering engine using media objects. In other cases, the "ViewInfo" parameter can represent the current position of one or more viewer poses. ViewInfo can be used to adjust the visibility of media access objects. For example, distant objects can be accessed at a lower level of detail (LoD). In some examples, ViewInfo can contain viewer pose information (e.g., pose information including the viewer's position and orientation). ViewInfo can also contain the position and orientation of objects as transformations (e.g., a 4×4 matrix defined by glTF 2.0). In some cases, ViewInfo and associated information can use the same coordinate system as the scene.

[0152] MAF can use ViewInfo to optimize the streaming of requested media and / or media objects. For example, MAF can use ViewInfo to adjust the level of detail rendered by the rendering engine based on the distance and orientation to the viewer. In some cases, the viewer's distance and orientation can be calculated with respect to the device being used by the user. In some examples, level of detail adjustments can be made to one or more polygons, one or more points, one or more texture resolutions, and / or one or more other properties. The device being used by the viewer can include, but is not limited to, mobile devices, head-mounted displays, or other devices used for viewing virtual reality (VR), augmented reality (AR), and / or mixed reality (MR) content, and / or other devices.

[0153] MAF (such as MAF 320 as previously described) and / or rendering engines can use buffer APIs (such as buffer API 335 as previously described). For example, buffer APIs can be used to allocate and control one or more buffers for data exchange between the rendering engine and the MAF (e.g., via media pipelines). In some examples, buffer APIs (e.g., buffer API 335) can provide or offer one or more of the following methods:

[0154]

[0155] Table 4

[0156] When allocating a buffer, buffer information can be provided (e.g., specifying the desired buffer configuration). In some examples, buffer information may include the maximum size of the buffer, static information in the buffer header, the number of frames in the buffer (e.g., for a circular buffer), the buffer update rate, any combination thereof, and / or other information.

[0157] Example interface definition language (IDL) descriptions for buffer APIs (such as Buffer API 335) are provided in Table 5 below:

[0158]

[0159]

[0160] Table 5

[0161] In some examples, the "count" parameter can indicate the number of frames contained in a given buffer. In some cases, the buffer can be a circular buffer, and the "count" parameter can include the number of frames contained in a given circular buffer. Frames in the buffer can hold data for a specific time instance and can be identified, for example, by an index in the range [0, count-1]. One or more of the frame's index, timestamp, and length can be included in the frame metadata.

[0162] In some examples, the "read_ptr" parameter can include the index of a frame that can currently be read from the buffer. In some cases, if read_ptr equals write_ptr, then no frame is currently available for reading. In some examples, the "write_ptr" parameter can include the index of a frame in the buffer at which MAF or the media pipe can perform a write operation (e.g., using the MAF API).

[0163] In some examples, the "headerLength" parameter can provide the length of the header included at the beginning of the buffer frame. A headerLength of 0 indicates that there is no buffer header.

[0164] In some examples, the "Frame" parameter can include information about the frame in the buffer. A buffer frame can include one or more attributes or fields of the buffer (for example, depicted in Table 5 as including an index attribute, a timestamp attribute, and a length attribute). The index attribute can indicate the position of the frame within the buffer. The timestamp attribute can correspond to the frame's rendering timestamp. The length attribute can correspond to the length of the buffer. In some examples, the timestamp format can be inherited from the accessed media or media object (for example, the timestamp format could be an NTP timestamp, a 0-offset timestamp, or another format).

[0165] More detailed examples of buffer frame attributes / fields are provided in Table 6 below:

[0166]

[0167]

[0168] Table 6

[0169] As previously described, MediaInfo can provide and / or contain a description of the information needed to create or initialize one or more media pipelines. In an illustrative example, MediaInfo provided for an HTML-5 media object (e.g., a video element) could be given as:

[0170] <video width＝"320"height＝"240"controls>

[0171] <source src＝"movie.mp4"type＝"video mp4">

[0172] <source src＝"movie.ogg"type＝"video ogg">

[0173] Your browser does not support the video tag.

[0174] < / video>

[0175] In the example above, <video>Tags can contain one or more for one or more corresponding video sources. <source> The `<src>` tag can specify the URL of a video file (e.g., an mp4 video file, an ogg video file, etc.). Although not depicted in the example above, additional attributes can include, but are not limited to: autoplay (e.g., specifying that the video starts playing as soon as it is ready); height / width (e.g., setting the height and / or width of the video); looping (e.g., specifying that the video will repeat from its beginning each time it ends); and / or preloading (e.g., specifying whether and how the video should be loaded on initial page load).

[0176] In some cases, Media Source Extension (MSE) can be used to attach a SourceBuffer to a media element track. For example, the following is a sequence of steps that can be performed to attach a SourceBuffer to a media element track using MSE. These steps can be performed in the order provided below or in another order:

[0177] 1. Define media codec details

[0178] var mimeCodec="video / mp4;codecs="avc1.4D401F"'

[0179] 2. Configure media source:

[0180] var mediaSource=new MediaSource()

[0181] 3. Attach the MediaSource to the media element:

[0182] video.src=URL.createObjectURL(mediaSource)

[0183] 4. Add SourceBuffer to MediaSource:

[0184] var sourceBuffer=mediaSource.addSourceBuffer(mimeCodec)

[0185] 5. Define the fetch and supplement functions

[0186] function fetchSegmentAndAppend(segmentUrl,sourceBuffer)

[0187] 6. Capture and supplement the initial segments:

[0188] fetchSegmentAndAppend("https: / / example.com / init.mp4",sourceBuffer)

[0189] 7. Capture and supplement the first media segment:

[0190] fetchSegmentAndAppend("https: / / example.com / media1.mp4",sourceBuffer)

[0191] 8. Start playback

[0192] Video.play();

[0193] 9. Capture and add the next media segment:

[0194] fetchSegmentAndAppend("https: / / example.com / media2.mp4",sourceBuffer)

[0195] In some cases, the information presented in the steps above can be summarized in a Media Presentation Description (MPD), such as the description in the DASH profile of CMAF media. In some cases, multiple source buffers can be provided for a single media source (e.g., to provide time alignment). In some examples, a Session Description Protocol (SDP) can be generated for WebRTC (Web Real-Time Communication) to provide real-time communication without plugins. In some cases, an SDP can also be generated to support peer-to-peer data sharing, as well as other media access, retrieval, and / or delivery tasks for MAF (and any associated buffers and / or media pipelines). For example, as part of an SDP give / response process, one or more portions of the media and buffer information previously described above can be exchanged.

[0196] Figure 7 This diagram illustrates an example of a media system 700 including rendering engine 740 and MAF 720 in the context of a glTF 2.0 rendering engine 740 requesting MPEG media. In some cases, rendering engine 740, MAF 720, or both rendering engine 740 and MAF 720 may be the same as or similar to the rendering engine and MAF previously described (e.g., including but not limited to those included in the previous description). Figure 3 The rendering engines are 340 and MAF 320.

[0197] As described in the previous examples of media information discussed above, it can be seen that the media pipeline (also known as the buffer pipeline, in...) Figure 7 The buffer pipe (shown as buffer pipe 770) can be equipped with different media access technologies. For example, for each of the MPEG loop buffers 732a, 732b, and 732c to be filled, the media access function (MAF) 720 can have sufficient information to select the appropriate source for the requested media object. In some examples, when multiple sources are specified (e.g., a primary source and one or more alternative sources), the MAF 720 can select among multiple sources. For example, the MAF 720 can select the appropriate source from among the multiple sources available to it based on factors such as user and / or system preferences, the capabilities of the rendering engine 740, the capabilities of the MAF 720, and current or planned network conditions and capabilities.

[0198] For the selected media source, the MAF 720 may have sufficient information to access the requested media / media object. For example, the MAF 720 may access the requested media object by using a media access protocol that corresponds to or is otherwise suitable for the file format, container, or type of the given media object. Figure 7 The diagram illustrates examples using different media access protocols, showing a first media access plugin 702 (e.g., for a DASH client), a second media access plugin 704 (e.g., for WebRTC), and a third media access plugin 706 (e.g., for disk access). Figure 7 As seen, media access protocols 702-706 can be used to access media and media objects stored in a location type. DASH plugin 702 is shown as being used to provide MAF 720 with access to remote storage 712; a second media access plugin 704 (e.g., for WebRTC) is shown as being used to provide MAF 720 with access to peering connection 716; and a third media access plugin 706 (e.g., a disk access plugin) is shown as being used to provide MAF 720 with access to local storage 714. In some examples, different plugins may be used to access media source locations 712-716. In some examples, MAF 720 may access media source locations without using plugins. In some examples, multiple plugins may be used to provide MAF 720 with access to a given media source location or a type of media source location.

[0199] For the selected media source, the MAF 720 also has sufficient information to set up one or more media pipelines to provide the requested media object to the rendering engine 740 (e.g., the correct buffer format requested by the rendering engine 740 in its calls to the MAF API 325 and / or the buffer API 335). Note that in Figure 7 In the media system 700, individual media pipelines (e.g., those previously discussed) are shown as buffer pipelines 770. In some examples, portions and / or steps of media pipeline functionality handled by one or more media pipelines may be provided as separate instances of each media pipeline. In some cases, portions and / or steps of media pipeline functionality handled by one or more media pipelines may be shared or accessed as common instances of multiple different media pipelines or some combination of both. In other words, the video decoding engine 782a, the audio decoding engine 782b, and the data compressor 782c may be provided as separate instances of each of the buffer pipelines 770 as needed, and / or may be provided as shared instances (or single instances, as shown) accessed by various different buffer pipelines 770 as needed.

[0200] In some examples, MAF 720 can query or retrieve additional information from rendering engine 740 to optimize the delivery of one or more requested media objects. For example, MAF 720 can retrieve additional information about the required quality of one or more output buffers (e.g., MPEG loop buffers 732a-c). Among other factors, MAF 720 can also retrieve timing information used by rendering engine 740 for one or more requested media objects initially identified in the media information / call to MAF API 325. MAF 720 can also retrieve GL Send Format (glTF) information, such as gITF accessors and buffer views 738 (e.g., identified by the bufferView variable mentioned herein). glTF is the standard file format for 3D scenes and models. gITF information can be stored in gITF buffer 736. Other information that can be used by the rendering engine 740 and / or the MAF 720 includes the MPEG visual timing accessor 742 and the MPEG visual timing accessor 794 (which can communicate with the timing scene extension and timing accessor synchronization 790, which can synchronize scene updates). In one example, the accessor in glTF 2.0 defines the type and layout of the data stored in the buffer viewed through the bufferView. For example, the data in the buffer may change dynamically over time when timing data is read from the buffer. Buffer elements can be extended to add support for circular buffers used for timing data. Examples of buffer views that the system 700 can use include buffer views 746a and 746b. Scene update information 748 can also be used by the MAF 720 and / or the rendering engine 740 to update the scene. For example, when a media sample becomes active, the media player can load sample data into the rendering engine 740, which can trigger a scene update performed by the rendering engine 740. If a scene update includes the addition of new information (e.g., glTF nodes and / or potential modifications to existing glTF nodes, such as one or more components), the rendering engine 740 can interact with the MAF 720 to fetch any new content associated with the scene update and render the new content accordingly.

[0201] Figure 8 This is a flowchart illustrating an example of a process 800 for fetching MPEG media objects using one or more media pipes and buffers. At box 802, the process includes starting the rendering engine. In an illustrative example, the rendering engine includes rendering engine 340 (such as...). Figure 3 As shown), rendering engine 440 (as shown) Figure 4 As shown), rendering engine 540 (as shown) Figure 5 As shown), rendering engine 640 (as shown) Figure 6 (as shown) and / or rendering engine 740 (as shown) Figure 7 (As shown in the image). In some examples, the rendering engine can be used to render, render, or otherwise display 3D media scenes or immersive media scenes.

[0202] At box 804, process 800 includes parsing an MPEG-1 scene description file or document. In an illustrative example, the scene description file may be based on the glTF 2.0 format and include... Figure 3 The scene description document 348 and / or shown are shown. Figure 4 The scene description and scene update document are shown. In some examples, parsing the scene description file may be performed by the rendering engine upon initial receipt or retrieval of the scene description file. In some cases, the scene description file may be parsed in an ongoing manner (e.g., where the rendering engine parses the scene description document to obtain media and / or buffer information needed to generate one or more calls to the MAF API and / or the Buffer API). In an illustrative example, the MAF API includes... Figure 3 , 4 And the MAF API 325 shown in Figure 7; and the buffer API includes Figure 3 , 4 The buffer API335 shown in Figure 7.

[0203] At box 806, process 800 includes completing the parsing of the MPEG-1 scene description. In an illustrative example, the MPEG-1 scene description contains information corresponding to one or more different media objects that are used by the rendering engine to render the MPEG media scene. In some examples, the parsing of the MPEG-1 scene description may be performed by the rendering engine to generate media information (e.g., MediaInfo as described in Tables 2 and 3) and / or buffer information (e.g., BufferInfo as described in Tables 2, 3, 5, and 6). The output of parsing the MPEG-1 scene description may be stored locally at the rendering engine, for example, for later padding or otherwise generating one or more MAF API or buffer API calls. In some examples, the output of parsing the MPEG-1 scene description may be stored remotely from the rendering engine. In some examples, the output of parsing the MPEG-I scene description may be stored remotely from the rendering engine, such that the rendering engine obtains or receives the output from parsing the MPEG-I scene description, rather than performing the parsing itself.

[0204] At box 808, procedure 800 includes initializing the Media Access Function (MAF). In an illustrative example, the MAF includes MAF 320 (as shown in the image). Figure 3 As shown), MAF 420 (as shown) Figure 4 (as shown) and / or MAF 720 (as shown) Figure 7 (As shown in the diagram). In some cases, the MAF and rendering engine can be provided on the same local media playback system or device. In other cases, the MAF and rendering engine can be provided separately. In some examples, initializing the MAF includes initializing one or more plugins for the MAF and / or acquiring and configuring one or more plugins for the MAF. Plugins may include protocol plugins (such as...). Figure 4 The protocol plugin 423 shown), format plugins (such as...) Figure 4 The format plugin 427 shown), and / or the MAF plugin of the rendering engine (such as Figure 4 The MAF plugin 443 shown is illustrated. In some examples, plugins at or for MAF initialization may include... Figure 7 One or more of the media access plugins 702-706 shown here.

[0205] At box 810, process 800 includes completing the initialization of the MAF. In an illustrative example, completing the initialization of the MAF includes establishing (and / or confirming the establishment capability) a communication session between the MAF and the rendering engine. The communication session between the MAF and the rendering engine is authorized, for example, via access to the MAF API (such as...). Figure 3 , 4 One or more calls to the MAF API (325) shown in Figure 7 are used to send and exchange the requested media object and its corresponding media information and / or buffer information. In some examples, completing the initialization of the MAF includes setting up (and / or confirming the setting capability) the MAF and the buffer API (such as...). Figure 3 , 4 The communication session between the MAF and the buffer API 335 shown in 7. Completing the initialization of the MAF may also include establishing (and / or confirming the establishment capability) a communication session between the MAF and one or more media sources or media source locations, using one or more plugins initialized at the MAF, as described above with respect to box 808. Media sources and media source locations may include, but are not limited to, remote or networked locations (such as cloud services or platforms), local storage (such as disk storage, solid-state storage, disks and other removable storage, etc.), and / or peer-to-peer connections.

[0206] At box 812, procedure 800 includes creating one or more media pipes. In an illustrative example, one or more media pipes include media pipes 372, 374 (e.g., Figure 3 (as shown), media channels 572-578 (e.g.) Figure 5 (as shown), media channels 672, 674 (such as) Figure 6 (as shown) and / or buffer pipe 770 (as shown) Figure 7 (As shown in the diagram). A media pipeline may include one or more internal decoding and / or processing operations / operators that can be applied as input to one or more tracks or media objects provided as input to a given media pipeline (e.g., retrieved by MAF from a media source and then provided as input to a given media pipeline). A media pipeline may provide output to one or more output buffers, which may provide storage for processed media objects requested by the rendering engine and may provide the rendering engine with access to the processed media objects. In some examples, a media pipeline may be created for each media object requested by the rendering engine or otherwise identified in MAF API calls as needing to be placed in one or more output buffers at some time and / or in a requested format.

[0207] At box 812, procedure 800 includes initializing one or more media pipes and buffers. In one illustrative example, initializing a media pipe includes the `initialize()` function shown in Table 1, and / or initializing a buffer includes the `allocate()` function shown in Table 4. Buffers may include output buffers used to exchange processed media objects between the MAF and the rendering engine, and may include one or more intermediate buffers. Intermediate buffers may be created within the media pipe to support one or more decoding, processing, or other operations performed on the media object by the media pipe before the final, processed media object is output to the output buffer. In some examples, one or more output buffers may be initialized to be contained within the media pipe. In some examples, one or more output buffers may be initialized outside the media pipe. In one illustrative example, a buffer may include buffers 332a-n ( Figure 3 As shown), buffer 432a-e ( Figure 4 As shown), buffer 532 ( Figure 5 (as shown), buffers 632 and 634a-d ( Figure 6 (as shown) and / or buffers 736 and / or 732a-c ( Figure 7 One or more of the following (shown in the diagram). Initialization of a newly created media pipeline can be performed (e.g., a MAF API call made by the rendering engine provides parameters for initializing the media pipeline to the requested media object; MAF creates each media pipeline and initializes each media pipeline using the parameters from the MAF API call). Initialization of a previously created, or otherwise existing and / or in-use media pipeline can be performed (e.g., a MAF API call made by the rendering engine can provide parameters for initializing an existing media pipeline; the parameters may include an identifier for each existing media pipeline, which MAF can use to subsequently reconfigure or initialize the existing media pipeline based on the parameters received in the MAF API call).

[0208] At box 816, procedure 800 includes initiating media fetching for each pipeline. In an illustrative example, media fetching may include the startFetching() function from Table 1. For example, after initializing the media pipeline and / or buffer at box 814, a given media pipeline may be set to the "READY" state. In response to the "READY" state, in some examples, procedure 800 may move to box 816 to begin media fetching for each pipeline that is in the "READY" state. Each media pipeline may perform discrete media fetching operations, for example, for one or more specific media objects identified by the rendering engine's MAF API call.

[0209] At box 818, process 800 includes setting the "ACTIVE" state for each media pipeline that is performing media fetching. In one example, the "ACTIVE" state can be set once the pipeline begins receiving the requested media object retrieved by MAF from the media source location as input. In another example, the "ACTIVE" state can be set for the media pipeline even if fetching has not yet started, once the rendering engine and / or MAF has sent a request for the media pipeline to start fetching (e.g., using the startFetching() function in Table 1).

[0210] At box 820, process 800 includes initiating a rendering loop at the rendering engine. In one example, the rendering engine can begin a rendering loop by accessing one or more output buffers to retrieve previously requested media objects for rendering the media scene. In some cases, MAF API calls may correspond to a specific rendering loop; for example, the MAF API may identify some or all of the media objects required by the rendering engine to execute a given rendering loop. In some examples, timing can be applied to one or more rendering loops executed by the rendering engine; for example, the rendering loop may be associated with a specific start time, and MAF may perform scheduling and pipeline management procedures such that all appropriate media pipelines and buffers are created, initialized, and triggered to begin fetching early enough to ensure that processed media objects are delivered to the output buffers at or before the rendering loop start time. In some examples, the rendering engine may request MAF to provide processed media objects to the output buffers, with the provision of processed media objects offset from the rendering loop start time. For example, the rendering engine can specify (or as a default / predefined parameter) in the MAF API call that the processed media object will be provided to the output buffer at least 500 milliseconds before the start time (other time amounts can also be specified, such as between 50 milliseconds and 3 seconds). In some examples, the rendering engine can include individual delivery time requirements or deadlines for one or more requested media objects. For example, the rendering engine can include additional parameters in the MAF API call to specify the delivery time of the requested media object.

[0211] At box 822, process 800 includes performing the rendering loop that begins at box 820 by iterating through the media objects contained in the output buffers provided by MAF to the rendering engine. In an illustrative example, iterating through the media objects may be based on an MPEG-1 scene description and / or the rendering engine's parsing of the MPEG-1 scene description (e.g., the parsing performed at boxes 804 and / or 806). Based on the scene description, the rendering engine determines each media object to be rendered for the rendering loop. For each media object, the rendering engine may iterate through its constituent attributes or media elements (which may themselves be media objects). The rendering engine can then fetch processed frames for each attribute by accessing and reading one or more corresponding output buffers. Processed frames can be obtained from the processed media elements provided to the output buffers by the media pipeline initialized by MAF. In some examples, the rendering engine may determine the corresponding output buffer(s) for a given frame because one or more MAF API calls of the rendering engine specify one or more output buffers(s) for each requested media object. In some examples, the rendering loop can continue after the above frame capture until the rendering engine binds property data and renders each object in the rendering loop.

[0212] Figure 9 This is a flowchart illustrating an example of a process 900 for processing media content. At box 902, process 900 may include acquiring media information and buffer information for media objects determined from a scene description of at least one media scene. In some examples, the media information may identify the intended format for storing the media object in one or more output buffers of a media pipeline. In some cases, buffer information may include one or more attributes of one or more output buffers. The media scene may include one or more immersive media objects, which may include (but are not limited to) three-dimensional (3D) mesh objects, 3D point cloud objects, virtual reality (VR) objects, augmented reality (AR) objects, and 6-DOF (DoF) objects. In some cases, the media information may include one or more source locations for acquiring the media object, and / or the media information may identify one or more rendering parameters for replaying the media object from one or more output buffers. In some examples, buffer information includes a list of buffers, each entry in the buffer list including at least a buffer identifier (e.g., where the buffer identifier is associated with one or more buffer attributes of the buffer identified by the buffer identifier). In some cases, media information and / or buffer information for media components that are part of a larger media object (e.g., a media object comprising multiple media components) may be acquired. In some examples, media information may include MediaInfo, as described with respect to Tables 2 and 3. In some examples, buffer information may include BufferInfo, as described with respect to Tables 2, 3, 5, and 6.

[0213] In an illustrative example, a rendering engine can be used to obtain media information and buffer information based on parsing a scene description of at least one media scene. The rendering engine may include rendering engine 340 ( Figure 3 As shown), rendering engine 440 ( Figure 4 As shown), rendering engine 540 ( Figure 5 As shown), rendering engine 640 ( Figure 6 As shown), rendering engine 740 ( Figure 7 (as shown) and / or other rendering engines. In some examples, a rendering engine may be used to render, render, or otherwise display a 3D media scene or immersive media scene that is at least partially described by a scene description (and / or subsequent scene updates). In some examples, the scene description may be an MPEG-1 scene description file or document, and / or the scene description file may be based on the glTF 2.0 format. The scene description may include Figure 3 The scene description document 348 and / or shown are shown. Figure 4 The scene description and scene update document are shown.

[0214] In some examples, parsing the scene description file and obtaining media and buffer information can be performed by the rendering engine upon initial receipt or retrieval of the scene description file. In some cases, the scene description file can be parsed in an ongoing manner, where the rendering engine parses the scene description document to obtain the media and / or buffer information needed to generate one or more calls to the MAF API and / or the Buffer API. In an illustrative example, the MAF API may include... Figure 3 , 4 And as shown in 7, the MAF API 325, and the buffer API may include Figure 3 , 4 And the buffered API 335 shown in Figure 7. The MAF API can be used to provide a communication interface (e.g., media access functionality) to the MAF and / or control the MAF. The MAF may include MAF 320 ( Figure 3 As shown), MAF 420 ( Figure 4 (as shown) and / or MAF 720 ( Figure 7 (As shown in the diagram). In some cases, the MAF and rendering engine can be provided on the same local media playback system or device. In other cases, the MAF and rendering engine can be provided separately.

[0215] At box 904, procedure 900 may include a media pipe for initializing a media object based on media information and / or buffer information. In an illustrative example, the media pipe may include media pipes 372, 374 ( Figure 3 (as shown), media channels 572-578 ( Figure 5 (as shown), media channels 672, 674 ( Figure 6 (as shown) and / or buffer pipe 770 ( Figure 7 One or more of the following (as shown). Initializing the media pipeline may include providing media information and / or buffer information as parameters for one or more MAF API calls to a media object describing the scene. For example, media information may include one or more source locations for obtaining the media object. In some examples, initializing the media pipeline may include initializing one or more plugins and / or obtaining and configuring one or more plugins such that the media pipeline and / or the plugin(s) used by the media pipeline are configured as needed. Plugins may include protocol plugins (such as...) Figure 4 The protocol plugin 423 shown), format plugins (such as...) Figure 4 The format plugin 427 shown), and / or the MAF plugin of the rendering engine (such as Figure 4 The MAF plugin 443 shown is illustrated. In some examples, plugins at or during MAF initialization may include... Figure 7 One or more of the media access plugins 702-706 shown.

[0216] A media pipeline can provide output to one or more output buffers, which can provide storage for processed media objects requested by the rendering engine. In some cases, one or more output buffers can provide the rendering engine with access to the processed media objects. In some examples, a media pipeline can be created for each media object or media component requested by the rendering engine or otherwise identified in MAF API calls as needing to be placed in one or more output buffers at some time and / or in a requested format. In an illustrative example, initializing a media pipeline may include using the initialize() function shown in Table 1. In some cases, initializing a media pipeline may also include initializing one or more buffers, for example, using the allocate() function shown in Table 4.

[0217] In some examples, media pipeline initialization can be performed for newly created media pipelines (e.g., a MAF API call made by the rendering engine provides parameters for initializing the media pipeline to the requested media object; MAF creates each media pipeline and initializes each media pipeline using the parameters from the MAF API call). In some examples, media pipeline initialization can be performed for media pipelines that have been previously created or otherwise exist and / or are in use (e.g., a MAF API call made by the rendering engine can provide parameters for initializing an existing media pipeline; the parameters may include an identifier for each existing media pipeline, which MAF can use to subsequently reconfigure or initialize the existing media pipeline based on the parameters received in the MAF API call). Media pipeline initialization may include allocating a new buffer in response to determining that a buffer identifier (e.g., a buffer identifier determined based on buffer information, a MAF API call, and / or a buffer API call) does not belong to any existing buffer. The allocation of the new buffer may then be performed based on one or more buffer properties associated with the buffer identifier.

[0218] At box 906, process 900 may include obtaining a media object in a delivery format. In an illustrative example, a media pipeline may be used to obtain a media object from a media source or media source location, where, for example, the delivery format is the format in which the media source provides the media object to the media pipeline. Media sources and media source locations may include, but are not limited to, remote or networked locations (such as cloud services or platforms), local storage (such as disk storage, solid-state storage, disk storage, and other removable storage), and / or peer-to-peer connections. In some cases, one or more media sources and / or media source locations may be identified in the media information using at least one of a Uniform Resource Locator (URL), a Uniform Resource Identifier (URI), and a local file path. In some cases, the media information may identify multiple media source locations, including a default media source location and one or more alternative media source locations different from the default media source location. In some examples, when multiple media source locations are identified, the selected source location may be determined by selecting from the multiple media source locations using one or more selection factors. Selection factors may include, but are not limited to, one or more of playback preferences, playback capabilities, and network characteristics. In some examples, media objects can be retrieved from a source location selected based on view information determined from one or more MAF API calls. View information may include the position of the media object relative to the viewer's location. In some cases, view information can be used to configure the selected source location for retrieving the media object. For example, view information may be used to adjust the level of detail of the media object (e.g., higher or lower resolution), adjust the bitrate of the media object, and / or adjust the visible portion or portions of the retrieved media object.

[0219] In an illustrative example, acquiring a media object may include initiating media fetching for a media pipeline. For example, media fetching can be performed using the `startFetching()` function from Table 1. In some cases, a media pipeline may be set to a "READY" state after initialization but before media fetching begins. In response to the "READY" state, media fetching can begin. Each media pipeline may perform discrete media fetching operations, such as for a specific one or more of the requested media object and / or media components identified by a MAF API call from the rendering engine. After media fetching begins to acquire a media object, process 900 may include setting an "ACTIVE" state for each media pipeline performing media fetching. In one example, the "ACTIVE" state can be set once the pipeline begins receiving a requested media object as input (e.g., a media object retrieved by MAF from a media source). In another example, the "ACTIVE" state can be set for a media pipeline even if fetching has not yet begun, once the rendering engine and / or MAF has sent a request to start fetching for the media pipeline (e.g., using the `startFetching()` function from Table 1).

[0220] In some cases, at box 906, process 900 may further include generating one or more processed media objects, at least in part, by processing the acquired media objects, such that each processed media object (e.g., one or more processed media objects) is converted from a delivery format to a desired format for storing each processed media object in one or more output buffers. In an illustrative example, a media pipeline may include one or more internal decoding and / or processing operations / operators that may be applied as input to one or more tracks, media objects, and / or media components provided to the media pipeline (e.g., retrieved from a media source by MAF and then provided as input to a given media pipeline). The desired format for storing each processed media object in the output buffer may be specified by rendering-generated MAF API calls and / or buffer API calls, for example, based on media information and / or buffer information, respectively. In some cases, the desired format may be specified or determined by the rendering engine, for example, based on one or more settings, preferences, or configurations. In some cases, the rendering engine may obtain the desired format for delivery or processed media objects to one or more output buffers based on parsing a scene description document associated with the media object.

[0221] At box 908, procedure 900 may include outputting one or more processed media objects to one or more output buffers using an initialized media pipeline. In one illustrative example, the buffer may include an output buffer used to exchange processed media objects between the MAF and the rendering engine. In some cases, the buffer may include at least one circular buffer. The buffer can be controlled by passing buffer information or by calling the buffer API one or more times. In some cases, the buffer may include one or more intermediate buffers. Intermediate buffers may be created within the media pipeline to support one or more decoding, processing, or other operations performed on the media object by the media pipeline before the final, processed media object is output to the output buffer. In some examples, one or more output buffers may be initialized to be included within the media pipeline. In some examples, one or more output buffers may be initialized outside the media pipeline. In one illustrative example, the buffer may include buffers 332a-n ( Figure 3 As shown), buffer 432a-e ( Figure 4 As shown), buffer 532 ( Figure 5 (shown), buffers 632 and 634a-d ( Figure 6 (as shown) and / or buffers 736 and / or 732a-c ( Figure 7 One or more of the ones shown.

[0222] In some examples, the processes described herein (e.g., process 800, process 900, and / or any other process described herein) may be performed by a computing device, apparatus, or system. In one example, process 800 and / or process 900 may be performed by a device having… Figure 12 The computing device architecture 1200 is used to perform the computing device or system. The computing device, apparatus, or system may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a connected watch or smartwatch, or other wearable device), a server computer, a computing device for an autonomous vehicle or autonomous vehicle, a robotic device, a laptop computer, a smart TV, a camera, and / or any other computing device with the resource capability to perform the processes described herein (including process 800, process 900, and / or any other process described herein). In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or one or more other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or one or more other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP) based data or other types of data.

[0223] Components of a computing device may be implemented in a circuit. For example, a component may include and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, graphics processing unit (GPU), digital signal processor (DSP), central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.

[0224] Processes 800, 900, and / or any other processes described herein are illustrated as logic flowcharts, whose operations represent sequences of operations that can be implemented using hardware, computer instructions, or a combination thereof. In the context of computer instructions, an operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific data type. The order in which the operations are described is not intended to be construed as limiting, and any number of the described operations can be combined in any order and / or in parallel to implement these processes.

[0225] Additionally, processes 800, 900, and / or any other processes described herein may be executed under the control of one or more computer systems configured with operable instructions, and may be implemented by hardware or a combination thereof as code (e.g., operable instructions, one or more computer programs, or one or more applications) that runs concurrently on one or more processors. As described above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0226] As mentioned above, a decoder (or decoding device) can be used to decode video data, for example by performing... Figure 5 The diagram shows one or more decoding operations 582. Examples of decoders include HEVC decoder 682a, HEVC decoder 682b, HEVC decoder 682c, and patch decoder 682d. Other examples of decoders include decoders configured to decode data encoded according to the Universal Video Codec (VVC) standard, the Advanced Video Codec (AVC) standard, the Open Media Consortium (AOMedia) Video 1 (AV1) format, etc.

[0227] Figure 10 and Figure 11 Details of the example encoding device 1004 and decoding device 1112 are shown in the figure. Figure 10 This is a block diagram illustrating an example encoding device 1004 that can implement one or more of the techniques described in this disclosure. Encoding device 1004 can, for example, generate the syntax structures described herein (e.g., syntax structures of video parameter sets (VPS), sequence parameter sets (SPS), picture parameter sets (PPS), and / or other syntax elements). Encoding device 1004 can perform intra-frame prediction and inter-frame prediction encoding / decoding on video blocks within a video stripe. As previously described, intra-frame encoding / decoding relies at least in part on spatial prediction to reduce or eliminate spatial redundancy within a given video frame or picture. Inter-frame encoding / decoding relies at least in part on temporal prediction to reduce or eliminate temporal redundancy within adjacent or surrounding frames of a video sequence. Intra-frame mode (I-mode) can refer to any of several spatially based compression modes. Inter-frame modes such as one-way prediction (P-mode) or two-way prediction (B-mode) can refer to any of several temporally based compression modes.

[0228] Encoding device 1004 includes a segmentation unit 35, a prediction processing unit 41, a filter unit 63, an image memory 64, an adder 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. The prediction processing unit 41 includes a motion estimation unit 42, a motion compensation unit 44, and an intra-frame prediction processing unit 46. For video block reconstruction, encoding device 1004 also includes an inverse quantization unit 58, an inverse transform processing unit 60, and an adder 62. Filter unit 63 is intended to represent one or more loop filters, such as deblocking filters, adaptive loop filters (ALF), and sample adaptive offset (SAO) filters. Although filter unit 63 is... Figure 10 The filter unit 63 is shown as an in-loop filter, but in other configurations, it may be implemented as a post-loop filter. The post-processing device 57 may perform additional processing on the encoded video data generated by the encoding device 1004. In some examples, the techniques of this disclosure may be implemented by the encoding device 1004. However, in other examples, one or more techniques of this disclosure may be implemented by the post-processing device 57.

[0229] like Figure 10 As shown, encoding device 1004 receives video data, and segmentation unit 35 segments the data into video blocks. This segmentation may also include segmentation into stripes, strip segments, slices, or other larger units, as well as video block segmentation, for example, according to a quadtree structure of LCUs and CUs. Encoding device 1004 generally illustrates components for encoding video blocks within a video stripe to be encoded. A stripe can be divided into multiple video blocks (and possibly into sets of video blocks referred to as slices). Prediction processing unit 41 can select one of several possible codec modes for the current video block based on error results (e.g., codec rate and distortion level), such as one of several intra-frame prediction codec modes or one of several inter-frame prediction codec modes. Prediction processing unit 41 can provide the resulting intra-frame or inter-frame encoded block to adder 50 to generate residual block data, and to adder 62 to reconstruct the encoded block for use as a reference picture.

[0230] The intra-prediction processing unit 46 within the prediction processing unit 41 can perform intra-prediction coding and decoding of the current video block relative to one or more adjacent blocks in the same frame or stripe as the current block to be encoded and decoded, to provide spatial compression. The motion estimation unit 42 and motion compensation unit 44 within the prediction processing unit 41 perform inter-prediction coding and decoding of the current video block relative to one or more prediction blocks in one or more reference pictures, to provide temporal compression.

[0231] Motion estimation unit 42 can be configured to determine the inter-frame prediction mode of video stripes based on a predetermined pattern of the video sequence. The predetermined pattern can assign video stripes in the sequence to P-stripes, B-strips, or GPB-strips. Motion estimation unit 42 and motion compensation unit 44 can be highly integrated, but are illustrated separately for conceptual purposes. Motion estimation performed by motion estimation unit 42 is the process of generating motion vectors that estimate the motion of video blocks. For example, motion vectors can indicate the displacement of the prediction unit (PU) of a video block within the current video frame or picture relative to the prediction block within a reference picture.

[0232] A predicted block is a block found to closely match the PU of the video block to be encoded or decoded in terms of pixel difference (or image sample difference), where the pixel difference can be determined by the sum of absolute differences (SAD), sum of squared differences (SSD), or other difference metrics. In some examples, encoding device 1004 can compute values ​​for sub-integer pixel positions of a reference image stored in image memory 64. For example, encoding device 1004 can interpolate values ​​for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of the reference image. Thus, motion estimation unit 42 can perform motion search relative to full-pixel positions and fractional pixel positions and output a motion vector with fractional pixel precision.

[0233] The motion estimation unit 42 calculates the motion vector of the PU by comparing the position of the PU of the video block in the inter-frame coding strip with the position of the predicted block of the reference picture. The reference picture can be selected from a first reference picture list (list 0) or a second reference picture list (list 1), each list identifying one or more reference pictures stored in the picture memory 64. The motion estimation unit 42 sends the calculated motion vector to the entropy coding unit 56 and the motion compensation unit 44.

[0234] Motion compensation performed by motion compensation unit 44 may involve capturing or generating prediction blocks based on motion vectors determined by motion estimation, which may perform interpolation down to subpixel precision. Upon receiving the motion vector of the PU for the current video block, motion compensation unit 44 can locate the prediction block to which the motion vector points in a list of reference images. Encoding device 1004 forms a residual video block by subtracting the pixel values ​​(or image sample values) of the prediction block from the pixel values ​​of the current video block being encoded / decoded, resulting in a pixel difference (or image sample difference). The pixel difference (or image sample difference) forms the residual data for the block and may include both luminance and chrominance difference components. Adder 50 represents one or more components performing this subtraction operation. Motion compensation unit 44 may also generate syntax elements associated with video blocks and video stripes for use by decoding device 1112 when decoding video blocks of video stripes.

[0235] As an alternative to the inter-frame prediction performed by the motion estimation unit 42 and motion compensation unit 44 as described above, the intra-frame prediction processing unit 46 can perform intra-frame prediction for the current block. Specifically, the intra-frame prediction processing unit 46 can determine the intra-frame prediction mode for encoding the current block. In some examples, the intra-frame prediction processing unit 46 can use various intra-frame prediction modes to encode the current block, for example, during a separate coding process, and the intra-frame prediction processing unit 46 can select an appropriate intra-frame prediction mode from the tested modes. For example, the intra-frame prediction processing unit 46 can use rate-distortion analysis for various tested intra-frame prediction modes to calculate rate-distortion values, and can select the intra-frame prediction mode with the best rate-distortion characteristics from the tested modes. Rate-distortion analysis typically determines the amount of distortion (or error) between the coded block and the original uncoded block encoded to produce the coded block, as well as the bit rate (i.e., number of bits) used to generate the coded block. Intra-prediction processing unit 46 can calculate the ratio based on the distortion and rate of various coded blocks to determine which intra-prediction mode displays the optimal rate-distortion value for the block.

[0236] In any case, after selecting an intra-prediction mode for a block, the intra-prediction processing unit 46 can provide the entropy coding unit 56 with information indicating the intra-prediction mode selected for the block. The entropy coding unit 56 can encode the information indicating the selected intra-prediction mode. The encoding device 1004 can include configuration data definitions of the encoding contexts for various blocks, as well as indications of the most likely intra-prediction modes, an intra-prediction mode index table, and a modified intra-prediction mode index table in the transmitted bitstream for each context. The bitstream configuration data may include multiple intra-prediction mode index tables and multiple modified intra-prediction mode index tables (also referred to as codeword mapping tables).

[0237] After prediction processing unit 41 generates a prediction block for the current video block via either inter-frame prediction or intra-frame prediction, encoding device 1004 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block can be included in a plurality of TUs and applied to transform processing unit 52. Transform processing unit 52 uses a transform (such as Discrete Cosine Transform (DCT) or a conceptually similar transform) to transform the residual video data into residual transform coefficients. Transform processing unit 52 can transform the residual video data from the pixel domain to the transform domain, such as the frequency domain.

[0238] Transform processing unit 52 can send the resulting transform coefficients to quantization unit 54. Quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process can reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be corrected by adjusting the quantization parameters. In some examples, quantization unit 54 can then perform a scan of a matrix including the quantized transform coefficients. Alternatively, entropy coding unit 56 can perform the scan.

[0239] After quantization, entropy coding unit 56 performs entropy coding on the quantized transform coefficients. For example, entropy coding unit 56 can perform context-adaptive variable-length codec (CAVLC), context-adaptive binary arithmetic codec (CABAC), syntax-based context-adaptive binary arithmetic codec (SBAC), probabilistic interval partitioned entropy (PIPE) codec, or another entropy coding technique. After entropy coding by entropy coding unit 56, the encoded bitstream can be sent to decoding device 1112, or archived for later transmission or retrieval by decoding device 1112. Entropy coding unit 56 can also perform entropy coding on the motion vectors and other syntax elements of the current video strip being encoded / decoded.

[0240] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel domain for use as a reference block later as a reference image. Motion compensation unit 44 can compute the reference block by adding the residual block to a predicted block of one of the reference images in the reference image list. Motion compensation unit 44 can also apply one or more interpolation filters to the reconstructed residual block to compute sub-integer pixel values ​​(or image sample values) used for motion estimation. Adder 62 adds the reconstructed residual block to the motion-compensated predicted block produced by motion compensation unit 44 to produce a reference block for storage in image memory 64. Motion estimation unit 42 and motion compensation unit 44 can use the reference block as a reference block to perform inter-frame prediction of blocks in subsequent video frames or images.

[0241] In this way, Figure 10 The encoding device 1004 represents an instance of a video encoder configured to perform any of the techniques described herein.

[0242] Figure 11 This is a block diagram illustrating an example decoding device 1112. Decoding device 1112 includes an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, an adder 90, a filter unit 91, and an image memory 92. The prediction processing unit 81 includes a motion compensation unit 82 and an intra-frame prediction processing unit 84. In some examples, decoding device 1112 can perform operations generally related to... Figure 10 The encoding process described by the encoding device 1004 is the inverse of the decoding process.

[0243] During the decoding process, decoding device 1112 receives an encoded video bitstream sent by encoding device 1004, the encoded video bitstream representing video blocks and associated syntax elements of an encoded video stripe. In some embodiments, decoding device 1112 may receive the encoded video bitstream from encoding device 1004. In some embodiments, decoding device 1112 may receive the encoded video bitstream from network entity 79, wherein network entity 79 is such as a server, a media-aware network element (MANE), a video editor / joiner, or other such device configured to implement one or more of the technologies described above. Network entity 79 may or may not include encoding device 1004. Some of the technologies described in this disclosure may be implemented by network entity 79 before it sends the encoded video bitstream to decoding device 1112. In some video decoding systems, network entity 79 and decoding device 1112 may be part of separate devices, while in other examples, the functionality described with respect to network entity 79 may be performed by the same device including decoding device 1112.

[0244] The entropy decoding unit 80 of the decoding device 1112 entropy decodes the bitstream to generate quantized coefficients, motion vectors, and other syntax elements. The entropy decoding unit 80 forwards the motion vectors and other syntax elements to the prediction processing unit 81. The decoding device 1112 can receive syntax elements at the video stripe level and / or video block level. The entropy decoding unit 80 can process and parse both fixed-length and variable-length syntax elements from one or more parameter sets (such as VPS, SPS, and PPS).

[0245] When a video strip is encoded / decoded into an intra-frame codec (I) strip, the intra-frame prediction processing unit 84 of the prediction processing unit 81 can generate prediction data for video blocks of the current video strip based on the intra-frame prediction mode notified by signaling and data from previously decoded blocks from the current frame or picture. When a video frame is encoded / decoded into an inter-frame codec (i.e., B, P, or GPB) strip, the motion compensation unit 82 of the prediction processing unit 81 generates prediction blocks for video blocks of the current video strip based on motion vectors and other syntax elements received from the entropy decoding unit 80. Prediction blocks can be generated from one of the reference pictures in the reference picture list. The decoding device 1112 can construct the reference frame list, i.e., list 0 and list 1, based on the reference pictures stored in the picture memory 92 using the default construction technique.

[0246] Motion compensation unit 82 determines the prediction information of video blocks in the current video strip by parsing motion vectors and other syntax elements, and uses this prediction information to generate prediction blocks for the decoded current video blocks. For example, motion compensation unit 82 can use one or more syntax elements in the parameter set to determine the prediction mode (e.g., intra-frame or inter-frame prediction), inter-frame prediction stripe type (e.g., B-strip, P-strip, or GPB-strip), construction information of one or more reference picture lists for the strip, motion vectors of each inter-frame encoded video block of the strip, inter-frame prediction status of each inter-frame encoded video block of the strip, and other information for decoding video blocks in the current video strip.

[0247] The motion compensation unit 82 can also perform interpolation based on an interpolation filter. The motion compensation unit 82 can use an interpolation filter, such as that used by the encoding device 1004 during the encoding of a video block, to calculate the interpolated values ​​of sub-integer pixels of the reference block. In this case, the motion compensation unit 82 can determine the interpolation filter used by the encoding device 1004 from the received syntax elements and can use the interpolation filter to generate a prediction block.

[0248] The inverse quantization unit 86 inverse-quantizes or dequantizes the quantized transform coefficients provided in the bitstream and decoded by the entropy decoding unit 80. The inverse quantization process may include determining the degree of quantization using quantization parameters calculated by the encoding device 1004 for each video block in the video strip, and similarly, determining the degree of inverse quantization that should be applied. The inverse transform processing unit 88 applies an inverse transform (e.g., inverse DCT or other suitable inverse transform), an inverse integer transform, or a conceptually similar inverse transform process to the transform coefficients to produce residual blocks in the pixel domain.

[0249] After the motion compensation unit 82 generates a prediction block for the current video block based on motion vectors and other syntax elements, the decoding device 1112 forms a decoded video block by adding the residual block from the inverse transform processing unit 88 to the corresponding prediction block generated by the motion compensation unit 82. Adder 90 represents one or more components performing this summation operation. If needed, loop filters (in or after the encoding / decoding loop) can also be used to smooth pixel transitions or otherwise improve video quality. Filter unit 91 is intended to represent one or more loop filters, such as deblocking filters, adaptive loop filters (ALF), and sample adaptive offset (SAO) filters. Although filter unit 91 is in Figure 11 The filter unit 91 is shown as an in-loop filter, but in other configurations, it can be implemented as a post-loop filter. The decoded video block in a given frame or image is then stored in image memory 92, which stores reference images for subsequent motion compensation. Image memory 92 also stores decoded video for later rendering on a display device.

[0250] In this way, Figure 11 The decoding device 1112 represents an example of a video decoder configured to perform any of the techniques described herein.

[0251] Figure 12 An example computing device architecture 1200 is illustrated, which can implement the various technologies described herein. For example, computing device architecture 1200 can implement… Figure 3 At least some portions of the media system 300 shown. Components of the computing device architecture 1200 are shown communicating electrically with each other using a connection 1205 such as a bus. The example computing device architecture 1200 includes a processing unit (CPU or processor) 1210 and a computing device connection 1205 that couples various computing device components, including computing device memories 1215 (such as read-only memory (ROM) 1220 and random access memory (RAM) 1225), to the processor 1210.

[0252] The computing device architecture 1200 may include a cache of high-speed memory that is directly connected to, closely adjacent to, or integrated into the processor 1210. The computing device architecture 1200 may copy data from memory 1215 and / or storage device 1230 to the cache 1212 for fast access by the processor 1210. In this way, the cache can provide performance improvements and avoid latency for the processor 1210 while waiting for data. These and other modules may control or be configured to control the processor 1210 to perform various actions. Other computing device memories 1215 may also be used. Memory 1215 may include multiple different types of memory with different performance characteristics. The processor 1210 may include any general-purpose processor and hardware or software services (e.g., service 1 1232, service 2 1234, and service 3 1236) stored in storage device 1230 and configured to control the processor 1210 and dedicated processors, wherein software instructions are incorporated into the processor design. Processor 1210 can be a self-contained system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors can be symmetric or asymmetric.

[0253] To enable users to interact with the computing device architecture 1200, input device 1245 can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice input, etc. Output device 1235 can also be one or more of a variety of output mechanisms known to those skilled in the art, such as a display, projector, television, speaker device. In some examples, multi-mode computing devices allow users to provide multiple types of input to communicate with computing device architecture 1200. Communication interface 1240 typically governs and manages user input and computing device output. There are no limitations on operation on any specific hardware arrangement, and therefore the basic features herein can be readily replaced by improved hardware or firmware arrangements developed by those skilled in the art.

[0254] Storage device 1230 is a non-volatile memory and may be a hard disk or other type of computer-readable medium capable of storing computer-accessible data, such as magnetic tape, flash memory cards, solid-state storage devices, digital multifunction disks, audio tapes, random access memory (RAM) 1225, read-only memory (ROM) 1220, and combinations thereof. Storage device 1230 may include services 1232, 1234, and 1236 for controlling processor 1210. Other hardware or software modules are also envisioned. Storage device 1230 may be connected to computing device connection 1205. In one aspect, a hardware module performing a specific function may include software components stored in a computer-readable medium, which are connected to the necessary hardware components (such as processor 1210, connection 1205, output device 1235, etc.) to perform that function.

[0255] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying one or more instructions and / or data. Computer-readable media may include non-transitory media on which data can be stored, but does not include carrier waves and / or transient electronic signals propagated wirelessly or via a wired connection. Examples of non-transitory media may include, but are not limited to, magnetic disks or magnetic tapes, optical storage media such as CDs or DVDs, flash memory, memory, or storage devices. Computer-readable media may store code and / or machine-executable instructions thereon, which may represent any combination of procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. Code segments may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, variables, parameters, or memory contents. Information, variables, parameters, data, etc., may be passed, forwarded, or transmitted by any suitable means, including memory sharing, messaging, token passing, network transmission, etc.

[0256] In some embodiments, computer-readable storage devices, media, and memories may include wired or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly exclude media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0257] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, those skilled in the art will understand that these embodiments can be practiced without these specific details. For clarity of explanation, in some examples, the technology may be presented as comprising individual functional blocks, including devices, device components, steps or routines in methods realized in software or a combination of hardware and software. Additional components may be used in addition to those shown in the figures and / or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form to avoid obscuring the embodiments with unnecessary details. In other examples, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details to avoid obscuring the embodiments.

[0258] Individual embodiments may be described above as processes or methods, wherein these processes or methods are depicted as flowcharts, diagrams, data flow diagrams, structural diagrams, or block diagrams. Although a flowchart can describe operations as a sequence of processes, many operations can be performed in parallel or simultaneously. Furthermore, the order of operations can be rearranged. When the operations of a process are completed, the process terminates, but there may be additional steps not included in the diagram. A process may correspond to a method, function, procedure, subroutine, subroutine, etc. When a process corresponds to a method, its termination may correspond to the function returning to the calling function or the main function.

[0259] The processes and methods described in the examples above can be implemented using computer-executable instructions stored in or otherwise available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, special-purpose computer, or processing device to perform certain functions or groups of functions. Some of the computer resources used may be accessible via a network. The computer-executable instructions may be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer-readable media that may be used to store instructions, information, and / or information created during the methods according to the described examples include hard disks or optical disks, flash memory, USB devices equipped with non-volatile memory, network storage devices, etc.

[0260] Devices implementing the processes and methods disclosed herein may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) that perform the necessary tasks may be stored in a computer-readable or machine-readable medium. One or more processors may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, etc. The functionality described herein may also be realized in peripheral devices or add-in cards. As a further example, such functionality may also be implemented on a circuit board in different chips or different processes running in a single device.

[0261] Instructions, media for transmitting such instructions, computing resources for running them, and other structures for supporting such computing resources are example components for providing the functionality described in this disclosure.

[0262] In the foregoing description, various aspects of this application have been described with reference to specific embodiments thereof; however, those skilled in the art will recognize that this application is not limited thereto. Therefore, although illustrative embodiments of this application have been described in detail herein, it should be understood that the concepts of the invention can be embodied and adopted in other diverse ways, and the appended claims are intended to be construed as including such variations, other than those limited by the prior art. Various features and aspects of the above-described applications can be used individually or in combination. Furthermore, embodiments can be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of this specification. Accordingly, the specification and drawings should be considered illustrative rather than restrictive. For illustrative purposes, methods are described in a specific order. It should be understood that in alternative embodiments, these methods may be performed in a different order than described.

[0263] Those skilled in the art will understand that, without departing from the scope of this specification, the less than ("<") and greater than (">") symbols or technical terms used herein can be replaced by the less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively.

[0264] When a component is described as being "configured" to perform certain operations, such configuration can be accomplished, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., microprocessors or other suitable electronic circuits), or any combination thereof.

[0265] The phrase "coupled to" means any component that is physically connected, directly or indirectly, to another component, and / or any component that communicates, directly or indirectly, with another component (e.g., via a wired or wireless connection and / or other suitable communication interface).

[0266] The language of the claims or other languages ​​containing "at least one" and / or "one or more" in the set of statements indicate that one or more members of the set (in any combination) satisfy the claim. For example, the claim language containing "at least one of A and B" indicates A, B, or A and B. In another example, the claim language containing "at least one of A, B, and C" indicates A, B, C, or A and B, or A and C, or B and C, or A and B and C. For example, the claim language containing "at least one of A and B" can indicate A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0267] The various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability between hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described above in accordance with their functions. Whether this function is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art can implement the described functions in different ways for each specific application; however, such implementation decisions should not be construed as departing from the scope of this application.

[0268] The techniques described herein can also be implemented in real time in electronic hardware, computer software, firmware, or any combination thereof. This technology can be implemented in any of a variety of devices, such as general-purpose computers, handheld receivers for wireless communication devices, or multi-purpose integrated circuit devices, including applications in handheld receivers and other devices for wireless communication devices. Any feature described as a module or component can be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the technology can be implemented at least in part by a computer-readable data storage medium comprising program code including instructions that, when run, perform one or more of the methods described above. The computer-readable data storage medium can form part of a computer program product, which may include packaging material. The computer-readable medium may include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, these technologies may be implemented at least in part by a computer-readable communication medium that carries program code in the form of communication instructions or data structures and can be accessed, read, and / or executed by a computer, such as a propagating signal or wave.

[0269] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor can be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, a combination of one or more microprocessors with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or apparatus suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein may be provided in dedicated software or hardware modules configured for encoding and decoding, or incorporated into a combined video encoder-decoder (codec).

[0270] The illustrative aspects of this disclosure include:

[0271] Aspect 1: A method for processing media content, the method comprising: acquiring media information and buffer information of a media object determined from a scene description of at least one media scene, wherein the media information identifies a desired format for storing the media object in one or more output buffers of a media pipeline, and wherein the buffer information includes one or more attributes of the one or more output buffers; initializing a media pipeline of the media object based on the media information and the buffer information; acquiring a media object of a delivery format; generating one or more processed media objects at least partially by processing the acquired media objects, wherein each of the one or more processed media objects is converted from the delivery format to the desired format to store each processed media object in an output buffer of one or more output buffers; and outputting the one or more processed media objects to one or more output buffers using the initialized media pipeline.

[0272] Aspect 2: The method of aspect 1, wherein media information and buffer information are provided as parameters of a Media Access Function (MAF) application programming interface (API) call to a media object describing a scene.

[0273] Aspect 3: The method of any of Aspects 1 or 2, wherein the media information includes one or more source locations for obtaining the media object.

[0274] Aspect 4: The method of any of Aspects 1 to 3, wherein the media information identifier is used to replay one or more rendering parameters of the media object from one or more output buffers.

[0275] Aspect 5: The method of aspect 2, wherein one or more calls to the MAF API are used to initialize the media pipeline.

[0276] Aspect 6: The method of aspect 5, wherein the media object is obtained from the source location selected based on view information determined from one or more MAF API calls.

[0277] Aspect 7: The method of any of Aspects 5 or 6, wherein the view information determined from one or more MAF API calls includes the position of the media object relative to the viewer's position.

[0278] Aspect 8: The method of any of Aspects 5 to 7 further includes: configuring source positioning based on view information; wherein the configuration includes adjusting the level of detail of the acquired media object, adjusting the bit rate of the acquired media object, and adjusting one or more of the visible portion of the acquired media object.

[0279] Aspect 9: The method of any of Aspects 5 to 8 further includes, at least in part, controlling each of the one or more output buffers by passing buffer information to a buffer application programming interface (API) and using one or more calls to the buffer API.

[0280] Aspect 10: The method of any of Aspects 1 to 9, wherein initializing the media pipeline includes allocating a new buffer in response to determining that a buffer identifier does not belong to an existing buffer, the allocation being based on one or more buffer attributes associated with the buffer identifier.

[0281] Aspect 11: The method of aspect 10, wherein the buffer information includes a buffer list, each entry of which includes a respective buffer identifier and is associated with one or more buffer attributes.

[0282] Aspect 12: The method of any one of Aspects 1 to 11, wherein one or more output buffers include at least one cyclic buffer.

[0283] Aspect 13: The method of any of Aspects 1 to 12, wherein the scene description includes a Graphical Language Transmission Format (glTF) scene description.

[0284] Aspect 14: The method of any of Aspects 1 to 13, wherein at least one media scene includes one or more immersive media objects, the one or more immersive media objects including at least one of a three-dimensional (3D) mesh object, a 3D point cloud object, a virtual reality (VR) object, an augmented reality (AR) object, and a 6-degree-of-freedom (DoF) object.

[0285] Aspect 15: The method of any of Aspects 1 to 14, wherein the media object is obtained from a source location in a delivery format, the source location being selected from one or more source locations identified in the media information.

[0286] Aspect 16: The method of aspect 15, wherein one or more source locations are identified in the media information using at least one of a Uniform Resource Locator (URL), a Uniform Resource Identifier (URI), and a local file path.

[0287] Aspect 17: The method of any of Aspects 15 or 16, wherein one or more source locations identified in the media information include a default media source location and one or more alternative media source locations that are different from the default media source location.

[0288] Aspect 18: The method of any of Aspects 15 to 17, wherein the selected source location is determined based on one or more selection factors identified in the media information, including one or more of playback preferences, playback capabilities, and network characteristics.

[0289] Aspect 19: An apparatus for processing media content, comprising: a memory; and a processor coupled to the memory, the processor being configured to: acquire media information and buffer information of a media object determined from a scene description of at least one media scene, wherein the media information identifies an expected format for storing the media object in one or more output buffers of a media pipeline, and wherein the buffer information includes one or more attributes of the one or more output buffers; initialize a media pipeline of the media object based on the media information and the buffer information; acquire a media object of a delivery format; generate one or more processed media objects at least partially by processing the acquired media objects, wherein each of the one or more processed media objects is converted from the delivery format to the expected format to store each processed media object in an output buffer of one or more output buffers; and output the one or more processed media objects to one or more output buffers using the initialized media pipeline.

[0290] Aspect 20: The apparatus of aspect 19, wherein media information and buffer information are provided as parameters of a Media Access Function (MAF) application programming interface (API) call to a media object describing a scene.

[0291] Aspect 21: An apparatus as described in any of Aspects 19 or 20, wherein the media information includes one or more source locations for obtaining a media object.

[0292] Aspect 22: An apparatus as described in any of Aspects 19 to 21, wherein media information identifiers are used to replay one or more rendering parameters of a media object from one or more output buffers.

[0293] Aspect 23: The apparatus of aspect 20, wherein one or more calls to the MAF API are used to initialize the media pipeline.

[0294] Aspect 24: The apparatus of aspect 23, wherein the media object is obtained from a source location selected based on view information determined from one or more MAF API calls.

[0295] Aspect 25: The apparatus of aspect 24, wherein the view information determined from one or more MAF API calls includes the position of a media object relative to the viewer's position.

[0296] Aspect 26: An apparatus as in any of Aspects 24 or 25, wherein the processor is configured to: configure source positioning based on view information; wherein the configuration includes adjusting the level of detail of the acquired media object, adjusting the bit rate of the acquired media object, and adjusting one or more of the visible portion of the acquired media object.

[0297] Aspect 27: An apparatus of any of Aspects 23 to 26, wherein the processor is configured to control each of one or more output buffers at least in part by passing buffer information to a buffer application programming interface (API) and by using one or more calls to the buffer API.

[0298] Aspect 28: An apparatus of any of Aspects 19 to 27, wherein, in order to initialize a media pipeline, the processor is configured to allocate a new buffer in response to determining that a buffer identifier does not belong to an existing buffer, the allocation being based on one or more buffer attributes associated with the buffer identifier.

[0299] Aspect 29: The apparatus of aspect 28, wherein the buffer information includes a buffer list, each entry of which includes a respective buffer identifier and is associated with one or more buffer attributes.

[0300] Aspect 30: An apparatus of any one of aspects 19 to 29, wherein one or more output buffers include at least one cyclic buffer.

[0301] Aspect 31: An apparatus as described in any of Aspects 19 to 30, wherein the scene description includes a Graphical Language Transmission Format (glTF) scene description.

[0302] Aspect 32: A device of any of Aspects 19 to 31, wherein at least one media scene includes one or more immersive media objects, the one or more immersive media objects including at least one of a three-dimensional (3D) mesh object, a 3D point cloud object, a virtual reality (VR) object, an augmented reality (AR) object, and a 6-degree-of-freedom (DoF) object.

[0303] Aspect 33: An apparatus of any of Aspects 19 to 32, wherein the media object is obtained from a source location in a delivery format, the source location being selected from one or more source locations identified in the media information.

[0304] Aspect 34: The apparatus of aspect 33, wherein one or more source locations are identified in media information using at least one of a Uniform Resource Locator (URL), a Uniform Resource Identifier (URI), and a local file path.

[0305] Aspect 35: A device as described in any of Aspects 33 or 34, wherein one or more source locations identified in the media information include a default media source location and one or more alternative media source locations that are different from the default media source location.

[0306] Aspect 36: An apparatus of any of Aspects 33 to 35, wherein the selected source location is determined based on one or more selection factors identified in the media information, the one or more selection factors including one or more of playback preferences, playback capabilities, and network characteristics.

[0307] Aspect 37: A non-transitory computer-readable storage medium having instructions stored thereon, which, when run by one or more processors, cause the one or more processors to perform any of the operations described in aspects 1 to 36.

[0308] Aspect 38: An apparatus comprising components for performing operations as described in any of aspects 1 to 36.< / video>

Claims

1. A method for processing media content, the method comprising: Obtain media information and buffer information of a media object determined from a scene description of at least one media scene, wherein the media information identifies the expected format for storing the media object in one or more output buffers of a media pipeline, wherein the buffer information includes one or more attributes of the one or more output buffers, and wherein the media information and the buffer information are provided as parameters for a Media Access Function (MAF) application programming interface (API) call to the media object of the scene description. The media pipeline of the media object is initialized based on the media information and the buffer information, wherein one or more calls to the MAF API are used to initialize the media pipeline; Obtain the media object in the delivery format; One or more processed media objects are generated, at least in part, by processing the acquired media objects, wherein each of the one or more processed media objects is converted from the delivery format to the expected format, and each processed media object is stored in an output buffer of the one or more output buffers; as well as The one or more processed media objects are output to the one or more output buffers using an initialized media pipeline, wherein the one or more output buffers are controlled at least in part by passing buffer information to the buffer application programming interface API and by using one or more calls to the buffer API.

2. The method of claim 1, wherein the media information includes one or more source locations for obtaining the media object.

3. The method of claim 1, wherein the media information identifier is used to replay one or more rendering parameters of the media object from the one or more output buffers.

4. The method of claim 1, wherein the media object is obtained from a source location selected based on view information determined from the one or more MAF API calls.

5. The method of claim 4, wherein the view information determined from the one or more MAF API calls includes the position of the media object relative to the viewer's position.

6. The method of claim 4, further comprising: Configure the source location based on the view information; The configuration includes adjusting one or more of the following: adjusting the level of detail of the acquired media object, adjusting the bit rate of the acquired media object, and adjusting the visible portion of the acquired media object.

7. The method of claim 1, wherein initializing the media pipeline includes allocating a new buffer in response to determining that a buffer identifier does not belong to an existing buffer, the allocation being based on one or more buffer attributes associated with the buffer identifier.

8. The method of claim 7, wherein the buffer information includes a buffer list, each entry in the buffer list including a respective buffer identifier, and each entry is associated with one or more buffer attributes.

9. The method of claim 1, wherein the one or more output buffers include at least one cyclic buffer.

10. The method of claim 1, wherein the scene description includes a scene description in the graphical language transmission format glTF.

11. The method of claim 1, wherein the at least one media scene comprises one or more immersive media objects, the one or more immersive media objects comprising at least one of a 3D mesh object, a 3D point cloud object, a virtual reality (VR) object, an augmented reality (AR) object, and a 6-DOF (DoF) object.

12. The method of claim 1, wherein the media object is obtained from a source location in the delivery format, the source location being selected from one or more source locations identified in the media information.

13. The method of claim 12, wherein at least one of a Uniform Resource Locator URL, a Uniform Resource Identifier URI, and a local file path is used to identify the one or more source locations in the media information.

14. The method of claim 12, wherein the one or more source locations identified in the media information include a default media source location and one or more alternative media source locations different from the default media source location.

15. The method of claim 12, wherein the selected source location is determined based on one or more selection factors identified in the media information, the one or more selection factors including one or more of playback preferences, playback capabilities, and network characteristics.

16. An apparatus for processing media content, comprising: Memory; as well as A processor coupled to the memory, the processor being configured to: Obtain media information and buffer information of a media object determined from a scene description of at least one media scene, wherein the media information identifies the expected format for storing the media object in one or more output buffers of a media pipeline, wherein the buffer information includes one or more attributes of the one or more output buffers, and wherein the media information and the buffer information are provided as parameters for a Media Access Function (MAF) application programming interface (API) call to the media object of the scene description. The media pipeline of the media object is initialized based on the media information and the buffer information, wherein one or more calls to the MAF API are used to initialize the media pipeline; Obtain the media object in the delivery format; One or more processed media objects are generated, at least in part, by processing the acquired media objects, wherein each of the one or more processed media objects is converted from the delivery format to the expected format, and each processed media object is stored in an output buffer of the one or more output buffers; as well as The one or more processed media objects are output to the one or more output buffers using an initialized media pipeline, wherein the one or more output buffers are controlled at least in part by passing buffer information to the buffer application programming interface API and by using one or more calls to the buffer API.

17. The apparatus of claim 16, wherein the media information includes one or more source locations for obtaining the media object.

18. The apparatus of claim 16, wherein the media information identifier is used to replay one or more rendering parameters of the media object from the one or more output buffers.

19. The apparatus of claim 16, wherein the media object is obtained from a source location selected based on view information determined from the one or more MAF API calls.

20. The apparatus of claim 19, wherein the view information determined from the one or more MAF API calls includes the position of the media object relative to the viewer's position.

21. The apparatus of claim 19, wherein the processor is configured to: Configure the source location based on the view information; The configuration includes adjusting one or more of the following: adjusting the level of detail of the acquired media object, adjusting the bit rate of the acquired media object, and adjusting the visible portion of the acquired media object.

22. The apparatus of claim 16, wherein, in order to initialize the media pipeline, the processor is configured to allocate a new buffer in response to determining that a buffer identifier does not belong to an existing buffer, the allocation being based on one or more buffer attributes associated with the buffer identifier.

23. The apparatus of claim 22, wherein the buffer information comprises a buffer list, each entry in the buffer list comprising a respective buffer identifier, and each entry being associated with one or more buffer attributes.

24. The apparatus of claim 16, wherein the one or more output buffers include at least one cyclic buffer.

25. The apparatus of claim 16, wherein the scene description includes a graphical language transmission format glTF scene description.

26. The apparatus of claim 16, wherein the at least one media scene comprises one or more immersive media objects, the one or more immersive media objects comprising at least one of a 3D mesh object, a 3D point cloud object, a virtual reality (VR) object, an augmented reality (AR) object, and a 6-DOF (DoF) object.

27. The apparatus of claim 16, wherein the media object is obtained from a source location in the delivery format, the source location being selected from one or more source locations identified in the media information.

28. The apparatus of claim 27, wherein the one or more source locations are identified in the media information using at least one of a Uniform Resource Locator (URL), a Uniform Resource Identifier (URI), and a local file path.

29. The apparatus of claim 27, wherein the one or more source locations identified in the media information include a default media source location and one or more alternative media source locations different from the default media source location.

30. The apparatus of claim 27, wherein the selected source location is determined based on one or more selection factors identified in the media information, the one or more selection factors including one or more of playback preferences, playback capabilities, and network characteristics.

31. An apparatus for processing media content, comprising components for the steps of the method according to any one of claims 1-15.

32. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1-15.

33. A program product containing stored instructions, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1-15.

Images