Data Structures for Multimedia Applications

Abstract

The embodiments described herein provide a unified container format for delivering different multimedia applications. An embodiment provides a data structure utilized to implement a plurality of multimedia applications. The data structure includes a first metadata level including low level metadata used to perform an operation related to media data in a bitstream. The data structure includes a second metadata level including mid level metadata used to apply the operation metadata to render the media data. The data structure includes a third metadata level including higher level metadata used to utilize the low level metadata and the mid level metadata to deliver the plurality of multimedia applications. The first metadata level further includes synchronization metadata for transforming the media data, the low level metadata, the mid level metadata, and the higher level metadata from a first multimedia application of the plurality of multimedia applications to a second multimedia application of the plurality of multimedia applications.

Description

[Technical field]

[0001] 1. CROSS-REFERENCE TO RELATED APPLICATIONS This application represents a national stage entry of PCT application PCT / US22 / 53418, filed on December 19, 2022, and claims the benefit of priority from the following priority applications: U.S. Provisional Application No. 63 / 301,467, filed on January 20, 2022, and No. 63 / 399,871, filed on August 22, 2022, and EP Application No. 22155345.6, filed on February 7, 2022.

[0002] 2. Field of disclosure The present application relates generally to a data structure for implementing a plurality of multimedia applications, and a controller for implementing the data structure. [Background technology]

[0003] 3. Background Multimedia experiences such as imaging and video applications utilize container formats (e.g., metafile, file formats) that allow multiple data streams to be embedded in a single file. These container formats contain metadata to identify the data streams and detail how the functions of the data streams are implemented. Containers include the High Efficiency Image File Format (HEIF), which uses a video codec to encode images using intraframe coding, and the Joint Photographic Experts Group (JPEG) format (also known as EXIF ​​and JFIF), which utilizes application segment markers (APP markers) to store information. However, there is an increasing need for container formats for multiple multimedia applications in high dynamic range (HDR) image quality. Summary of the Invention [Means for solving the problem]

[0004] The embodiments described herein provide a unified container format for delivering different multimedia experiences. The multimedia experiences include, for example, a still photo application, a video application, a live photo application, and the like. In addition, the different multimedia experiences may be different versions of the same application (e.g., an original photo application and an updated photo application). The unified container format is backward compatible with existing formats. Multiple experiences are encapsulated into one single bitstream. Using the unified container format, a playback system can determine which application to run depending on computational resources, device capabilities, and / or user preferences. Additionally, interference and dependencies between different multimedia applications should be minimized.

[0005] In one exemplary aspect of the present disclosure, a data structure is provided for implementing a plurality of multimedia applications. The data structure includes a first metadata level including low level metadata used to perform operations related to media data in a bitstream. The data structure includes a second metadata level including mid level metadata used to apply the low level metadata to render the media data. The data structure includes a third metadata level including higher level metadata used to utilize the low level metadata and the mid level metadata to deliver the plurality of multimedia applications. The first metadata level further includes synchronization metadata for transforming the media data, the low level metadata, the mid level metadata, and the higher level metadata from a first multimedia application of the plurality of multimedia applications to a second multimedia application of the plurality of multimedia applications.

[0006] Thus, various aspects of the present disclosure provide for the implementation of multimedia applications, such as photographic and video applications having high dynamic range and high or standard resolution, and provide improvements in at least the fields of image projection, signal processing, and image display. [Brief description of the drawings]

[0007] These and other more detailed and specific features of the various embodiments are more fully disclosed in the following description, which refers to the accompanying drawings.

[0008] [Figure 1] 1 illustrates an exemplary data structure for implementing a multimedia experience.

[0009] [Diagram 2] 1 illustrates exemplary playback operations for a multimedia experience.

[0010] [Figure 3A] 1 illustrates exemplary in-experience editing operations for a multimedia experience.

[0011] [Figure 3B] 1 illustrates exemplary cross-experience editing operations for a multimedia experience.

[0012] [Figure 4A] 1 illustrates an example in-experience transformation operation for a multimedia experience.

[0013] [Figure 4B] 1 illustrates an example inter-experience conversion operation for a multimedia experience.

[0014] [Diagram 5] 1 illustrates an example of a base media file format according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] The present disclosure and aspects thereof may be embodied in various forms, including hardware, devices or circuits controlled by computer-implemented methods, computer program products, computer systems and networks, user interfaces, and application programming interfaces; as well as hardware-implemented methods, signal processing circuits, memory arrays, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. The foregoing is intended only to give a general idea of ​​various aspects of the present disclosure and is in no way intended to limit the scope of the present disclosure.

[0016] Additionally, while this disclosure will primarily focus on examples in which the various circuits are used in digital projection systems, it will be understood that these are merely examples. The disclosed systems and methods may be implemented in display devices having OLED displays, LCD displays, quantum dot displays, etc. It should be further understood that the disclosed systems and methods may be used in any device that needs to project light, such as cinema, consumer, and other commercial projection systems, heads-up displays, virtual reality displays, etc.

[0017] 3-Level Imaging Format 1 provides a three-level data structure 100 (e.g., metadata architecture, container, file structure) for implementing a multimedia experience (e.g., a multimedia application). The data structure 100 includes low-level metadata 102, mid-level metadata 104, and high-level metadata 106. The low-level metadata 102 and mid-level metadata 104 serve as building blocks that are implemented by the high-level metadata 106 that comprises the multimedia experience.

[0018] The low-level metadata 102 consists of basic metadata for describing how to perform operations on standard media data in a bitstream. The low-level metadata 102 includes Prediction Metadata (PM) 110, Display Management Metadata (DM) 112, Photo Metadata (OM) 114, Immersive Metadata (IM) 116, Exchangeable Image File Format (EXIF) / International Color Consortium (ICC) Metadata (EM) 118, Synchronization Metadata (SM) 120, and Null Metadata (NM) 122. In some implementations, the low-level metadata 102 may include additional metadata used to implement a multimedia experience.

[0019] The predictive metadata 110 includes polynomial prediction, multivariate multiple regression (MMR) prediction, and tensor product B-spline (TPB) prediction. The display management metadata 112 includes different levels of metadata syntax ranging from L0 to L255. The photo metadata 114 is used for new still images and may have the same L1 syntax as the display management metadata 112. The EXIF / ICC metadata 118 includes camera image related metadata such as thumbnail information. The immersive metadata 116 includes object maps, depth maps, and response curves. The null metadata 122 is reserved and used when baseline media data is needed without additional metadata. Some metadata groups may include some of the same or similar syntax. Additionally, in the decoder, a dispatcher component may convert one syntax in one metadata group to another syntax in another metadata group, reusing existing code and hardware and reducing code size.

[0020] Sync metadata 120 is structured similarly to display management metadata 112. Each level of sync metadata 120 specifies a source experience, a sink experience, and a designed behavior. The table below provides an example syntax; however, other syntaxes may be used. In some implementations, sync metadata 120 is updated and edited using a variety of user-accessible tools.

[0021] The payload header indicates the number of extension blocks to include in the synchronization metadata 120, an example of which is shown in Table 1. [Table 1]

[0022] Each extension block contains a simple header indicating its length, its level, and a container that contains its own payload, an example of which is shown in Table 2. [Table 2]

[0023] An example of the payload format itself is provided in Table 3 below. [Table 3]

[0024] The source_experience_id and sink_experience_id shown in Table 3 may be defined and mapped at a higher level. The operation_value metadata in Table 3 is a placeholder for various usages such as:

[0025] Temporal Synchronization: When editing a video in a video application, the corresponding still photo may be updated in a photo application. In such cases, the operation_value metadata may be a frame index or a timestamp.

[0026] Spatial resolution: Different multimedia experiences may have different scales for the same attribute. For example, a still photo often has a larger resolution than a video. The operation_value metadata can define the resolution mapping between different experiences.

[0027] Propagation options / intensity: The synchronization metadata 120 may indicate whether editing or conversion of data between different multimedia experiences is allowed and the corresponding "intensity" within or between experiences. For example, the operation_value metadata may be a simple binary flag to indicate whether a workflow from standard dynamic range (SDR) photos to HDR video is allowed. If a workflow from SDR photos to HDR video is allowed, the operation_value metadata may define which type of SDR-HDR conversion algorithm and their corresponding parameters should be used. For example, which TPB enum to choose may be specified so that backward reshaping from SDR to HDR with different color volumes can be specified. Furthermore, in the case of HDR video to SDR photos, the intensity may indicate which TPB enum to choose and use to perform forward reshaping to generate SDR photos.

[0028] Metadata Transformation: Synchronization metadata 120 may indicate how to transform metadata from one multimedia experience to another, such as from a photo application to a video application or vice versa.

[0029] Auxiliary Information: Synchronization metadata 120 may indicate necessary metadata for performing actions related to the multimedia experience.

[0030] When multiple multimedia experiences are provided in a single bitstream, multiple enhancement blocks with the same level may be used, for example:

[0031] Multiple sinks: A transformation from a first experience A to a second experience B may affect a third experience C. In such a scenario, two extension blocks with the same level may indicate an operation. For example, a bitstream includes SDR photos, HDR video, and immersive HDR / SDR photos. Updating the HDR video affects both SDR photos and immersive HDR / SDR photos.

[0032] Multiple Sources: The bitstream can include SDR photos, HDR videos, and immersive HDR photos. One application can be converted to another application. For example, SDR photos can be converted from HDR videos, and SDR photos can be converted from immersive HDR photos.

[0033] The mid-level metadata 104 describes which media data to provide as input to the processing modules and how to apply operation metadata to render the media data. In other words, the mid-level metadata 104 consists of the media data bitstream and the corresponding required low-level metadata 102 for the rendering to be used. The media data may be packed together with the experience metadata in the bitstream or may be stored in different locations in the same bitstream. In some implementations, storing the media data in different locations may enable format backward compatibility.

[0034] Each multimedia experience may include zero, one, two, or more low level metadata 102, and different low level metadata 102 may be encapsulated in rpu_type formats. In some implementations, there are 64 rpu_types available for allocation. In the illustrated example, the mid level metadata 104 includes rpu_type_2 130, rpu_type_4 132, rpu_type_5 134, and rpu_type_63 136. rpu_type_2 130 includes media data as a base layer bitstream and two lower level metadata, namely prediction metadata 110 and display management metadata 112. rpu_type_4 132 includes picture metadata 114. rpu_type_5 134 includes immersive metadata 116. rpu_type_63 136 may be reserved for TPB payload.

[0035] The high level metadata 106 includes several applications that utilize the mid level metadata 104 and the synchronization metadata 120, such as a first application 150, a second application 152, and a third application 154, each application being a different multimedia experience (photo experience, video experience, SDR photo experience, HDR photo experience, SDR video experience, HDR video experience, immersive SDR / HDR video experience, etc.). Multiple experiences of the same type may be present in the same bitstream. Each experience may also include side information indicating whether the media data and metadata are packed together or located in different locations.

[0036] The playback system decides what to play according to playback computational resources, device capabilities, and end-user preferences. The playback computational resources indicate what level of experience the system can support. Device capabilities include whether the end device (e.g., output device) can display HDR images and videos, the sensors available on the end device, and whether the end device has a touch screen. The end-user preferences include what experience the user has requested or prefers to be rendered.

[0037] In some embodiments, the high-level metadata 106 includes a default presentation application 156 (e.g., a raw application). Alternatively, the presentation mode (or application) may be selected by the user. In some embodiments, the bitstream may be encapsulated in an existing container that has the format of the default presentation application 156. For example, a photo experience may include JPEG thumbnails as the default presentation without being associated with any low-level metadata 102 or mid-level metadata 104. However, the high-level metadata 106 may use the synchronization metadata 120 to describe the mechanism between different applications that have a default presentation. In other embodiments, the bitstream is encapsulated in a container that does not have a default presentation. In such embodiments, the required application is activated by user or environmental input.

[0038] Media playback A device implementing the data structure 100 may play different experiences depending on the computational resources, display capabilities, sensor availability, and user preferences available to the device. For example, FIG. 2 provides a playback operation 200 for a multimedia experience, such as a second application 152. The second application 152 may be, for example, a photo or image experience. When the second application 152 is stored in a legacy container, such as a JPEG container (e.g., as defined by the JPEG standard, EXIF ​​and JFIF file formats, etc.), and the legacy container file is passed to an end device, only JPEG SDR still images are provided.

[0039] However, if a second application 152 is provided to an end device implementing the data structure 100, the media data can be viewed in SDR or HDR still images using photo playback 204 implementing rpu_type_4 132, or as HDR live photos (e.g., video) using the predictive metadata 110 and display management metadata 112 included in rpu_type_2 130 implemented by video playback 202. Whether the media data is viewed using video playback 202 or photo playback 204 is based on a user preference or selection.

[0040] MediaEdit The media data may be edited to modify the experience defined by its rpu_type (e.g., its color grading, its resolution, its frame rate, etc.). Figure 3A shows an example intra-experience editing operation 300 that allows each experience (e.g., each rpu_type) to perform edits without affecting another experience (e.g., another rpu_type). Figure 3B shows an example cross-experience editing operation 350 in which edits in one experience are propagated to other experiences. Whether to perform intra-experience editing operation 300 or cross-experience editing operation 350 and how to propagate between experiences is defined by synchronization metadata 120.

[0041] The in-experience editing operations 300 of FIG. 3A allow editing in one experience (e.g., the second application 152) using an editing tool 310. The edits may be propagated to other experiences, but the media data and its corresponding metadata are edited in a single experience. The in-experience editing operations 300 of FIG. 3A may perform several different operations based on whether the edits in one experience are automatically reflected in the other experiences. The operations may include individually editing in one experience without updating the other experiences, editing one experience and automatically updating the other experiences with the edits, and editing one experience and optionally updating the other experiences with the edits. The edit updates are controlled by the synchronization metadata 120.

[0042] 3A , the second application 152 includes rpu_type_m 315, rpu_type_n 320, and sync metadata 120. An rpu_type_m edit module 335 performs any edits on rpu_type_m 315. An rpu_type_n edit module 330 performs any edits on rpu_type_n 320. An sync metadata update module 325 updates the sync metadata 120.

[0043] As an example of an in-experience editing operation 300, the synchronization metadata 120 controls the propagation of edits between the SDR still image and the HDR still image. In some embodiments, the SDR still image is edited independently of the HDR still image without updating the HDR still image. In some embodiments, editing the HDR still image automatically updates the SDR still image. In some embodiments, the SDR still image is edited and the HDR still image is updated conditionally using a backward revertible TPB to upconvert SDR to HDR. When edits to the SDR still image include new content such as graphics and text, object-based (e.g., masking) side information may be used. The added information may be upconverted differently and then blended in the HDR domain.

[0044] Cross-experience editing operations 350 allow several experiences to be edited collaboratively using editing tools 360. For example, in some embodiments, one experience with references from other experiences is edited and the edits automatically update the other experiences. In other embodiments, one experience with references from other experiences is edited and the edits optionally update the other experiences. Edit updates are controlled by synchronization metadata 120.

[0045] 3B example, the second application 152 includes rpu_type_m 315, rpu_type_n 320, and sync metadata 120. An edit rpu_type_m module 375 performs any edits to rpu_type_m 315. An update sync metadata module 365 updates sync metadata 120. The changes are then propagated to an update rpu_type_n module 370, which updates rpu_type_n 320.

[0046] As an example of a cross-experience editing operation 350, the synchronization metadata 120 controls the propagation of edits between an SDR still image and an HDR live photo (e.g., video). In some embodiments, the HDR live photo is edited and the SDR still photo is automatically updated. In such embodiments, the metadata indicates which frame in the HDR video to edit. In other embodiments, the SDR still image is edited and the HDR live photo is conditionally updated using a backward-revertible TPB. In such embodiments, the inverse metadata indicates how to propagate the edits, such as cropping, resizing, and tone mapping.

[0047] Experience conversion Experience conversion involves converting one experience to another and converting content between profiles and levels, even within the same experience. Figure 4A illustrates an example intra-experience conversion operation 400. Figure 4B illustrates an example inter-experience conversion operation 450.

[0048] FIG. 4A includes application x 405, application y 410, and conversion tool 415. The conversion tool 415 converts from application x 405 to application y 410. Both application x 405 and application y 410 may be different profiles (or versions) of the same or similar experience, such as converting from one photo experience to another. The intra-experience conversion operations 400 may include media data conversion, in which media data, such as a High Efficiency Video Coding (HEVC) encoded bitstream, is decoded into a common workspace and encoded into another workspace for conversion. Different settings, such as resolution, color space, frame rate, SEI, and VUI, may be converted. This function is performed by intra-type conversion rpu_type_m module 430, which converts rpu_type_m 420 from application x 405 to application y 410. The intra-experience conversion operations 400 may also include metadata conversion, in which a synchronization metadata update tool 435 updates metadata, such as synchronization metadata 120, according to the conversion. For example, the presentation management metadata 112 may be transformed into new content.

[0049] FIG. 4B includes application x 455, application y 460, and conversion tool 465. Conversion tool 465 converts from application x 455 to application y 460. Application x 455 may be, for example, a video experience, and application y 460 may be, for example, a photo experience. Inter-experience conversion operations 450 may include media data conversion, where media data is converted from one type of media to another type of video. For example, media data may be converted from photo to video or vice versa. Media data conversion may be performed by rpu_type_m to rpu_type_n type conversion module 480, which converts rpu_type_m 470 to rpu_type_n 485. Inter-experience conversion operations 450 may also include metadata conversion, where metadata of one type is converted to metadata of another type. For example, photo metadata 114 may be converted to video metadata (included in display management metadata 112), or vice versa. In some embodiments, predicted metadata 110 is converted. The update sync metadata block 490 can update the sync metadata 120 .

[0050] Container Implementation The data structure 100 may be integrated into existing containers such as HEIF or JPEG. As an example, HEIF uses a video codec to encode images using intraframe coding. The syntax of HEIF is based on the ISO BMFF (Base Media File Format). ISO BMFF uses "boxes" to structure different categories of data. Each box is preceded by a four-character type (4CC). Boxes may be nested and hierarchical (e.g., a box inside another box). Each box has a size, which is an integer that specifies the number of bytes in the box. Each box also has a box type, such as compact.

[0051] Two different definitions are used to separate still images from video. First, still images are stored as items. All image items are independently coded and do not depend on other items for their decoding. Any number of image items may be included in the same file. Image sequences (e.g. videos) are stored as tracks. Image sequence tracks are used when there are coding dependencies between images or when the playback of images is timed. Unlike video tracks, timing in image sequence tracks is advisory.

[0052] A HEIF bitstream contains two main components: metadata with 4CC as "meta" and media data with 4CC as media data box "mdat" or item data box "idat". The metadata part represents side information (e.g. the structure of a multimedia application) and the media data stores the multimedia data.

[0053] The components of the data structure 100 may be implemented as boxes in a HEIF bitstream. For example, the low-level metadata 102 may be defined as the box "dbom."

number

[0054] The metadata_type may be, for example, "prdm" for predictive metadata 110, "dipm" for display management metadata 112, "phom" for photo metadata 114, or "synm" for synchronization metadata 120. Additional components of low-level metadata 102 may also be defined.

[0055] The mid-level metadata 104 may be defined as a box “dbex” that contains the low-level metadata 102 .

number

[0056] The association of media data to each experience and behavior metadata may be linked by an ItemProperties Box ("iprp") included in the HEIF bitstream. For the top-level metadata 106, "iref" boxes define the metadata and media data required for each application. In some implementations, boxes for synchronization metadata 120 are stored at the top level.

[0057] As another example, JPEG-XT is backward compatible with JPEG. JPEG-XT uses an APP marker (APP11) to store additional information for extension. There are 16 APP markers in JPEG. To utilize data structure 100, the format can be extended to, but is not limited to, APP11. A JPEG marker (16 bits of 0xff followed by a 16-bit ID) has 16 bits of information indicating the number of bytes in the current marker, so each marker is 2 16 = Only carries 64KB of information. If the metadata is larger than 64KB then multiple APP markers are required.

[0058] Additionally, to avoid false positives of the marker byte "0xff", for non-marker byte 0xff, every 0xff should be followed by a padding byte 0x00. Using the box definition and box structure used in ISO BMFF, headers and media data may be packed into multiple APP markers in JPEG. Thus, the live experience is provided as a JPEG still image, but other devices may provide a similar experience.

[0059] Enabling HDR still images The described box can be further extended to provide an HDR experience for still images. Table 4 gives example codec options for the imaging experience. [Table 4]

[0060] As an example, in a photography application, HDR HEVC encoded still images may be stored with photo metadata 114. Alternatively, a hybrid log-gamma (HLG) transfer function may be used instead of a perceptual quantizer (PQ) transfer function in HEVC encoded still images, as shown in the following pseudocode:

number

[0061] In another example, for a photography application, SDR JPEG encoded still images may be stored with EXIF ​​metadata and HDR HEVC encoded still images may be stored with photo metadata, as shown in the following pseudocode:

number

[0062] FIG. 5 illustrates an example of a Base Media File Format (BMFF) according to an embodiment. This BMFF is for a hypothetical new HDR photo format called, but not limited to, "Dolby Imaging" or "DI". It is based on the JPEG file format using the APP11 marker, although any other APP marker may be used as well. As shown in FIG. 5, the BMFF includes the following: APP marker (505) (e.g., APP11) (2 bytes) Payload length (510) (2 bytes) Identification String (515) (e.g., "DI") Null Byte (520) Dolby imaging payload data (525), including HDR (e.g., HEVC) image data (529) and rpu metadata (527).

[0063] The above systems and methods can provide a data structure for a multimedia experience. The systems, methods, and devices according to the present disclosure can take any one or more of the following configurations:

[0064] (1) A data structure utilized to implement a plurality of multimedia applications, the data structure including: a first metadata level including low level metadata used to perform operations related to media data in a bitstream; a second metadata level including mid level metadata used to apply the low level metadata to render the media data; and a third metadata level including higher level metadata used to utilize the low level metadata and the mid level metadata to deliver the plurality of multimedia applications, the first metadata level further including synchronization metadata for transforming the media data, the low level metadata, the mid level metadata, and the higher level metadata from a first multimedia application of the plurality of multimedia applications to a second multimedia application of the plurality of multimedia applications.

[0065] (2) The data structure of (1), wherein the low-level metadata includes: prediction metadata including polynomial prediction, multivariate multiple regression prediction, and tensor product B-spline prediction; display management metadata for displaying video using the media data; photo metadata for displaying images using the media data; exchangeable image file format metadata including camera image related metadata; null metadata for implementing baseline media data; and immersive metadata including object maps, depth maps, and response curves for the media data.

[0066] (3) A data structure described in any one of (1) to (2), wherein the synchronization metadata includes a payload header indicating the number of extension blocks, and each extension block includes a header indicating the length, level, and container for its respective payload.

[0067] (4) The data structure of (3), wherein the payload header includes source_experience_ID metadata, sink_experience_ID metadata, and operation_value metadata.

[0068] (5) The data structure of (4), wherein the operation_value metadata is one selected from the group consisting of a frame index for a still photo, a timestamp for a video frame, a resolution mapping function, and a binary flag indicating whether a standard dynamic range (SDR) photo to high dynamic range (HDR) video workflow is allowed.

[0069] (6) The data structure according to any one of (1) to (5), wherein the plurality of multimedia applications includes a still photo application and a video application.

[0070] (7) The data structure of (6), wherein when the media data is manipulated in the still photo application, the synchronization metadata updates the media data, the low level metadata, the mid level metadata, and the high level metadata for the video application.

[0071] (8) The data structure of any one of (1) to (7), wherein the low-level metadata and the mid-level metadata are implemented as boxes within a High Efficiency Image Format (HEIF) container.

[0072] (9) The data structure of (8), wherein the box is implemented in an application segment marker (APP marker) within a Joint Photographic Experts Group (JPEG) container.

[0073] (10) A controller for implementing the data structure described in any one of (1) to (9), wherein the controller is configured to edit a video in a first application included in the plurality of multimedia applications and propagate the edits to a corresponding photo in a second application included in the plurality of multimedia applications, and the synchronization metadata includes a frame index for the corresponding photo.

[0074] (11) The data structure according to any one of (1) to (9), wherein the synchronization metadata defines a resolution mapping between the first multimedia application and the second multimedia application.

[0075] (12) The data structure of any one of (1) to (9) or (11), wherein the synchronization metadata includes a binary flag indicating whether a workflow from a standard dynamic range (SDR) photography application to a high dynamic range (HDR) video application is allowed.

[0076] (13) The data structure according to (12), wherein the synchronization metadata includes an SDR-HDR conversion algorithm and its corresponding parameters.

[0077] (14) A controller for implementing the data structure described in any one of (1) to (9) or (11) to (13), the controller being configured to receive user input indicating one of the plurality of multimedia applications; and based on the user input, provide the media data as either an SDR still photo application or an HDR video application.

[0078] (15) A controller for implementing the data structure described in any one of (1) to (9) or (11) to (13), the controller being configured to perform an in-multimedia-application editing operation in which editing of media data in a first multimedia application is performed without affecting a second multimedia application.

[0079] (16) A controller for implementing the data structure described in any one of (1) to (9) or (11) to (13), the controller being configured to perform an inter-multimedia application editing operation in which editing of media data in a first multimedia application is propagated to a second multimedia application.

[0080] (17) A controller for implementing the data structure described in any one of (1) to (9) or (11) to (13), the controller being configured to perform an intra-multimedia application conversion operation in which media data is converted from a first multimedia application of a first type to a second multimedia application of the first type.

[0081] (18) A controller for implementing the data structure described in any one of (1) to (9) or (11) to (13), the controller being configured to perform an inter-multimedia application conversion operation in which media data is converted from a first multimedia application of a first type to a second multimedia application of a second type.

[0082] (19) A controller for implementing the data structure described in any one of (1) to (9) or (11) to (13), configured to provide the media data as either an SDR still photo application or an HDR video application according to at least one of playback computational resources, device capabilities, and end user preferences.

[0083] (20) A controller for implementing the data structure described in any one of (1) to (9) or (11) to (13), configured to provide an imaging application included in the plurality of multimedia applications, the controller being configured to include at least one of: (a) a first codec including a perceptual quantizer transfer function for HDR in a HEIF container; (b) a second codec including a hybrid log-gamma transfer function for HDR in a HEIF container; (c) a third codec including JPG and HDR as a single inventory in a JPEG container; and (d) a fourth codec including SDR and HDR as HEVC and as a single inventory in a HEIF container.

[0084] With respect to processes, systems, methods, heuristics, etc. described herein, steps of such processes, etc. are described as occurring according to a certain ordered sequence, but it should be understood that such processes may be practiced with the described steps performed in orders other than the order described herein. Additionally, it should be understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. In other words, the process descriptions herein are provided for the purpose of illustrating certain embodiments and should not be construed as limiting the scope of the claims in any way.

[0085] Therefore, it should be understood that the above description is intended to be illustrative and not restrictive. Many embodiments and applications other than the examples provided will be apparent upon reading the above description. The scope should not be determined with reference to the above description, but instead with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. It is anticipated and intended that future developments will occur in the technology described herein, and that the disclosed systems and methods will be incorporated into such future embodiments. In short, it should be understood that this application is capable of modification and variation.

[0086] All terms used in the claims are intended to be given their broadest reasonable interpretation and their ordinary meaning as understood by one skilled in the art described herein, unless expressly indicated to the contrary herein. In particular, the use of singular articles such as "a," "the," "said," etc., should be read as describing one or more of the indicated elements, unless the claim describes an express limitation to the contrary.

[0087] The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. The Abstract is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it will be seen that various features are grouped together in various embodiments for the purpose of facilitating flow of the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed embodiments incorporate more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as separately claimed subject matter.

Claims

1. 1. A data structure utilized to implement a plurality of multimedia applications, the data structure comprising: a first metadata level that includes low-level metadata used to perform operations related to the media data in the bitstream; a second metadata level including intermediate level metadata used to apply the low level metadata to render the media data; a third metadata level including higher level metadata used to utilize the lower level metadata and the intermediate level metadata to deliver the plurality of multimedia applications; the first metadata level further includes synchronization metadata for transforming the media data, the low level metadata, the mid level metadata, and the high level metadata from a first multimedia application of the plurality of multimedia applications to a second multimedia application of the plurality of multimedia applications; Data structure.

2. The low level metadata includes: prediction metadata including polynomial predictions, multivariate multiple regression predictions, and tensor product B-spline predictions; display management metadata for displaying a video using the media data; photo metadata for displaying an image using the media data; an exchangeable image file format metadata including camera image related metadata; null metadata for implementing baseline media data; immersive metadata including object maps, depth maps, and response curves for the media data; The data structure of claim 1 .

3. 3. The data structure of claim 1, wherein the synchronization metadata includes a payload header indicating a number of extension blocks, each extension block including a header indicating a length, a level, and a container for its respective payload.

4. The data structure of claim 3 , wherein the payload header includes source_experience_ID metadata, sink_experience_ID metadata, and operation_value metadata.

5. 5. The data structure of claim 4, wherein the operation_value metadata is selected from the group consisting of a still photo frame index, a video frame timestamp, a resolution mapping function, and a binary flag indicating whether a standard dynamic range (SDR) photo to high dynamic range (HDR) video workflow is allowed.

6. 6. The data structure of claim 1, wherein the plurality of multimedia applications includes a still photography application and a video application.

7. 7. The data structure of claim 6, wherein the synchronization metadata updates the media data, the low level metadata, the mid level metadata, and the high level metadata for the video application when the media data is manipulated in the still photo application.

8. The data structure of claim 1 , wherein the low-level metadata and the mid-level metadata are implemented as boxes within a High Efficiency Image Format (HEIF) container.

9. 9. The data structure of claim 8, wherein the box is implemented in an Application Segment Marker (APP Marker) within a Joint Photographic Experts Group (JPEG) container.

10. The data structure of claim 1 , wherein the synchronization metadata defines a resolution mapping between the first multimedia application and the second multimedia application.

11. 11. The data structure of claim 1, wherein the synchronization metadata includes a binary flag indicating whether a workflow from a standard dynamic range (SDR) photo application to a high dynamic range (HDR) video application is allowed.

12. The data structure of claim 11 , wherein the synchronization metadata includes an SDR-HDR conversion algorithm and its corresponding parameters.

13. A controller for implementing a data structure according to any one of claims 1 to 12, the controller comprising: editing a video in a first application of the plurality of multimedia applications; propagating the edits to a corresponding photo in a second application of the plurality of multimedia applications; the synchronization metadata includes a frame index for the corresponding photograph; controller.

14. the controller receiving a user input indicating one of the plurality of multimedia applications; configured to provide the media data as either an SDR still photo application or an HDR video application based on the user input. The controller of claim 13.

15. 15. The controller of claim 14, wherein the controller is configured to perform in-multimedia application edit operations in which editing of the media data in a first multimedia application is performed without affecting a second multimedia application.

16. 16. The controller of claim 13, wherein the controller is configured to perform inter-multimedia application editing operations in which editing of the media data in a first multimedia application is propagated to a second multimedia application.

17. 17. The controller of claim 13, wherein the controller is configured to perform an intra-multimedia application conversion operation in which the media data is converted from a first multimedia application of a first type to a second multimedia application of the first type.

18. 18. The controller of claim 13, wherein the controller is configured to perform an inter-multimedia application conversion operation in which the media data is converted from a first multimedia application of a first type to a second multimedia application of a second type.

19. The controller in question is: and providing the media data as either an SDR still photo application or an HDR video application according to at least one of a regenerative computational resource, a device capability, and an end-user preference. A controller according to any one of claims 13 to 18.

20. To provide an imaging application in the plurality of multimedia applications, the controller: (a) a first codec including a perceptual quantizer transfer function for HDR in a HEIF container; (b) a second codec including a hybrid log-gamma transfer function for HDR within the HEIF container; (c) a third codec that includes JPG and HDR as a single inventory within a JPEG container; (d) a fourth codec, including SDR and HDR, as HEVC and as a single inventory within said HEIF container; 20. A controller as claimed in any one of claims 13 to 19, configured to include at least one of:

Images