Data Structure for Multimedia Applications

Abstract

The embodiments described herein provide a unified container format for delivering different multimedia applications. An embodiment provides a data structure utilized to implement a plurality of multimedia applications. The data structure includes a first metadata level including low level metadata used to perform an operation related to media data in a bitstream. The data structure includes a second metadata level including mid level metadata used to apply the operation metadata to render the media data. The data structure includes a third metadata level including higher level metadata used to utilize the low level metadata and the mid level metadata to deliver the plurality of multimedia applications. The first metadata level further includes synchronization metadata for transforming the media data, the low level metadata, the mid level metadata, and the higher level metadata from a first multimedia application of the plurality of multimedia applications to a second multimedia application of the plurality of multimedia applications.

Description

Technical Field

[0001] 1. Cross - Reference to Related Applications This application represents the national stage entry of PCT application PCT / US22 / 53418, filed on December 19, 2022, and claims the benefit of priority from the following priority applications: U.S. Provisional Application No. 63 / 301,467, filed on January 20, 2022, and No. 63 / 399,871, filed on August 22, 2022, and EP Application No. 22155345.6, filed on February 7, 2022.

[0002] 2. Field of Disclosure This application generally relates to data structures for implementing multiple multimedia applications and controllers for implementing the data structures.

Background Art

[0003] 3. Background Multimedia experiences such as imaging and video applications utilize container formats (e.g., metafiles, file formats) that allow multiple data streams to be embedded in a single file. These container formats include metadata for identifying the data streams and detailing how the functionality of the data streams is implemented. Containers include the High Efficiency Image File Format (HEIF), which uses a video codec to encode images using intra - frame coding, and the Joint Photographic Experts Group (JPEG) format (also known as EXIF and JFIF), which uses application segment markers (APP markers) to store information. However, there is a growing need for container formats for multiple multimedia applications in high - dynamic - range (HDR) image quality.

Summary of the Invention

Means for Solving the Problems

[0004] The embodiments described herein provide a unified container format for delivering different multimedia experiences. Multimedia experiences include, for example, still photo applications, video applications, live photo applications, and the like. Additionally, different multimedia experiences may be different versions of the same application (e.g., the original photo application and an updated photo application). The unified container format is backward compatible with existing formats. Multiple experiences are encapsulated in a single bitstream. Using the unified container format, a playback system can determine which application to execute depending on computing resources, device capabilities, and / or user preferences. Further, interference and dependencies between different multimedia applications need to be minimized.

[0005] In an exemplary aspect of the present disclosure, a data structure is provided for implementing multiple multimedia applications. The data structure includes a first metadata level that includes low-level metadata used to perform operations related to media data within a bitstream. The data structure includes a second metadata level that includes intermediate-level metadata used to apply the low-level metadata for rendering the media data. The data structure includes a third metadata level that includes high-level metadata used to utilize the low-level metadata and the intermediate-level metadata for delivering multiple multimedia applications. The first metadata level further includes synchronization metadata for converting media data, low-level metadata, intermediate-level metadata, and high-level metadata from a first multimedia application among the multiple multimedia applications to a second multimedia application among the multiple multimedia applications.

[0006] In this way, various aspects of the present disclosure provide implementations of multimedia applications such as photo applications and video applications having high dynamic range and high or standard resolution, resulting in improvements in at least technical fields such as image projection, signal processing, and image display.

Brief Description of the Drawings

[0007] These and other more detailed and specific features of the various embodiments are more fully disclosed in the following description with reference to the accompanying drawings.

[0008]

Figure 1

[0009]

Figure 2

[0010]

Figure 3A

[0011]

Figure 3B

[0012]

Figure 4A

[0013]

Figure 4B

[0014]

Figure 5

Modes for Carrying Out the Invention

[0015] The present disclosure and aspects thereof can be embodied in various forms including hardware, devices or circuits controlled by a computer-implemented method, computer program products, computer systems and networks, user interfaces, and application programming interfaces; as well as hardware implementation methods, signal processing circuits, memory arrays, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. The above is only intended to give a general concept of various aspects of the present disclosure and is in no way intended to limit the scope of the present disclosure.

[0016] Furthermore, the present disclosure mainly focuses on examples where various circuits are used in a digital projection system, but it will be understood that these are merely examples. The disclosed systems and methods can be implemented in display devices having OLED displays, LCD displays, quantum dot displays, etc. It should be further understood that the disclosed systems and methods can be used in any device that needs to project light, such as in movie theaters, consumer, and other commercial projection systems, head-up displays, virtual reality displays, etc.

[0017] 3 - level imaging format FIG. 1 provides a three-level data structure 100 (e.g., metadata architecture, container, file structure) for implementing a multimedia experience (e.g., a multimedia application). The data structure 100 includes low-level metadata 102, intermediate-level metadata 104, and high-level metadata 106. The low-level metadata 102 and the intermediate-level metadata 104 serve as building blocks implemented by the high-level metadata 106 including the multimedia experience.

[0018] The low-level metadata 102 consists of basic metadata for describing how to perform operations on standard media data within a bitstream. The low-level metadata 102 includes predictive metadata (PM) 110, display management metadata (DM) 112, photo metadata (OM) 114, immersive metadata (IM) 116, exchangeable image file format (EXIF) / International Color Consortium (ICC) metadata (EM) 118, synchronization metadata (SM) 120, and null metadata (NM) 122. In some implementations, the low-level metadata 102 may include additional metadata used to implement a multimedia experience.

[0019] The predictive metadata 110 includes polynomial prediction, multivariate multiple regression (MMR) prediction, and tensor product B-spline (TPB) prediction. The display management metadata 112 includes different levels of metadata syntax in the range from L0 to L255. The photo metadata 114 is used for new still images and may have the same L1 syntax as the display management metadata 112. The EXIF / ICC metadata 118 includes camera image-related metadata such as thumbnail information. The immersive metadata 116 includes object maps, depth maps, and response curves. The null metadata 122 is reserved and is used when baseline media data is required without additional metadata. Some metadata groups may include some of the same or similar syntax. Further, in a decoder, a dispatcher component may convert one syntax within one metadata group to another syntax in another metadata group, reusing existing code and hardware and reducing the code size.

[0020] The synchronous metadata 120 is configured in the same way as the display management metadata 112. Each level of the synchronous metadata 120 specifies a source experience, a sink experience, and a designed operation. The following table provides an exemplary syntax. However, other syntaxes may also be used. In some implementations, the synchronous metadata 120 is updated and edited using various user-accessible tools.

[0021] The payload header indicates the number of extension blocks to be included in the synchronous metadata 120, and an example thereof is shown in Table 1.

Table 1

[0022] Each extension block includes a simple header indicating a container that includes a length, a level, and its own payload, and an example thereof is shown in Table 2.

Table 2

[0023] An example of the payload format itself is provided in Table 3 below.

Table 3

[0024] The source_experience_id [source experience ID] and sink_expierence_id [sink experience ID] shown in Table 3 may be defined and mapped at a higher level. The operation_value [operation value] metadata in Table 3 is a placeholder for various usages as follows.

[0025] Temporal synchronization: When editing a video in a video application, the corresponding still photo may be updated in the photo application. In such cases, the operation_value metadata may be a frame index or a timestamp.

[0026] Spatial resolution: Different multimedia experiences may have different scales for the same attribute. For example, still images often have a higher resolution than videos. The operation_value metadata can define the resolution mapping between different experiences.

[0027] Propagation options / intensity: The synchronization metadata 120 can indicate whether editing or conversion of data between different multimedia experiences is allowed and the corresponding "intensity" within or between experiences. For example, the operation_value metadata may be a simple binary flag to indicate whether a workflow from a standard dynamic range (SDR) photo to an HDR video is allowed. If a workflow from an SDR photo to an HDR video is allowed, the operation_value metadata may define which type of SDR-HDR conversion algorithms and their corresponding parameters should be used. For example, which TPB enum should be selected can be specified so that a backward reshaping from SDR to HDR with different color volumes can be specified. Further, in the case of an HDR video to an SDR photo, the intensity can indicate which TPB enum should be selected and used to perform the forward reshaping to generate the SDR photo.

[0028] Metadata conversion: The synchronization metadata 120 can indicate how to convert metadata from one multimedia experience to another, such as from a photo application to a video application or vice versa.

[0029] Auxiliary information: The synchronization metadata 120 can indicate the necessary metadata for performing operations related to the multimedia experience.

[0030] When multiple multimedia experiences are provided in a single bitstream, multiple extension blocks with the same level can be utilized. For example:

[0031] Multiple sinks: The conversion from the first experience A to the second experience B can affect the third experience C. In such scenarios, two extension blocks having the same level can exhibit operations. For example, the bitstream can include SDR photos, HDR videos, and immersive HDR / SDR photos. Updating the HDR video affects both the SDR photo and the immersive HDR / SDR photo.

[0032] Multiple sources: The bitstream can include SDR photos, HDR videos, and immersive HDR photos. One application can be converted to another application. For example, the SDR photo may be converted from the HDR video, or the SDR photo may be converted from the immersive HDR photo.

[0033] The intermediate level metadata 104 describes which media data to provide as input to the processing module and how to apply the operation metadata for rendering the media data. In other words, the intermediate level metadata 104 consists of the media data bitstream and the corresponding low level metadata 102 required for the rendering to be used. The media data may be packed together with the experience metadata in the bitstream or stored at different positions in the same bitstream. In some implementations, storing the media data at different positions can enable format backward compatibility.

[0034] Each multimedia experience includes zero, one, two, or more low-level metadata 102, and different low-level metadata 102 can be encapsulated in the rpu_type format. In some implementations, there are 64 rpu_types available for assignment. In the illustrated example, the intermediate-level metadata 104 includes rpu_type_2 130, rpu_type_4 132, rpu_type_5 134, and rpu_type_63 136. rpu_type_2 130 includes media data as a base layer bitstream and two lower-level metadata, namely prediction metadata 110 and presentation management metadata 112. rpu_type_4 132 includes photo metadata 114. rpu_type_5 134 includes immersive metadata 116. rpu_type_63 136 can be reserved for the TPB payload.

[0035] The high-level metadata 106 includes several applications that utilize the intermediate-level metadata 104 and synchronization metadata 120, such as a first application 150, a second application 152, and a third application 154. Each application is a different multimedia experience (photo experience, video experience, SDR photo experience, HDR photo experience, SDR video experience, HDR video experience, immersive SDR / HDR video experience, etc.). Multiple of the same type of experience may be present in the same bitstream. Each experience may also include side information indicating whether the media data and metadata are packed together or located at different positions.

[0036] The playback system determines what to play according to playback computing resources, device capabilities, and end-user preferences. The playback computing resources indicate the level of experience the system can support. The device capabilities include whether an end device (e.g., an output device) can display HDR images and videos, sensors available on the end device, and whether the end device has a touch screen. End-user preferences include what experience the user has requested or preferred to render.

[0037] In some embodiments, the high-level metadata 106 includes a default presentation application 156 (e.g., a raw application). Alternatively, the presentation mode (or application) may be selected by the user. In some embodiments, the bitstream may be encapsulated in an existing container having the format of the default presentation application 156. For example, a photo experience may include a JPEG thumbnail as the default presentation without being associated with any low-level metadata 102 or mid-level metadata 104. However, the high-level metadata 106 can describe the mechanism between different applications with default presentations using the synchronization metadata 120. In other embodiments, the bitstream is encapsulated within a container that does not have a default presentation. In such embodiments, the required application is activated by user or environmental input.

[0038] Media playback Devices implementing the data structure 100 can reproduce different experiences depending on the computing resources, display capabilities, sensor availability, and user preferences available to the device. For example, FIG. 2 provides a playback operation 200 for a multimedia experience such as a second application 152. The second application 152 may be, for example, a photo or image experience. When the second application 152 is stored in a legacy container such as a JPEG container (such as defined by, for example, the JPEG standard, EXIF, and JFIF file formats) and the legacy container file is passed to the end device, only JPEG SDR still images are provided.

[0039] However, when the second application 152 is provided to an end device implementing the data structure 100, the media data can be viewed in SDR or HDR still images using photo playback 204 implementing rpu_type_4 132, or as an HDR live photo (e.g., video) using the prediction metadata 110 and display management metadata 112 included in rpu_type_2 130 implemented by video playback 202. Whether the media data is viewed using video playback 202 or photo playback 204 is based on user preferences or user selection.

[0040] Media editing Media data can be edited to modify the experience defined by its rpu_type (e.g., its color grading, its resolution, its frame rate, etc.). FIG. 3A shows an exemplary in-experience editing operation 300 that allows an edit to be performed in one experience (e.g., each rpu_type) without affecting another experience (e.g., another rpu_type). FIG. 3B shows an exemplary cross-experience editing operation 350 in which edits in one experience are propagated to other experiences. Whether to perform the in-experience editing operation 300 or the cross-experience editing operation 350, and how to propagate between experiences, is defined by the synchronization metadata 120.

[0041] The in-experience editing operation 300 of FIG. 3A allows an edit to be made in one experience (e.g., a second application 152) using an editing tool 310. The edit result can be propagated to other experiences, but the media data and its corresponding metadata are edited in a single experience. The in-experience editing operation 300 of FIG. 3A can perform several different operations based on whether an edit in one experience is automatically reflected in other experiences. The operations can include editing individually in one experience without updating another experience, editing one experience and automatically updating another experience by that edit, and editing one experience and optionally updating another experience by that edit. The edit update is controlled by the synchronization metadata 120.

[0042] In the example of FIG. 3A, the second application 152 includes an rpu_type_m 315, an rpu_type_n 320, and synchronization metadata 120. An rpu_type_m editing module 335 performs any edits to the rpu_type_m 315. An rpu_type_n editing module 330 performs any edits to the rpu_type_n 320. A synchronization metadata update module 325 updates the synchronization metadata 120.

[0043] As an example of the in-experience editing operation 300, the synchronization metadata 120 controls the propagation of edits between the SDR still image and the HDR still image. In some embodiments, the SDR still image is edited independently of the HDR still image without updating the HDR still image. In some embodiments, editing the HDR still image automatically updates the SDR still image. In some embodiments, when the SDR still image is edited, the HDR still image is conditionally updated using a backward revertible TPB to upconvert the SDR to HDR. When the edit to the SDR still image includes new content such as graphics and text, object-based (e.g., masking) side information may be used. The added information may be upconverted in different ways and then fused in the HDR domain.

[0044] The cross-experience editing operation 350 allows several experiences to be edited jointly using the editing tool 360. For example, in some embodiments, one experience with references from other experiences is edited and the edit result automatically updates the other experiences. In other embodiments, one experience with references from other experiences is edited and the edit result optionally updates the other experiences. The edit update is controlled by the synchronization metadata 120.

[0045] In the example of FIG. 3B, the second application 152 includes rpu_type_m 315, rpu_type_n 320, and the synchronization metadata 120. The rpu_type_m editing module 375 performs any edits to rpu_type_m 315. The synchronization metadata update module 365 updates the synchronization metadata 120. The changes are then propagated to the rpu_type_n update module 370, which updates rpu_type_n 320.

[0046] As an example of the experience editing operation 350, the synchronization metadata 120 controls the propagation of edits between an SDR still image and an HDR live photo (e.g., video). In some embodiments, the HDR live photo is edited and the SDR still image is automatically updated. In such embodiments, the metadata indicates which frames within the HDR video are edited. In other embodiments, the SDR still image is edited and the HDR live photo is conditionally updated using a backward recoverable TPB. In such embodiments, the reverse metadata indicates how to propagate the edit results, such as cropping, resizing, and tone mapping.

[0047] Experience conversion Experience conversion involves converting one experience to another and converting content between profiles and levels even within the same experience. FIG. 4A shows an exemplary in-experience conversion operation 400. FIG. 4B shows an exemplary between-experience conversion operation 450.

[0048] Figure 4A includes application x 405, application y 410, and conversion tool 415. The conversion tool 415 converts from application x 405 to application y 410. Both application x 405 and application y 410 can be different profiles (or versions) of the same or similar experiences, such as converting from one photo experience to another. The in-experience conversion operation 400 can include media data conversion where media data, such as a High Efficiency Video Coding (HEVC) encoded bitstream, is decoded into a common workspace and then encoded into another workspace for conversion. Different settings such as resolution, color space, frame rate, SEI, and VUI can be converted. This function is performed by the in-type conversion rpu_type_m module 430 that converts rpu_type_m 420 from application x 405 to application y 410. The in-experience conversion operation 400 can also include metadata conversion where the synchronization metadata update tool 435 updates metadata such as synchronization metadata 120 according to the conversion. For example, the display management metadata 112 may be converted to new content.

[0049] Figure 4B includes application x 455, application y 460, and conversion tool 465. Conversion tool 465 converts from application x 455 to application y 460. Application x 455 may be, for example, a video experience, and application y 460 may be, for example, a photo experience. Experience conversion operation 450 may include media data conversion in which media data is converted from one type of media to another type of video. For example, the media data may be converted from a photo to a video or vice versa. The media data conversion may be performed by type conversion module 480 from rpu_type_m 470 to rpu_type_n 485 that converts rpu_type_m to rpu_type_n. Experience conversion operation 450 may also include metadata conversion in which one type of metadata is converted to another type of metadata. For example, photo metadata 114 may be converted to video metadata (included in display management metadata 112), or vice versa. In some embodiments, prediction metadata 110 is converted. Synchronous metadata update block 490 can update synchronous metadata 120.

[0050] Container implementation Data structure 100 may be integrated into an existing container such as HEIF or JPEG. As an example, HEIF uses a video codec to encode an image using intra-frame coding. The syntax of HEIF is based on ISO BMFF (Base Media File Format). ISO BMFF uses "boxes" to structure different categories of data. Each box is preceded by a 4-character type (4CC). Boxes may be nested and hierarchical (e.g., a box inside another box). Each box has a size that is an integer specifying the number of bytes in the box. Also, each box has a box type such as compact.

[0051] Two different definitions are used to separate still images and videos. First, still images are stored as items. All image items are encoded independently and do not depend on other items for their decoding. Any number of image items may be included in the same file. Image sequences (e.g., videos) are stored as tracks. If there is an encoding dependency between images or the timing of image playback is taken into account, an image sequence track is used. Unlike video tracks, the timing in an image sequence track is advisory.

[0052] The HEIF bitstream contains two main components. Metadata with a 4CC as "meta", and media data with a 4CC as the media data box "mdat" or the item data box "idat". The metadata part represents side information (e.g., the structure of a multimedia application), and the media data stores multimedia data.

[0053] The components of the data structure 100 can be implemented as boxes in the HEIF bitstream. For example, the low-level metadata 102 may be defined as the box "dbom".

Number

[0054] The above metadata_type can be, for example, "prdm" for predictive metadata 110, "dipm" for display management metadata 112, "phom" for photo metadata 114, "synm" for synchronization metadata 120. Additional components of the low-level metadata 102 can also be defined.

[0055] The intermediate-level metadata 104 may be defined as the box "dbex" that contains the low-level metadata 102.

Number

[0056] The association of media data with each experience metadata and operation metadata can be linked by the ItemProperties Box ("iprp") included in the HEIF bitstream. For the upper-level metadata 106, the "iref" box defines the metadata and media data required for each application. In some implementations, the box for the synchronization metadata 120 is stored at the top level.

[0057] As another example, JPEG-XT is backward compatible with JPEG. JPEG-XT uses the APP marker (APP11) to store additional information for extension. There are 16 APP markers in JPEG. For using the data structure 100, the format can be extended to APP11, although not limited. Since the JPEG marker (16-bit 0xff followed by 16-bit ID) has 16-bit information indicating the number of bytes in the current marker, each marker can carry only 16 = 64 kilobytes of information. If the metadata is larger than 64KB, multiple APP markers are required.

[0058] Furthermore, to avoid misdetection of the marker byte "0xff", for non-marker byte 0xff, stuffing byte 0x00 should follow each 0xff. Using the box definition and box structure used in ISO BMFF, the header and media data may be put into multiple APP markers in JPEG. Thus, the raw experience is provided as a JPEG still image, but other devices can provide a similar experience.

[0059] Activation of still images in HDR The boxes described can be further extended to provide an HDR experience for still images. Table 4 gives exemplary codec options for the imaging experience.

Table 4

[0060] As an example, in a photo application, a still image encoded with HDR HEVC can be stored together with photo metadata 114. Alternatively, as shown in the following pseudocode, in a still image encoded with HEVC, a hybrid log-gamma (HLG) transfer function can be used instead of a perceptual quantizer (PQ) transfer function.

Number

[0061] In another example, for a photo application, as shown in the following pseudocode, a still image encoded with SDR JPEG may be stored together with EXIF metadata, and a still image encoded with HDR HEVC may be stored together with photo metadata.

Number

[0062] FIG. 5 shows an example of a base media file format (BMFF) according to an embodiment. This BMFF is for a virtual new HDR photo format called, but not limited to, "Dolby Imaging" or "DI". It is based on the JPEG file format using the APP11 marker, but any other APP marker can be used as well. As shown in FIG. 5, the BMFF includes the following. · APP marker (505) (e.g., APP11) (2 bytes) · Payload length (510) (2 bytes) · Identification string (515) (e.g., "DI") · Null byte (520) Dolby Imaging Payload Data (525) including · HDR (e.g., HEVC) image data (529) and rpu metadata (527).

[0063] The above systems and methods can provide data structures for multimedia experiences. The systems, methods, and devices according to the present disclosure can take any one or more of the following configurations.

[0064] (1) A data structure used to implement a plurality of multimedia applications, comprising: a first metadata level including low-level metadata used to perform operations related to media data in a bitstream; a second metadata level including intermediate-level metadata used to apply the low-level metadata for rendering the media data; and a third metadata level including high-level metadata used to utilize the low-level metadata and the intermediate-level metadata for delivering the plurality of multimedia applications, wherein the first metadata level further includes synchronization metadata for converting the media data, the low-level metadata, the intermediate-level metadata, and the high-level metadata from a first multimedia application among the plurality of multimedia applications to a second multimedia application among the plurality of multimedia applications.

[0065] (2) The low-level metadata includes: prediction metadata including polynomial prediction, multivariate multiple regression prediction, and tensor product B-spline prediction; display management metadata for displaying videos using the media data; photo metadata for displaying images using the media data; exchangeable image file format metadata including camera image-related metadata; null metadata for implementing baseline media data; and immersive metadata including object maps, depth maps, and response curves of the media data, the data structure according to (1).

[0066] (3) The synchronization metadata includes a payload header indicating the number of extension blocks, and each extension block includes a header indicating a container for its length, level, and respective payload, the data structure according to any one of (1) to (2).

[0067] (4) The payload header includes source_expierience_ID metadata, sink_experience_ID metadata, and operation_value metadata, the data structure according to (3).

[0068] (5) The operation_value metadata is one selected from the group consisting of a frame index of a still photo, a time stamp of a video frame, a resolution mapping function, and a binary flag indicating whether a workflow from a standard dynamic range (SDR) photo to a high dynamic range (HDR) video is permitted, the data structure according to (4).

[0069] (6) The plurality of multimedia applications includes a still photo application and a video application, the data structure according to any one of (1) to (5).

[0070] (7) When the media data is operated in the still photo application, the synchronization metadata updates the media data for the video application, the low-level metadata, the intermediate-level metadata, and the high-level metadata, the data structure according to (6).

[0071] (8) The low-level metadata and the intermediate-level metadata are implemented as boxes in a High Efficiency Image Format (HEIF) container, the data structure according to any one of (1) to (7).

[0072] (9) The box is implemented in an Application Segment Marker (APP marker) in a Joint Photographic Experts Group (JPEG) container, the data structure according to (8).

[0073] (10) A controller for implementing the data structure according to any one of (1) to (9), wherein the controller edits video in a first application included in the plurality of multimedia applications and propagates the edit to a corresponding photo in a second application included in the plurality of multimedia applications, and the synchronization metadata includes a frame index for the corresponding photo, the controller.

[0074] (11) The synchronization metadata defines a resolution mapping between the first multimedia application and the second multimedia application, the data structure according to any one of (1) to (9).

[0075] (12) The synchronization metadata includes a binary flag indicating whether a workflow from a Standard Dynamic Range (SDR) photo application to a High Dynamic Range (HDR) video application is permitted, the data structure according to any one of (1) to (9) or (11).

[0076] (13) The synchronization metadata is the data structure according to (12), including an SDR-HDR conversion algorithm and its corresponding parameters.

[0077] (14) A controller for implementing the data structure according to any one of (1) to (9) or (11) to (13), the controller receiving a user input indicating one of the plurality of multimedia applications; and configured to provide the media data as either an SDR still image application or an HDR video application based on the user input.

[0078] (15) A controller for implementing the data structure according to any one of (1) to (9) or (11) to (13), configured to execute an in-multimedia-application editing operation in which editing of media data in a first multimedia application is executed without affecting a second multimedia application.

[0079] (16) A controller for implementing the data structure according to any one of (1) to (9) or (11) to (13), configured to execute an inter-multimedia-application editing operation in which editing of media data in a first multimedia application is propagated to a second multimedia application.

[0080] (17) A controller for implementing the data structure according to any one of (1) to (9) or (11) to (13), configured to execute an in-multimedia-application conversion operation in which media data is converted from a first multimedia application of a first type to a second multimedia application of the first type.

[0081] A controller for implementing the data structure according to any one of (1) to (9) or (11) to (13), the controller being configured to perform an inter - multimedia - application conversion operation in which media data is converted from a first - type first multimedia application to a second - type second multimedia application.

[0082] A controller for implementing the data structure according to any one of (1) to (9) or (11) to (13), the controller being configured to provide the media data as either an SDR still - image application or an HDR video application according to at least one of playback computing resources, device capabilities, and end - user preferences.

[0083] A controller for implementing the data structure according to any one of (1) to (9) or (11) to (13), the controller being configured to include at least one of: (a) a first codec including a perceptual quantizer transfer function for HDR in a HEIF container; (b) a second codec including a hybrid log - gamma transfer function for HDR in a HEIF container; (c) a third codec including JPG and HDR as a single inventory in a JPEG container; and (d) a fourth codec including SDR and HDR as HEVC and as a single inventory in a HEIF container, in order to provide an imaging application included in the plurality of multimedia applications.

[0084] Regarding the processes, systems, methods, heuristics, etc. described herein, although the steps of such processes, etc. are described as occurring according to a certain ordered sequence, it should be understood that such processes can be practiced using the steps described herein in an order other than the order described. Further, it should be understood that certain steps can be executed simultaneously, other steps can be added, or certain steps described herein can be omitted. In other words, the description of the processes herein is provided for the purpose of exemplifying certain embodiments and should in no way be construed as limiting the scope of the claims.

[0085] Therefore, it should be understood that the above description is intended to be illustrative and not limiting. Many embodiments and applications other than the examples provided will be apparent upon reading the above description. The scope should not be determined with reference to the above description, but rather, the appended claims should be referred to, along with the full scope of equivalents to which such claims are entitled. Future developments are expected and intended in the technologies described herein, and the disclosed systems and methods will be incorporated into such future embodiments. In short, it should be understood that this application is capable of modification and variation.

[0086] All terms used in the claims are intended to be given their broadest reasonable interpretation and their ordinary meaning as understood by one of ordinary skill in the art to which the technologies described herein pertain, unless an explicit contrary indication is made herein. In particular, the use of singular articles such as "a", "the", "said", etc. should be read as reciting one or more of the indicated elements, unless the claim describes an explicit limitation to the contrary.

[0087] The summary of the present disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. The summary is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Additionally, in the foregoing detailed description, for the purpose of bettering the flow of the present disclosure, it can be seen that various features are grouped together in various embodiments. This method of disclosure should not be interpreted as reflecting an intention that the claimed embodiments incorporate more features than are expressly recited in each claim. Rather, as reflected by the following claims, the subject matter of the invention lies in less than all of the features of a single disclosed embodiment. Thus, the following claims are hereby incorporated by reference into the detailed description herein, and each claim stands on its own as a separately claimed subject matter.

Claims

1. A controller for implementing a data structure utilized for implementing a plurality of multimedia applications, each multimedia application providing at least one playback type, the data structure comprising: A first metadata level including low-level metadata used to perform operations related to media data in a single bitstream, the bitstream including media data for the plurality of multimedia applications; A second metadata level including intermediate-level metadata, each intermediate-level metadata being related to one of the at least one playback type and describing how to apply the low-level metadata to render the media data for that playback type; A third metadata level including high-level metadata for delivering the plurality of multimedia applications, each high-level metadata being related to one of the plurality of multimedia applications and used to implement the low-level metadata and the intermediate-level metadata related to the at least one playback type provided by that multimedia application; The first metadata level further including synchronization metadata for converting the media data, the low-level metadata, and the intermediate-level metadata related to a first multimedia application among the plurality of multimedia applications into those related to a second multimedia application among the plurality of multimedia applications; The controller comprising: Editing video in a first application included in the plurality of multimedia applications; Configured to propagate the editing to corresponding photos in a second application included in the plurality of multimedia applications; The synchronization metadata including a frame index for the corresponding photos, the controller.

2. The low-level metadata comprises: Prediction metadata including polynomial prediction, multivariate multiple regression prediction, and tensor product B-spline prediction, Display management metadata for displaying a video using the media data, Photo metadata for displaying an image using the media data, Exchangeable image file format metadata including camera image-related metadata, Null metadata for implementing baseline media data, Including object maps, depth maps, and immersive metadata including response curves for the media data, The controller according to claim 1.

3. The synchronization metadata includes a payload header indicating the number of extension blocks, and each extension block includes a header indicating the length of the extension block, the level of the extension block, and a container for each payload of the extension block. The controller according to claim 1.

4. The payload header includes source_expierience_ID metadata, sink_experience_ID metadata, and operation_value metadata. The controller according to claim 3.

5. The operation_value metadata is selected from the group consisting of a frame index of a still photo, a time stamp of a video frame, a resolution mapping function, and a binary flag indicating whether a workflow from a standard dynamic range (SDR) photo to a high dynamic range (HDR) video is permitted. The controller according to claim 4.

6. The plurality of multimedia applications includes a still photo application and a video application. The controller according to claim 1.

7. When the media data is operated in the still photo application, the synchronization metadata updates the media data, the low-level metadata, and the intermediate-level metadata related to the video application. The controller according to claim 6.

8. The low-level metadata and the intermediate-level metadata are implemented as boxes within a high efficiency image format (HEIF) container. The controller according to claim 1.

9. The box is the controller according to claim 8, implemented in an application segment marker (APP marker) within a Joint Photographic Experts Group (JPEG) container.

10. The synchronization metadata is the controller according to claim 1, which defines a resolution mapping between the first multimedia application and the second multimedia application.

11. The synchronization metadata is the controller according to claim 1, including a binary flag indicating whether a workflow from a standard dynamic range (SDR) photo application to a high dynamic range (HDR) video application is permitted.

12. The synchronization metadata is the controller according to claim 11, including an SDR-HDR conversion algorithm and its corresponding parameters.

13. The controller receives a user input indicating one of the plurality of multimedia applications; Based on the user input, it is configured to play the media data as either an SDR still photo or an HDR video. The controller according to claim 1.

14. The controller is configured to execute an in-multimedia-application editing operation in which editing of the media data in a first multimedia application is performed without affecting a second multimedia application, according to claim 1.

15. The controller is configured to execute an inter-multimedia-application editing operation in which editing of the media data in a first multimedia application is propagated to a second multimedia application, according to claim 1.

16. The controller is configured to execute an in-multimedia-application conversion operation in which the media data is converted from a first multimedia application of a first type to a second multimedia application of the first type, according to claim 1.

17. The controller according to claim 1, wherein the controller is configured to perform an inter - multimedia - application conversion operation in which the media data is converted from a first multimedia application of a first type to a second multimedia application of a second type.

18. The controller is: configured to play the media data as either an SDR still image or an HDR video according to at least one of playback computing resources, device capabilities, and end - user preferences; The controller according to claim 1.

19. To provide an imaging application included in the plurality of multimedia applications, the controller is: (a) a first codec including a perceptual quantizer transfer function for HDR within a HEIF container; (b) a second codec including a hybrid log - gamma transfer function for HDR within the HEIF container; (c) a third codec including JPG and HDR as a single inventory within a JPEG container; (d) a fourth codec including SDR and HDR as a single inventory within the HEIF container and as HEVC The controller according to claim 1, configured to include at least one of them.

Images