Method, device and medium for generating or parsing media files or generating reconstructed videos

Abstract

The present invention relates to a method, apparatus, and medium for generating or parsing a media file or generating a reconstructed video. Furthermore, the present invention relates to a method for generating a media file from a video sequence, the method comprising, using a server, obtaining video data comprising a plurality of samples from the video sequence; generating video tracks based on the obtained video data, each video track comprising a sample of the video sequence, and the video tracks being associated with descriptive metadata, the descriptive metadata comprising: spatial information related to one or more samples in the associated video track; and composite information for organizing the generated video tracks to obtain a complete picture when displayed by a client; and generating a media file comprising the generated video tracks.

Description

[0001] (This application is a divisional application of application No. 2018800225834, filed on March 20, 2018, entitled “Method for Generating Media Data.”) Technical Field

[0002] The present invention generally relates to the field of encapsulating and parsing timed media data, e.g. according to the ISO Base Media File Format as defined by the MPEG standards organization, to provide a flexible and extensible format that facilitates interchange, management, editing and presentation of media data and improves stream management. Background Art

[0003] The International Organization for Standardization Base Media File Format (ISO BMFF, ISO / IEC 14496-12) is a well-known, flexible, and extensible format for describing coded, timed media data bitstreams for local storage or transmission over a network or via a bitstream delivery mechanism. This file format is object-oriented. It consists of building blocks called boxes, which are organized sequentially or hierarchically and define parameters of the coded, timed media data bitstream, such as timing and structure parameters. In this file format, an entire presentation is referred to as a movie. An animation is logically divided into tracks. Each track represents a timed sequence of media data (e.g., a frame of video). Within each track, each timed unit of data is called a sample; this can be a frame of video or audio. Samples are implicitly numbered sequentially. An animation can be organized as a list of animation and track fragments. An animation fragment extends the presentation in time. These animation fragments provide information that previously existed in a box called a MovieBox. The actual samples are in a box called a MediaDataBox. Within an animation clip, there is a collection of track fragments (zero or more track fragments for each track). A track fragment contains zero or more track runs, each of which records a continuous run of samples for that track.

[0004] The file format has another part (Part-5, ISO / IEC 14496-15) that describes encapsulation tools for various video coding formats based on NAL (Network Abstraction Layer) units. Examples of such coding formats are AVC (Advanced Video Coding), SVC (Scalable Video Coding), HEVC (High Efficiency Video Coding), or L-HEVC (Layered HEVC).

[0005] HEVC and similar video coding formats define different spatial subdivisions of samples (e.g., pictures): tiles, slices, and slice segments. A tile defines a rectangular region of a picture containing an integer number of coding tree units (CTUs) or coding blocks (hereinafter collectively referred to as coding units). As such, a tile is a good candidate for representing a region of interest (ROI). However, the organization of the coded video data (bitstream) in terms of syntax and its encapsulation into NAL units (or NALUs) is based on slices and slice segments (as in AVC).

[0006] A slice in HEVC is a set of slice segments, where at least the first slice segment is an independent slice segment and the other slice segments (if present) are dependent slice segments. A slice segment contains an integer number of consecutive (in raster scan order) CTUs. A slice does not necessarily have to have a rectangular shape (and therefore is not as suitable for ROI performance as a block). A slice segment is encoded in the HEVC bitstream as: slice_segment_header followed by slice_segment_data. The difference between an independent slice segment (ISS) and a dependent slice segment (DSS) lies in their headers: a dependent slice segment has a shorter header due to the reuse of information from the header of the independent slice segment. Both independent and dependent slice segments contain a list of entry points in the bitstream.

[0007] When encoding a video bitstream using blocks, there are tools in Part 15 of the MPEG file format to describe the mapping of NAL units to blocks. This Part 15 describes tools for encapsulating blocks as follows:

[0008] All chunks in a single video track; or

[0009] Each HEVC tile in the tile track with common information stored in the tile base track; or

[0010] A set of HEVC tiles forming a rectangular area in the tile track with common information in the tile track stored in the tile base track.

[0011] Media content creators want to provide different versions of a video, i.e., several videos generated from the same original video by compositing, image effects, or manipulations applied to recorded video samples. The problem is providing these different renderings of the media content without increasing the size of the resulting media file too much. Summary of the Invention

[0012] The present invention is designed to solve one or more of the above problems. The proposed solution performs a standard encapsulation of media tracks and further constructs one or more special tracks called generated tracks that contain only generation rules and parameters based on the input media tracks. Thus, the media data is stored once and referenced in the generated tracks. Each generated track consists of a set of transformation operators and parameters described in its track header, with a pointer to the original track used. Depending on whether the transformation parameters are static (same for all samples) or dynamic (changing based on the sample), there are alternative descriptions and storage of the transformation parameters. For example, one possibility is to store the operators or transformation parameters at the sample level in the "mdat" box (media data box).

[0013] According to a first aspect of the present invention, there is provided a method for encoding media data comprising a plurality of samples, the method comprising: utilizing a server device,

[0014] - encoding the media data to obtain encoded media data;

[0015] - encapsulating the encoded media data in a set of at least one first track;

[0016] -Generate the second track;

[0017] in,

[0018] - the second track describes samples representing the result of applying a set of at least one transformation operator to at least one sample of the first track;

[0019] - said second track comprises a reference to at least one of said first tracks; and

[0020] - The second track includes the description of the set of transformation operators in the metadata section.

[0021] In an embodiment, the method further comprises:

[0022] - encapsulating one or more parameters characterizing at least one transformation operator in a metadata portion of said second track.

[0023] According to another aspect of the present invention, a method for decoding media data comprising a plurality of samples is provided, the method comprising: utilizing a client device,

[0024] - receiving a first track comprising a reference to at least one second track and a description of at least one set of transformation operators;

[0025] - receiving said at least one second rail; and

[0026] - generating samples of said media data by applying said set of transformation operators to samples received in said at least one second track.

[0027] According to another aspect of the present invention, a server device is provided for encoding media data comprising a plurality of samples, the server device comprising a processor, the processor being configured to:

[0028] - encoding the media data to obtain encoded media data;

[0029] - encapsulating the encoded media data in a set of at least one first track;

[0030] -Generate the second track;

[0031] in,

[0032] - the second track describes samples representing the result of applying a set of at least one transformation operator to at least one sample of the first track;

[0033] - said second track comprises a reference to at least one of said first tracks; and

[0034] - The second track includes the description of the set of transformation operators in the metadata section.

[0035] According to another aspect of the present invention, a client device is provided for decoding media data comprising a plurality of samples, the client device comprising a processor, the processor being configured to:

[0036] - receiving a first track comprising a reference to at least one second track and a description of at least one set of transformation operators;

[0037] - receiving said at least one second rail; and

[0038] - generating samples of said media data by applying said set of transformation operators to samples received in said at least one second track.

[0039] According to another aspect of the present invention, there is provided a computer program product for a programmable device, the computer program product comprising an instruction sequence for implementing the method according to the present invention when the instruction sequence is loaded into and executed by the programmable device.

[0040] According to another aspect of the present invention, a computer-readable storage medium storing instructions of a computer program is provided, wherein the instructions are used to implement the method according to the present invention.

[0041] As mentioned earlier, HEVC and similar video coding formats define different spatial subdivisions of samples (e.g., pictures): tiles, slices, and slice segments. HEVC provides implicit rules for composing the tiles that the client wants to display. Other similar video coding formats (such as AVC or OMAF) do not benefit from these implicit rules and need to provide composition information so that the client can display a video composed of sub-parts. The composition information is coupled with spatial information (position and size) about the sub-parts to be composed.

[0042] However, the methods and devices provided in the prior art are static and do not allow the spatial information to be adjusted over time as often as needed. This can be quite problematic for some applications (particularly those that allow the synthesis of spatial video parts) when the position and size of the video parts change (for example, when tracking moving objects in a video sequence).

[0043] The present invention is also designed to solve the above concerns.

[0044] According to another aspect of the present invention, a method for generating a media file from a video sequence is provided, the method comprising: utilizing a server,

[0045] - obtaining video data comprising a plurality of samples from the video sequence;

[0046] - generating video tracks based on the obtained video data, each video track comprising a sample of a video sequence, and the video tracks being associated with descriptive metadata, the descriptive metadata comprising:

[0047] spatial information relating to one or more samples in the associated video track; and

[0048] Composition information, used to organize the generated video tracks to obtain a complete picture when displayed by the client; and

[0049] - Generating a media file including the generated video track.

[0050] In other words, the present invention proposes implementing spatial information so that it can be associated with one or more samples. The number of samples of interest can be adapted to the situation, thus providing greater flexibility. Furthermore, in cases where the coding format already provides spatial information associated with one or more samples, the present invention makes it possible to avoid duplication of this spatial information by reusing it to construct a video sub-portion.

[0051] According to another aspect of the present invention, a method for parsing a media file is provided, wherein the media file includes video data corresponding to a video sequence, and the video data includes a plurality of samples. The method comprises: utilizing a client,

[0052] - obtaining video tracks from the video data, each video track comprising a sample of a video sequence, and the video tracks being associated with descriptive metadata, the descriptive metadata comprising:

[0053] spatial information relating to one or more samples in the associated video track; and

[0054] Composition information used to organize the generated video tracks to get a complete picture when displayed by the client.

[0055] According to another aspect of the present invention, a program is provided. When the program is executed by a computer or a processor, the computer or the processor is caused to execute the above method.

[0056] According to another aspect of the present invention, there is provided a computer-readable storage medium storing the above-mentioned program.

[0057] According to another aspect of the present invention, there is provided an apparatus for generating a media file from a video sequence, the apparatus comprising:

[0058] - means for obtaining video data comprising a plurality of samples from said video sequence;

[0059] - means for generating video tracks based on the obtained video data, each video track comprising a sample of a video sequence, and said video tracks being associated with descriptive metadata comprising:

[0060] spatial information relating to one or more samples in the associated video track; and

[0061] Composition information, used to organize the generated video tracks to obtain a complete picture when displayed by the client; and

[0062] - Means for generating a media file comprising the generated video track.

[0063] According to another aspect of the present invention, an apparatus for parsing a media file is provided, wherein the media file includes video data corresponding to a video sequence, the video data including a plurality of samples, the apparatus comprising:

[0064] - means for obtaining video tracks from said video data, each video track comprising a sample of a video sequence, and said video tracks being associated with descriptive metadata, said descriptive metadata comprising:

[0065] spatial information relating to one or more samples in the associated video track; and

[0066] Composition information used to organize the generated video tracks to get a complete picture when displayed by the client.

[0067] At least a portion of the method according to the present invention can be implemented by a computer. Thus, the present invention can take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects that may all generally be referred to herein as "circuits," "modules," or "systems." Furthermore, the present invention can take the form of a computer program product embodied in any tangible medium having a representation of computer-usable program code embedded in the medium.

[0068] Since the present invention can be implemented in software, the present invention can be embodied as computer-readable code on any suitable carrier medium for provision to a programmable device. Tangible, non-transitory carrier media may include storage media such as floppy disks, CD-ROMs, hard drives, magnetic tape devices, or solid-state memory devices. Transient carrier media may include signals such as electrical signals, electronic signals, optical signals, acoustic signals, magnetic signals, or electromagnetic signals such as microwave or RF signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Embodiments of the present invention will now be described, by way of example only, and with reference to the following drawings, in which:

[0070] Figure 1 shows a typical client-server system in which the present invention may be used;

[0071] Figure 2 shows different ways of generating tracks from visual tracks according to embodiments of the present invention;

[0072] Figure 3 FIG. 1 shows a generated visual track design according to an embodiment of the present invention;

[0073] Figure 4 A flowchart representing a method for packaging a media file containing one or more generated visual tracks according to an embodiment of the present invention is shown;

[0074] Figure 5 The following illustrates parsing of a media file or media segment according to an embodiment of the present invention;

[0075] Figure 6 Showing the generation of samples in an embodiment of the present invention;

[0076] Figure 7 A block diagram representing a server or client device that can implement the steps of one or more embodiments;

[0077] Figure 8a and Figure 8b The present invention illustrates a packaging process for a video corresponding to a wide-view scene and extraction of the wide-view scene in an embodiment of the present invention;

[0078] Figure 9 shows the encapsulation of sub-picture tracks forming a track group according to the present invention;

[0079] Figure 10 Shows the different types of orbital groups used in spatial compounding;

[0080] Figure 11 Show implicit and explicit composition;

[0081] Figure 12 shows a parsing process for a media file utilizing implicit compositing; and

[0082] Figure 13 Subsets of track groups as well as overlapping track groups according to the present invention are shown. DETAILED DESCRIPTION

[0083] Figure 1 A typical client server system is shown in which the present invention may be used. The invention is not limited to such systems as the invention may relate to the generation of media files that may be distributed in any manner, not just by streaming over a communications network.

[0084] On the server side, the system includes: a media encoder 100 (particularly a video encoder), a media packager 110, and a media server 120. The media packager 110 includes a NALU parser 111, a memory 112, and an ISOBMFF (ISOBMFF) writer 113. The media server 120 can generate a manifest file (MPD) 121 and media segments 122. On the client side, the system also includes a media client 150 having the following different modules: an ISOMBFF parser 152; a media decoder 153 (particularly a video decoder); a display 154; and an HTTP client 151 for supporting adaptive HTTP streaming, in particular, parsing a streaming manifest 159 to control the streaming of media segments 190. The system also includes a module capable of operating on the encoded bitstream and / or decoded pictures: a transformation module 155.

[0085] Typically, a media client 150 requests a manifest file 121 in order to obtain a description of the different media representations available on the media server 120. Thus, the media client 150 then requests the media segments 122 that it is interested in. These requests are made via an HTTP module 151. The received media segments are then parsed by an ISOBMFF parser 152, decoded by a video decoder 153, and optionally transformed by a transformation unit 155 for playback on a display 154.

[0086] The video sequence is typically encoded by a video encoder 100 in order to be packaged into one or several files by a media packager 110. A media server 120 makes the generated files available to clients.

[0087] According to some embodiments of the present invention, another module is included in the system, either as part of the media packager or external to the media packager: a generation unit 130. This unit allows the user to view the packaged media tracks and edit and modify them by applying various image operations to the samples. The generation unit interacts with the media packager when the user wants to insert his composited result as an additional track into the media file produced by the ISOBMFF writer.

[0088] The media server is optional in the sense that the present invention primarily deals with media encapsulation and construction / description of tracks produced by compounding, derivation, synthesis, transformation, or aggregation of one or more media tracks. For the media server, the transport portion (HTTP module and manifest parser) is optional in the sense that the present invention is also applicable to media clients including simple media players to which the encapsulated media files are provided for rendering. The media files can be provided by full download, by progressive download, by adaptive streaming, or simply by reading the media files on disk or from memory.

[0089] The method according to the present invention is performed by the media packager module 110, and more specifically by the ISOBMFF writer module 113 in cooperation with the composition unit 130, wherein the method is performed by the media packager module 110 and more specifically by the ISOBMFF writer module 113 in cooperation with the composition unit 130. Figure 7 The server device is shown as including software code when executed by its CPU 704.

[0090] Typically, the encapsulation module is responsible for reading the high-level syntax of the video bitstream comprising the compressed video to extract and identify the different NAL units, and to organize the coded data in the ISOBMFF file or ISOBMFF segment 122 containing the video bitstream into one or more tracks with descriptive metadata according to the ISOBMFF box hierarchy. The ISOBMFF file or ISOBMFF segment 122 may also contain one or more other synchronized media tracks, such as audio, subtitles, text, metadata... According to one aspect of the present invention, the ISOBMFF writer generates additional tracks for the media tracks, which are produced by applying compositing or transformation or derivation or special effects to the media tracks by the compositing unit 130. In order to avoid duplicating data (i.e., original data + generated data), the ISOBMFF writer 113 constructs one or more special tracks called generated tracks, which, instead of providing the results of the transformation, describe the transformation rules and parameters to construct the generated tracks. A generated track is created for each composite or derivation. This has the advantage of keeping both the original track and the transformed track in the same media file. Reference Figure 4 To further detail the encapsulation process. Figure 5 and Figure 6 To detail the parsing, reading, and rendering of media files or media segments.

[0091] Figure 2 Different ways of generating tracks from visual tracks are shown. Generating visual tracks may be generated by applying one or more operations to one or more video tracks.

[0092] Figure 2 (a) describes a generated visual track 202 obtained from a single input video track 200 via a single transformation operator 201 .

[0093] Figure 2 (b) describes a generated visual track 212 obtained from a single input video track 210 through multiple transformation operators 211.

[0094] Figure 2 (c) describes a generated visual track 222 obtained from multiple video tracks 220 through a single operation 221.

[0095] Figure 2 (d) describes a generated visual track 232 obtained from the plurality of video tracks 230 through the plurality of operations 231 .

[0096] There is a case where the samples of the generated track are calculated using the same transformation operator and the same transformation operator parameters for all samples. This transformation is called a static transformation. In contrast, there is a case where each sample can have different transformation operator parameters from each other. In this case, dynamic transformation is discussed. Although the current design for generating visual tracks is suitable for dynamic transformations, the design incurs some description overhead for static transformations (repeating transformation operators and parameters in each generated sample). In addition, the current design implies describing the transformation in the media data box. This may be sensitive to track editing, for example, when adding or removing tracks from a media file, the offset of the generated samples may be affected. The present invention proposes to describe the transformation in the metadata box to the greatest extent possible, thereby avoiding mixing with the media data. To this end, Figure 3 The design shown suggests defining transformation rules in the metadata of the generated track and, where possible, declaring parameters in the track metadata, at least for static transformations. For dynamic transformations, the transformation parameters can be stored in an external track (e.g., a timed metadata track) or in the generated samples. The external track can be in the encapsulated media file, but can also be contained within another media file. The design according to the present invention supports both static and dynamic transformation parameter declaration.

[0097] Figure 3 The generated vision track design according to an embodiment of the present invention is shown.

[0098] A generated visual track (GVT) (or derived visual track (DVT)) is described as a video track. The handler for the GVT is set to a reserved four-character code: "vide". The tracks from which the GVT is generated are referenced via track reference types with the value "gtrk" or ("dtrk") starting from the GVT 310 and pointing to the respective input video tracks. For example, GVT 310 is generated from input video tracks 301 and 309, and track 305 is not used in the generation process. For any track, the GVT 310 contains a data reference box 312 indicating whether the data is in this file or in a remote file. In Figure 3 In the example shown, data related to the GVT means that the samples are in the media data box "mdat" 320. The data reference box 312 can also be used to reference metadata tracks that provide, for example, transformation parameters stored in other media files. The GVT 310 contains a SampleTableBox "stbl" 313 that describes the samples.

[0099] The generated visual track contains a TransformPropertyContainer 314 in its descriptive metadata box. This TransformPropertyContainer 314 contains a list of transforms or operations 341-344 to derive the generated visual track from the input tracks. Each of these transforms or operations is declared with a parameter indicating whether the transform or operation is a static transform or a dynamic transform. The transform or operation also contains a parameter indicating whether the transform or operation applies to a single input media track or a list of input media tracks. Another parameter of the TransformProperty indicates whether the operation or transform is applied after or before decoding of the input samples.

[0100] For example, reference Figure 2 (a), generate visual track 310 and reference only one video track through track reference type 311. Then, TransformPropertyContainer 314 contains a single TransformProperty type. Figure 2 (b) Generate visual track 310 and reference a single video track through track reference type 311. Then, TransformPropertyContainer 314 contains a list of TransformProperty types. Figure 2 (c) Generate visual track 310 and reference multiple video tracks through track reference type 311. TransformPropertyContainer 314 contains a single TransformProperty type. Figure 2 (d) Generate visual track 310 to reference multiple video tracks through track reference type 311. Then, TransformPropertyContainer 314 contains a list of TransformProperty types.

[0101] The TransformProperty box can be specialized into a single input track TransformProperty box and a multi-track TransformProperty box. In a preferred embodiment, this is done by specifying different values ​​for the "version" parameter of the TransformProperty box. When the "version" parameter of the TransformProperty box takes the value 1, this means that a single input track is provided as a transform operand. When the "version" parameter of the TransformProperty box takes the value 2, this means that multiple input tracks are treated as transform operands; that is, the generated samples are calculated from many input video samples.

[0102] Then, it is proposed to generate a visual track design, thereby introducing a transform container box 314 and defining a TransformProperty box as follows:

[0103] Declaration of the transform used to generate the track:

[0104] Box Type: 'tpcb'

[0105] Container: TrackBox(′trak′), only in a track that declares

[0106] at least one track reference type of type `gtrk′,

[0107] Mandatory (per track): No

[0108] Quantity(per track):Zero or one

[0109] This box provides a list of transformations to be applied to the samples used for track generation:

[0110]

[0111] has the following semantics:

[0112] The flags parameter of this box is used to describe the kind of transformations. It also provides an indication of the complexity of these transformations. The flag values ​​(with the names and values ​​provided below as examples) allow Figure 2 A high-level description in the file of the different configurations is shown. This way, an ISOBMFF parser or reader can easily check whether it can handle the generated track.

[0113] All_static: Indicates that all included TransformProperty boxes correspond to static transforms. The flag value is 0x000001. This means that the generated track does not contain generated samples, because the samples can be generated entirely from TransformProperty instructions.

[0114] All_single_input: Indicates that all contained TransformProperty boxes refer to a single input track. The flag value is 0x000002. When the all_single flag value is set, the TransformPropertyContainer only contains TransformProperty boxes with version = 1. When the all_single flag value is not set, no conclusion can be drawn about the type of contained TransformProperty boxes. This can be all dynamic TransformProperty boxes (with version = 2) or a mix of static and dynamic transforms.

[0115] The num_entries parameter indicates the number of consecutive transforms that are applied to the input track samples to construct the generated samples of the generated track.

[0116] In an alternative embodiment, the version parameter of the TransformPropertyContainer can be used to indicate the Figure 2 Different configurations. For example:

[0117] In version 1, a TransformPropertyContainer box declares only one TransformProperty box. Thus, there is no need to specify the number of entries.

[0118] Combining version=1 with the all_single_input flag indicates that Figure 2 Configuration (a) is in use. When the all_single_input flag is not set, this signaling indicates Figure 2 Configuration (a) or (c) above.

[0119] When version=2, the TransformPropertyContainer box declares more than one TransformProperty box. In this case, the number of entries (the number of TransformProperties used to construct the generated track) should be provided. When combined with the value of flag all_single_input, this indicates Figure 2 When the all_single_input flag is not set, this signaling indicates Figure 2 Configuration (a) or (c) above.

[0120] The box is then rewritten as:

[0121]

[0122] In an alternative embodiment, the information contained in the flag value "all_static" can be given as a special VisualSampleEntry in the SampleTableBox of the generated track, indicating that the sample of this track is actually a null sample. For example, the reserved code "gtse" is used, which means the generated track sample is null. The transformations applied to the media samples are described by the TransformProperty box as follows (TransformProperty is a general box used to declare common parameters for all specific types of transformations (other TransformProperty boxes that extend this general TransformProperty box can provide transformation parameters)).

[0123]

[0124] For the case where the TransformProperty has dynamic parameters declared in the sample (the dedicated parameter "params_in_samples" in the TransformProperty box is set to 1), the transform parameters are declared in the generated sample and stored in the MediaData box:

[0125] aligned(8)class GeneratedSample(){TransformationParametersparameter;}

[0126] has the following semantics:

[0127] num_inputs specifies the number of input samples to the transform.

[0128] input_trackID indicates the track_ID of the source track that the input samples to the transform come from. When set to 0, indicates the output of the previous transform or operation.

[0129] params_in_samples indicates whether the transform parameters are declared in the derived samples.

[0130] Optionally, an additional flag may indicate whether the transformation parameters are in another track of the same media file or in another track of a remote media file when params_in_samples = 0. In the latter case, the track may be reached, for example, via a data URL box in the DataEntryBox of the generated visual track.

[0131] param_track_ID indicates the track_ID of the track that declares the transform parameters (for example, parameter track 340). This track can be referenced by the generated visual track via a track reference type with a specific value (for example, "dtrp" for derived track parameters). (The names and codes here are only examples; any reserved codes that do not conflict with other track reference types are for convenience.)

[0132] In an alternative embodiment, instead of referencing track_ID directly, the track_ref_index_or_zero parameter can be used: this parameter is a 1-based index into a TrackReferenceTypeBox of type "gtrk", or the value 0 to indicate the output of a previous ImageOperation. When using this way of referencing the input track, the generic TransformProperty with version=1 can even be empty, because for version=1 there is only one input track, and it can be easily retrieved from the track reference box of the generated track. The TransformProperty box is then rewritten as follows:

[0133]

[0134] Similarly, the param_track_ID parameter may be an index to a track reference type 341 with the value "dtrp" that references the track that provides transform parameters. Using a track reference type encoded on 8 bits results in a shorter description than track_ID. One advantage of track_ID is that it arises when the track is contained in an external or remote media file.

[0135] The transform_ID is a unique identifier that can be used to reference the declared TransformProperty across all transforms.

[0136] In addition, still according to an embodiment of the present invention, the flag of TransformProperty can take the following values ​​to provide more information about the transformation to construct the generated visual track:

[0137] Transform_is_static: Indicates that the TransformProperty has fixed parameters for a given track or track fragment. The flag value is 0x000001. A static TransformProperty contains the values ​​of the parameters of the transformation to be applied to the input sample. These are specified by one of the special TransformProperty boxes called TransformationParameters. When the flag value is not set, the TransformProperty box will indicate where the transform parameters are defined: in the generated sample, in a track within the media file, or in a track outside the media file.

[0138] Transform_post_decoding: Indicates that the TransformProperty is applied after the samples are decoded for generation. The flag value is 0x000002. Post-decoding TransformProperty requires that the input samples be decoded before the transform is applied. When this flag is not set, the TransformProperty is applied to the compressed samples.

[0139] Transform_preserve_samples: Indicates that (after applying the TransformProperty) the number of samples obtained in the resulting track will be the same as the number in the input track. The flag value is 0x000004. When this flag is not set, the resulting track will contain a different number of samples than the number of samples in the input track.

[0140] In an alternative embodiment using the "version" parameter, various TransformProperty boxes are defined that inherit from the abstract TransformProperty box. For example, the StaticTransformProperty box and the DynamicTransformProperty box each define one trackID for a transform operand (when the track_ID is used to reference an input track) or nothing (when the input track is referenced implicitly by the track reference box of the generated track), and define an ordered list of trackIDs for multiple transform operands (either a 1-based index into a TrackReferenceTypeBox of type "gtrk", or a value of 0 to indicate the output of a previous ImageOperation when the track_reference box is used).

[0141] When the transform used to generate the visual track is a static transform, the generated visual track may not contain generated samples, or at least contain empty samples. For this case, the SampleTableBox "stbl" 313 then becomes optional, and the generated visual track has a minimal set of metadata boxes, thereby becoming more compact. In a preferred embodiment, the generated visual track contains only the initialization data in the moov and track boxes and no other data, where the transform is described in these boxes. In this preferred embodiment, the track header contains a specific value for the "flags" parameter to indicate that the track is an empty track (no sample data). This information can also be provided by any dedicated parameter in the track header box or subboxes, such as the all_static value interpreted using the semantics of the TransformProperty box. An alternative embodiment is to define more explicit and dedicated flag values ​​for the track header box, for example:

[0142] Empty_samples: Indicates that this track contains no sample data in the Media Data Box. The flag value is represented in an unused bit of the 24-bit Flags parameter (so as not to conflict with existing flag values). For example, the flag value could be 0x000016. When this unused bit is set to 1, it means that the track contains no sample data in the Media Data Box. The default value of this bit is 0. When this value is set, the track can declare a special VisualSampleEntry in its SampleTableBox indicating that the sample is actually empty.

[0143] There may be cases where a track (regardless of the track, not just the generated track) has some samples that are empty (not all empty), for example, when the generated track does not take into account all samples from the input media track, or when one or more samples are lost or damaged during transmission and then no data is stored for these samples. To represent this configuration at the track header level, the following flag values ​​are defined (the value names are just examples):

[0144] Partial_samples: Indicates that some samples of the track may be empty or damaged. The flag value is represented in an unused bit of the 24-bit flag parameter (so as not to conflict with existing flag values). For example, the flag value can be 0x000032. When this unused bit is set to 1, it means that the track contains samples with no data in the media data frame. The default value of this bit is 0. Indicating this information at the track level is useful for media players: in order to correctly process a media file that includes a track with this flag value, the track must be robust to missing data for some samples.

[0145] These flags are not limited to generating tracks. When the configuration of a track changes over time, these flags can also be redefined in the track fragment (for example in the tf_flags value of the TrackFragmentHeaderBox "tfhd").

[0146] This informs the media player that the TransformationPropertyContainer 314 and its list of TransformProperty boxes (341-344) are sufficient to construct the generated sample. Figure 3 In the example above, the media data box 320 does not have data for generating samples.

[0147] In an alternative embodiment, the generated visual track 310 with static transformations still has an "stbl" box 313. This "stbl" box may contain a VisualSampleEntry indicating that the sample is empty, i.e., the sample does not have any data in the media data box 320. In this embodiment, the stbl box also contains a decoder configuration record.

[0148] In another alternative embodiment, the generated visual track 310 contains an stbl box 313 in which all mandatory boxes are present for the sample description (this is to accommodate players that are not robust to missing mandatory boxes). In this case, the generated samples are declared as references to specific VisualSampleEntries. Each specific VisualSampleEntry with its own four-character code indicates whether the generated sample is actually an empty sample ("dtre"), a complete sample that contains a transform parameter ("dtrf") sample in its data, or a sample for which transform parameters are declared in another track rather than in the input media file ("dtri") or in a remote track ("dtro").

[0149] When a generated visual track contains a SampleDescription box with a "dtre"VisualSampleEntry, this means that there is no data for the derived samples, i.e. all parameters are static and declared in the metadata box of the derived or generated visual track. As such, the size of these samples is set to 0. If the transform_preserve_samples flag value is set to 1, the decoding and composite timestamps are those from the input track. This particular VisualSampleEntry is declared as follows:

[0150] aligned(8)class DerivedEmptySampleEntry extends VisualSampleEntry

[0151] (′dtre′){

[0152] EmptyDerivedTrackCorfigRecord derived_config_rec; / / mandatory

[0153] }

[0154] Where the derived_config_rec parameter is the following box:

[0155]

[0156] where transfo_list is a TransformPropertyContainer box that declares a list of static TransformProperty boxes with types and parameters.

[0157] The sample referencing this VisualSampleEntry is constructed using the transform list declared in this EmptyDerivedTrackConfigRecord. Alternatively, the transform_list can be declared directly at the visual sample entry level without any intermediate EmptyDerivedTrackConfigRecord. When the TransformPropertyContainer box is declared at the VisualSampleEntry level, it is no longer necessary to declare the TransformPropertyContainer box at the media handler or track header level.

[0158] When a generated visual track contains a SampleDescription box (VisualSampleEntry) with "dtrf", this means that the samples of that track contain in their body or data a declaration of all the transform parameters to be applied to the input samples to generate them. Store this sample data in the media data box of the track. Set the sample to byte offset and sample to block offset appropriately in the appropriate boxes of the SampleTableBox. Declare this particular VisualSampleEntry as follows:

[0159]

[0160] Where the config_record parameter is the following box:

[0161]

[0162] Where transform_list is an array of TransformProperty boxes with type but no parameters, as the box is dynamic. At least one of these TransformProperty boxes is dynamic. Static TransformProperty boxes embed the transform parameters, while dynamic TransformProperty boxes embed the transform parameters in the sample data, which is then stored in the media data box:

[0163] aligned(8)class DerivedSample(){

[0164] Trans formation Parameter params[];

[0165] }

[0166] The sample that references this VisualSampleEntry is constructed using the transform list declared in this FullDerivedTrackConfigRecord, and also using the transform parameters declared in the sample body. Note that in this case, derived samples can mix static and dynamic transforms. Static transforms can be referenced by a unique transform_ID, and dynamic transforms are followed by the appropriate parameters to be applied.

[0167] When a generated visual track contains a SampleDescription box with a "dtri"VisualSampleEntry, this means that the transform parameters for the samples of the track to be constructed, derived, or generated are contained in a track present in the media file. For each sample to be constructed, the transform parameters can be found in the corresponding sample in the indicated track (via the track_ID or index of the track in the track reference type 341 indicating the track reference for the transform parameters).

[0168] Declare this particular VisualSampleEntry as follows:

[0169]

[0170] Among them, the config_record parameter parameter is the following box:

[0171]

[0172] Here, transform_list is an array of TransformProperty boxes of type and eventually some parameters (due to being dynamic). At least one of these TransformProperty boxes is dynamic and indicates that the parameter in the sample is always false, which means that the transform parameters should be retrieved from a track in the media file. The Id of this track can be included in the TransformProperty. This means that to generate or derive a sample, the input sample is transformed using the transform parameters of the corresponding sample in the parameter track (same decoding time).

[0173] When a generated visual track contains a SampleDescription box with a "dtro"VisualSampleEntry, this means that the transform parameters for the samples of that track to be constructed, derived, or generated are contained in an out-of-band track present in the remote file. For each sample to be constructed, the transform parameters can be found in the corresponding sample of the indicated track (via the track_ID in the remote media file). Alternatively, if a media data box with an identifier is available, that identifier can be used in the sample data to reference the transform parameter value in the remote track.

[0174] Declare this particular VisualSampleEntry as follows:

[0175]

[0176] Where the config_record parameter is the following box:

[0177]

[0178] Where transform_list is an array of TransformProperty boxes with a type and eventually some parameters. At least one of these TransformProperty boxes is dynamic and indicates that the parameter in the sample is always false, which means that the transform parameters should be retrieved from a track external to the media file. The Id of this track can be included in the TransformProperty. This means that to generate or derive the sample, the input sample is transformed using the transform parameters of the corresponding sample in the external parameter track (same decoding time).

[0179] The last approach has the advantage of supporting both static and dynamic transformations, in particular for dynamic transformations where some generated samples are empty, i.e. where the generated visual track does not have a one-to-one mapping with the input samples (the generated visual track may only have a subset of the transformed input samples). In this case, the generated visual track 310 has a SampleTableBox 313 that defines at least two different VisualSampleEntries: one VisualSampleEntry for declaring generated samples, for example, with a retained "gtrk" code, and one VisualSampleEntry for declaring an empty sample as described above. Each generated sample is then declared to contain data in 320 or to be empty, by reference to one of these at least two different VisualSampleEntries. This configuration can be represented by the flag value of the TransformProperty box: Transform_preserve_samples. In order to skip samples in the generated visual samples, a special TransformationParameter "no-op" (no operation) is defined that simply involves placing an empty sample at a given time.

[0180] No-Operation Transform Type

[0181] Box Type: 'nopt'

[0182] Mandatory(per sample):No

[0183] Quantity(per sample):Zero or one

[0184] A "no-op" transform involves generating an empty sample. When a no-op transform attribute is present in a sample, the other transform attributes are ignored.

[0185] syntax:

[0186] aligned(8)class NoOpTransform extends TransformProperty(′nopt′){

[0187] }

[0188] Finally, for dynamic transformations, in combination with or without static transformations, there are the following partial reciprocal alternatives to the transformation description:

[0189] - All transformation rules and parameters are static in the metadata box of the track, and the track has no generated samples (the track is then marked as having empty samples).

[0190] - The transformation parameters are reflected in the specific VisualSampleEntry listed in the SampleTableBox 313, each generated sample references one of these VisualSampleEntry.

[0191] - Simply declare the transformation parameters in the sample placed in the media data box 320.

[0192] - Declare transform parameters in an external track (eg, a timed metadata track).

[0193] Figure 3 The flexibility of the design shown and disclosed above allows combining various description possibilities and allows covering both static and dynamic transformations and mixing the two in the same generated vision track.

[0194] The goal of the Transform_post_decoding value of the TransformProperty "flags" parameter is to reduce the number of decoding operations. When packaging content, the editor knows which transform to apply to the content and how the samples are constructed for the content: using this flag, the content editor can apply as many transforms as possible in the compressed domain and then decode the latest generated samples as possible, avoiding compositing in the pixel domain and having to decode and store intermediate samples multiple times.

[0195] Depending on the complexity of the track transformations declared in the media file, for example via the "version" parameter of the TransformProperty or TransformPropertyContainer box, a brand (branch) or compatible brand at the beginning of the media file (e.g., in the ftyp box) can quickly indicate to the media player the complexity of the generated tracks in the file. For example, two brands can be defined as "dtrs" and "dtra" for single track generation or transformation and for advanced (i.e., multi-track) generation or transformation, respectively. The former corresponds to a media file (or segment) that contains only TransformProperty boxes with version=1 or any equivalent parameter, which indicates that the transformation or operation for deriving samples only takes a single track as input. The latter corresponds to a media file (or segment) that contains a TransformProperty box with version=2 or any equivalent parameter, which indicates that the transformation or operation for deriving samples can contain more than one input track (i.e., a multi-track transformation). An alternative definition of brand can be Figure 2one brand for each configuration. For example, brand "dtr1" indicates a media file containing a simple derived track computed from a single input track involving a single operation (configuration 2(a)). This means that, according to the previous embodiment, the TransformationPropertyContainer box is a version=1 box (or any parameter indicating that the box contains a single transformation or operation). Following the same principle, other brands can be defined based on the number of input tracks and the number of operations or transformations applied to the input tracks. Depending on the capabilities, the client can then decide to process only media tracks (no generated tracks), a subset of the generated tracks (e.g., generated tracks with simple operations), or all generated tracks can be supported. The client's processing can involve filtering tracks while parsing the media file, or streaming the scene without ever selecting such complex or unsupported generated tracks.

[0196] An alternative embodiment that is particularly relevant when different transforms are applied to different samples, but generally respects a given sample mode when the value of the transform_preserve_samples flag of the generated track is not set, uses sample groups and associates each sample group with a transform or a transform set. A pattern of samples (or sample groups) is defined, and different transform properties (VisualSampleTransfoEntry) are applied to these sample groups or sample patterns. A SampleToTransformBox is defined with a transformation_type. The SampleToTransformBox and the SampleTransfoDescriptionBox are associated by this transformation_type. The VisualSampleTransfoEntry then contains at least an operator and a parameter of the operator. For example: operator = rotation and parameter (angle) = 90°. In other words, a VisualSampleTransfoEntry can correspond to one of the specific TransformProperty boxes.

[0197] Figure 4 A flow chart representing a method for packaging a media file containing one or more generated visual tracks according to an embodiment of the present invention is shown.

[0198] This involves the Figure 1The encapsulation of media streams generated by a media encoder 100 (in particular by a video encoder) according to the ISO base media file format. During step 400, the input media streams are received by the media packager 101. During step 401, the media packager 110 encapsulates the various media streams into one or more ISOBMFF tracks or track fragments. As a result of step 401, a media file, a fragmented media file or a collection of media segments is generated. The user or the content creator himself can then examine the generated media and create additional tracks representing alternative or modified views of the media content. This step is summarized as a composite step 402. This step involves selecting one or more input tracks from the encapsulated media tracks and defining the special effects, transformations or operations to be applied to the selected tracks in order to generate or produce different tracks.

[0199] During compositing, the user has many possibilities and options when transforming media tracks, such as:

[0200] - Select input samples from one or more media tracks

[0201] - Select one or more operations to apply to the input samples

[0202] -Apply the same or different transformation parameters to the selected input samples

[0203] - Perform a transformation on each selected sample or a subset of selected samples.

[0204] When the user has finished compositing, he may decide to record the results of his compositing in the same file as the input media tracks.

[0205] It is to be noted that the composition unit 130 and the composition building step 402 may also be integrated in the media packager 110 and pre-recorded as automatic transformations to be applied to the recorded media stream: for example, rotation of the video track, predefined cropping or resizing of the video, or any operation pre-planned or provided by the packaging device.

[0206] When the user decides to record, or if the media packager is configured to automatically create a generated track, the ISOBMFF writer 113 initiates a new track in the media file by inserting a "trak" box below the main "moov" box in step 404. This track box will contain a description of the composited generated track. A track header box "tkhd" is created with a new track_ID assigned to the new track. The ISOBMFF writer then describes the creation time and the dimensions (width and height) of the generated track in this track header box. This information can be obtained from the composite unit and / or parameterized by the user.

[0207] Then, in step 405, a track reference box "tref" is created to link the generated track to one or more input media tracks whose track reference type is equal to "gtrk" and from which the generated track is generated, to indicate that the track is "generated from" the reference track. Optionally, in the same step 405, the generated track and the input media track are also associated with the same track group via a TrackGroupBox (track group box) "trgr" with a specific grouping type value, which indicates that the tracks in the group are variants of the original media tracks.

[0208] Then, in step 406, the ISOBMFF writer describes the media data and how to handle the generated track. This is done in the track media "mdia" box, and specifically in the HandlerBox "hdlr" subbox. The handler_type is set to "vide" if the generated track is generated from a visual track, or to "sound" if the generated track is generated from an audio input track. The handler_type can even be extended to other media types, and the handler should then be set to the handler type of the input track used for the generation. Optionally, the name is set to "generated" in the HandlerBox so that the purpose of the track is clear to those who inspect the track. Depending on the handler_type value, media-specific handler information is set during the same step 406. When the generated track is generated from an input video track, a special VideoMediaHeaderBox "vmhd" is added to the track description and extended to declare the list of transforms involved in the track generation. To distinguish it from the existing VideoMediaHeaderBox and maintain backward compatibility, the VideoMediaHeaderBox for the generated track uses version="1":

[0209]

[0210] A specific VideoMediaHeaderBox actually requires two parameters: an indication that this specific VideoMediaHeaderBox is not a regular VideoMediaHeaderBox (the example of the version=1 value above), and a TransformPropertyContainer box (which declares the transform list used for the generation of the track).

[0211] Based on the user's selections related to composite builds, Figure 3The flags and version values ​​of the TransformPropertyContainer box and the list of TransformPropertyContainer boxes are set according to the relevant instructions. This is done in step 407. In the case of the all_static transform tested in step 408, the description of the generated track ends here in step 415, which means that no SampleTableBox is declared. If at least one of the transforms used to generate the track is not static, a SampleTableBox is created to describe how the generated samples can be calculated in step 409. This involves: in the case of dynamic parameters, setting the appropriate VisualSampleEntry in the sample description box, and when the transform parameters are stored in the generated sample, calculating the data of the generated sample (the transform parameters of the given sample) and storing this data in the memory of the media packager in step 410.

[0212] The ISOBMFF writer iterates over the list of operations or transformations declared in the generation track in test step 411. Whenever the next operation or transformation is a dynamic transformation (corresponding to test 412 being false), the generation samples are updated with the current operation parameters.

[0213] To support both static and dynamic parameters, whenever a static operation is encountered, the payload of the generated sample is updated with the identifier of the static transformation to be applied in step 413. Once the operation list is reached for a sample, the sample description (size, byte and / or block offset, decoding and / or compounding time in the data box) is updated in the appropriate sample description related boxes in step 414.

[0214] For really simple static transforms targeting a single input track that can be handled by the transform matrix in the track header, the resulting track is declared with empty_samples and the identity TransformProperty "idtt" is defined as a single operation. The transform parameters are then declared in the transform matrix in the track header box of the resulting track.

[0215] It is recommended to declare the transforms in the order of operations starting with the pre-decoding transform first, so that the number of decoding operations and storage of intermediate samples is kept as limited as possible.

[0216] The generated files or segments may further be placed on a streaming server for distribution, eg using adaptive streaming over HTTP.

[0217] When preparing content for adaptive streaming over HTTP, part or all of the generated visual track can be exposed in the streaming manifest. For example, when mapped to the manifest (Media Presentation Description) of the MPEG DASH streaming protocol, the generated visual track is exposed as a representation with the attribute dependencyId that references all input tracks to indicate the representation required to decode and reconstruct the generated track. Optionally, the "gtrk" reference type can be utilized to define the relationship between the representation of the generated visual track and the representations of its input video tracks. This can be done using the dependencyType attribute or by declaring an associationId and associationType between these representations. This allows media streaming clients to select the representation corresponding to the generated visual track for rendering in order to be informed that segments of the input tracks should also be streamed.

[0218] Figure 5 The parsing of a media file or a media segment according to an embodiment of the present invention is shown.

[0219] This description applies both when the file is fully available to the media client 150 and when the file is streamed to the media client 150. The processing of the media file begins with the receipt of initialization data for the media file in step 500. Based on this initialization data, the ISOBMFF parser 152 parses the moov box and determines in step 501 the number of tracks contained in the file of a type (video, audio, metadata, text, etc.).

[0220] Then, in step 502, the ISOBMFF parser 152 parses the relationship between all these tracks: if at least one track reference contains at least one "gtrk" track reference type to another track, then the file contains not only media data tracks, but also generated tracks. The parser then needs to decide whether to reveal these generated tracks to the user. Note that some generated tracks may be indicated as not intended to be displayed (e.g., when the flag value in the track header does not have a set Track_in_movie value), for example when used as an intermediate step in a complex generation. In this case, these generated tracks are not revealed to the client for selection and playback. To determine which tracks to reveal to the user, the parser looks at the list of TransformProperty boxes declared for the generated tracks. Only by parsing the type of transform in step 504 can the parser decide in step 505 whether the parser will be able to reconstruct the generated tracks.

[0221] In case a transform type is not supported or available in the media client, the generated track is not processed. A warning may be displayed to the user via a graphical user interface or sent to the application using a media parser, for example, which may then decide to select or switch to a streaming application that supports the track. Note that in the case of fragmented tracks, the initialization data may be updated in the moof box and these initial steps may be performed again. Note that when describing the media file in a streaming manifest such as a DASH MPD, the support for generated tracks may also be obtained via an ISOBMFF specific brand or streaming profile or any descriptor that signals the presence of a transform.

[0222] When the media player or media client supports the transforms declared in the generated track, the parser 152 checks whether all transforms are static in step 507. This information can be obtained by parsing the flag value in the Track Header box of the generated track, or by checking the version or flag value of the TransformPropertyContainer box, or by traversing the list of declared TransformProperty boxes of the generated track and checking their flag and version values.

[0223] For static transformations, the parser stores the operators and transformation parameters in client memory in step 510. When all transformations are static and the generation track is marked as empty_samples, the parser can start building generation samples by sequentially applying the transformation operators using the parameters to the input samples in step 511. The input samples are determined by parsing the TransformProperty box of the generation track.

[0224] In the case that at least some of the transformations are dynamic, which means that the test step 507 is false, the parser checks whether the transformation parameters are in the generated sample, in another track contained in the media file, or in an external track. The parser then starts building samples one by one in step 521 until the last input sample to be processed is tested in step 522.

[0225] Figure 6 The generation of samples in the embodiment of the present invention is shown.

[0226] The method assumes that the input track has been identified and the set of transform operators has been read. The processing is assumed to loop over consecutive samples to be generated and focus on the processing of a given sample.

[0227] At the start of the process an input sample is read and stored in step 600. It is worth noting that when complex numbers are used there may be only one input track and therefore one input sample.

[0228] In step 601, a first transformation operator is read.

[0229] In step 602, it is tested whether the current transformation operator is static or dynamic.

[0230] In the case where the transform operator is dynamic, the parameters of the operator must be obtained in step 603. These parameters can be obtained from the associated samples in the generation track or in the reference parameter track (local or remote). In the case where the transform operator is static, the parameters of the transform operator are the same for all samples and are assumed to be known at this stage of the processing.

[0231] In step 604, it is tested whether the transform operator is supposed to be applied to the encoded samples or the decoded samples. It is assumed here that the list of transform operators can contain only "pre-decoding" operators, only "post-decoding" operators, or both. When both are included in the list, the list advantageously first contains only "pre-decoding" transform operators and then only "post-decoding" transform operators. Thus, the input sample will be decoded only once.

[0232] In the case where the current transform operator is a "pre-decoding" operator, the input coded samples are read in step 605. For the second and further "pre-decoding" transform operators to be applied, the input coded samples are actually the generated coded samples stored by applying the previous transform operator. Then, in step 606, the current transform operator is applied to the input coded samples to obtain the generated coded samples stored in step 607.

[0233] In case the current transform operator is a "post-decoding" operator, the input samples are read in step 608. For the second and further "pre-decoding" transform operators to be applied, the input decoded samples are actually the generated decoded samples stored by applying the previous transform operator.

[0234] Then, only for the first "post-decoding" transform operator, the input samples are encoded and must be decoded in step 609. This is because, by definition, a "post-decoding" transform operator must be applied to decoded samples. In subsequent cycles, the decoding step 609 is not performed.

[0235] Then, in step 610 , the current transform operator is applied to the input decoded samples to obtain generated decoded samples which are stored in step 611 .

[0236] In step 612, it is tested whether the end of the list of transformation operators has been reached. If the end of the list of transformation operators has not been reached, the process loops back to the next operator in step 601.

[0237] When all transform operators have been applied, it is tested in step 613 whether only a "pre-decoding" operator is present. In this case, the current generated sample is encoded and needs to be decoded in step 614 before being ready for output in step 615.

[0238] When the input media track to a transform operator has dependencies on other tracks, such as through an extractor (explicit reconstruction) or implicit reconstruction of its samples, the input samples provided to the transform refer to reconstructed samples after reconstruction. This applies to both post-decoding operations and transforms. For pre-decoding operations, the input samples are formed by the concatenation of sample data including the sample itself plus data derived from the reconstruction rule resolution. For example, in the case of scalable video with dependencies between enhancement layer samples and base layer samples, when the transform is a pre-decoding transform, the input samples will refer to the concatenation of enhancement sample data plus base sample data. Similarly, the input samples to a post-decoding transform will be the reconstructed / decoded input samples consisting of the base samples plus the enhancement samples.

[0239] Figure 7 A block diagram represents a server or client device 700 that may implement the steps of one or more embodiments.

[0240] Preferably, the device 700 includes: a communication bus 702; a central processing unit (CPU) 704, which is capable of executing instructions from a program ROM 706 when the device is powered on, and instructions related to software applications from a main memory 708 after power is turned on. The main memory 708 is, for example, a random access memory (RAM) type used as a working area for the CPU 704 via the communication bus 702, and its storage capacity can be expanded using an optional RAM connected to an expansion port (not shown). For example, instructions related to software applications can be loaded into the main memory 708 from a hard disk (HD) 710 or the program ROM 706. When such software applications are executed by the CPU 704, reference is made to the server. Figures 4 to 6 The packaging step.

[0241] Reference numeral 712 is a network interface that enables the device 700 to be connected to a communication network 714. When executed by the CPU 704, the software application is configured to react to requests received via the network interface and provide data streams and requests to other devices via the network.

[0242] Reference numeral 716 denotes a user interface for displaying information to a user and / or receiving input from a user.

[0243] Here, it should be noted that, as a variation, the apparatus 700 for managing the reception or transmission of multimedia bitstreams may include a device capable of implementing the method described in reference Figures 4 to 6 One or more application-specific integrated circuits (ASICs) of the method described are integrated, for example and without limitation, into a device for generating or displaying video sequences and / or for listening to audio sequences.

[0244] Figure 8a The embodiment of the present invention shows a packaging process for a video corresponding to a wide-view scene and extraction of the wide-view scene.

[0245] Wide-view scenes are typically spherical 360° media content.

[0246] The wide view scene is encoded and packed into at least one track 801. The video representing the extraction from the wide view is encoded and packed into a special track 802 containing a reference 803 to the wide view track and a descriptor 804 of the extracted portion of the wide view video.

[0247] This descriptor can be used as one of the transformation operators that allow to represent the spatial part of the captured projection.

[0248] The descriptor 804 typically gives positional parameters that identify the extracted portion 806 in the wide-view video 805. Typically, these parameters are related to yaw, pitch, and roll parameters.

[0249] For example, the descriptor 804 may employ the following syntax as a specific VisualSampleGroupEntry:

[0250]

[0251] It has the following semantics:

[0252] yaw_center specifies the yaw value relative to the center of the projected image in increments of 0.01 degrees. The value of yaw_center should be in the range -18000 to 17999. If not present, the value of yaw_center should be inferred to be equal to 0.

[0253] pitch_center specifies the pitch value relative to the center of the projected image in increments of 0.01 degrees. The value of pitch_center should be in the range -9000 to 9000. In the absence of this parameter, the value of pitch_center should be inferred to be equal to 0.

[0254] roll_center specifies the roll value relative to the center of the projected image in increments of 0.01 degrees. The value of roll_center should be in the range -18000 to 18000. If not present, the value of roll_center should be inferred to be equal to 0.

[0255] yaw_range specifies the range of yaw values ​​present in the projected image, in increments of 0.01 degrees. The value of yaw_range should be in the range 1 to 36000. If not present, the value of yaw_range should be inferred to be 36000.

[0256] pitch_range specifies the range of pitch values ​​present in the projected image, in increments of 0.01 degrees. The value of pitch_range should be in the range 1 to 18000. If not present, the value of pitch_range should be inferred to be equal to 18000.

[0257] When a track is a viewport for another track, the track contains a "tref" box with a "chunk" reference type to the complete track representing the entire sphere.

[0258] In a preferred embodiment, the descriptor 804 contains more descriptive parameters:

[0259]

[0260] It has the following semantics (matching the semantics in the Omnidirectional Projection Indication SEI message in HEVC):

[0261] region_id is a unique identifier for all spherical region sample group descriptions related to the same visual source. All spherical region sample group descriptions with the same region_id share the same reference spherical space.

[0262] full_picture, when set, indicates that this spherical region sample group description actually represents a full spherical picture, which presents at least the same visual content formed by the union of all sample group descriptions of type "srif" having the same region_id.

[0263] When dynamic_viewport is set, the coordinates of the spherical area are provided via the metadata track identified by track_ID. Otherwise, the coordinates of the rectangular area are provided by yaw_center, pitch_center, roll_center, yaw_range, and pitch_range.

[0264] track_ID indicates a metadata track containing timed spherical coordinates describing the coordinates of each sample of the spherical region represented by the spherical region sample group description.

[0265] yaw_center specifies the yaw value relative to the center of the projected image in increments of 0.01 degrees. The value of yaw_center should be in the range of -18000 to 17999.

[0266] pitch_center specifies the pitch value relative to the center of the projected image in increments of 0.01 degrees. The value of pitch_center should be in the range of -9000 to 9000.

[0267] roll_center specifies the roll value relative to the center of the projected image in increments of 0.01 degrees. The value of roll_center should be in the range of -18000 to 18000.

[0268] yaw_range specifies the range of yaw values ​​present in the projected image in increments of 0.01 degrees. The value of yaw_range should be in the range 1 to 36000.

[0269] pitch_range specifies the range of pitch values ​​present in the projected image in increments of 0.01 degrees. The value of pitch_range should be in the range of 1 to 18000.

[0270] Note that the format of the spherical coordinate timed metadata track is redefined as a special MetadataSampleEntry as follows:

[0271] Spherical Coordinates Sample Entries

[0272] Sample Entry Type: `spcc'

[0273] Container: Sample Description Box(`stsd′)

[0274] Mandatory: No

[0275] Quantity: 0 or 1

[0276] The spherical coordinate sample entry provides spatial information about a reference track expressed in a spherical coordinate system.

[0277] syntax

[0278] A sample entry for spherical coordinates would be as follows:

[0279]

[0280] Semantics

[0281] projection_type indicates a specific mapping of rectangular decoder picture output samples onto a coordinate system.

[0282] Spherical coordinate sample format

[0283] syntax

[0284] Spherical coordinate samples should conform to the following syntax:

[0285]

[0286] The sync samples of the ROI metadata track are samples with an interpolate value of 0.

[0287] Semantics

[0288] yaw_center specifies the yaw value relative to the center of the projected image in increments of 0.01 degrees. The value of yaw_center should be in the range of -18000 to 17999.

[0289] pitch_center specifies the pitch value relative to the center of the projected image in increments of 0.01 degrees. The value of pitch_center should be in the range of -9000 to 9000.

[0290] roll_center specifies the roll value relative to the center of the projected image in increments of 0.01 degrees. The value of roll_center should be in the range of -18000 to 18000.

[0291] yaw_range specifies the range of yaw values ​​present in the projected image in increments of 0.01 degrees. The value of yaw_range should be in the range of 1 to 36000.

[0292] pitch_range specifies the range of pitch values ​​present in the projected image in increments of 0.01 degrees. The value of pitch_range should be in the range of 1 to 18000.

[0293] interpolate indicates the temporal continuity of consecutive samples. If true, the application can linearly interpolate the values ​​of the ROI coordinates between the previous sample and the current sample. If false, there should be no interpolation of values ​​between the previous sample and the current sample.

[0294] Similarly, for 2D images, a descriptor for a rectangular region is defined as a RectRegionSampleGroupEntry ("rrif"). (The names and codes here are examples only; they could also be "block" or "part," or any other reserved 4-character code.) This descriptor is identified by a specific reserved code (e.g., "rrif"). Rectangular region sample group descriptions are used to describe the spatial relationship between video or image media tracks. This allows identification of decoded samples from one track as spatially corresponding to a given rectangular region in another track.

[0295]

[0296] has the following semantics:

[0297] region_id is a unique identifier for all rectangular region sample group descriptions related to the same visual source. All rectangular region sample group descriptions with the same region_id share the same reference rectangular space.

[0298] full_picture, when set, indicates that the rectangular region is actually a full picture representing the same visual content formed by at least the union of all sample groups of type "rrif" with the same region_id, in which case region_width and region_height should be set to the width and height of the reference region.

[0299] dynamic_roi, when set, indicates that the coordinates of the rectangular region are provided via the metadata track identified by track_ID. The format of the 2D Cartesian coordinate timed metadata track as specified in clause 6 of ISO / IEC 23001-10 shall be used, in which case region_width and region_height shall be set to reference_width and reference_height as provided in the 2D Cartesian coordinate sample entry. Otherwise, the coordinates of the rectangular region are provided by horizontal_offset, vertical_offset, region_width and region_height.

[0300] track_ID indicates the metadata track containing timed 2D Cartesian coordinates describing the coordinates of each sample of the rectangular region represented by the rectangular region sample group description.

[0301] horizontal_offset and vertical_offset give the horizontal and vertical offsets of the upper left coordinate of the rectangular region represented by the rectangular region sample group description relative to the upper left coordinate of the reference region. The reference region is the region formed by the union of all sample group descriptions of type "rrif" with the same region_id.

[0302] region_width and region_height give the width and height of the rectangular region represented by the rectangular region sample group description in integer coordinates, or the width and height of the reference rectangular space when full_picture or dynamic_roi is set.

[0303] Note that the units used to describe region sizes are arbitrary and may correspond to video pixel resolution, but need not.

[0304] In an alternative embodiment, the "srif" and "rrif" descriptors also contain a unique identifier as an additional parameter. This identifier allows applications to reference the descriptor via the value of the identifier. The identifier may be a group_ID as defined below.

[0305] Then, in addition to simple cropping or rotation operations, these new descriptors from ISO / IEC 14496_15 and the existing tile descriptor "trif" can be used in a new special TransformProperty. This can be useful for ROI or viewport selection or extraction. For example, a special TransformProperty is proposed to use the trif descriptor for static ROI extraction from a packed HEVC track:

[0306] Static ROI selection

[0307] Box Type: ′sroi′

[0308] Mandatory(persample):No

[0309] Quantity (per sample):Any

[0310] Static ROI selection is to define the transformation properties of ROI extraction.

[0311]

[0312] The transform is declared as a static TransformProperty applied to a single input track. The parameter tileRegion_ID instructs the parser to extract the NAL units or samples associated with the tile region descriptor with that tileRegion_ID. This also applies to static viewports in spherical content. Depending on the pre-decoding or post-decoding flag value, the operation involves sub-bitstream extraction from the input track, or decoding and cropping operations. This depends on the independence of the encapsulated tile regions described by the trif. In the independent case, the TransformProperty can be set to the pre-decoding transform, otherwise it is set to post-decoding.

[0313] Instead of tileRegion_ID values, other parameters such as x, y, w, h can be given. However, this is less convenient for a parser that has to match parameters to those in different trif sample groups.

[0314] For multi-track TransformProperty, the parameter can be present in a list of TileRegionId. For example:

[0315]

[0316] To generalize these kinds of transformations, the selection operator can be defined as follows:

[0317]

[0318] Here, grouping_type indicates the criteria used to select NAL units or samples from one or more input tracks. This can be rrif, srif, trif, rap (to select on random access samples within a track), etc., and group_ID provides the exact group ID from which to extract. Static / dynamic information and single / multi-track and pre-decoding / post-decoding parameters should be appropriately set by the content creator during the multiplexing step within the compositing unit 130.

[0319] In other words, a method is proposed for providing, using a first device, packaged media data comprising samples, the media data comprising first media data representing a photographic projection of a scene in a wide view, and second media data corresponding to a spatial portion of the photographic projection, the method comprising:

[0320] encoding the first media data and the second media data;

[0321] encapsulating the encoded first media data into a first track;

[0322] encapsulating the encoded second media data into a second track, where the second track includes a second header;

[0323] A parameter for referencing the first track is provided in the second header.

[0324] The parameter may be associated with a type representing a spatial portion (eg, a tile).

[0325] This parameter may be included in a box of type "Block" reference type.

[0326] The parameter may be a descriptor, for example a VisualSampleGroupEntry descriptor like SphericalRegionGroupEntry or RectRegionGroupEntry.

[0327] Tracks corresponding to partial picture tracks and full picture tracks are all set and defined in, for example, a track group box "TrackGroupTypeBox" (related to "track group") with a new track grouping type. The flag used to indicate a full picture may be a flag associated with a specific track_group_type:

[0328]

[0329] Among them, full_track indicates whether the track belonging to the track group corresponds to a complete video track, and "part" is the track grouping type (the names here are just examples, and any 4-character code reserved for this purpose can be used).

[0330] "part" (or any other reserved 4-character code) indicates that the track with this track group box in the "trak" box belongs to a composite consisting of several tracks (multi-track composite). Tracks with the same value of track_group_id within a TrackGroupTypeBox of track_group_type "part" are mapped to spatial parts, or sub-pictures from the same source. For example, an ultra-high definition recording of a video may have several video tracks in high definition, each for a different spatial part of the ultra-high resolution video to be encoded. Then, video tracks with track_grouping_type="part" will have the same value of track_group_id.

[0331] More generally, a new metadata or new track grouping type for track group descriptions can be used to describe a collection of tracks that are intended to be composed, rendered, or displayed together. According to an embodiment of the present invention, it is proposed to provide different kinds of track group signaling depending on whether the content creator packages the video tracks into the media file. This enables media players to:

[0332] - Render the tracks belonging to the group at their original positions and sizes (as in Figure 11 1161 in): implicit composition (1141). Implicit composition is implicit because the composition parameters (mainly position and size) are values ​​that are already stored in the track's descriptive metadata 1142 (e.g. in the track header, sample entry, sample group description, etc.). The purpose of the track group description is to describe the relationship between different tracks in their original context. Reference Figure 10 1001 in the figure describes an example of such a track group.

[0333] Render tracks at any position, potentially changing their position and / or size (e.g. Figure 11 1162 in): explicit composition (1151). The purpose of the track group description is then to describe the purpose of the track group (via track_grouping_type) and also to describe the composition parameters 1152. Figure 10 An example of such a track group is described at 1002 in

[115] . This can be considered an alternative to a derived track. The difference from a derived track is that a generated visual track does not physically exist in the encapsulated file, but only exists in the instructions (1152) on how to build the generated visual track.

[0334] exist Figure 8a When describing the encapsulation of a wide view area, Figure 8b The generation and packaging of a video track representing a portion of a geometric projection (2D projected image) in the context of a 360° panoramic video is described in detail. (360° content 850 from a sphere, without poles). Typically, this content is recorded by a camera setup that provides a 360° field of view of the recorded scene. The recorded views are then stitched together into a single image: sometimes referred to as a panoramic 2D projected image 851. The projection is typically an equirectangular projection, but can also be onto faces of a cube or other regular polyhedron, where each face is projected as a 2D image. For example, a selected spherical region 850 is divided into three regions, sections, or spatial portions, each recorded by a camera. This operation can produce very high-resolution video, typically wide video greater than 8K. The panoramic images can be rearranged (or "packed") into regions on the 2D panorama, as shown at 852. For example, a region corresponds to or contains a projected spherical viewport from the sphere 850, or a region corresponds to a face of a cube when used as the projection type. These regions may or may not match the camera view. Each of these packed or reordered images 852 may be encoded / compressed into one or more video bitstreams (not shown) and encapsulated in an ISOBMFF file 860. The encoding may be done in HEVC or AVC or any equivalent compression scheme. The encapsulated file may contain one or more tracks such as 853, 854, 855. For example, in Figure 8bIn the example above, emphasis is placed on area "2" compared to areas "1" and "3". This may be the case for areas with high detail, or because this is a recommended viewpoint by the content editor or any other criteria. Each track 853-855 may contain descriptive metadata that provides the position and size of the relevant area in the 2D projected picture and / or the position in the 3D reference associated with the wide view or 360° scene 850. This position and size information may be described in sample description groups with specific descriptors as described below: VisualSampleGroupEntry: RectRegionGroupEntry ("rrif") and SphericalRegionGroupEntry ("srif"), respectively. When generating the packed frame 852 from the panorama 851, the sample description tables of tracks 853-855 contain a mapping between areas in the 2D projected picture to their corresponding areas in the packed image. After obtaining the stitching for the panoramic structure 851, if packaging is not performed, tracks 853-855 contain their positions and sizes within the 2D projection image 851 in a special descriptor, VisualSampleGroupEntry. Setting this information when generating the ISOBMFF file 860 is the responsibility of the mp4 writer. The segmentation into regions and the final packaging are provided by the user through the media recorder, or pre-programmed in the application controlling the camera device, or even defined during the camera device setup. This information can be used during media track packaging or even stored as descriptive metadata in the user data box.

[0335] Figure 9 An example of a collection of tracks (901-904) to be composed, rendered or displayed together is provided. A collection of input tracks may contain zero, one or more derived tracks that use these input tracks. A collection of input tracks may also be tracks of a media type other than video, and tracks of different media types may be mixed. For the special case of a track group describing a spatial composite of input tracks, the input tracks may also be referred to as sub-picture tracks. This is the case where each sub-picture track covers a spatial portion or spatial region of a (larger) picture. The term picture here should be interpreted as each picture in a video sequence; i.e. a sub-picture track contains samples that each represent a sub-portion or spatial region or sub-picture of the complete video. In Figure 9 In the example of FIG, the track grouping describes four input tracks 901-904 (video tracks in this example) that can be simultaneously composed, rendered, or displayed as a higher resolution video 900. This can be used for virtual reality panoramas (e.g., projecting a wide / 360° view of 850 onto a 2D picture on 851 or 852) or for ultra-high resolution video.

[0336] In an embodiment, each of the video tracks 901-904 is encoded in an independent bitstream, for example using a video compression scheme such as H.264 / AVC or H.265 / HEVC or any other video compression method. Each track 901-904 is further encapsulated in an ISOBMFF track (i.e., containing descriptive metadata of the media samples), for example, in a "trak" box such as 905.

[0337] Description of samples typically occurs in a sample table box such as 910, which may contain one or more sample groups, and a sample group description box (generally 911).

[0338] For example, each input track conveys its (two-dimensional) position and size 912 in its sample group description box (typically "sgpd" and "sbgp"). The position and size can be declared in a VisualSampleGroupEntry 912 such as a RectRegionGroupEntry ("rrif") or a TileRegionGroupEntry ("trif" in the case of the HEVC format), or in any dedicated descriptor in the sample table box. These positions and sizes can also be declared in the visual sample entry as part of the sample description 913. This is the case, for example, of a "packed frame" obtained by projecting a 360° video scene onto a 2D panorama and further segmenting, reassembling, or packing it into regions 914 on another 2D picture before encoding (e.g., as in the RegionWisePackingStruct of the MPEG Omni Media Application format). When all samples of a track share the same position and size, the position and size are declared in the VisualSampleGroupEntry represented as the default sample description index in the "sgpd" box. This allows to have a more compact description, thereby avoiding the declaration of sample groups. When declaring the position and size in a VisualSampleGroupEntry of type "rrif" as follows or in a simplified version (same semantics as described above);

[0339]

[0340] Set the corresponding Track Group Type box 952 as follows:

[0341]

[0342] in:

[0343] composition_width specifies the width of the composite image in luma samples.

[0344] composition_height specifies the height of the composite image in luma samples.

[0345] track_group_id inherited from TrackGroupTypeBox 951 provides a unique identifier for a group of tracks.

[0346] The track_group_type value ("spco" in this example) indicates that the track is a group of tracks that are spatially related.

[0347] The embodiment described above differs from the initial embodiment in the "rrif" because the compound identifier (previous region_id in "rrif") is now described in the track group type box (track_group_id) instead of in the sample or sample group description box. This allows a track to be part of zero, one or more compounds. Note that the indication of the "full picture" (also called "full_track" in the "part" track group example above) can also be placed in the track group type box. The RectRegionGroupEntry then becomes even simpler:

[0348]

[0349] Track Group 950 and Track Group Type boxes 951-953 indicate that the group of input tracks 901-904 has a specific relationship. Here, the specific grouping type "spco" (of "Track Composite Group" 952) indicates that these four input tracks can be implicitly composited together for simultaneous display and rendering as in 900. The corresponding Track Group Type box 952 can still be written as described above. Optionally, when the content creator encapsulates the full video as an alternative track to the sub-picture track in the same media file, the Track Group Type box 952 can include a flag indicating whether the current track is a "full video" track, a "full picture" track, or a full track. In this case, only the full video track represents the composite width and height of the sub-picture track.

[0350]

[0351] In an alternative embodiment of the track group type used for spatial multiplexing, the "full_picture" flag is systematically present. This has no additional cost in terms of signaling (still 1 byte) and allows the parser to infer whether the video track is a sub-picture track (the "full_picture" flag is set to 0) or a full picture track (the "full_picture" flag is set to 1). It can be inferred that the track is not intended to be displayed alone (in contrast to the "full_picture" track):

[0352]

[0353] In order to make it easy for a media player to determine whether a track in a track group is intended to be displayed, the track group type box may contain, for example, a parameter called "output_track", which, when set to 1, indicates that the track can be selected for presentation to the user, and, when set to 0, indicates that the track is only used here for combination with other tracks in the track group and is not intended to be rendered or displayed alone. Another possibility to indicate this track property is to define a specific flag in the track header box, for example "Track_non_displayable":=0x000010 indicating that the track is enabled and used in presentation, but is not allowed as a preview and is not sufficient for rendering on its own.

[0354] In an alternative embodiment, a track group ("TrackGroupBox") contains a collection of input tracks whose input tracks correspond to a region of interest (e.g., a specific viewpoint in 360° content or a specific object in 2D video). For this case, the track group type box ("TrackGroupTypeBox") can contain additional flags instead of, or in combination with, the full_picture flag for "spco" from the previous embodiment:

[0355]

[0356] Again, this ROI indication feature has no additional cost in terms of signaling (using one of the reserved bits) and enables the parser to infer that the video track is a sub-picture track corresponding to the ROI (the "roi_picture" flag is set to 1). It can be inferred that this sub-picture track can be rendered separately by exposing it to the user via a graphical user interface. When used in streaming or transmitted over a network, this track can be marked or indicated as having some priority relative to other sub-picture tracks.

[0357] In another alternative embodiment, another flag is used in the SubPictureCompositionBox to mix tracks or subpicture tracks with static positions and sizes and tracks or subpicture tracks with dynamic positions and sizes into the track group as follows:

[0358]

[0359] Note that this additional flag, called dynamic_coordinates, can be combined with other flags for spatially composite track groups in various embodiments. This flag is set to 0 or false when the tracks or sub-picture tracks in the track group have static positions and sizes. Conversely, when a track or sub-picture track has dynamic positions and sizes in another track (usually a timed metadata track), this flag is set to 1 or true, and the track identifier of the other track is indicated in the track_ID parameter.

[0360] Figure 10 Different kinds of track group type boxes ("TrackGroupTypeBox") are shown that are more or less descriptive and prescriptive in relation to the tracks contained in the track group. Figure 10 The track group type box 950 in is a track group type box as standardized in ISO / IEC 14496-12 (ISOBMFF). The present invention defines different kinds of track group type boxes ("TrackGroupTypeBox") to handle sub-picture tracks.

[0361] The first track group type box ("TrackCompositionGroupBox") is the track group type box described at 1001. This track group type box inherits the track_group_id from box 951 and declares a specific track_grouping_type: "implicit compo_type". Note that in order to comply with ISOBMFF rules, a 4-character code should be reserved to represent this "implicit compo_type", for example: "spci" for "spatial composite implicit". This means that this track group 1001 relies on information already available in the sub-picture tracks (in the track header part of 905, in the sample entry 913 or in the sample group description 911...) to render or display these sub-picture tracks as they are spatially arranged in the source that is encoded and encapsulated into this track set. This box 1001 also contains the size of the composite of the sub-picture tracks so obtained. Optionally, this box 1001 may also contain one or all of the flags described in the alternative to the "spco" track group type box. As Figure 9 As shown, this Track Grouping Type box does not contain the position and size of each sub-picture track. Instead, the Track Grouping Type box implicitly refers to (links 920 to) descriptive metadata describing the track or samples that already exist in each track. This allows duplication of descriptive metadata to be avoided. In addition, it allows the position and size of the sub-picture tracks and the spatial arrangement to change over time. Since the sample grouping mechanism is used, the Track Grouping Type box can be changed at the track fragment granularity. This is not the case where the position and size of each sub-picture track are declared in the Track Group Type box itself. According to Figure 12 To illustrate the processing of 952 and 920.

[0362] Another track group type box 1002 defines another track_grouping_type for an "explicit composite type." As mentioned above, a 4-character code should be reserved for this track grouping type, such as "spce" for "spatial composite explicit." This box 1002, in addition to declaring the composite width and height (ultimately combined with one or all of the flags mentioned for "spco"), also declares the new spatial arrangement of the sub-picture tracks. This can be considered a variant of a derived track. In particular, the new box 1002 declares the 2D translation parameters of the sub-picture track to change the position of the sub-picture track in the resulting presentation. This declaration is optional and can instead be set to (0,0). Similarly, optional rotation and scaling factors can be declared to change the arrangement and / or size of the sub-picture track in the resulting composite or presentation. Note that a composite matrix, such as that defined in the track or animation header, can be declared for this purpose. A player encountering this track grouping type box has the positioning and transformation information to render each sub-picture track according to the media editor's specifications.

[0363] Alternative embodiments of explicit compositing 1002 are depicted as 1004 and 1005. Another track_grouping_type box 1004 indicates grid-based compositing ("grid_compo_type"). A specific four-character string should be reserved and declared for this purpose, for example, "spcg" for "spatial composite grid." The parameters contained in this new type of box 1004 are grid operators, as described in 1005. These parameters declare where each sub-picture track of the track group can be placed to render a grid of mosaicked video. The position of the sub-picture track within the mosaicked video is given by the last grid_idc parameter. For other variants, one or all flags available for "spco" may also be included in box 1005. Such a track group box is processed on the player side as follows: The input images of each sub-picture track are inserted into the 1-based index of the grid in row-major order (starting from the top row and working from left to right). If the size of the sub-picture track does not match the size of the grid cell, it is implicitly scaled to fit the cell size. An optional flag may be set to indicate whether the pixel aspect ratio should be preserved. Another optional flag may indicate whether splicing is desired.

[0364] The reconstructed image is formed by composing the decoded input pictures from the sub-picture tracks into a grid. The grid has a column width in luma pixels equal to output_width divided by num_of_columns, and a row height in luma pixels equal to output_height divided by num_of_rows, without gaps or overlaps. When splicing is not desired, crops are made on the right and bottom to the indicated output_width and output_height. This grid operator 1005 can also be used as a TransformProperty or transform operator for derived tracks. When used, this grid operator 1005 is stored in the TransformPropertyContainer 314 and may not require track_idc. In practice, the grid can treat the input tracks indicated by the track reference type "dtrp" 341 as input. In this case, the order of these references indicates the position in the grid.

[0365] Another embodiment of track groups used for spatial compounding is the new box 1003. This box has a special track_grouping_type value indicating that the compounding is dynamic. (A 4-character code should also be defined and reserved for this purpose). It is a special kind of explicit compounding in the sense that compounding parameters (mainly position and size) are provided in an external metadata track (e.g. a timed metadata track as in Amendment 1 of ISO / IEC 23001-10). Special parameters are provided in the track group: a track identifier for the metadata track that provides the compounding parameters. This is useful for associating video tracks with metadata tracks, since the track references linking the metadata track to the video track are unidirectional (a "cdsc" track reference type to the video track or a "cdtg" track reference type to the tracks in the group). Having the track_ID directly in the track group makes it easier for the media player to get the compounding parameters. With respect to other kinds of spatial compounding track groups, box 1003 contains the compounding width and height so obtained. Optionally, box 1003 may also contain one or all of the flags declared in the variants of the "spco" track group type box. Although Figure 10The different kinds of track group type boxes used for spatial compounding are shown (because the different boxes are identified by specific track_grouping_type and have reserved 4-character codes), but note that these different kinds can also be represented in other ways. One possible way is to use a single 4-character code (e.g., an "spco" code and a "version" parameter for the box). Instead of inheriting from a track group type box with version=0 as in 951, the version parameter can be changed. For example, version=0 corresponds to implicit compounding 1001, while version=1 indicates explicit compounding 1002 and version=2 corresponds to dynamic compounding 1003. Version=3 can indicate grid-based compounding 1004. The specific parameters of each box are only included for a specific value of the version parameter. For example:

[0366]

[0367] In an alternative embodiment, (instead of the version parameter described above) a flags parameter is used to indicate different kinds of track group type boxes. Thus, a single track_grouping_type is used, such as "spco". A specific spatial composite track group type box of type "spco" inherits from TrackGroupTypeBox with flags not set to 0 as in 951, but may take a different predefined value such as tg_explicit_composition = 0x000001 to indicate that the composition is explicit composition (similar to version = 1 in the previous embodiment). By default, composition is considered implicit (version = 0 and flags = 0). Other flags may be defined such as tg_dynamic_composition = 0x000002 and tg_grid_composition = 0x000004. These may be used as different values ​​of the version parameter as described in the above embodiments.

[0368] Another aspect of the overloading of the TrackGroupType box is the possibility to define trackgroup identifiers that are not limited to integers as is currently the case in ISOBMFF. This is useful for declaring trackgroups that contain tracks that are not necessarily contained in the same media file. This enables associating tracks in a media file with remote tracks in other files or on different servers. This can be done with specific values ​​for the "flags" or "version" parameters. The TrackGroupType box can then be redefined as follows (here using the version parameter):

[0369]

[0370] Where global_track_group_id is a URN entry, such as the DataEntryUrnBox defined in ISOBMFF. The parameter "name" in the DataEntryUrnBox is required. The parameter "location" is optional. Each of the "name" and "location" parameters is a null-terminated string using UTF-8 characters.

[0371] Figure 13 Another feature of track groups is shown: the concept of subsets. In 1300, a track group containing four input tracks is provided. In practice, there are two sub-picture tracks (video 1 and video 2), but each uses a different resolution. These tracks can then be aggregated into two subsets, each with a different resolution: for example, subset 1310 contains high-definition (HD) video, while subset 1320 contains simple definition (SD) video. This means that, implicitly, tracks 1311 and 1312 can be spatially combined into a larger video (2 times the HD resolution), and in addition, tracks 1321 and 1322 can also be combined into a larger video (2 blocks of SD resolution). To allow subset descriptions in track groups, TrackGroupTypeBox is modified as follows. First, TrackGroupTypeBox is a new version of the box (a version number that does not conflict with the composite type described above) or a new flags value (for example: contains_subset = 0x000008). The following example is shown with the version number set to 1. This can use any kind of grouping, not just spatial compounding.

[0372]

[0373] Among them, the TrackGroupSubsetBox is defined as follows:

[0374] aligned(8)class TrackGroupSubsetBox extends FullBox(`trgs′, version,

[0375] flags){

[0376] unsigned int(8)subset_idc;

[0377] / / parameters describing the subset

[0378] / / For example in 1310, 1320, the composition sizes

[0379] }

[0380] Where subset_idc indicates the index of the subset in the track group. subset_idc is a 1-based number whose range is limited to the track groups it contains. No more than 256 subsets can be defined in a track group. Each track belonging to a track group and a specific subset conveys the index of the subset to which the track corresponds. For example, in 1300, tracks 1311 and 1312 will have a TrackGroupSubsetBox with subset_idc=1 (the number of track group subsets contained in the track), while tracks 1321 and 1322 will have a TrackGroupSubsetBox with subset_idc=2. Each subset will declare its own composite size (no longer an attribute of the track group). This allows track groups to be described more finely by placing the parameters of the entire group in the TrackGroupTypeBox and the subset-specific information in the TrackGroupSubsetBox. Subset parameters may even overload the parameters declared in the TrackGroupTypeBox. This allows default parameter values ​​to be defined for all tracks within a track group, and allows default parameter values ​​to be specified only for a subset of tracks belonging to a specific subset.

[0381] To allow a track to be part of more than one subset, TrackGroupTypeBox can use version 5 to provide an array of subsets instead of just one instance. A parameter to specify the number of subsets also exists in TrackGroupTypeBox, as follows:

[0382]

[0383] The subset can also be used to define some composition restrictions: for example, HD tracks are only combined with other HD tracks, but not with SD tracks. The subset can also be used to describe another composition of the tracks in the track group, which results in a video that can also be used for rendering or display. In an alternative embodiment, the subset feature can be specified in the TrackGroupBox itself instead of the TrackGroupTypeBox according to the same principle: version or flags value. Another focus of the subset feature is for describing media files in streaming manifests. For example, the MPEG DASH standard supports spatial relationship descriptions (SRDs) in its MPD manifests. The SRD has a parameter called spatial_set_id that provides an identifier for a set of spatial objects. Sub-picture tracks with the same subset_idc as 1300 can then be described in the DASH MPD using an SRD descriptor with the value of the subset_idc parameter as the spatial_set_id value.

[0384] For the version of the track group type box ("TrackGroupTypeBox") without TrackGroupSubsetBox, the subset_idc or spatial_set_id parameter may be included directly as a parameter of the track group type box ("TrackGroupTypeBox"). This subset_idc or spatial_set_id parameter may be included in both the implicit and explicit track group boxes used for spatial compounding, i.e. the parameter may be combined with compounding information such as position and size and optionally transformation parameters. Still in Figure 13 In

[1350] , video 1350 is segmented into six spatial regions, each represented by a video track (sub-picture track). Although video 1350 is a very high-definition video that may exceed the profile and level (maximum allowed size) of the available video codec, each sub-picture track or subset of sub-picture tracks conforms to a given profile or level. Providing media players with some granularity regarding spatial access within video sequence 1350 can be advantageous. Track groups can be used to represent this by defining the profile and level required to process one or more regions. For example, in 1350, several track groups are defined that group three sub-picture tracks. Each track group then conveys profile_tier_level information resulting from spatial composition of the three sub-picture tracks, in addition to composition parameters. Furthermore, by sending the tracks of a track group rather than the entire video 1350, this organization can be used to facilitate navigation within immersive content, such as 360-degree content or virtual reality content. Furthermore, handling overlap between track groups is a good approach to avoid defining video tracks that each encapsulate three sub-picture tracks, as it avoids data duplication.

[0385] One advantage of implicit composite track group 1141 is that implicit composite track group 1141 references and uses (e.g., according to Figure 9 Parameters stored in the sample description (shown and described in detail). This brings some flexibility in sub-picture organization, as sample entries and sample group descriptions can be updated over time, for example in units of track fragments. This allows track groups to be defined independently of track properties (e.g. position, size). This is of particular interest for block-based coding where the block structure changes over time (this can be done in HEVC by updating picture parameter sets along the video sequence). For example, in video surveillance applications, the block structure can be adapted to match an object of interest in a video sequence that is being tracked. Having a block surround the object enables ROI-based streaming and saves transmission bytes. This is handled by the trif or rrif VisualSampleGroupEntry and sample grouping mechanism. This is not possible with track groups. To allow this, the present invention also proposes to extend track groups to allow track segment groups. This is useful for explicit compositing where track groups or track segment groups contain composite parameters. Using the track segment group box, these parameters can be updated over time. The number of tracks in a group can also change over time. The new box can be declared as follows:

[0386] Box Type: `tfgr′

[0387] Container: TrackFragmentBox(′traf′)

[0388] Mandatory: No

[0389] Quantity: Zero or one

[0390] This box enables the indication of track groups that are valid for a track fragment, where the groups share certain characteristics or the tracks within a group have certain relationships. This box contains zero or more boxes, and the specific characteristics or relationships are indicated by the box type of the contained boxes. The contained boxes include identifiers that can be used to infer that tracks belong to the same track fragment group. Tracks that contain containing boxes of the same type within a TrackFragmentGroupBox and have the same identifier value within those containing boxes belong to the same track fragment group.

[0391] Track fragment groups should not be used to indicate dependencies between track fragments.

[0392]

[0393] track_fragment_group_type indicates the grouping type and should be set to a registered value or a value from a derived specification or registry to 1. track_fragment_group_type may be, for example, a "part" value defined in the present invention to describe the spatial organization of sub-picture tracks.

[0394] track_fragment_group_id is a unique identifier in the entire media file. When the value of track_fragment_group_id is the same for each fragment of the same track, the value corresponds to the same track group. When the track in the track fragment group changes for each fragment, the track_fragment_group_id should also change to inform the media player. When track fragment groups are used, it is recommended that fragments be aligned (in time) between the tracks belonging to the track fragment group. Optionally, the 'trgr' box at the trak level can be parameterized to indicate that the box is overloaded in some fragments or that the box will remain constant along the entire media duration. This can be done using a specific flags value for the 'trgr' box.

[0395] Figure 12 The processing of a media file by a media player or an ISOBMFF parser according to one aspect of the present invention is described as follows: a track group describes a situation where a track or a sub-picture track is spatially composited. Figure 12 The first step not shown in the figure is to parse each trak box of the track present in the media file. During parsing, a track list containing a track group box "trgr" is established, and a track list contained in the group is established for each group.

[0396] Figure 12 The processing of the tracks of a track group corresponding to an implicit spatial composite track group is illustrated starting at 1200. First, in step 1201, the position of the current track is defaulted to (x=0, y=0) and its size is set to the width and height declared in the Track Header box. Then, in 1202, the Sample Table box is parsed to find the sample entry declared in the track. If the sample entry has the value "resv" (test 1203), then in step 1204 the parser looks for a SchemeTypeBox. If the scheme type is set to a value indicating that the track contains projected omnidirectional video (test 1305 is true), then in 1206 the parser looks for region-level packing information, typically a RegionWisePackingBox.

[0397] If none is found, the parser searches the user data box for stitching or setup information in step 1207. If none is found (test 1207 is false), the track's size is set to the width and height set in the track's raw sample entry or in the projected_picture width and height, and composited with these positions and sizes, in step 1210. If the user data box provides stitching information for the camera setup information, and thus the track's position and size, this stitching information is used in step 1208 to set the appropriate position and size, and the track is then composited with in step 1210. If packing information is present in step 1206, this information is used in step 1209 to obtain the track's packing position and size, and the track is composited with these values ​​in step 1210. If the scheme type is unknown or does not correspond to omnidirectional content or candidate media content for composite, default positions and sizes are used in step 1210. When test 1203 returns false, the sample entry value is tested against "hvt1", "lht1" or any sample entry value indicating that the current track is a tile track (step 1211). If this is the case, the position and size are parsed from the "trif" box in 1212.

[0398] Otherwise, the parser looks for a general region descriptor such as "rrif" in 1213. If one is found, the parameters in the "rrif" or equivalent descriptor are used to set the position and size of the track in 1214. This process iterates until all tracks of the track group have been processed. Step 1210 involves associating the positions and sizes of the tracks in the track list belonging to the track group to be used for compositing and rendering during decoding of the corresponding bitstream. Figure 12 The processing of 1200 is part of the player initialization step. This can also happen during decoding if track fragment groups are in use. The composite parameters can then be updated periodically by performing the processing from 1200 onwards whenever a new track fragment occurs.

[0399] Track groups used for spatial multiplexing also apply to input tracks coded as independent HEVC tiles, which are further encapsulated into tile tracks according to ISO / IEC 14496-15 version 4. In this case, the tracks or sub-picture tracks belonging to the track group are tile tracks. The tile base tracks of these tile tracks should also be part of the same track group. Figure 12As described, implicit compositing uses the positions and sizes given in the "trif" box. The "trif" descriptor conveys the position and size of the tiles in their original sequence. The tile base track and each tile track are all associated to the same track group (the "trak" box of the tile base track and each tile track contains the "trgr" box). One advantage of using tile tracks in implicit spatial compositing, or using grid operators for derived tracks or in spatially composited track groups, is that compositing can be done in the compressed domain (by, for example, bitstream concatenation or by optical transcoding of slice positions). When used in derived tracks and grid operators, the Transform_post_decoding flag in the derived track, or any equivalent means, is set to 0 to indicate that the operation does not require decoding of the tracks. This is the case for HEVC bitstreams after encoding with motion constraints and / or independent tiles. Typically, this operation is performed after decoding when grid or spatial compositing uses independent video bitstreams.

[0400] Another embodiment involves declaring the top-level track associations of a media file directly in the "moov" box, rather than in individual "trak" boxes. A new structure is then defined that provides a unique identifier for the track association, a type that provides information about the association or its purpose, and a list of tracks to be included in the association (track_ID). This declaration has the benefit of avoiding duplication of information in the individual tracks that are part of the track association. Such track associations can, for example, reflect the MPEG DASH concept of preselection or bundling at the file format level: a collection of tracks that improve the user experience when played together. Alternatively, this new box can be declared as part of the user data in the "udat" box, for example, in the same way as the track selection box, but for complementary or associated tracks. The common parameters of the tracks in the track association are provided as a list of attributes.

[0401] Box Type: 'tass'

[0402] Container: UserDataBox(′udta′)or MovieBox(`moov′)

[0403] Mandatory: No

[0404] Quantity: Zero, One or More

[0405] The track association box is contained in the user data box or directly in the animation box of the media file described by the track association box.

[0406]

[0407] has the following semantics:

[0408] track_association_id is an integer that provides a unique identifier for a track association that specifies a group or set of tracks.

[0409] track_association_type is a generalization of the KindBox originally defined in ISOBMFF for a track but extended to the description of any part of a movie or media file.

[0410] track_ID_list provides a list of tracks associated with this track association.

[0411] attribute_list is a list of attributes until the end of the box. The attributes in this list are common attributes that apply to all tracks associated with the track.

[0412] Figure 4 、 Figure 5 and Figure 6 Any step of the algorithm shown may be implemented in software by execution of an instruction set or program by a programmable computing machine, such as a PC ("personal computer"), a DSP ("digital signal processor"), or a microcontroller, or in hardware by a machine or dedicated component, such as an FPGA ("field programmable gate array") or an ASIC ("application-specific integrated circuit"), or the like.

[0413] Although the present invention has been described above with reference to specific embodiments, the present invention is not limited to these specific embodiments, and modifications within the scope of the present invention will be apparent to those skilled in the art.

[0414] Numerous other modifications and changes will be apparent to those skilled in the art when referring to the foregoing exemplary embodiments, which are given by way of example only and are not intended to limit the scope of the invention, and which are determined solely by the appended claims. In particular, different features from different embodiments may be interchanged where appropriate.

[0415] In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. The mere fact that different features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be used to advantage.

Claims

1. A method for generating an ISOBMFF-based media file based on video data, the video data comprising a plurality of samples, the method comprising: generating a plurality of video tracks based on the video data, the plurality of video tracks having a spatial relationship; grouping the plurality of video tracks into video track groups that can be spatially arranged to construct a spatial composite of the plurality of video tracks; as well as Generate the media file including the video track group and descriptive metadata, wherein the descriptive metadata includes both of the following: (i) a first box in each video track of the video track group, describing the video track group and comprising first spatial information describing reconstructed video data corresponding to spatial composite of the plurality of video tracks, and (ii) one or more second frames other than the first frame, for describing second spatial information, wherein the second spatial information is used to describe the spatial relationship of the multiple video tracks within the spatial composite of the multiple video tracks, wherein the second spatial information includes a position and size of a video of each video track in the spatial composite.

2. The method according to claim 1, wherein The size in the second spatial information represents a spatial width and a spatial height of each video corresponding to the plurality of video tracks, and the first spatial information represents a spatial width and a spatial height of the reconstructed video data.

3. The method according to claim 1, wherein The size in the second spatial information represents the spatial width and spatial height of each video corresponding to the plurality of video tracks.

4. The method according to claim 1, wherein The first spatial information includes sizes representing a spatial width and a spatial height of the reconstructed video data.

5. The method according to claim 1, wherein The descriptive metadata further includes an identifier of a reference rectangular space defining the second spatial information.

6. The method according to claim 1, wherein The first spatial information represents a spatial width and a spatial height of the reconstructed video data.

7. The method according to claim 1, wherein The positions in the second spatial information represent spatial offsets of the videos corresponding to the multiple video tracks.

8. The method according to claim 1, wherein The descriptive metadata further includes grouping information related to grouping of the plurality of samples, and the second spatial information is described in the second box per sample group.

9. The method according to claim 1, wherein The spatial compounding renders the plurality of video tracks at their original positions and sizes in the video data.

10. The method according to claim 1, wherein The spatial compounding renders the plurality of video tracks at any position by changing the position and / or size of the plurality of video tracks.

11. The method according to claim 1, wherein The descriptive metadata includes a flag indicating that tracks of the plurality of video tracks are not intended to be presented individually.

12. The method according to claim 11, wherein The first box is the Track Group Type box.

13. An apparatus for generating an ISOBMFF-based media file based on video data, the video data comprising a plurality of samples, the apparatus comprising a processor configured to: generating a plurality of video tracks based on the video data, the plurality of video tracks having a spatial relationship; grouping the plurality of video tracks into video track groups that can be spatially arranged to construct a spatial composite of the plurality of video tracks; as well as Generate the media file including the video track group and descriptive metadata, wherein the descriptive metadata includes both of the following: (i) a first box in each video track of the video track group, describing the video track group and comprising first spatial information describing reconstructed video data corresponding to spatial composite of the plurality of video tracks, and (ii) one or more second frames other than the first frame, for describing second spatial information, wherein the second spatial information is used to describe the spatial relationship of the multiple video tracks within the spatial composite of the multiple video tracks, wherein the second spatial information includes a position and size of a video of each video track in the spatial composite.

14. A method for generating a reconstructed video from an ISOBMFF-based media file based on video data, the video data comprising a plurality of samples, the method comprising: obtaining a set of a plurality of video tracks from the media file, the set of the plurality of video tracks being spatially arrangeable to construct a spatial composite of the plurality of video tracks, the plurality of video tracks being based on the video data and having a spatial relationship; Descriptive metadata is obtained from the media file, the descriptive metadata including both: (i) a first box in each of the plurality of video tracks, describing a group of the plurality of video tracks and comprising first spatial information describing reconstructed video data corresponding to spatial composite of the plurality of video tracks, and (ii) one or more second frames other than the first frame, for describing second spatial information, wherein the second spatial information is used to describe the spatial relationship of the plurality of video tracks within the spatial composite of the plurality of video tracks, wherein the second spatial information includes a position and a size of a video of each video track in the spatial composite; and The reconstructed video is generated from the plurality of video tracks based on the first spatial information and the second spatial information.

15. The method according to claim 14, wherein The size in the second spatial information represents a spatial width and a spatial height of each video corresponding to the plurality of video tracks, and the first spatial information represents a spatial width and a spatial height of the reconstructed video data.

16. The method according to claim 14, wherein The size in the second spatial information represents the spatial width and spatial height of each video corresponding to the plurality of video tracks.

17. The method according to claim 14, wherein: The first spatial information includes sizes representing a spatial width and a spatial height of the reconstructed video data.

18. The method according to claim 14, wherein The descriptive metadata further includes an identifier of a reference rectangular space defining the second spatial information.

19. The method according to claim 14, wherein The first spatial information represents a spatial width and a spatial height of the reconstructed video data.

20. The method according to claim 14, wherein The positions in the second spatial information represent spatial offsets of the videos corresponding to the multiple video tracks.

21. The method according to claim 14, wherein The descriptive metadata further includes grouping information related to grouping of the plurality of samples, and the second spatial information is described in the second box per sample group.

22. The method according to claim 14, wherein The spatial compounding renders the plurality of video tracks at their original positions and sizes in the video data.

23. The method according to claim 14, wherein The spatial compounding renders the plurality of video tracks at any position by changing the position and / or size of the plurality of video tracks.

24. The method according to claim 14, wherein The descriptive metadata includes a flag indicating that tracks of the plurality of video tracks are not intended to be presented individually.

25. The method according to claim 24, wherein The first box is the Track Group Type box.

26. An apparatus for generating a reconstructed video from an ISOBMFF-based media file based on video data, the video data comprising a plurality of samples, the apparatus comprising a processor configured to: obtaining a set of a plurality of video tracks from the media file, the set of the plurality of video tracks being spatially arrangeable to construct a spatial composite of the plurality of video tracks, the plurality of video tracks being based on the video data and having a spatial relationship; Descriptive metadata is obtained from the media file, the descriptive metadata including both: (i) a first box in each of the plurality of video tracks, describing a group of the plurality of video tracks and comprising first spatial information describing reconstructed video data corresponding to spatial composite of the plurality of video tracks, and (ii) one or more second frames other than the first frame, for describing second spatial information, wherein the second spatial information is used to describe the spatial relationship of the plurality of video tracks within the spatial composite of the plurality of video tracks, wherein the second spatial information includes a position and a size of a video of each video track in the spatial composite; as well as The reconstructed video is generated from the plurality of video tracks based on the first spatial information and the second spatial information.

27. A non-transitory computer-readable storage medium storing instructions of a computer program for implementing the method according to any one of claims 1 to 12 and 14 to 25.

Images