Fusion-friendly file format

Abstract

To provide a file format that enables extraction or merging of spatial subsets of coded video using compressed domain processing.SOLUTION: Video data for deriving a spatially varying portion of a scene is formatted in a file format and includes a set of two or more source tracks, each source track includes encoded video data representing a spatial portion of the video showing a scene, the set of two or more source tracks further includes one or more group indicators indicating source tracks belonging to a respective source track group, and one or more active source track indicators indicating the number of two or more active source tracks in the source track group.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] This application relates to a file format that enables extraction or merging of spatial subsets of coded video using compressed domain processing. Specifically, this application relates to video data for deriving spatially varying portions of a scene, a method and apparatus for creating video data for deriving spatially varying portions of a scene, and a method and apparatus for deriving spatially varying portions of a scene from video data formatted in a specific file format. This application also relates to corresponding computer programs, computer-readable media, and digital storage media.

[0002] 1. Introduction Typically, coded video data, such as data video coded with AVC (Advanced Video Coding), HEVC (High Efficiency Video Coding), or the currently under development VVC (Versatile Video Coding), is stored or transmitted in a specific container format, such as the ISO Base Media File Format and its various extensions, as specified in, for example, ISO / IEC 14496-12 (Coding of audiovisual objects—Part 12: ISO base media file formats), ISO / IEC 14496-15 (Coding of audiovisual objects—Part 12: Carrying Network Abstraction Layer (NAL) unit structured video in the ISO base media file format), and ISO / IEC 23008-12 (High-efficiency coding and media delivery in heterogeneous environments—Part 12: Image file formats). Such container formats include special provisions targeted at applications that rely on compressed-domain processing to extract or combine spatial subsets of the coded video, for example, for the purpose of using a single decoder on an end device. A non-exhaustive list of example such applications follows: Region of Interest (RoI) streaming, which transmits varying spatial subsets of a video; Multi-party conferencing, where encoded video streams from multiple participants are jointly decoded by a single decoder, or Tile-based streaming for 360-degree video playback, e.g. in VR applications.

[0003] 1.1 Tiled Streaming of 360-Degree Video In the latter, a 360-degree video of a scene is spatially segmented, as shown in FIG. 1, and each spatial segment is provided to a streaming client in multiple representations at different spatial resolutions. The figure shows cube-map projected 360-degree video divided into 6x4 spatial segments (including left, front, right, back, bottom, and top) at two resolutions (high resolution and low resolution). For simplicity, these independently decodable spatial segments are referred to herein as tiles. Depending on the video coding technology selected, structures such as tiles, bricks, and slices can be used to achieve independent encoding of different spatial segments. For example, when encoding each tile with Versatile Video Coding (VVC), which is currently under development, this can be achieved by dividing the picture using a suitable tile / brick / slice structure so that intra- or inter-prediction is not performed between different tiles / bricks of the same or different pictures. For example, a single tile can be used as a separate slice to encode each independently decodable spatial segment, or the brick concept can be further used to achieve more flexible tiling.

[0004] Typically, when using a modern head-mounted display (HMD), a user sees only a subset of the tiles that make up the entire 360-degree video through a solid viewport boundary representing a 90x90-degree field of view (FoV), as shown in the top of Figure 2. The corresponding tiles shown as shaded areas in the top of Figure 2 (in this example, the four tiles on the right, the two tiles on the bottom, one tile on the front, and one side on the back) are downloaded at the highest resolution (also shaded in the bottom left of the figure).

[0005] However, to accommodate sudden changes in the user's orientation, the client application also needs to download and decode representations of other tiles outside the current viewport (not shaded at the top of Figure 2), shown with different shading in the lower right of Figure 2. Thus, a client of such an application downloads tiles covering the current viewport at the highest resolution and tiles outside the current viewport at a lower resolution, while the tile resolution selection is always adapted to the user's orientation. After client-side download, fusing the downloaded tiles into a single bitstream to be processed by a single decoder is a way to address the constraints of typical mobile devices with limited computational and power resources. Figure 3 shows a possible tile arrangement in the joint bitstream for the above example. The fusing operation to generate the joint bitstream must be performed at the bitstream level through compressed-domain processing to avoid complex pixel-domain processing, such as transcoding or decoding separate tiles independently before synchronously rendering them on a cube.

[0006] A metadata description in the form of a so-called supplemental enhancement information (SEI) message in the coded video bitstream describes how the samples of the coded image relate to their positions in the original projection (a cube map in this example) to enable the reconstruction of a cube in 3D space (or a sphere, depending on the projection used). This metadata description, called region-wise packing (RWP), is essential for post-decoding renderers that render the viewport of media consumption devices such as head-mounted displays (HMDs). The RWP SEI message indicates the mapping from the projected video (e.g., shown on the left side of Figure 1, conceptually required for further processing after decoding) and one particular combination of packed coded video (as shown in Figure 3, or on the right side of Figure 4, obtained by decoding the combined bitstream) by defining a rectangular region and the displacement / transformation between that projected video and the packed video.

[0007] While the examples in Figures 1-3 show content for all resolution versions being tiled identically, with all tiles (high-resolution and low-resolution) covering the entire 360-degree space and no tiles repeatedly covering the same area, other tilings can be used, as shown in Figure 4. The entire low-resolution version of the video can be blended with high-resolution tiles that cover a subset of the 360-degree video. The entire low-resolution fallback video can be coded as a single tile, while the high-resolution tiles are rendered as an overlay on the low-resolution portion of the video at the end of the rendering process.

[0008] 1.2 Tiled Streaming with HEVC and File Format Issues In codecs such as HEVC, the need for a fusion operation from the perspective of the video bitstream is related to the tile structure of a picture and the CTU (coding tree unit) address signaling of individual tiles (i.e., slices). On the server side, these tiles exist (and are therefore downloaded) as individual, independent HEVC bitstreams, each of which contains a single tile and slice per picture (e.g., first_slice_in_pic_flag equal to 1 in all slice headers, parameter sets describing bitstreams with only a single tile). The fusion operation requires merging these individual bitstreams into a single bitstream by inserting the correct parameter sets and slice headers to reflect the tile structure and their position within the merged picture plane. Rather than leaving the details of fusion (deriving and replacing parameter sets and slice headers) to the client implementation, MPEG OMAF (Coded Representation of Immersive Media—Part 2: Omnidirectional Media Format; ISO / IEC 23090-2) specifies a state-of-the-art scheme that enables clients to fuse bitstreams through: Generating the correct parameter sets and slice headers during the packaging stage, and Copying the slice payload using a file formatting tool called an extractor.

[0009] These extractors are actually an extension of the file format, i.e., NAL units of a special NAL (Network Abstraction Layer) unit type defined in ISO / IEC 14496-15 that contain a pointer to another NAL unit (e.g., containing data for one tile) packaged in a different track. The extractors themselves are stored in a special extractor file format track (the "hvc2" track) that carries only the parameter set and modified slice header data (e.g., reflecting new tile positions, adjusted quantization step size values ​​relative to the parameter set base value, etc.), while the slice payload (i.e., the entropy-coded data that constitutes the actual sample values ​​of the picture when decoded) is referenced by the extractor by pointing to (part of) a NAL unit in another track, which is copied when such a file format track is read.

[0010] In a 360-degree video tile-based streaming system, this extractor tool typically results in a design where each tile is packaged and provided as an independent HEVC stream in a separate file format track that can be decoded by a compatible HEVC decoder to yield a respective spatial subset of the full picture. A set of such extractor tracks, each targeting a specific viewing direction (i.e., a combination of tiles at a specific resolution that focuses decoding resources, such as sample budgets, on tiles within the viewport), are provided, with the file format tool then performing a fusion process to yield a single compatible HEVC bitstream containing all the necessary tiles upon loading. The client can select the extractor track that best suits the current viewport and download the track containing the referenced tiles.

[0011] Each extractor track stores a parameter set in the HEVCConfigurationBox contained in the HEVCSampleEntry. These parameter sets are generated during the file format packaging process and are only available in the sample entries; that is, the parameter set is delivered out-of-band (using the initialization segment) when a client selects an extractor track, and therefore the parameter set cannot change over time while playing the same extractor track. In addition to the required sample entries, the initialization segment of an extractor track also contains a fixed list of dependent trackIDs in a track reference container ('tref'). The extractor (contained in the extractor track's media segment) contains an index value that references this 'tref' to determine which trackID is referenced by the extractor.

[0012] However, this design has a number of drawbacks. Each view direction (or combination of tiles) needs to be represented through a separate extractor track that explicitly references the tiles (i.e. tracks) it contains, which introduces significant overhead. Clients may be better able to choose tile resolutions (i.e. create their own combinations) that better suit their needs (e.g. client FoV and latency considerations). Also, the data contained in such extractor tracks is often very similar throughout the timeline (inline and sample constructors remain the same). Typically, all slice headers need to be adjusted through the extractor, which introduces a significant additional overhead. This results in many pointers to dependent tracks, which means that a lot of buffer copying needs to be performed, which can be particularly expensive in web applications using JavaScript. The file format parser cannot resolve extractor tracks unless all data has been completely downloaded beforehand, which can add additional delays to the system when, for example, all video data (tiles) has been downloaded and the client is waiting to fetch the extractor tracks. The complexity of general encryption of such extractor tracks increases, since partial encryption has to be applied (the slice payload has to be encrypted independently of the slice header).

[0013] 1.3 VVC Design and File Format Impact on Tiled Streaming For the next codec generations, such as VVC, two main efforts were made to simplify the extraction / derivation operations of the compressed domain.

[0014] 1.3.1 Tiling Syntax in VVC While in HEVC the subdivision of a picture into slices (NAL units) was ultimately signaled at the slice header level, i.e., by having multiple slices in a tile or multiple tiles in a slice, in VVC the subdivision of a picture into slices (NAL units) is described solely in the parameter set. After the first division level is signaled through the rows and columns of tiles, the second division level is signaled through the so-called brick division of each tile. A tile that does not contain further brick division is also called a single brick. The number of slices per picture and the associated bricks are explicitly indicated in the parameter set.

[0015] 1.3.2 Slice Address Signaling in VVC For example, previous codecs such as HEVC relied on slice position signals through slice addresses in CTU raster scan order in each slice header, specifically, first_slice_in_pic_flag and slice_address, which have coded lengths that depend on the picture size. VVC features indirection of these addresses instead of these two syntax elements, where slice headers carry, instead of explicit CTU positions, identifiers (e.g., brick_id, tile_id, or subpic_id) that are mapped to specific picture positions by the associated parameter set as slice addresses. Thus, if tiles are to be relocated in an extraction or merging operation, only the indirection of the parameter set needs to be adjusted instead of each slice header.

[0016] 1.3.3 VVC Syntax and Semantics Figure 5 shows relevant excerpts of currently envisioned VVC picture parameter set and slice header syntax taken from the VVC specification (Draft 6, 11th edition), with line numbers preceding the relevant syntax. The syntax elements on lines 5 to 49 of the picture parameter set syntax relate to the tiling structure, and the syntax elements on lines 54 to 61 of the picture parameter set syntax and the slice_address syntax element of the slice header syntax relate to slice / tile placement.

[0017] The semantics of the syntax elements related to slicing / tiling are as follows: slice_id[i] specifies the slice ID of the i-th slice. The length of the slice_id[i] syntax element is signalled_slice_id_length_minus1 + 1 bits. If not present, the value of slice_id[i] is inferred to be equal to i for each i in the range 0 to num_slices_in_pic_minus1, inclusive. slice_address specifies the slice address of the slice. If not present, the value of slice_address is inferred to be equal to 0. If rect_slice_flag is 0, The slice address is the brick ID defined by equation (7-59), The length of slice_address is Ceil(Log2(NumBricksInPic)) bits, The value of slice_address is in the range 0 to NumBricksInPic-1 inclusive. Otherwise (rect_slice_flag is equal to 1), The slice address is the slice ID of the slice, The length of the slice_address is signalled_slice_id_length_minus1+1 bits, If signalled_slice_id_flag is 0, the value of slice_address is in the range of 0 to num_slices_in_pic_minus1. Otherwise, the value of slice_address is in the range of 0 to 2. (signalled_slice_id_length_minus1+1) The range is -1.

[0018] The requirements for bitstream conformance are that the following constraints apply: The value of slice_address must not be equal to the value of slice_address of any other coded slice NAL unit of the same coded picture. If rect_slice_flag is 0, the slices of the picture are in ascending order of their slice_address values. The shape of a picture slice is such that, when decoded, the entire left and entire top boundary of each brick consists of the picture boundary or the boundary of the previously decoded brick(s).

[0019] For example, future container format unification designs, such as future file format extensions to which the present invention pertains, may facilitate changes in the VVC high-level syntax relative to the HEVC high-level syntax. More particularly, the present invention includes aspects dealing with: Basic classification of source tracks (tile tracks) into fusion-ready sets, Configurable parameter sets and / or templates for SEI messages, configurable parameter sets and / or extended classifications for SEI messages; and Random access point indication in track combinations.

[0020] According to an aspect of the present invention, there is provided video data for deriving a spatially varying portion of a scene, the video data being formatted in a file format and comprising: a set of two or more source tracks, each source track containing encoded video data representing a spatial portion of the video showing a scene; The set of two or more source tracks comprises a source track group, and the formatted video data further comprises one or more group indicators indicating the source tracks belonging to each source track group, and one or more active source track indicators indicating the number of two or more active source tracks in the source track group.

[0021] According to another aspect of the present invention, there is provided video data for deriving a spatially varying portion of a scene, the video data being formatted in a file format and comprising: a set of two or more source tracks, each source track containing coded video data representing a spatial portion of video showing a scene; gathering information including fusion information for merging subsets of two or more sets of source tracks to generate a section-specific video data stream; and the formatted video data further includes a template for a configurable parameter set and / or SEI message, the template indicating one or more values ​​of the parameter set or SEI message that need to be adapted to generate a parameter set or SEI message specific to the section-specific video stream.

[0022] According to another aspect of the present invention, there is provided video data for deriving a spatially varying portion of a scene, the video data being formatted in a file format and comprising: a set of one or more source tracks containing encoded video data representing spatial portions of a video showing a scene; The encoded video data is encoded using random access points, and the formatted video data further includes one or more random access point alignment indicators that indicate whether the random access points in the encoded video data for all spatial portions are aligned.

[0023] According to another aspect of the present invention, there is provided a method of preparing video data for deriving a spatially varying portion of a scene, the video data being formatted in a file format and comprising: a set of two or more source tracks, each source track containing encoded video data representing a spatial portion of the video showing a scene; the set of two or more source tracks comprises a source track group, and the formatted video data further comprises one or more group indicators indicating source tracks belonging to each source track group, and one or more active source track indicators indicating the number of the two or more active source tracks in the source track group; The method is: Determining a source track group and the number of two or more active source tracks in the group, creating one or more group indicators and one or more active source track indicators, and writing them to the formatted video data.

[0024] According to another aspect of the present invention, there is provided a method of preparing video data for deriving a spatially varying portion of a scene, the video data being formatted in a file format and comprising: a set of two or more source tracks, each source track containing coded video data representing a spatial portion of video showing a scene; collected information including fusion information for fusion of subsets of two or more sets of source tracks to generate a section-specific video data stream; the collected information further includes a template of a configurable parameter set and / or SEI message, the template indicating one or more values ​​of the parameter set or SEI message that need to be adapted to generate a parameter set or SEI message specific to the section-specific video stream; The method is: This involves creating a template and writing it into a collection of formatted video data.

[0025] According to another aspect of the present invention, there is provided a method of preparing video data for deriving a spatially varying portion of a scene, the video data being formatted in a file format and comprising: a set of one or more source tracks containing encoded video data representing spatial portions of a video showing a scene; the encoded video data is encoded using random access points, and the formatted video data further includes one or more random access point alignment indicators that indicate whether the random access points in the encoded video data for all spatial portions are aligned; The method is: Creating one or more random access point alignment indicators and writing them into the formatted video data.

[0026] According to another aspect of the present invention there is provided an apparatus for producing video data for deriving a spatially varying portion of a scene, the video data being formatted in a file format, the apparatus being adapted to perform a method according to any of claims 38 to 55.

[0027] According to another aspect of the present invention, there is provided a method of deriving spatially varying portions of a scene from video data, the video data being formatted in a file format and comprising: a set of two or more source tracks, each source track containing encoded video data representing a spatial portion of the video showing a scene; the set of two or more source tracks comprises a source track group, and the formatted video data further comprises one or more group indicators indicating source tracks belonging to each source track group, and one or more active source track indicators indicating the number of the two or more active source tracks in the source track group; The method is: Reading from the formatted video data one or more group indicators, one or more active source track indicators, and encoded video data from the indicated number of two or more active source track groups, and deriving spatially varying portions of the scene based thereon.

[0028] According to another aspect of the present invention, there is provided a method of deriving spatially varying portions of a scene from video data, the video data being formatted in a file format and comprising: a set of two or more source tracks, each source track containing coded video data representing a spatial portion of video showing a scene; collected information including fusion information for fusion of subsets of two or more sets of source tracks to generate a section-specific video data stream; the collected information further includes a template of a configurable parameter set and / or SEI message, the template indicating one or more values ​​of the parameter set or SEI message that need to be adapted to generate a parameter set or SEI message specific to the section-specific video stream; The method is: This includes reading a template from the collection of formatted video data and adapting one or more values ​​of a parameter set or SEI message indicated by the template to generate a parameter set or SEI message specific to the section-specific video stream.

[0029] According to another aspect of the present invention, there is provided a method of deriving spatially varying portions of a scene from video data, the video data being formatted in a file format and comprising: a set of one or more source tracks containing encoded video data representing spatial portions of a video showing a scene; the encoded video data is encoded using random access points, and the formatted video data further includes one or more random access point alignment indicators that indicate whether the random access points in the encoded video data for all spatial portions are aligned; The method is: The method includes reading one or more random access point indicators from the formatted video data and accessing the encoded video data based thereon.

[0030] According to another aspect of the present invention there is provided an apparatus for deriving a spatially varying portion of a scene from video data, the video data being formatted in a file format, the apparatus being adapted to perform a method according to any of claims 57 to 74.

[0031] According to another aspect of the present invention there is provided a computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method of claims 38 to 55 or 57 to 74.

[0032] According to another aspect of the present invention there is provided a computer readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of claims 38 to 55 or 57 to 74.

[0033] According to another aspect of the present invention there is provided a digital storage medium having stored thereon video data according to any one of claims 1 to 37.

[0034] It is to be understood that the video data of claims 1 to 37, the method of claims 38 to 55, the apparatus of claim 56, the method of claims 57 to 74, the apparatus of claim 75, the computer program of claim 76, the computer readable medium of claim 77 and the digital storage medium of claim 78 have similar and / or identical preferred embodiments, in particular as defined in the dependent claims.

[0035] It is to be understood that a preferred embodiment of the invention can also be any combination of the dependent claims or the above embodiments with the respective independent claim.

[0036] Hereinafter, embodiments of the present invention will be described in more detail with reference to the accompanying drawings. [Brief explanation of the drawings]

[0037] [Figure 1] A diagram showing 360-degree video in cubemap projection tiling into 6x4 tiles at two resolutions (high resolution and low resolution). [Figure 2] FIG. 1 illustrates how a user typically sees only a subset of the tiles that make up a full 360-degree video when using a modern head-mounted display (HMD). [Figure 3]FIG. 3 illustrates possible tile arrangements within the merged bitstream for the examples of FIGS. 1 and 2. [Figure 4] FIG. 10 illustrates another tiling scheme that can blend an entire low-resolution version of the video with high-resolution tiles that cover a subset of the 360-degree video. [Figure 5A] 1 shows relevant excerpts of currently envisioned VVC picture parameter set and slice header syntax taken from the VVC specification (Draft 6, 11th edition). [Figure 5B] 1 shows relevant excerpts of currently envisioned VVC picture parameter set and slice header syntax taken from the VVC specification (Draft 6, 11th edition). [Figure 5C] 1 shows relevant excerpts of currently envisioned VVC picture parameter set and slice header syntax taken from the VVC specification (Draft 6, 11th edition). [Figure 6A] FIG. 10 illustrates the XML schema carried within the file format box used to provide the template creation rules. [Figure 6B] FIG. 10 illustrates the XML schema carried within the file format box used to provide the template creation rules. [Figure 6C] FIG. 10 illustrates the XML schema carried within the file format box used to provide the template creation rules. [Figure 7] FIG. 10 illustrates the concept of template gaps, storing parameter set templates in a VVCDecoderConfigurationRecord and signaling gaps using offset and size values. [Figure 8] FIG. 1 illustrates two types of decoder configuration processes allowed by the file format specification: the use of out-of-band parameter sets and in-band parameter sets. [Figure 9]FIG. 1 illustrates a new concept of generation parameter sets, where a collection track contains a parameter set template stored out-of-band (in a sample entry), which is used to create a "generation parameter set" that can exist in-band when all required media segments have been selected by the client. DETAILED DESCRIPTION OF THE INVENTION

[0038] The following description of embodiments of the present invention with reference to the figures first focuses on embodiments related to basic classification of source tracks (tracks of tiles) into fusible sets. Then, embodiments related to configurable parameter sets and / or templates in SEI messages are described, followed by embodiments related to advanced classification for random access point indication in configurable parameter sets and / or SEI messages and track merging. In certain applications, all four types of embodiments can be used together to exploit each of these concepts.

[0039] To motivate and facilitate understanding of the embodiments, an example of a 360-degree video playback application will be described based on a cube-map projection of the scene shown in FIGS. 1-3 tiled into 6×4 spatial segments at two resolutions (high resolution and low resolution). Such a cube-map projection constitutes video data arranged to derive spatially varying portions of the scene. For example, as shown in the top part of FIG. 2 , a user may view a 90°×90° field of view using a head-mounted display (HMD). In the example of FIG. 2 , the subset of tiles required to represent the FoV are four tiles on the right side of the cube-map projection, two tiles below, one tile in front, and one tile in the back. Of course, depending on the user's viewing direction, other subsets of tiles may be required to represent the user's current FoV. In addition to these tiles, which the client application can download and decode at high resolution, the client application may also need to download other tiles outside the viewport to accommodate sudden changes in the user's orientation. These tiles can be downloaded and decoded by the client application at low resolution. As mentioned above, after download at the client side, it may be desirable to fuse the downloaded tiles into a single bitstream to be processed by a single decoder, e.g., to address the constraints of typical mobile devices with limited computational resources and power.

[0040] In this example, we assume that each tile is coded independently using the Versatile Video Coding (VVC) currently under development. This coding can be achieved by partitioning the picture using a suitable tile / brick / slice structure, e.g., so that intra- or inter-prediction is not performed between different tiles / bricks of the same or different pictures. As can be seen from Figure 5, which shows an excerpt of the currently envisioned VVC picture parameter set and slice header syntax taken from the VVC specification (Draft 6, 11th Edition), VVC extends the concept of tiles and slices known from HEVC by so-called bricks, which specify a rectangular region of CTU (coding tree unit) rows within a particular tile in a picture. Thus, a tile can be divided into multiple bricks, each consisting of one or more CTU rows within the tile. Using this extended tile / brick / slice structure, a tile arrangement such as that shown in Figure 3 can be easily formed, in which a 4x2 spatial segment of high-resolution video and a 4x4 spatial segment of low-resolution video are merged into a combined bitstream through compressed domain processing.

[0041] According to the present invention, this merging process is supported by a specific "merging-friendly" file format in which the video data is formatted. In this example, this file format is an extension of MPEG OMAF (ISO / IEC 23090-2), which is further based on the ISO Base Media File Format (ISO / IEC 14496-12), which defines the general structure of time-based multimedia files such as video and audio. In this file format, independently decodable video data corresponding to different spatial segments are contained in different tracks, also referred to herein as source tracks or tile tracks.

[0042] Although this example assumes VVC as the basic video codec, the present invention is not limited to the application of VVC, and other video codecs such as HEVC (High Efficiency Video Coding) can also be used to realize different aspects of the present invention. Furthermore, while this example assumes the file format to be an extension of MPEG OMAF, the present invention is not limited to such an extension, and other file formats or extensions of other file formats can also be used to realize different aspects of the present invention.

[0043] 2. Basic classification of source tracks into a fusion-ready set According to a first aspect of the present invention, a basic classification mechanism allows a file format parser to indicate that several source tracks belong to the same group and that a given number of tiles belonging to that group should be reproduced.

[0044] In this regard, the formatted video data includes a set of two or more source tracks, each of which includes encoded video data representing a spatial portion of a video showing a scene. The set of two or more source tracks includes source track groups, and the formatted video data further includes one or more group indicators indicating source tracks belonging to each source track group and one or more active source track indicators indicating the number of two or more active source tracks in the source track group. In this example, a first source track group includes 6x4 high-resolution tiles of a cube map projection, and a second source track group includes 6x4 low-resolution tiles. This can be indicated by one or more group indicators. Furthermore, as described above, if the user's expected FoV is 90 degrees by 90 degrees, eight of the 24 high-resolution tiles need to be reproduced to represent the user's current field of view, while 16 of the low-resolution tiles need to be transmitted to allow for sudden changes in the user's orientation. The eight source tracks in the first group and the 16 source tracks in the second group can be referred to as "active" source tracks, and their respective numbers can be indicated by one or more active source track indicators.

[0045] In one embodiment, this can be achieved by using a first box in the file format that contains one or more group indicators, e.g., a Track Group Type box. A possible syntax and semantics based on the Track Group box concept from the ISO Base Media File Format can be as follows: JPEG2025138676000002.jpg87168

[0046] track_group_type indicates the classification type and is set to one of the following values ​​or registered values, or values ​​from a derived specification or registration. [...] 'aaaa' indicates that this track belongs to a group of tracks with the same track_group_ID value, of which a subset of num_active_tracks should be played. num_active_tracks must be greater than 1.

[0047] In this case, one or more group indicators are realized by the syntax element track_group_ID, and one or more active source track indicators are realized by the syntax element num_active_tracks. Also, a new track_group_type is defined ('aaaa' is just an example) that indicates that the track group type box contains the syntax element num_active_tracks. This kind of track group type box can be signaled within each respective source track belonging to a group.

[0048] Because both source tracks belonging to the first group and source tracks belonging to the low-resolution group are required to realize a 360-degree video playback application, the present application further anticipates the possibility of indicating to a file format parser that two or more groups of source tracks are bundled together, and in this regard, the formatted video data further includes one or more group bundle indicators that indicate such bundling.

[0049] In another embodiment, in combination with the above signaling per source track, this can be achieved by using another second box, for example a track reference type box, to bundle together multiple groups to be used in one combination (e.g. using one track_group_ID value for high resolution tiles and one track_group_ID value for low resolution tiles).

[0050] In the TrackGroupTypeBox of type 'aaaa' which indicates the uniqueness of the track_group_ID, the value of (flags&1) shall be equal to 1 so that the group can be referenced via 'tref'.

[0051] As implied by the general semantics of track references to track_group_ID, the num_active_tracks tracks in the 'aaaa' source track group are used to resolve 'tref'.

[0052] Alternatively, in another embodiment, the Source Track Group does not indicate the number of tracks to play, but instead expresses this characteristic through an extension of the Track Reference Type box as follows: JPEG2025138676000003.jpg100167

[0053] In this example, the file format contains one or more group indicators indicating the source tracks that belong to each source track group, one or more active source track indicators indicating the number of active source tracks in the source track group, and one or more group bundle indicators indicating that two or more source track groups are bundled together, all in a single box, in this example the track reference type box.

[0054] The syntax element num_track_group_IDs indicates the number of source track groups bundled together in the track reference type box, and the syntax elements track_group_IDs[i] and num_active_tracks_per_track_group_IDs[i] indicate the track group ID and the number of active tracks per group. In other words, in this embodiment, each source track group is indicated by a respective group ID (e.g., track_group_ID), and two or more source track groups that are bundled together are indicated by an indicator of the number of two or more source track groups that are bundled together (e.g., num_track_group _IDs) and an array of the respective group IDs (e.g., track_group_IDs[i]).

[0055] In the latter two embodiments, the formatted video data may further include a collection track containing fusion information for merging subsets of a set of two or more source tracks to generate a section-specific video data stream, and the track reference box is included in the collection track.

[0056] Alternatively, in yet another embodiment, source track signaling is used to bundle together (sub)groups of source tracks that collect tiles of the same resolution (e.g., high and low resolution). Again, this embodiment can be based on the concept of a track group box from ISO base media files, whose possible syntax and semantics are as follows: JPEG2025138676000004.jpg133168

[0057] track_group_type indicates the classification type and is set to one of the following values ​​or registered values, or values ​​from a derived specification or registration. [...] 'bbbb' indicates that this track belongs to a track group with the same value of track_group_ID and a subgroup with the same value of track_subgroup_ID, of which a subset of num_active_tracks_per_track_subgroup_IDs[i] tracks should be played, and track_subgroup_IDs[i] is equal to track_subgroup_ID.

[0058] Thus, in this example, each source track group is indicated as being a subgroup of the source track by its respective subgroup ID (e.g., track_subgroup_ID), and two or more source track subgroups bundled together are indicated by a common group ID (e.g., track_group_ID), an indicator of the number of two or more source track subgroups bundled together (e.g., num_track_subgroup_IDs), and an array of their respective subgroup IDs (e.g., track_subgroup_IDs[i]).

[0059] Alternatively, in yet another embodiment of the present invention, further group-specific level signaling allows the client to select a group / subgroup combination that matches the level capabilities of the supported decoder. For example, an extension of the last embodiment using the Track Group Type box could be as follows: JPEG2025138676000005.jpg143168

[0060] track_group_type indicates the classification type and is set to one of the following values ​​or registered values, or values ​​from a derived specification or registration. [...] 'cccc' indicates that this track belongs to a track group with the same value of track_group_ID and a subgroup with the same value of track_subgroup_ID, of which a subset of num_active_tracks_per_track_subgroup_IDs[i] tracks should be played, where track_subgroup_IDs[i] is equal to track_subgroup_ID, playback of the group with track_group_ID corresponds to the level of level_idc of the bitstream corresponding to that group, and the resulting bitstream will have the indicated number of num_active_tracks_per_track_subgroup_IDs[i] tracks for each of the num_track_subgroup_IDs subgroups.

[0061] In other words, in this case, the formatted video data further includes a level indicator (e.g., level_idc) that indicates the encoding level of the source track group or bundle of two or more source track groups when the indicated number of tracks are played together.

[0062] It should be noted that level indicators may also be provided in other described embodiments. Furthermore, two or more source track groups do not necessarily have to differ (only) in resolution, but rather may additionally or alternatively differ in encoding fidelity. For example, a first source track group may include source tracks including encoded video data at a first resolution and / or fidelity, and a second source track group may include source tracks including encoded video data at a second resolution and / or encoding fidelity that is different from the first resolution and / or encoding fidelity.

[0063] 3. Configurable Parameter Sets and / or Templates for SEI Messages As mentioned above, certain applications require parameter sets or SEI message variants depending on the playout context (joint decoding of tiles in a fused bitstream with varying tile positions and tile neighbors), and therefore, in many cases, it is neither easy nor possible to have a single parameter set that applies to multiple combinations.

[0064] One embodiment consists of signaling the classification mechanism, e.g., as described above, to further indicate that some values ​​of the parameter set template need to be changed. For example, referring to the example of varying only the tile selection, as described above, the classification mode used indicates that the slice_address (HEVC terminology) or slice_id (current VVC terminology used in the picture parameter set syntax table shown in Figure 5) need to be modified. Another classification mode value indicates that the RWP SEI message needs to be adjusted, or that syntax elements related to tiling also need to be adjusted.

[0065] The drawback of such an approach is that it requires signaling a different group type or similar indication for each use case that may require a different syntax element to be modified (sometimes the slice_id, in other use cases a different syntax element such as the tiling parameters may need to be replaced). A more flexible and generic approach that allows for the modification of any syntax element and indicates which syntax elements need to be modified would be beneficial.

[0066] For this purpose, in another embodiment, a representation of the unaffected parameter set values, i.e., a parameter set template, is carried within the box of the file format, which the client can use to generate the correct parameter set depending on its tile / track selection.

[0067] Thus, according to this second aspect of the present invention, the formatted video data includes a set of two or more source tracks, each including encoded video data representing a spatial portion of a video showing a scene, and collection information including fusion information for fusing a subset of the set of two or more source tracks to generate a section-specific video data stream. The collection information further includes a template of a configurable parameter set and / or SEI message, the template indicating one or more values ​​of the parameter set or SEI message that need to be adapted to generate a parameter set or SEI message specific to the section-specific video stream. In some embodiments, the formatted video data includes a collection track that includes the collection information. Different embodiments of this aspect are described below.

[0068] 3.1 XML / JSON Templates In one embodiment, a parameter set template and / or SEI message template is an XML or JSON description of the encoding structure of a parameter set or SEI message, including syntax element names and values, and possibly their encoding. From this XML / JSON description, a client (file format parser) can generate a bitstream representation of the parameter set / SEI message by encoding each individual syntax element in its respective form, concatenating the results, and performing emulation prevention. For syntax elements that need to be adjusted by the file format parser, such as the syntax element slice_id or equivalent information for adjusting the position of a tile in a tiled layout, the respective field is preferably marked in the XML / JSON description as follows: JPEG2025138676000006.jpg2689

[0069] In another embodiment, rules for template creation are provided using an XML or JSON schema carried within a file format box. Figure 6 illustrates one embodiment of such a schema using XML. An advantage of using an XML / JSON schema is that a receiving file format parser can generate a conforming parameter set / SEI message bitstream without prior knowledge of the underlying codec, as long as the syntax element encoding options (e.g., fixed-length vs. variable-length encoding, exponential-Golomb coding, etc.) are known. An additional advantage is that once a single schema is defined, it can be used to easily validate all generated parameter set templates and / or SEI message templates. The XML / JSON description metadata with the corresponding parameter set templates is preferably stored in the track box ('trak') of the collection track located within the initialization segment.

[0070] 3.2 Bitstream Templates without Emulation Prevention In another embodiment, the parameter set template and / or SEI message template is based on the coded bitstream form of the parameter set / SEI message, i.e., the individual syntax element values ​​are coded according to a specification (e.g., fixed-length vs. variable-length code, exponential-Golomb code, etc.) and concatenated according to their specified order. However, this form does not include emulation prevention bytes. Therefore, emulation prevention must be performed before such parameter sets are used in a video bitstream.

[0071] In one embodiment, the parameter set template and / or SEI message template carries an indication of the gap in which to insert a syntax element value, i.e., its encoded representation, such as slice_id.

[0072] Thus, in a general sense, a template may include concatenated encoding syntax elements of a parameter set or SEI message in which values ​​that do not need to be matched are validly encoded in the template, and further includes one or more gap indicators that indicate gaps in the template that should be filled with validly encoded values ​​that do need to be matched. The one or more gap indicators that indicate gaps preferably include an offset and a size of the gap in the template.

[0073] Figure 7 illustrates the concept of template gaps, where parameter set templates are stored within a VVCDecoderConfigurationRecord and gaps are signaled using corresponding offset and size values. Gaps can be signaled, for example, by defining a bitstream blob position (offset) relative to the start of the VVCDecoderConfigurationRecord and the size of the gap, signaling which element of the parameter set or SEI message is the next element of that blob according to the specification. In one embodiment, a slice_id value (see Figure 5) can be inserted into such a template gap. In another embodiment, tiling structure syntax values ​​(see Figure 5) are inserted into the parameter set template gap.

[0074] Preferably, generating a section-specific video stream-specific parameter set or SEI message includes, after filling gaps in the template, performing emulation prevention on the concatenated coding syntax elements to generate a coded bitstream of the parameter set or SEI message.

[0075] 3.3 Templates with placeholder values In another embodiment, the parameter set templates and / or SEI message templates stored in the VVCDecoderConfigurationRecord are fully decodable, i.e., they are stored in a bitstream format with emulation prevention, just like a normal non-template parameter set or SEI message, but the fields to be adjusted are filled with valid placeholder values ​​per encoding. Such template parameter sets are fully specification-compliant and can be parsed by a standard corresponding VVC parser. The idea of ​​using such parameter set templates and / or SEI message templates is that when a parser processes these parameter sets / SEI messages, it can easily use the instances to overwrite the necessary values ​​to complete the definition of the generated parameter set / SEI message.

[0076] Thus, in a general sense, a template may comprise a parameter set or encoded bitstream of an SEI message that includes an emulation prevention byte in which one or more values ​​that need to be matched are filled with placeholder values ​​that are validly encoded in the encoded bitstream. In a variation of this embodiment described in section 3.2 above, it is understood that the one or more gap indicators correspond to placeholder value indicators that indicate placeholder values ​​that need to be matched, and that the one or more placeholder value indicators that indicate placeholder values ​​include the offset and size of the placeholder values ​​within the template.

[0077] 3.4 Possible realizations The following shows a possible implementation of the decoder configuration record box within a sample entry that includes the above-described embodiment, i.e., the new sample entry type 'vvcG'. Here, within the loop "for(i=0;i<numNalus;i++)", a NAL unit can include, for example, a parameter set template or an SEI message template, or a bitstream that forms an XML / JSON base64 encoded representation of a parameter set template or an SEI message template. JPEG2025138676000007.jpg240160

[0078] In this implementation, the template is included in the decoder configuration record (e.g., VvcDecoderConfigurationRecord), but it can also be included at another location within the initialization segment, such as at another location within the sample description box or at another location within the sample entry box. Further, the presence of the template in the NAL unit is preferably indicated by the NAL unit type (e.g., by defining a specific NAL unit type that indicates the NAL unit including the template).

[0079] In addition to indicating a parameter set template or an SEI message template within a sample entry of type 'vvcG', the presence of a parameter set template or an SEI message template is preferably also indicated by an additional flag templateNalu within the decoder configuration record of a normal 'vvc1' sample entry. This flag can be provided, for example, for each NAL unit within the loop "for(i=0;i<numNalus;i++)".

[0080] Thus, in a general sense, a template may be included in a sample entry box, preferably in a decoder configuration record, and the presence of a template in a NAL unit is indicated by a sample entry type (e.g., 'vvcG') and / or one or more template indicators (e.g., templateNalu) in the sample entry box.

[0081] In these embodiments, additional NAL unit types, such as supplemental enhancement information (SEI) messages, may be carried in any of the template forms described above and modified accordingly depending on the particular combination selected at the client side. One such SEI message is the RWP SEI message specified by AVC and HEVC.

[0082] To facilitate parameter / syntax element substitution within a parameter set or SEI message, there is additional information required through a classification mechanism that is partially signaled within the collection information, e.g., collection tracks and source tracks selected to be combined. This aspect is further described below in Section 4.

[0083] 3.5 Truck-by-truck vs. sample-by-sample transport The described methods for configurable parameter sets and / or SEI messages can be present, for example, in the decoder configuration record of the initialization segment as in the above embodiment, or can be present in the track in a particular sample. If the parameter set template is included in the track, for example as a media sample, then the new sample format can be defined as a parameter set template or SEI message template, for example in XML / JSON format.

[0084] In another embodiment, a NAL unit with a NAL unit type reserved for external use in VVC is used, and the body of the NAL unit (i.e., the NAL unit payload) is filled with some (somehow distinguishable) parameters and placeholder values ​​that need to be modified according to some values ​​in the sample group information or the like. For this purpose, one of the described methods (templates in XML / JSON or bitstream format with the "to be modified" fields identified) can be inserted into the NAL unit payload of this special NAL unit structure.

[0085] FIG. 8 illustrates two types of decoder configuration processes allowed by the file format specification. · An out-of-band parameter set contained only in a sample entry in the corresponding decoder configuration record box in the initialization segment. In-band parameter sets, contained in sample entries but which can also be sent in the media samples themselves, allowing the decoder configuration to be changed over time while playing the same file format track.

[0086] OMAF Version 1 only allows out-of-band signals for 360-degree video, and each extractor track contains a predetermined set of parameters generated by the file format packager for a fixed tiling configuration. Therefore, every time a client wants to change this tiling configuration, they must change the extraction track and reinitialize the decoder with the corresponding parameter set.

[0087] As already explained in the previous section, having such a predetermined set of parameters for a particular tiling configuration is a major drawback, as the client can only work on the predetermined extractor tracks for the particular tiling scheme and cannot flexibly blend the necessary tiles itself (without the extractor NAL units).

[0088] Therefore, the idea of ​​the present invention is to combine the concepts of in-band and out-of-band parameter sets to create a solution that includes both concepts. Figure 9 illustrates the new concept of generated parameter sets. The corresponding collection track contains a parameter set template stored out-of-band (in a sample entry), which is used to create a "generated parameter set" that will be available in-band when all required media segments are selected by the client. A file format track classification mechanism is used to provide information on how to update the parameter set template based on the selected subset of downloaded tiles.

[0089] In one embodiment, the collection track does not contain the media segments themselves, so that the media segments are implicitly defined as the sum of the media segments of the selected tiles ('vvcG' in Figure 9). Thus, the initialization segment (e.g., sample entry) of the collection track contains the entire metadata required to create the generated parameter set.

[0090] In another embodiment, the collection track also includes media segments that provide additional metadata for parameter set generation, allowing the behavior of parameter set generation to change over time rather than relying solely on metadata from the sample entries.

[0091] Thus, in a general sense, the template may be included in an initialization segment of a collection track, preferably in a sample description box, more preferably in a sample entry box, and most preferably in a decoder configuration record, and the fusion information includes media segments including references to coded video data of a subset of a set of two or more source tracks, one or more of the media segments further including i) a template for a configurable parameter set and / or SEI message, or ii) an indicator indicating that the media segment of the generated section-specific video data stream includes the parameter set and / or SEI message generated together with the template.

[0092] Note that in all embodiments relating to the use of configurable parameter sets and / or templates in SEI messages, slices can be used to encode the coded video data contained in each source track, and the generation of section-specific video data streams does not require adapting the values ​​of the slice headers of the slices.

[0093] Preferably, the coded video data included in each source track is i) coded using tiles, and the values ​​that need to be adapted relate to the tile structure, and / or ii) coded using bricks, and the values ​​that need to be adapted relate to the brick structure, and / or iii) coded using slices, and the values ​​that need to be adapted relate to the slice structure. In particular, the values ​​that need to be adapted may represent the positions of tiles and / or bricks and / or slices within the pictures of the video and / or the coded video data.

[0094] The parameter set is preferably a video parameter set (VPS), a sequence parameter set (SPS), or a picture parameter set (PPS), and / or the SEI message is preferably a region-wise-packing (RWP) SEI message.

[0095] 4. Configurable Parameter Sets and / or Extended Classification for SEI Messages As explained in the introduction, the current state-of-the-art way to indicate that a group of source tracks can be decoded together is through the above-mentioned extractor tracks, which explicitly reference each track that carries the appropriate parameter set to form one particular valid combination, as shown in Figure 2. To reduce the overhead of this state-of-the-art solution (one track per viewport), the present invention provides more flexibility in indicating which tracks can be combined and the rules for combination. Thus, as part of the present invention, a set of two or more source tracks can include one or more boxes in a file format, each box containing additional information to describe syntax elements that identify characteristics of the source tracks; this additional information allows for the generation of section-specific video stream-specific parameter sets or SEI messages without the need to parse the coded video data.

[0096] In one embodiment, the additional information describes syntax elements that identify a slice ID or other information used in a slice header that identifies the slice structure of the associated VCL NAL unit to identify the slice in the integrated bitstream and its location in the combined picture.

[0097] In another embodiment, the additional information describes i) syntax elements identifying the width and height of the coded video data contained in each source track, and / or ii) syntax elements identifying projection mapping, transformation information, and / or guard band information related to the generation of the region-wise packing (RWP) SEI message. For example, the width and height of the coded video data may be identified in units of coded samples or in units of largest coded blocks. For the RWP SEI message, the syntax element identifying the projection mapping includes the width and height of a rectangular region within the projection mapping, as well as top and left positions. Additionally, the syntax element identifying the transformation information may include rotation and mirroring.

[0098] Furthermore, in another embodiment, the additional information further includes the coding length and / or coding mode of each syntax element (e.g., u(8), u(v), ue(v)) to facilitate creation of the configurable parameter set or SEI message.

[0099] In one embodiment, the syntax of the above box is as follows: As mentioned above, each initialization segment of each source track contains a 'trgr' box (track classification indication) inside a 'trak' box (track box) with an extended Track Group Type box. As a result, new syntax such as the following can be carried in the extension of the Track Group Type box: JPEG2025138676000008.jpg250166 JPEG2025138676000009.jpg90170

[0100] 5. Random Access Point Instructions for Track Combining In VVC, NAL unit types may be mixed within the same access unit, where IDR NAL units may be mixed with non-IDR NAL units, i.e., some regions may be coded using inter prediction, while other regions in the picture may be intra-coded, resetting the prediction chain for this particular region. In such samples, the client may change its tile selection in some parts of the picture, and for this reason it is essential that these samples are marked in the file format signaling mechanism to indicate, for example, a sub-picture random access point (RAP), which indicates that even non-IDR NAL units have the instantaneous decoder refresh (IDR) property when extracted.

[0101] In this aspect of the invention, different spatial portions of the video showing a scene may also be provided in a single source track. Thus, video data for deriving the spatially varying portions of the scene is predicted, the video data being formatted in a file format and including a set of one or more source tracks containing encoded video data representing the spatial portions of the video showing the scene. The encoded video data is encoded using random access points, and the formatted video data further includes one or more random access point alignment indicators that indicate whether the random access points in the encoded video data for all spatial portions are aligned.

[0102] For example, in one embodiment, different regions of a picture are separated into multiple source tracks. The classification mechanism preferably signals whether RAPs are aligned or not. This signaling can be done, for example, by ensuring that wherever a RAP is present in a source track, the RAP is present in a corresponding access unit of another source track containing another spatial portion of the picture, or by having an additional track (similar to the master track) used to signal the RAP. In the second case, only the RAP signaled in a "master" track, such as the collection track described above, indicates the RAP in the other source tracks. If the classification mechanism indicates that the RAPs are not aligned, all RAP signaling in the other source tracks needs to be analyzed. In other words, in this embodiment, coded video data representing different spatial portions are contained in different source tracks, and the formatted video data further includes a common track containing one or more random access point indicators indicating random access points in all source tracks.

[0103] In another embodiment, all spatial portions are included in the same source track. Nevertheless, in some use cases (e.g., zooming), it may be desirable to extract a portion of the entire image (e.g., a region of interest (RoI) in the middle). In such scenarios, the RAPs in the entire picture and the RAPs in the RoI may not always coincide. For example, there may be more RAPs in the RoI than in the entire picture.

[0104] In these embodiments, the formatted video data may further include one or more partial random access point indicators that indicate that an access unit of the video has random access points for spatial portions of the video but not for the entire access unit. Additionally, the formatted video data may further include partial random access point information that describes the location and / or shape of the spatial portions that have random access points.

[0105] In one realization, this information can be provided using so-called sample groups, which are used in the ISO Base Media File Format to indicate certain characteristics of a picture (e.g., synchronization samples, RAPs, etc.). In the present invention, sample groups can be used to indicate that an access unit has partial RAPs, i.e., sub-picture (region-specific) random access points. Furthermore, signaling can be added to indicate that a region can be indicated without any drift for each picture, and the dimension of the region can be signaled. The following shows the syntax of the existing sample to group box: JPEG2025138676000010.jpg96166

[0106] In this embodiment, a specific classification type 'prap' (partial rap) is used to define the sample group for the SampleToGroupBox.

[0107] The sample group description can be defined as follows, for example: JPEG2025138676000011.jpg32169

[0108] A sample description shows the following randomly accessible region dimensions, for example: JPEG2025138676000012.jpg49135

[0109] In a further embodiment, different regions are mapped to separate NAL units, i.e., only some NAL units of an access unit can be decoded. It is part of the invention to show that a particular NAL unit can be treated as a RAP if only the subset corresponding to this region is decoded for the bitstream. For this purpose, the sub-sample classification information for sub-pic RAPs can be derived using the existing concept of sub-sample information box, e.g., as follows: JPEG2025138676000013.jpg179137

[0110] The codec_specific_parameters can indicate which subsamples are RAPs and which are not.

[0111] 6. Further Embodiments

[0013] Up to this point, the description of embodiments of the present invention below with reference to the figures has focused on video data for deriving spatially varying portions of a scene and the particular file formats in which this portion is formatted. However, the present invention also relates to methods and apparatus for creating video data for deriving spatially varying portions of a scene, as well as methods and apparatus for deriving spatially varying portions of a scene from video data formatted in a particular file format. Furthermore, the present invention also relates to corresponding computer programs, computer-readable media, and digital storage media.

[0112] More specifically, the present invention also relates to the following embodiments:

[0113] 1. A method of preparing video data for deriving spatially varying portions of a scene, the video data being formatted in a file format and comprising: a set of two or more source tracks, each source track containing encoded video data representing a spatial portion of a video showing the scene; the set of two or more source tracks includes a source track group, and the formatted video data further includes one or more group indicators indicating source tracks belonging to each source track group, and one or more active source track indicators indicating the number of two or more active source tracks in the source track group; The method comprises: determining the number of source track groups and two or more active source tracks within a group, creating the one or more group indicators and the one or more active source track indicators, and writing them to the formatted video data.

[0114] In an embodiment of this method, the formatted video data further comprises one or more group bundling indicators that indicate two or more source track groups are bundled together, and the method further comprises: Determining the two or more source track groups that are bundled together, creating the one or more bundle indicators, and writing them into the formatted video data.

[0115] In an embodiment of this method, the one or more group indicators indicating the source tracks belonging to the respective source track groups and the one or more active source track indicators indicating the number of active source tracks in the source track groups are included in a first box of a file format that is separate from a second box of a file format that includes the one or more group bundle indicators indicating that the two or more source track groups are bundled together.

[0116] In an embodiment of this method, the first box is a TrackGroup type box and the second box is a TrackReference type box.

[0117] In an embodiment of this method, the one or more group indicators indicating the source tracks belonging to the respective source track groups, the one or more active source track indicators indicating the number of active source tracks in the source track groups, and the one or more group bundle indicators indicating that two or more source track groups are bundled together are contained in a single box of the file format.

[0118] In an embodiment of this method, the single box is a TrackGroup type box or a TrackReference type box.

[0119] In an embodiment of this method, the Track Group Type box is included in a source track, and / or the formatted video data further includes an acquisition track including merging information for merging subsets of the set of two or more source tracks to generate a section-specific video data stream, and the Track Reference Box is included in the acquisition track, and the method further comprises: determining the subset of the set of two or more source tracks, creating the collected track including the fusion information, and writing the same to the formatted video data.

[0120] In an embodiment of this method, each source track group is indicated by a respective group ID, and the two or more source track groups bundled together are indicated by an indicator indicating the number of the two or more source track groups bundled together and an array of the respective group IDs; or each source track group is indicated as being a sub-group of source tracks by a respective sub-group ID, and the two or more sub-groups of source tracks bundled together are indicated by a common group ID, an indicator indicating the number of the two or more sub-groups of source tracks bundled together, and an array of the respective sub-group IDs.

[0121] In an embodiment of the method, the formatted video data further comprises a level indicator indicating an encoding level of the source track group or of a bundle of two or more source track groups, the method comprising: determining the source track group or the bundle of two or more source track groups, creating the level indicator, and writing it to the formatted video data;

[0122] In an embodiment of this method, a first source track group includes source tracks containing encoded video data at a first resolution and / or fidelity, and a second source track group includes source tracks containing encoded video data at a second resolution and / or encoding fidelity that is different from the first resolution and / or encoding fidelity.

[0123] 1. A method of preparing video data for deriving spatially varying portions of a scene, the video data being formatted in a file format and comprising: a set of two or more source tracks, each source track containing encoded video data representing a spatial portion of a video showing the scene; collected information including fusion information for merging subsets of the two or more sets of source tracks to generate a section-specific video data stream; the collected information further includes a template of a configurable parameter set and / or an SEI message, the template indicating one or more values ​​of the parameter set or the SEI message that need to be adapted to generate a parameter set or an SEI message specific to a section-specific video stream; The method comprises: creating said template and writing it into said collection of said formatted video data.

[0124] In an embodiment of this method, the formatted video data includes a collection track containing the collection information.

[0125] In an embodiment of this method, the template comprises an XML or JSON description of the encoding structure of the parameter set or the SEI message.

[0126] In an embodiment of this method, the formatted video data further comprises an XML or JSON schema providing rules for the creation of the template, and the method further comprises: Creating the XML or JSON schema and writing it into the formatted video data.

[0127] In an embodiment of this method, the template includes concatenated encoding syntax elements of the parameter set or the SEI message, within which values ​​that do not need to be matched are validly encoded, and the template further includes one or more gap indicators that indicate gaps within the template that should be filled with validly encoded values ​​that need to be matched.

[0128] In an embodiment of this method, the one or more gap indicators that indicate the gap include an offset and a size of the gap within the template.

[0129] In an embodiment of this method, the generating the parameter set or the SEI message specific to the section-specific video stream includes, after filling the gaps in the template, performing emulation prevention on the concatenated coding syntax elements to generate a coded bitstream for the parameter set or the SEI message.

[0130] In an embodiment of this method, the template includes an encoded bitstream of the parameter set or the SEI message that includes an emulation prevention byte, and the one or more values ​​in the encoded bitstream that need to be matched are filled with validly encoded placeholder values.

[0131] In an embodiment of this method, said template is included in an initialization segment of said collection track, preferably in a sample description box, more preferably in a sample entry box, and most preferably in a decoder configuration record.

[0132] In an embodiment of this method, the template is included in an NAL unit, and the presence of the template in the NAL unit is indicated by the NAL unit type.

[0133] In an embodiment of this method, the template is included in a sample entry box, preferably in a decoder configuration record, and the presence of the template in a NAL unit is indicated by the sample entry type and / or by one or more template indicators in the sample entry box.

[0134] In an embodiment of this method, the fusion information is included in an initialization segment of the collection track, preferably in a sample description box, more preferably in a sample entry box, and most preferably in a decoder configuration record, and the fusion information includes media segments including references to the encoded video data of the subset of the set of two or more source tracks, one or more of the media segments including i) a template for a configurable parameter set and / or SEI message, or ii) an indicator indicating that a parameter set and / or SEI message generated using a template is included in the media segment of the generated section-specific video data stream.

[0135] In an embodiment of this method, the encoded video data contained by each source track is encoded using slices, and the generation of the section-specific video data stream does not require adapting values ​​in the slice headers of the slices.

[0136] In embodiments of this method, the encoded video data contained by each source track is i) encoded using tiles, and the values ​​that need to be adapted are associated with the tile structure, and / or ii) encoded using bricks, and the values ​​that need to be adapted are associated with the brick structure, and / or iii) encoded using slices, and the values ​​that need to be adapted are associated with the slice structure.

[0137] In an embodiment of this method, the values ​​that need to be adapted represent the positions of tiles and / or bricks and / or slices within a picture of the video and / or the coded video data.

[0138] In an embodiment of this method, the parameter set is a video parameter set (VPS), a sequence parameter set (SPS), or a picture parameter set (PPS), and / or the SEI message is a region-wise packing (RWP) SEI message.

[0139] In an embodiment of this method, the set of two or more source tracks includes one or more boxes of the file format, each box containing additional information for describing syntax elements that identify the characteristics of a source track, and the additional information enables the generation of the parameter set or SEI message specific to the section-specific video stream without the need to parse the encoded video data.

[0140] In an embodiment of this method, the additional information describes i) syntax elements that identify the width and height of the encoded video data contained by each source track, and / or ii) syntax elements that identify projection mapping, transformation information and / or guard band information associated with the generation of a region-wise packing (RWP) SEI message.

[0141] In an embodiment of this method, the coded video data contained by each source track is coded using slices, and the additional information describes syntax elements that identify slice IDs or other information used in the slice header to identify the slice structure.

[0142] In an embodiment of this method, said additional information further comprises a coding length and / or a coding mode of said respective syntax element.

[0143] In an embodiment of this method, the one or more boxes are extensions of a box of type TrackGroup.

[0144] 1. A method of preparing video data for deriving spatially varying portions of a scene, the video data being formatted in a file format and comprising: a set of one or more source tracks containing encoded video data representing spatial portions of a video showing the scene; the encoded video data is encoded using random access points, and the formatted video data further includes one or more random access point alignment indicators that indicate whether the random access points in the encoded video data for all spatial portions are aligned; The method comprises: creating the one or more random access point alignment indicators and writing them into the formatted video data.

[0145] In an embodiment of this method, the formatted video data further comprises one or more partial random access point indicators that indicate that an access unit of the video has random access points for a spatial portion of the video but not for the entire access unit, and the method further comprises: Creating the one or more partial random access point indicators and writing them into the formatted video data.

[0146] In an embodiment of this method, the formatted video data further comprises partial random access point information representing the position and / or shape of the spatial portion having the random access point, and the method further comprises: Creating the partial random access point information and writing it to the formatted video data.

[0147] In an embodiment of this method, different spatial parts of an access unit are contained in different NAL units, and the partial random access point information describes which NAL units are random access points for each spatial part, and the partial random access point information is contained in a box of the file format, preferably in a sub-sample information box.

[0148] In an embodiment of the method, the encoded video data representing the different spatial portions are contained in different source tracks, and the formatted video data further comprises a common track including one or more random access point indicators indicating the random access points of all source tracks.

[0149] 56. An apparatus for creating video data for deriving spatially varying parts of a scene, the video data being formatted in a file format, the apparatus being adapted to perform the method of any of claims 38 to 55 or any of the above embodiments.

[0150] 1. A method of deriving spatially varying portions of a scene from video data, the video data being formatted in a file format, the method comprising: a set of two or more source tracks, each source track containing encoded video data representing a spatial portion of a video showing the scene; the set of two or more source tracks includes a source track group, and the formatted video data further includes one or more group indicators indicating source tracks belonging to each source track group, and one or more active source track indicators indicating the number of two or more active source tracks in the source track group; The method comprises: reading from the formatted video data the one or more group indicators, the one or more active source track indicators, and the encoded video data from the indicated number of two or more active source track groups, and deriving spatially varying portions of the scene based thereon.

[0151] In an embodiment of this method, the formatted video data further comprises one or more group bundling indicators that indicate two or more source track groups are bundled together, and the method further comprises: The method includes reading from the formatted video data the one or more bundle indicators and the two or more source track groups bundled together, and deriving spatially varying portions of the scene based thereon.

[0152] In an embodiment of this method, the one or more group indicators indicating the source tracks belonging to the respective source track groups and the one or more active source track indicators indicating the number of active source tracks in the source track groups are included in a first box of a file format that is separate from a second box of a file format that includes the one or more group bundle indicators indicating that the two or more source track groups are bundled together.

[0153] In an embodiment of this method, the first box is a TrackGroup type box and the second box is a TrackReference type box.

[0154] In an embodiment of this method, the one or more group indicators indicating the source tracks belonging to the respective source track groups, the one or more active source track indicators indicating the number of active source tracks in the source track groups, and the one or more group bundle indicators indicating that two or more source track groups are bundled together are contained in a single box of the file format.

[0155] In an embodiment of this method, the single box is a TrackGroup type box or a TrackReference type box.

[0156] In an embodiment of this method, the Track Group Type box is included in a source track, and / or the formatted video data further includes an acquisition track including merging information for merging subsets of the set of two or more source tracks to generate a section-specific video data stream, and the Track Reference Box is included in the acquisition track, and the method further comprises: reading the fusion information and the subset of the set of two or more source tracks from the formatted video data, and fusing the subset of the set of two or more source tracks based on the fusion information to generate the section-specific video data stream.

[0157] In an embodiment of this method, each source track group is indicated by a respective group ID, and the two or more source track groups bundled together are indicated by an indicator indicating the number of the two or more source track groups bundled together and an array of the respective group IDs; or each source track group is indicated as being a sub-group of source tracks by a respective sub-group ID, and the two or more sub-groups of source tracks bundled together are indicated by a common group ID, an indicator indicating the number of the two or more sub-groups of source tracks bundled together, and an array of the respective sub-group IDs.

[0158] In an embodiment of the method, the formatted video data further comprises a level indicator indicating an encoding level of the source track group or of a bundle of two or more source track groups, the method comprising: reading the level indicator and the source track group or the bundle of two or more source track groups from the formatted video data and deriving spatially varying portions of the scene based thereon.

[0159] In an embodiment of this method, a first source track group includes source tracks containing encoded video data at a first resolution and / or fidelity, and a second source track group includes source tracks containing encoded video data at a second resolution and / or encoding fidelity that is different from the first resolution and / or encoding fidelity.

[0160] 1. A method of deriving spatially varying portions of a scene from video data, the video data being formatted in a file format, the method comprising: a set of two or more source tracks, each source track containing encoded video data representing a spatial portion of a video showing the scene; collected information including fusion information for merging subsets of the two or more sets of source tracks to generate a section-specific video data stream; the collected information further includes a template of a configurable parameter set and / or an SEI message, the template indicating one or more values ​​of the parameter set or the SEI message that need to be adapted to generate a parameter set or an SEI message specific to a section-specific video stream; The method comprises: a parameter set or SEI message specific to the section-specific video stream; a parameter set or SEI message specific to the section-specific video stream; a parameter set or SEI message specific to the section-specific video stream; a parameter set or SEI message specific to the section-specific video stream;

[0161] In an embodiment of this method, the template comprises an XML or JSON description of the encoding structure of the parameter set or the SEI message.

[0162] In an embodiment of this method, the formatted video data further comprises an XML or JSON schema providing rules for the creation of the template, and the method further comprises: reading the XML or JSON schema and using it in the generation of the parameter set or the SEI message.

[0163] In an embodiment of this method, the template includes concatenated encoding syntax elements of the parameter set or the SEI message, within which values ​​that do not need to be matched are validly encoded, and the template further includes one or more gap indicators that indicate gaps within the template that should be filled with validly encoded values ​​that need to be matched.

[0164] In an embodiment of this method, the one or more gap indicators that indicate the gap include an offset and a size of the gap within the template.

[0165] In an embodiment of this method, the generating the parameter set or the SEI message specific to the section-specific video stream includes, after filling the gaps in the template, performing emulation prevention on the concatenated coding syntax elements to generate a coded bitstream for the parameter set or the SEI message.

[0166] In an embodiment of this method, the template includes an encoded bitstream of the parameter set or the SEI message that includes an emulation prevention byte, and the one or more values ​​in the encoded bitstream that need to be matched are filled with validly encoded placeholder values.

[0167] In an embodiment of this method, said template is included in an initialization segment of said collection track, preferably in a sample description box, more preferably in a sample entry box, and most preferably in a decoder configuration record.

[0168] In an embodiment of this method, the template is included in an NAL unit, and the presence of the template in the NAL unit is indicated by the NAL unit type.

[0169] In an embodiment of this method, the template is included in a sample entry box, preferably in a decoder configuration record, and the presence of the template in a NAL unit is indicated by the sample entry type and / or by one or more template indicators in the sample entry box.

[0170] In an embodiment of this method, the template is included in an initialization segment of the collection track, preferably in a sample description box, more preferably in a sample entry box, and most preferably in a decoder configuration record, and the fusion information includes media segments including references to the encoded video data of the subset of the set of two or more source tracks, one or more of the media segments including i) a template for a configurable parameter set and / or SEI message, or ii) an indicator indicating that a parameter set and / or SEI message generated using the template is included in the media segment of the generated section-specific video data stream.

[0171] In an embodiment of this method, the encoded video data contained by each source track is encoded using slices, and the generation of the section-specific video data stream does not require adapting values ​​in the slice headers of the slices.

[0172] In embodiments of this method, the encoded video data contained by each source track is i) encoded using tiles, and the values ​​that need to be adapted are associated with the tile structure, and / or ii) encoded using bricks, and the values ​​that need to be adapted are associated with the brick structure, and / or iii) encoded using slices, and the values ​​that need to be adapted are associated with the slice structure.

[0173] In an embodiment of this method, the values ​​that need to be adapted represent the positions of tiles and / or bricks and / or slices within a picture of the video and / or the coded video data.

[0174] In an embodiment of this method, the parameter set is a video parameter set (VPS), a sequence parameter set (SPS), or a picture parameter set (PPS), and / or the SEI message is a region-wise packing (RWP) SEI message.

[0175] In an embodiment of this method, the set of two or more source tracks includes one or more boxes of the file format, each box containing additional information for describing syntax elements that identify the characteristics of a source track, and the additional information enables the generation of the parameter set or SEI message specific to the section-specific video stream without the need to parse the encoded video data.

[0176] In an embodiment of this method, the additional information describes i) syntax elements that identify the width and height of the encoded video data contained by each source track, and / or ii) syntax elements that identify projection mapping, transformation information and / or guard band information associated with the generation of a region-wise packing (RWP) SEI message.

[0177] In an embodiment of this method, the coded video data contained by each source track is coded using slices, and the additional information describes syntax elements that identify slice IDs or other information used in the slice header to identify the slice structure.

[0178] In an embodiment of this method, said additional information further comprises a coding length and / or a coding mode of said respective syntax element.

[0179] In an embodiment of this method, the one or more boxes are extensions of a box of type TrackGroup.

[0180] 1. A method of deriving spatially varying portions of a scene from video data, the video data being formatted in a file format, the method comprising: a set of one or more source tracks containing encoded video data representing spatial portions of a video showing the scene; the encoded video data is encoded using random access points, and the formatted video data further includes one or more random access point alignment indicators that indicate whether the random access points in the encoded video data for all spatial portions are aligned; The method comprises: reading the one or more random access point indicators from the formatted video data and accessing the encoded video data based thereon.

[0181] In an embodiment of this method, the formatted video data further comprises one or more partial random access point indicators that indicate that an access unit of the video has random access points for a spatial portion of the video but not for the entire access unit, and the method further comprises: reading the one or more partial random access point indicators from the formatted video data and accessing the encoded video data based thereon.

[0182] In an embodiment of this method, the formatted video data further comprises partial random access point information representing the position and / or shape of the spatial portion having the random access point, and the method further comprises: The method includes reading the partial random access point information and accessing the encoded video data based thereon.

[0183] In an embodiment of this method, different spatial parts of an access unit are contained in different NAL units, and the partial random access point information describes which NAL units are random access points for each spatial part, and the partial random access point information is contained in a box of the file format, preferably in a sub-sample information box.

[0184] In an embodiment of the method, the encoded video data representing the different spatial portions are contained in different source tracks, and the formatted video data further comprises a common track including one or more random access point indicators indicating the random access points of all source tracks.

[0185] 75. An apparatus for deriving spatially varying portions of a scene from video data, the video data being formatted in a file format, the apparatus being adapted to perform a method according to any of claims 57 to 74 or any of the preceding embodiments.

[0186] A computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method of claims 38 to 55 or 57 to 74 or any of the above embodiments.

[0187] A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to perform the method of claims 38-55 or 57-74 or any of the above embodiments.

[0188] A digital storage medium having stored thereon video data according to any one of claims 1 to 37.

[0189] These methods, apparatus, computer programs, computer readable media and digital storage media may have corresponding features as described with respect to the formatted video data.

[0190] In general, a method for creating video data for deriving spatially varying portions of a scene can include creating different types of indicators, such as one or more group indicators, one or more active source track indicators, one or more group bundle indicators, a level indicator, one or more partial random access point indicators, templates, such as configurable parameter sets and / or templates for SEI messages, and information, such as i) syntax elements identifying the width and height of the coded video data contained in each source track, and / or ii) syntax elements identifying projection mapping, transformation information, and / or guard band information related to the generation of region-wise packing (RWP) SEI messages, partial random access point information, etc. In this context, it may be necessary to determine the specific information signaled in the file format, the source track groups, and the number of two or more active source tracks in a group. In some cases, this determination can be performed through an interface that allows a user to input the required information or can be derived partially or completely from the coded video data (e.g., RAP information).

[0191] Similarly, a method for deriving spatially varying portions of a scene from video data may include reading different types of indicators, templates, and information and performing different tasks using the read data, which may include deriving spatially varying portions of a scene based thereon, and / or generating a section-specific video stream-specific parameter set or SEI message, and / or accessing encoded video data based on the read RAP information.

[0192] The embodiments of the present invention can be implemented in hardware or software depending on specific implementation requirements. The implementation can be performed using a digital storage medium storing electronically readable control signals, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM or FLASH memory, which cooperates (or can cooperate) with a programmable computer system to execute the respective methods. The digital storage medium can therefore be computer-readable.

[0193] In some embodiments, a programmable logic device (eg, a field programmable gate array) may be used to perform some or all of the functionality of the methods described herein.

[0194] In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein, although in general these methods are preferably performed by any hardware apparatus.

[0195] The apparatus described herein may be implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0196] The devices described herein, or any components of the devices described herein, may be implemented at least in part in hardware and / or software.

[0197] The methods described herein can be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0198] The methods described herein, or any components of the apparatus described herein, may be implemented at least in part by hardware and / or software.

[0199] The above-described embodiments merely illustrate the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be apparent to those skilled in the art. It is therefore the intention to be limited only by the scope of the appended claims, and not by the specific details shown in the description and interpretation of the embodiments herein.

Claims

1. Video data for deriving a spatially varying portion of a scene, the video data being formatted in a file format; a set of two or more source tracks, each source track containing encoded video data representing a spatial portion of a video showing the scene; the set of two or more source tracks includes a source track group, and the formatted video data further includes one or more group indicators indicating source tracks belonging to each source track group, and one or more active source track indicators indicating the number of two or more active source tracks in the source track group. Video data.

2. the formatted video data further includes one or more group bundling indicators that indicate two or more source track groups are bundled together; 2. Video data according to claim 1.

3. the one or more group indicators indicating the source tracks belonging to the respective source track groups and the one or more active source track indicators indicating the number of active source tracks in the source track groups are included in a first box of a file format separate from a second box of a file format including the one or more group bundle indicators indicating that the two or more source track groups are bundled together; 3. Video data according to claim 2.

4. The first box is a TrackGroup type box and the second box is a TrackReference type box.

4. Video data according to claim 3.

5. the one or more group indicators indicating the source tracks belonging to the respective source track groups, the one or more active source track indicators indicating the number of active source tracks in the source track groups, and the one or more group bundle indicators indicating two or more source track groups bundled together are contained in a single box in the file format; 3. Video data according to claim 2.

6. The single box is a Track Group type box or a Track Reference type box.

6. Video data according to claim 5.

7. the Track Group Type box is included in a source track, and / or the formatted video data further includes a collection track containing merging information for merging subsets of the set of two or more source tracks to generate a section-specific video data stream, and the Track Reference box is included in the collection track.

7. Video data according to claim 4 or 6.

8. each said source track group is indicated by a respective group ID, and the two or more source track groups bundled together are indicated by an indicator indicating the number of the two or more source track groups bundled together and an array of the respective group IDs; or each source track group is indicated as being a sub-group of source tracks by a respective sub-group ID, and the two or more sub-groups of source tracks bundled together are indicated by a common group ID, an indicator indicating the number of the two or more source track sub-groups bundled together, and an array of the respective sub-group IDs; Video data according to any one of claims 2 to 7.

9. the formatted video data further includes a level indicator indicating an encoding level of the source track group or of a bundle of two or more source track groups; Video data according to any one of claims 2 to 8.

10. a first group of source tracks including source tracks containing encoded video data at a first resolution and / or fidelity, and a second group of source tracks including source tracks containing encoded video data at a second resolution and / or encoding fidelity different from said first resolution and / or encoding fidelity; Video data according to any one of claims 1 to 9.

11. Video data for deriving a spatially varying portion of a scene, the video data being formatted in a file format; a set of two or more source tracks, each source track containing encoded video data representing a spatial portion of a video showing the scene; collected information including fusion information for merging subsets of the two or more sets of source tracks to generate a section-specific video data stream; the formatted video data further comprising templates of configurable parameter sets and / or SEI messages, the templates indicating one or more values ​​of the parameter sets or SEI messages that need to be adapted to generate a section-specific video stream-specific parameter set or SEI message. Video data.

12. the formatted video data includes a collection track containing the collection information; Video data according to claim 11.

13. the template includes an XML or JSON description of the encoding structure of the parameter set or the SEI message; Video data according to claim 11 or 12.

14. the formatted video data further includes an XML or JSON schema that provides rules for the creation of the template. Video data according to any one of claims 11 to 13.

15. the template includes concatenated encoding syntax elements of the parameter set or the SEI message, within the template values ​​that do not need to be matched are validly encoded, and the template further includes one or more gap indicators that indicate gaps within the template that should be filled with validly encoded values ​​that need to be matched. Video data according to claim 11 or 12.

16. the one or more gap indicators indicating the gap include an offset and a size of the gap within the template; 16. Video data according to claim 15.

17. generating the parameter sets or the SEI messages specific to the section-specific video stream includes, after filling the gaps in the template, performing emulation prevention on the concatenated coding syntax elements to generate a coded bitstream of the parameter sets or the SEI messages. Video data according to claim 15 or 16.

18. the template includes an encoded bitstream of the parameter set or the SEI message that includes an emulation prevention byte, and the one or more values ​​in the encoded bitstream that need to be matched are filled with validly encoded placeholder values. Video data according to claim 11 or 12.

19. the template further includes one or more placeholder value indicators indicating placeholder values ​​that need to be matched, the one or more placeholder value indicators indicating the placeholder values ​​including an offset and a size of the placeholder values ​​within the template; 20. Video data according to claim 18.

20. said template being included in an initialization segment of said collection track, preferably in a sample description box, more preferably in a sample entry box, most preferably in a decoder configuration record; Video data according to any one of claims 12 to 19.

21. the template is contained in a NAL unit, and the presence of the template in the NAL unit is indicated by the NAL unit type. Video data according to any one of claims 11 to 19.

22. the template is included in a sample entry box, preferably in a decoder configuration record, and the presence of the template in a NAL unit is indicated by the sample entry type and / or by one or more template indicators in the sample entry box.

22. Video data according to claim 21.

23. the template is included in an initialization segment of the collection track, preferably in a sample description box, more preferably in a sample entry box, and most preferably in a decoder configuration record, and the fusion information comprises media segments comprising references to the coded video data of the subset of the set of two or more source tracks, one or more of the media segments comprising i) a template for a configurable parameter set and / or SEI message, or ii) an indicator indicating that a parameter set and / or SEI message generated using a template is included in the media segment of the generated section-specific video data stream. Video data according to any one of claims 12 to 21.

24. the encoded video data contained by each source track is encoded using slices, and the generation of the section-specific video data stream does not require adapting values ​​of slice headers of the slices; Video data according to any one of claims 11 to 23.

25. the coded video data contained by each source track is i) coded using tiles, and the values ​​that need to be adapted are related to the tile structure, and / or ii) coded using bricks, and the values ​​that need to be adapted are related to the brick structure, and / or iii) coded using slices, and the values ​​that need to be adapted are related to the slice structure; Video data according to any one of claims 11 to 24.

26. the values ​​that need to be adapted represent the positions of tiles and / or bricks and / or slices within a picture of the video and / or the coded video data, 26. Video data according to claim 25.

27. the parameter set is a video parameter set (VPS), a sequence parameter set (SPS), or a picture parameter set (PPS), and / or the SEI message is a region-wise packing (RWP) SEI message; Video data according to any one of claims 11 to 26.

28. the set of two or more source tracks includes one or more boxes in the file format, each box including additional information for describing syntax elements that identify the characteristics of a source track, the additional information enabling the generation of the parameter set or the SEI message specific to the section-specific video stream without having to parse the coded video data. Video data according to any one of claims 11 to 27.

29. the additional information describes i) syntax elements that identify the width and height of the coded video data contained by each source track, and / or ii) syntax elements that identify projection mapping, transformation information and / or guard band information related to the generation of a Region Wise Packing (RWP) SEI message.

29. Video data according to claim 28.

30. the coded video data contained by each source track is coded using slices, and the additional information describes a syntax element that identifies a slice ID or other information used in the slice header to identify the slice structure.

29. Video data according to claim 28.

31. the additional information further includes a coding length and / or a coding mode of the respective syntax elements. Video data according to any one of claims 28 to 30.

32. The one or more boxes are extensions of a box of type TrackGroup. Video data according to any one of claims 28 to 31.

33. Video data for deriving a spatially varying portion of a scene, the video data being formatted in a file format; a set of one or more source tracks containing encoded video data representing spatial portions of a video showing the scene; the encoded video data is encoded using random access points, and the formatted video data further includes one or more random access point alignment indicators that indicate whether the random access points in the encoded video data for all spatial portions are aligned. Video data.

34. the formatted video data further includes one or more partial random access point indicators that indicate that an access unit of the video has random access points for a spatial portion of the video but not for the entire access unit.

34. Video data according to claim 33.

35. the formatted video data further includes partial random access point information representing the position and / or shape of the spatial portion having the random access point; 35. Video data according to claim 34.

36. different spatial parts of an access unit are contained in different NAL units, and the partial random access point information describes which NAL units are random access points for each spatial part, and the partial random access point information is contained in a box of the file format, preferably a subsample information box.

36. Video data according to claim 35.

37. the encoded video data representing the different spatial portions are included in different source tracks, and the formatted video data further includes a common track including one or more random access point indicators indicating the random access points of all source tracks.

34. Video data according to claim 33.

38. 1. A method of preparing video data for deriving spatially varying portions of a scene, the video data being formatted in a file format and comprising: a set of two or more source tracks, each source track containing encoded video data representing a spatial portion of a video showing the scene; the set of two or more source tracks comprises a source track group, and the formatted video data further comprises one or more group indicators indicating source tracks belonging to each source track group, and one or more active source track indicators indicating the number of two or more active source tracks in the source track group; The method comprises: determining the number of source track groups and two or more active source tracks within a group, creating the one or more group indicators and the one or more active source track indicators, and writing them to the formatted video data. method.

39. The formatted video data further includes one or more group bundling indicators that indicate two or more source track groups are bundled together, and the method further comprises: determining the two or more source track groups that are bundled together, creating the one or more bundle indicators, and writing them into the formatted video data; 39. The method of claim 38.

40. the one or more group indicators indicating the source tracks belonging to the respective source track groups, the one or more active source track indicators indicating the number of active source tracks in the source track groups, and the one or more group bundle indicators indicating two or more source track groups bundled together are contained in a single box in the file format; 40. The method of claim 39.

41. each source track group is indicated by a respective group ID, and the two or more source track groups that are bundled together are indicated by an indicator indicating the number of the two or more source track groups that are bundled together, and an array of the respective group IDs; or each source track group is indicated as being a sub-group of source tracks by a respective sub-group ID, and the two or more sub-groups of source tracks that are bundled together are indicated by a common group ID, an indicator indicating the number of the two or more sub-groups of source tracks that are bundled together, and an array of the respective sub-group IDs; 41. The method of claim 39 or 40.

42. The formatted video data further includes a level indicator indicating an encoding level of the source track group or a bundle of two or more source track groups, and the method further comprises: determining a source track group or said bundle of two or more source track groups, creating said level indicator and writing same to said formatted video data; 42. The method of any one of claims 39 to 41.

43. 1. A method of preparing video data for deriving spatially varying portions of a scene, the video data being formatted in a file format and comprising: a set of two or more source tracks, each source track containing encoded video data representing a spatial portion of a video showing the scene; collected information including fusion information for merging subsets of the two or more sets of source tracks to generate a section-specific video data stream; the collected information further includes templates of configurable parameter sets and / or SEI messages, the templates indicating one or more values ​​of the parameter sets or SEI messages that need to be adapted to generate a section-specific video stream-specific parameter set or SEI message; The method comprises: creating said template and writing it into said collection of said formatted video data; method.

44. the formatted video data includes a collection track containing the collection information; 44. The method of claim 43.

45. the template includes an encoded bitstream of the parameter set or the SEI message that includes an emulation prevention byte, and the one or more values ​​in the encoded bitstream that need to be matched are filled with validly encoded placeholder values.

45. The method of claim 43 or 44.

46. the template further includes one or more placeholder value indicators indicating placeholder values ​​that need to be matched, the one or more placeholder value indicators indicating the placeholder values ​​including an offset and a size of the placeholder values ​​within the template; 46. ​​The method of claim 45.

47. the template is included in an initialization segment of the collection track, preferably in a sample description box, more preferably in a sample entry box, and most preferably in a decoder configuration record, and the fusion information comprises media segments comprising references to the coded video data of the subset of the set of two or more source tracks, one or more of the media segments comprising i) a template for a configurable parameter set and / or SEI message, or ii) an indicator indicating that a parameter set and / or SEI message generated using a template is included in the media segment of the generated section-specific video data stream.

47. A method according to any one of claims 44 to 46.

48. the encoded video data contained by each source track is encoded using slices, and the generation of the section-specific video data stream does not require adapting values ​​of slice headers of the slices; 48. The method of any one of claims 43 to 47.

49. the set of two or more source tracks includes one or more boxes in the file format, each box including additional information for describing syntax elements that identify the characteristics of a source track, the additional information enabling the generation of the parameter set or the SEI message specific to the section-specific video stream without having to parse the coded video data.

49. The method of any of claims 43 to 48.

50. the coded video data contained by each source track is coded using slices, and the additional information describes a syntax element that identifies a slice ID or other information used in the slice header to identify the slice structure.

50. The method of claim 49.

51. the additional information further includes a coding length and / or a coding mode of the respective syntax elements.

51. The method of claim 49 or 50.

52. 1. A method of preparing video data for deriving spatially varying portions of a scene, the video data being formatted in a file format and comprising: a set of one or more source tracks containing encoded video data representing spatial portions of a video showing the scene; the encoded video data is encoded using random access points, and the formatted video data further includes one or more random access point alignment indicators that indicate whether the random access points in the encoded video data for all spatial portions are aligned; The method comprises: creating the one or more random access point alignment indicators and writing them into the formatted video data; method.

53. The formatted video data further includes one or more partial random access point indicators that indicate that an access unit of the video has random access points for a spatial portion of the video but not for the entire access unit, and the method further comprises: creating the one or more partial random access point indicators and writing them into the formatted video data.

53. The method of claim 52.

54. The formatted video data further includes partial random access point information representing the position and / or shape of the spatial portion having the random access point, and the method further comprises: creating the partial random access point information and writing it to the formatted video data; 54. The method of claim 53.

55. different spatial parts of an access unit are contained in different NAL units, and the partial random access point information describes which NAL units are random access points for each spatial part, and the partial random access point information is contained in a box of the file format, preferably a subsample information box.

55. The method of claim 54.

56. 56. An apparatus for producing video data for deriving a spatially varying portion of a scene, the video data being formatted in a file format, the apparatus being adapted to perform a method according to any of claims 38 to 55. Device.

57. 1. A method of deriving spatially varying portions of a scene from video data, the video data being formatted in a file format, the method comprising: a set of two or more source tracks, each source track containing encoded video data representing a spatial portion of a video showing the scene; the set of two or more source tracks comprises a source track group, and the formatted video data further comprises one or more group indicators indicating source tracks belonging to each source track group, and one or more active source track indicators indicating the number of two or more active source tracks in the source track group; The method comprises: reading from the formatted video data the one or more group indicators, the one or more active source track indicators, and the encoded video data from the indicated number of two or more active source track groups, and deriving spatially varying portions of the scene based thereon; method.

58. The formatted video data further includes one or more group bundling indicators that indicate two or more source track groups are bundled together, and the method further comprises: reading from the formatted video data the one or more bundle indicators and the two or more source track groups bundled together, and deriving spatially varying portions of the scene based thereon; 58. The method of claim 57.

59. the one or more group indicators indicating the source tracks belonging to the respective source track groups, the one or more active source track indicators indicating the number of active source tracks in the source track groups, and the one or more group bundle indicators indicating two or more source track groups bundled together are contained in a single box in the file format; 59. The method of claim 58.

60. each said source track group is indicated by a respective group ID, and the two or more source track groups bundled together are indicated by an indicator indicating the number of the two or more source track groups bundled together and an array of the respective group IDs; or each source track group is indicated as being a sub-group of source tracks by a respective sub-group ID, and the two or more sub-groups of source tracks bundled together are indicated by a common group ID, an indicator indicating the number of the two or more source track sub-groups bundled together, and an array of the respective sub-group IDs; 60. The method of claim 58 or 59.

61. The formatted video data further includes a level indicator indicating an encoding level of the source track group or a bundle of two or more source track groups, and the method further comprises: reading the level indicator and the source track group or the bundle of two or more source track groups from the formatted video data and deriving spatially varying portions of the scene based thereon; 61. The method of any of claims 58 to 60.

62. 1. A method of deriving spatially varying portions of a scene from video data, the video data being formatted in a file format, the method comprising: a set of two or more source tracks, each source track containing encoded video data representing a spatial portion of a video showing the scene; collected information including fusion information for merging subsets of the two or more sets of source tracks to generate a section-specific video data stream; the collected information further includes templates of configurable parameter sets and / or SEI messages, the templates indicating one or more values ​​of the parameter sets or SEI messages that need to be adapted to generate a section-specific video stream-specific parameter set or SEI message; The method comprises: reading the template from the collection of formatted video data and adapting the one or more values ​​of the parameter set or SEI message indicated by the template to generate the parameter set or SEI message specific to the section-specific video stream. method.

63. the formatted video data includes a collection track containing the collection information; 63. The method of claim 62.

64. the template includes an encoded bitstream of the parameter set or the SEI message that includes an emulation prevention byte, and the one or more values ​​in the encoded bitstream that need to be matched are filled with validly encoded placeholder values.

64. The method of claim 62 or 63.

65. the template further includes one or more placeholder value indicators indicating placeholder values ​​that need to be matched, the one or more placeholder value indicators indicating the placeholder values ​​including an offset and a size of the placeholder values ​​within the template; 65. The method of claim 64.

66. the template is included in an initialization segment of the collection track, preferably in a sample description box, more preferably in a sample entry box, and most preferably in a decoder configuration record, and the fusion information comprises media segments comprising references to the coded video data of the subset of the set of two or more source tracks, one or more of the media segments comprising i) a template for a configurable parameter set and / or SEI message, or ii) an indicator indicating that a parameter set and / or SEI message generated using a template is included in the media segment of the generated section-specific video data stream.

66. The method of claim 64 or 65.

67. the encoded video data contained by each source track is encoded using slices, and the generation of the section-specific video data stream does not require adapting values ​​of slice headers of the slices; 67. The method of any of claims 63 to 66.

68. the set of two or more source tracks includes one or more boxes in the file format, each box including additional information for describing syntax elements that identify the characteristics of a source track, the additional information enabling the generation of the parameter set or the SEI message specific to the section-specific video stream without having to parse the coded video data.

68. The method of any of claims 63 to 67.

69. the coded video data contained by each source track is coded using slices, and the additional information describes a syntax element that identifies a slice ID or other information used in the slice header to identify the slice structure.

69. The method of claim 68.

70. the additional information further includes a coding length and / or a coding mode of the respective syntax elements.

70. The method of claim 68 or 69.

71. 1. A method of deriving spatially varying portions of a scene from video data, the video data being formatted in a file format, the method comprising: a set of one or more source tracks containing encoded video data representing spatial portions of a video showing the scene; the encoded video data is encoded using random access points, and the formatted video data further includes one or more random access point alignment indicators that indicate whether the random access points in the encoded video data for all spatial portions are aligned; The method comprises: reading the one or more random access point indicators from the formatted video data and accessing the encoded video data based thereon. method.

72. The formatted video data further includes one or more partial random access point indicators that indicate that an access unit of the video has random access points for a spatial portion of the video but not for the entire access unit, and the method further comprises: reading the one or more partial random access point indicators from the formatted video data and accessing the encoded video data based thereon.

72. The method of claim 71.

73. The formatted video data further includes partial random access point information representing the position and / or shape of the spatial portion having the random access point, and the method further comprises: reading the partial random access point information and accessing the encoded video data based thereon; 73. The method of claim 72.

74. different spatial parts of an access unit are contained in different NAL units, and the partial random access point information describes which NAL units are random access points for each spatial part, and the partial random access point information is contained in a box of the file format, preferably a subsample information box.

74. The method of claim 73.

75. 75. An apparatus for deriving spatially varying parts of a scene from video data, the video data being formatted in a file format, the apparatus being adapted to perform a method according to any of claims 57 to 74. Device.

76. comprising instructions that, when executed by a computer, cause the computer to perform the method of claims 38 to 55 or 57 to 74, Computer program.

77. comprising instructions that, when executed by a computer, cause the computer to perform the method of claims 38 to 55 or 57 to 74, Computer-readable medium.

78. 38. The video data according to any one of claims 1 to 37, Digital storage media.

Images