Method and device for generating a file, method and device for processing a file

Abstract

To provide a method, a device, and a program for extracting a region of a target in a still image stored using an HEVC still image file format.SOLUTION: In a method of encapsulating a coded bitstream that represents one or more images, the encapsulated bitstream includes a data part and a metadata part. The method includes: a step of including image item information identifying a portion of the data part representing an image or a sub-image of a single image; a step of including image description information including a group of parameters including display parameters and / or transformation operators for one or more images, the image description information being stored in the metadata part; and a step of outputting the bitstream, together with the included information, as an encapsulated data file.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention provides a method for storing data such as still images, bursts of still images, or video data in a media container with descriptive metadata. item Such metadata relates to the storage of item Provides easy access to [Background technology]

[0002] Some of the approaches described in this section may be approaches that have been pursued, but are not necessarily approaches that have been previously conceived or pursued. Thus, the approaches described in this section do not necessarily predate the claims in this application, and are not admitted to be prior art by inclusion in this section.

[0003] The HEVC standard defines a profile for coding of still images and describes specific tools for compressing single still images or bursts of still images. An extension of the ISO Base Media File Format (ISOBMFF) for this type of image data is proposed for inclusion in Part 12 of the ISO / IEC 23009 standard under the name "Image File Format". The standard covers two modes of storage corresponding to different usage scenarios: - storage of a sequence of images corresponding to timings optionally used by the decoder, where the images can depend on other images; and - Single image storage, a collection of individually coded images.

[0004] In the first case, the encapsulation is close to the encapsulation of video tracks in the ISO Base Media File Format (see document <<Information technology -- Coding of audiovisual objects -- Part 12: ISO Base Media File Format >>, ISO / IEC 14496-12:2008, 3rd edition, October 2008), and the same tools and concepts are used for description, such as "trak" boxes and sample groupings. A "trak" box is a file format box that contains sub-boxes for describing a track, in other words a timed sequence of related samples.

[0005] In the second case, a set of ISOBMFF boxes is used: "meta" boxes. These boxes and their hierarchy offer fewer descriptive tools than "track" boxes and relate to "information items" or "items" instead of related samples.

[0006] The image file format can be used to display multimedia files locally or to stream multimedia presentations. HEVC still images have many uses which raise many questions.

[0007] Image bursts are one application. An image burst is a sequence of still pictures captured by a camera and stored as a single representation (where many picture items refer to a block of data). Users may want to do some kind of manipulation to these pictures, such as selecting them as thumbnails or cover pages, or applying effects to them.

[0008] Thus, descriptive metadata is needed to identify the list of pictures in a block of data along with their corresponding bytes.

[0009] Computational photography is another application, where a user has access to different resolution levels of the same picture (different exposures, different focus, etc.) These different resolution levels need to be stored as metadata so that one can be selected and the corresponding piece of data can be found and extracted for processing (rendering, editing, sending, etc.).

[0010] As the resolution level of pictures increases relative to their size, there is a need to have sufficient descriptions so that only some spatial portions of these large pictures can be easily identified and extracted.

[0011] Another class of applications involves accessing specific pictures of a video sequence, for example for video summarization, forensic images in video surveillance data, and the like.

[0012] For these types of applications, in addition to the compressed video data and video tracker metadata, there is a need for image metadata that allows easy access to the key images.

[0013] Furthermore, professional cameras are reaching high spatial resolution levels. Video or images with 4K2K resolution levels are now commonplace. Even 8K4K video or images are common today. In parallel, video is increasingly being played on mobile and connected devices with video streaming capabilities. Thus, dividing video into tiles becomes important when a user of a mobile device wants to view or focus on a subportion of the video while maintaining or even improving quality. By using tiles, a user can interactively request a spatial subportion of the video.

[0014] Thus, there is a need to describe these spatial sub-portions of a video in a compact form in a file format so that they can be accessed without further processing beyond simply parsing the metadata boxes, and access to the spatial sub-portions is also of interest to users for the images corresponding to the video so described.

[0015] The ISO / IEC 23009 standard addresses two ways of encapsulating still images into a recently discussed file format.

[0016] One approach is based on the concept of a timed sequence of related samples of "track" boxes and related description tools, another is based on "meta" boxes based on information items instead of samples, providing less description tools, especially for describing regions of interest and supporting tiling.

[0017] Thus, there is a need to provide support for tiling in new image file formats.

[0018] The use of tiles, especially during compression, is well known in the art. For their indexing in ISO Base Media File Formats, tiling descriptors also exist in the revised draft of Part 15 of the ISO / IEC 14496 standard "Carriage of NAL unit structured video in the ISO Base Media File Format."

[0019] However, these descriptors rely on "track" boxes and sample grouping tools and cannot be used in still image file formats when using a "meta"-based approach, without which selecting and extracting tiles from coded pictures stored in this file format becomes cumbersome.

[0020] FIG. 1 shows the description of a still image coded using tiles in the "meta" box (100) of the ISO Base Media File Format, as disclosed in MPEG contribution m32254.

[0021] In addition to the individual information items for each tile picture (102, 103, 104, and 105), an information item for the overall picture 101 is defined. These information items are stored in a box called "ItemInfoBox" (iinf). A box called "ItemReferenceBox" (106) from the ISO BMFF standard is used to indicate the existence of a "tile" relationship (107) between the information item for the overall picture and the four information items (108) corresponding to the tile pictures. A box called "ItemLocationBox" (109) uses the identifier of each information item to specify the byte range(s) in the coded data (110) that represent each information item. Another box called "ItemReferenceBox" (112) is used to associate EXIF metadata (111) with the information item for the overall picture (101), and a corresponding data block (111) is generated in the media data box (110). Also, a further information item (113) is generated to identify the EXIF metadata. Summary of the Invention [Problem to be solved by the invention]

[0022] Even if the whole picture and its tiles are introduced as information items, no tiling information is provided here. Furthermore, when associating additional metadata with an information item (such as EXIF), no data blocks are generated that are referenced using additional "ItemReferenceBoxes".

[0023] The reuse of tiling information from EXIF and the mechanisms defined in the Still Image File Format draft would not allow for the description of irregular grids using existing EXIF tags.

[0024] Thus, there is a need for further improvements in file formats for still images, particularly HEVC still images, and specifically for methods for extracting regions of interest in still images stored using this file format.

[0025] The present invention is within the above context. [Means for solving the problem]

[0026] According to a first aspect of the present invention, there is provided a method of encapsulating a coded bitstream representing one or more images, the method comprising: - providing tile description information including spatial parameters for dividing an image area into one or more tiles; - providing tile picture item information identifying portions of the bitstream representing tiles of a single image; - providing reference information linking the tile picture items to the tile description information; - outputting the bitstream together with the provided information as an encapsulated data file; Includes.

[0027] This output can be according to a defined standard and is readable and decodable.

[0028] The method according to the first aspect allows for easy identification, selection and extraction of tiles from, for example, ultra-high resolution level images (4K2K, 8K4K...) by parsing syntax elements without tedious calculations.

[0029] These description tools of the ISO Base Media File Format Metadata Box can be extended, in particular to allow tile descriptions to be associated with information items.

[0030] Portions of the "meta" box hierarchy can be extended to provide additional description tools and in particular to support tile-based access within still images.

[0031] The method according to the first aspect allows for easy extraction of HEVC tile-based regions of interest from an encoded HEVC still image.

[0032] Embodiments of the present invention provide tile description support and tile access for still images encoded according to the HEVC standard.

[0033] This allows regions of interest features to be kept available for video tracking against still images. In general, portions of a still picture that correspond to user-defined regions of interest can be easily identified and extracted for rendering or transmission to a media player.

[0034] For example, the encapsulated coded bitstream also includes information identifying a timed portion of the data stream that corresponds to a video sequence.

[0035] Thus, for some still images that are part of this video, dual indexing can be provided on a single piece of data that provides the same access facilities to the video.

[0036] For example, the tile description information includes a set of spatial parameters for each tile picture item.

[0037] For example, the tile description information includes spatial parameters that are common to multiple tile picture items.

[0038] For example, the tile description information is embedded in the bitstream.

[0039] For example, the tile description information is provided as metadata.

[0040] For example, the reference information includes the type of reference and additional descriptive metadata including the tile description information.

[0041] For example, the reference information includes a type of reference and reference parameters for the tile description information.

[0042] The method may further comprise the step of providing a metadata item for referencing the tile description information in the bitstream.

[0043] For example, tile picture items may be grouped and provided with reference information to link the group of tile picture items to the tile description information.

[0044] For example, all references linking a metadata item to another item are contained within a single reference box in the encapsulated data file.

[0045] For example, all relationships from one item of any kind are stored in a single item information descriptor.

[0046] For example, said output is performed by a server module for adaptive streaming.

[0047] For example, the output is made for storage in a memory.

[0048] For example, the output is directed to a display module for display.

[0049] For example, said output is performed by a communication module for transmission.

[0050] For example, the encapsulated data file corresponds to a standardized file format.

[0051] For example, the encapsulated data file is decodable and playable.

[0052] According to a second aspect of the present invention, there is provided a method of processing an encapsulated data file comprising an encoded bitstream corresponding to one or more images and information containing tile description information comprising spatial parameters for dividing an area of the image into one or more tiles, the method comprising: - selecting an image region of interest; - identifying a tile from said tile description information that corresponds to a selected area of interest; - selecting one or more tile picture items linked to the identified tiles, each tile picture item identifying a portion of the bitstream representing one tile of a single image; - extracting the portion of the bitstream identified by the selected tile picture item(s); - outputting the extracted bitstream portion; Includes.

[0053] For example, said output is performed by a server module for adaptive streaming.

[0054] For example, the output is made for storage in a memory.

[0055] For example, the output is directed to a display module for display.

[0056] For example, said output is performed by a communication module for transmission.

[0057] For example, the encapsulated data file corresponds to a standardized file format.

[0058] For example, the encapsulated data file is decodable and playable.

[0059] According to a third aspect of the present invention, there is provided a method of processing image data representing at least one image for encapsulation in an encapsulation file, the method comprising: - obtaining a spatial subdivision of said at least one image into a plurality of image portions; - determining at least one portion identification data identifying a data portion within said image data representing an image portion of said plurality; - said image data is at least subdivision description data describing said subdivision of said at least one image, said partial identification data, and Reference data linking said subdivision description data with said part identification data, and encapsulating the data in the encapsulation file; Includes.

[0060] For example, the image data represents a plurality of images of a video sequence, and the method further includes determining at least one time identification data identifying a data portion within the image data that represents a time portion of the video sequence, the image data being encapsulated together with the time identification data.

[0061] For example, a plurality of portion identification data may be determined to each represent the same image portion of an image of the time portion of the video sequence.

[0062] For example, at least the subdivision description data is encapsulated as metadata for the image data.

[0063] For example, the spatial subdivision is embedded in a bitstream containing the image data.

[0064] For example, for each image portion, respective portion identification data is determined.

[0065] For example, common portion identification data for multiple image portions is determined.

[0066] The method may further include outputting, by a server device, the encapsulated file into a bitstream for adaptive streaming.

[0067] The method may further include outputting the encapsulated file into a bitstream for transmission to a display device for displaying the image data.

[0068] The method may further include outputting the encapsulated file into a bitstream for transmission to a client device.

[0069] The method may further include storing the encapsulated file in a storage device.

[0070] For example, the reference data includes the type of reference and additional descriptive metadata including said subdivision descriptive data.

[0071] For example, the reference data includes a type of reference and a reference parameter for said subdivision description data.

[0072] For example, the subdivision description data is referenced in a metadata item.

[0073] For example, partial identification data may be grouped and reference data may link groups of partial identification data to said partial identification data.

[0074] For example, the encapsulation file contains a single reference box that contains all the reference data for the image data.

[0075] For example, the encapsulation file includes a description that includes a representation of the relationship between the subdivision description data, part identification data, and reference data.

[0076] According to a fourth aspect of the present invention, there is provided a method for processing an encapsulated file, the file comprising: - image data representing at least one image; - subdivision description data representing a spatial subdivision of said at least one image into a plurality of image portions; - at least one portion identification data identifying a data portion within the image data representing an image portion of the plurality; - reference information linking said subdivision description data with said partial information; Including, The method comprises: - determining a region of interest in said at least one image; - determining at least one image portion that belongs to the region of interest based on the subdivision description data; - accessing, based on said reference data, at least one portion identification data that identifies a data portion in said image data that represents said at least one image portion belonging to said region of interest; - extracting the data portion within the image data; Includes.

[0077] For example, the image data includes a plurality of images of a video sequence, the encapsulation file further includes at least one time identification data identifying a data portion within the image data representing a time portion of the video sequence, an area of interest for an image of the time portion of the video sequence is determined, and a data portion corresponding to the area of interest in the plurality of images of the time portion of the video sequence is extracted.

[0078] For example, a plurality of portion identification data each represent the same image portion of an image of the time portion of the video sequence.

[0079] For example, at least the sub-division data is encapsulated as metadata for the image data.

[0080] For example, for each image portion, respective portion identification data is determined.

[0081] For example, common portion identification data is determined for multiple image portions.

[0082] The method may further include receiving, by a server device, the encapsulated file as an adaptively streamed bitstream.

[0083] The method may further include displaying the region of interest.

[0084] For example, the reference data includes the type of reference and further descriptive metadata that encompasses said subdivision descriptive data.

[0085] For example, the reference data includes a type of reference and a reference parameter for said subdivision description data.

[0086] For example, the subdivision description data is referenced in a metadata item.

[0087] For example, partial identification data may be grouped and reference data may link groups of partial identification data to said partial identification data.

[0088] For example, the encapsulation file contains a single reference box that contains all the reference data for the image data.

[0089] For example, the encapsulation file includes a description that includes a representation of the relationship between the subdivision description data, part identification data, and reference data.

[0090] According to a fifth aspect of the present invention, there is provided a device configured to implement a method according to the first aspect above.

[0091] This device is - a processing unit configured to provide tile description information including spatial parameters for dividing an image area into one or more tiles, to provide tile picture item information identifying portions of a bitstream representing tiles of a single image, and to provide reference information linking said tile picture items to said tile description information; a communication unit configured to output said bitstream together with said provided information as an encapsulated data file; It is good to include.

[0092] According to a sixth aspect of the present invention there is provided a device configured to implement a method according to the second aspect above.

[0093] The device may be configured to process an encapsulated data file including an encoded bitstream corresponding to one or more images and information including tile description information including spatial parameters for dividing an image area into one or more tiles. - a processing unit configured to select an image region of interest, identify from said tile description information tiles corresponding to the selected area of interest, select one or more tile picture items linked to said identified tiles, each tile picture item identifying a bitstream portion representing a tile of a single image, and extract the bitstream portion identified by the selected tile picture item(s); a communication unit configured to output the extracted bitstream portions.

[0094] According to a seventh aspect of the present invention there is provided a device configured to implement a method according to the third aspect above.

[0095] The device may be configured to process image data representing at least one image for encapsulation into an encapsulation file, the device obtaining a spatial subdivision of the at least one image into a plurality of image portions, determining at least one portion identification data in the image data identifying a data portion representing one of the plurality of image portions, and converting the image data into at least: - subdivision description data describing said subdivision of said at least one image; - the partial identification data; - a processing unit adapted to encapsulate in said encapsulation file said subdivision description data together with reference data linking said part identification data.

[0096] According to an eighth aspect of the present invention there is provided a device configured to implement a method according to the fourth aspect above.

[0097] This device is - image data representing at least one image; - subdivision description data representing a spatial subdivision of said at least one image into a plurality of image portions; - at least one portion identification data identifying a data portion within the image data representing an image portion of the plurality; - reference data linking said subdivision description data with said partial information; The system may be configured to process encapsulated files containing:

[0098] The device may also include a processing unit configured to determine a region of interest in the at least one image, determine at least one image portion belonging to the region of interest based on the subdivision description data, access at least one portion identification data identifying a data portion in the image data representing the at least one image portion belonging to the region of interest based on the reference data, and extract the data portion in the image data.

[0099] According to a ninth aspect of the present invention, - a first device according to any one of the fifth to seventh aspects; - a second device according to any one of the sixth to eighth aspects for processing a file from the first device; A system is provided that includes:

[0100] According to a tenth aspect of the present invention there is provided a computer program and computer program product comprising instructions for implementing the methods according to the first, second, third and / or fourth aspect(s) of the present invention when loaded and executed on computer means of a programmable device.

[0101] According to an eleventh aspect of the present invention, there is provided a method for generating a file containing an encoded bitstream, the method comprising the steps of: One or more items of one or more types Data corresponding to and generating a file in which a data portion stores an encoded bitstream including the one or more items, and a metadata portion stores identification information for identifying each of the one or more items and one or more properties, Output the generated file, The metadata portion includes association information indicating an association between one or more properties and identification information of an item corresponding to the one or more properties. and included in the "meta" box in ISOBMFF. A method is provided, characterized in that:

[0102] In one embodiment, each parameter included in the image description information is: - Type information, and / or - an identifier used to link image item information to said parameters, associated with additional data including

[0103] In one embodiment, the metadata portion is contained in the "meta" data box of the ISOBMFF.

[0104] In some embodiments, the additional data is a header.

[0105] In some embodiments, the additional data is a virtual item.

[0106] In another embodiment, each transformation operator included in the image description information is associated with additional data including an identifier used to link the transformed item to said transformation operator.

[0107] In one embodiment, a box stored in the metadata portion contains at least one transformation operator.

[0108] In one embodiment, the data portion of the encapsulated bitstream includes transformed items associated with one or more transformation operators, and the metadata portion includes: - information identifying the original image to which the transformation operator was applied; - information for localizing the converted item in the data part; Further includes:

[0109] In one embodiment, the transformed item comprises at least one transformation index, ie an index that allows to identify one of the transformation operators in the metadata portion.

[0110] According to a twelfth aspect of the present invention, there is provided a method for processing a file containing an encoded bitstream, the method comprising the steps of: One or more items of one or more types Data corresponding to a file in which a data portion stores an encoded bitstream including the one or more items, and a metadata portion stores identification information for identifying each of the one or more items and one or more properties; outputting data corresponding to said one or more items; The metadata portion includes association information indicating an association between one or more properties and identification information of an item corresponding to the one or more properties. 、 The metadata part is contained in the "meta" box in ISOBMFF. The method is characterized in that:

[0111] In one embodiment, the method further comprises, before the identifying step, the step of reading additional data contained in the set of parameters, the additional data being: - Type information, and / or - an identifier used to link image or sub-image item information to said parameters, Includes.

[0112] In one embodiment, the metadata portion is contained in the ISOBMF "meta" data box.

[0113] In some embodiments, the additional data is a header.

[0114] In some embodiments, the additional data is a virtual item.

[0115] In another embodiment, each transformation operator included in the image description information is associated with additional data including an identifier used to link the transformed item to said transformation operator.

[0116] In one embodiment, a box stored in the metadata portion contains at least one transformation operator.

[0117] In one embodiment, the data portion of the encapsulated bitstream includes transformed items associated with one or more transformation operators, and the metadata portion includes: - information identifying the original image to which the transformation operator was applied; - information for localizing the converted item in the data part; Further includes:

[0118] In some embodiments, the transformed item includes at least one transformation index, which allows identifying one of the transformation operators in the metadata portion.

[0119] According to a thirteenth aspect of the present invention, there is provided a server device for encapsulating coded bitstreams representing one or more images, the server device being configured to implement the encapsulation method according to the eleventh aspect of the present invention.

[0120] According to a fourteenth aspect of the present invention, there is provided a client device for encapsulating coded bitstreams representing one or more images, the client device being configured to implement a processing method according to the twelfth aspect of the present invention.

[0121] According to a fifteenth aspect of the present invention there is provided a computer program and a computer program product comprising instructions for implementing the methods according to the eleventh and twelfth aspects of the present invention when loaded and executed on computer means of a programmable device.

[0122] Other features and advantages of the present invention will become apparent from the following description of non-limiting exemplary embodiments, which proceeds with reference to the accompanying drawings. [Brief explanation of the drawings]

[0123] [Figure 1] 1 shows a description of a still image coded using tiles in the "meta" box of the ISO Base Media File Format, as disclosed in MPEG contribution m32254. [Figure 2] 1 shows an example of a tiled video. [Figure 3] 1 illustrates various tile / slice configurations in HEVC. [Figure 4] Shows tile encapsulation in ISO Base Media File Format using "track" boxes. [Figure 5] This shows the standard metadata for describing information items in the ISOBMFF "meta" box. [Figure 6] 1 shows an exemplary extension to an information item description. [Figure 7] It provides a mechanism for referencing between information items. [Figure 8] 1 illustrates the context for the implementation of embodiments of the present invention. [Figure 9] 1 is a schematic block diagram of a computing device for implementing one or more embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0124] Hereinafter, various embodiments of the present invention will be described.

[0125] To better understand the technical context, video tiling will be explained with reference to FIG. 2, which shows a video (200) with successive temporal frames. Each frame (201) is divided into eight (here rectangular) parts called "tiles" T1 to T8. The number and shape of these tiles can vary. In the following, the tiling will be considered to be the same regardless of the index of the video frames.

[0126] The result of this tiling is eight independent sub-pictures (202), which represent sections of the overall picture. Each independent sub-picture can be coded as an independent bitstream, for example according to the AVC or HEVC standard, or it can be part of one single picture bitstream, for example a tile in the HEVC standard or a slice in the AVC standard.

[0127] The HEVC standard defines various spatial subdivisions of a picture: tiles, slices, and slice segments. These various subdivisions (or partitions) are introduced for different purposes: slices are related to streaming streams, while tiles and slice segments are defined for parallel processing.

[0128] A tile defines a rectangular region of a picture that encompasses an integer number of Coding Tree Units (CTUs). Figure 3 shows the tiling of an image (300), defined by row and column boundaries (301, 302). This makes tiles good candidates for describing regions of interest in terms of location and size. However, the bitstream organization of the HEVC standard is based on slices (as in the AVC standard) rather than on syntax and its encapsulation into Network Abstract Layer (NAL) units.

[0129] According to the HEVC standard, a slice is a set of slice segments, where at least the first slice segment is an independent slice segment, and others, if present, are dependent slice segments. A slice segment contains an integer number of consecutive CTUs (in raster scan order), which do not necessarily have a rectangular shape (and thus are less suitable than tiles for representing regions of interest). It is coded in the HEVC bitstream as a header called "slice_segment_header," followed by data called "slice_segment_data." Independent and dependent slice segments differ in their headers; dependent slice segments have shorter headers because they reuse information from the headers of independent slice segments. Both independent and dependent slice segments contain a list of entry points into either tiles or entropy decoding synchronization points in the bitstream.

[0130] 3 shows images 310 and 320 with different configurations of slices, slice segments, and tiles. These configurations differ from that of image 300, in which one tile has one slice (containing only one independent slice segment). Image 310 is partitioned into two vertical tiles (311, 312) and one slice (with five slice segments). Image 320 is divided into two tiles (321, 322), with left tile 321 having two slices (each with two slice segments) and right tile 322 having one slice (with two slice segments). The HEVC standard defines organization rules between tiles and slice segments, which can be summarized as follows (one or both conditions must be met): - All CTUs in a slice segment belong to the same tile, and - All CTUs in a tile belong to the same slice segment.

[0131] To accommodate support and transport in the region of interest, the configuration 300, where one tile encompasses one slice with one independent segment, is preferred, although the encapsulation solution will also work with other configurations 310 or 320.

[0132] The tiles, which are appropriate supports for the region of interest, are actually packed into NAL units for transport over the network and are aggregated to form access units (coded pictures or samples at the file format level). According to the HEVC standard, the type of NAL unit is specified in the NAL unit header. For the NAL unit type "coded slice segment", the slice_segment_header indicates the address of the first coding tree block in the slice segment via the "slice_segment_address" syntax element. The tiling information is provided in the PPS (Picture Parameter Set) NAL unit. Thus, the relationship between slice segments and tiles can be derived from these parameters.

[0133] By definition, spatial prediction is reset on tile boundaries. However, this does not prevent a tile from using temporal predictors from a different tile in the reference frame(s). During encoding, to construct independent tiles, motion vectors for prediction units within a tile are constrained to remain within the co-located tile in the reference frame(s). Furthermore, in-loop filters (deblocking and SAO) need to be deactivated on tile boundaries to avoid introducing error drift when decoding only one tile. This control of in-loop filters is already available in the HEVC standard and is set in the slice segment header using a flag called "loop_filter_across_tiles_enabled_flag." By explicitly setting this flag to 0, pixels at tile boundaries do not depend on pixels located on the boundaries of adjacent tiles. When these two conditions for motion vectors and in-loop filters are met, the tiles are said to be "independently decodable" or "independent."

[0134] If a video sequence is coded as a set of independent tiles, it is possible to decode the video using tile-based decoding from one frame to another without risking loss of reference data or propagation of reconstruction errors. This configuration makes it possible to reconstruct only the spatial portion of the original video that corresponds, for example, to a region of interest.

[0135] In what follows, we consider independent tiles.

[0136] Referring to Figure 4, the encapsulation of tiles into the ISOBMFF file format is shown. For example, each tile is encapsulated in its own track. Setup and initialization information common to all tiles is encapsulated in a specific track, for example, called the "tile base track." The entire video is then encapsulated as a composite of all these tracks, i.e., the tile's base track and the set of tile tracks.

[0137] Figure 4 shows an exemplary encapsulation. One way to encapsulate tiled video according to the ISOBMFF standard is to divide each tile into its own track, encapsulate setup and initialization information common to all tiles in a specific track, e.g., called the "tile base track," and encapsulate the entire video as a composite of all these tracks, i.e., a set of base track plus tile track. This encapsulation is thus referred to as "multi-track tile encapsulation." An example of multi-track tile encapsulation is presented in Figure 4.

[0138] Box 401 represents the main ISOBMFF box "moov", which contains the complete list of tracks together with their identifiers. For example, boxes 411-414 represent tile tracks (four tiles in this example), and box 420 represents a tile-based track. Additional tracks, such as audio or text tracks, can be used and encapsulated within the same file; however, for the sake of simplicity, such additional tracks are not described here.

[0139] As shown in FIG. 4, tile data is divided into independently addressable tracks so that any combination of tile tracks can be easily reconstructed for decoding and display from the tile base tracks that reference these tile tracks. The tile base tracks are also sometimes referred to as "composite tracks" or "reference tracks" because they allow any combination of tiles, such as one, many, or all tiles. The tile base track 420 contains information common to all tile tracks and a list of samples 450 (only the first one is shown in FIG. 4) in the "mdat" box. Each sample 450 in the tile base track 420 is constructed by reference to each tile track through the use of extractors (each one of 451-454 represents one extractor for each tile). Each tile track 411-414 represents a spatial portion of the entire image or an entire frame image. Tile descriptions (e.g., position, size, bandwidth, etc.) are stored in the track header box (not shown) of each tile track 411-414. The tile base track and each tile track are cross-referenced using a "TrackReferenceBox" in each track (405). Each tile track 411-414 references the tile base track 420 as a "tbas" track ("tbas" is a specific code indicating the coding dependency from each tile track to the tile base track when finding the parameter "HEVCDecoderConfigurationRecord" that enables the setup of the video decoder that will process the elementary stream obtained from parsing the file format). Conversely, to enable the reconstruction of the entire video, the tile base track 420 indicates a dependency of type "scal" to each tile track (405). This indicates the coding dependency and reflects the definition of the tile base track's samples 450 as extractors for the tile track's data. These extractors are specific extractors that can handle missing data during parsing.In FIG. 4, to provide a streamable version of the file, each track is decomposed into media segments (431-434 for tile tracks and 460 for tile-based tracks). Each media segment contains one or more movie fragments, indicated by a "moof" box plus data. For tile tracks, the data portion corresponds to a spatial subportion of the video, while for tile-based tracks, it contains a set of parameters, SEI messages, if present, and a list of extractors. The "moov" box 401 would correspond to an initialization segment in the case of a streaming application. While FIG. 4 shows only one segment, the tracks can be decomposed into any number of segments; the only constraint is that the segments for tile tracks and tile-based tracks follow the same temporal decomposition (i.e., they are temporally aligned), allowing switching from the entire video to a tile or set of tiles. For simplicity, the granularity of this temporal decomposition is not described here.

[0140] This file format has descriptive metadata that describes the relationships between tracks (e.g., "VisualSampleGroupEntries," or track reference types in "tref" boxes) so that data corresponding to a single tile, a combination of tiles, or all tiles can be easily identified by parsing the descriptive metadata.

[0141] In the following, still images are described at the same level. In this way, user selection of any tile, combination of tiles, or all tiles of a picture is facilitated for identification and extraction. When pictures are mixed into video data, their description is provided in parallel with the descriptive metadata for the video. Thus, for the same data set, there is an additional indexing layer for pictures (in addition to the indexing layers for video and audio).

[0142] In still image file formats using the "meta" box, pictures with associated information are described as information items. As shown in Figure 5, these information items are listed in a dedicated sub-box of the "meta" box, "ItemInfoBox" 500. This sub-box indicates the number of information items present in the file. It also contains, for each item, descriptive metadata represented as "ItemInfoEntry" 501. According to the ISO BMFF standard evolution, there are several versions of this box 502 (0, 1, 2).

[0143] The "meta" item cannot be stored contiguously within a file, and there are no specific restrictions on the interleaving of the item data. Thus, two items in the same file may share one or several data blocks. This is particularly useful for HEVC tiles (tiles can be stored contiguously or non-contiguously), as it is easy to have one item per independently decodable tile. This item indicates, via the ItemLocationBox, the offset of the data in the main HEVC picture used for the tile, and the length of the slice(s).

[0144] According to embodiments, new item types describing tiled pictures can be added, for example named "hvct" or "tile", or the item type from ISO / IEC 14496-15 "hvt1" is reused. Each item representing a tiled picture (whatever the chosen four-letter code) can have a reference of type "tbas" to the "hvc1" item from which it was extracted. Each item has an identifier "item_ID" 503 and is further described in the box "ItemLocationBox" regarding the location and size of bytes in the media data box containing the compressed data for the picture.

[0145] Such a syntax allows a file format reader (or "parser") to determine, via a list of information items, how many information items are available using information about their type 504, such as "tile" to indicate that the information item is a tile picture of a whole picture.

[0146] This thus makes it possible to select a file of information items, a combination thereof, or a subset of information items in the entire set, to download only one tile of an image and the associated decoder settings, while skipping other tiles.

[0147] If an HEVC tile depends on another HEVC tile for decoding, said dependency shall be indicated by an item reference of type "dpnd" (or any specific four-letter code indicating a coding dependency) as described in ISO / IEC 14496-15:2013 AMD1 WD document w14123 "Enhanced carriage of HEVC and support of MVC with depth information", MPEG107 San Jose, January 2014.

[0148] This document defines a tool for associating HEVC tile NALUs with sample group descriptions indicating the spatial location of the tile (using the "TileRegionGroupEntry" descriptor), while there is no direct equivalent descriptor of sample groupings for metadata information items, which would allow for descriptor reuse for these.

[0149] Thus, according to embodiments, a tile description item is defined for each tile, and the tile is linked to its description using a modified version of the "ItemReferenceBox" box as described below.

[0150] According to another embodiment, only one tiling description is provided, preferably in a general way, so that the item list does not become too long.

[0151] This design can be done as follows: - Allows some items to describe a set of metadata similar to the sample group, but specific to each item type, - Adds the ability to specify one parameter for any item, for a given type of item reference, where this parameter will be interpreted according to the type of the referenced item (as well as the type of the grouping).

[0152] As will be explained below with reference to FIG. 6, it may be necessary to upgrade the descriptive metadata for an information item.

[0153] According to the ISOBMFF standard, the sample grouping mechanism is based on two principal boxes with the following "grouping_type" parameter: - The box "SampleGroupDescriptionBox" has a parameter "sgpd" that defines a list of properties (the list "SampleGroupEntry"). - The box "SampleToGroupBox" has a parameter "sbgp" that defines a list of sample groups and their mapping to properties.

[0154] The "grouping_type" parameter links a list of sample groups to a list of properties, and the mapping of one sample group to one property in the list is specified in the box "SampleToGroupBox".

[0155] To provide the same functionality for these information items, we need to describe a list of information item groups and a list of properties, and we need to be able to map each group of information items to a property.

[0156] In the following it is explained how such descriptive metadata can be incorporated into a still image file format, in other words how to link descriptors to image items. Even though an example of use for the HEVC still image file format is described, the following functionality can also be used in other standards such as ISO / IEC 14496-12 to associate any kind of information item with additional descriptive metadata.

[0157] According to embodiments, the existing "ItemInformationEntry" box 601 with the parameter "infe" is extended with new version numbers (602 and 603) to link each item to a property via a new parameter called "iref_type" 604 as shown in Figure 6. This avoids the creation of new boxes and makes it possible to improve the description while keeping it short.

[0158] The original definition of the ItemInformationEntry box was:

[0159]

number

[0160] is given by

[0161] A new version that links the tile picture to its description could look like this:

[0162]

number

[0163] According to another embodiment, closer to the box "SampleToGroupBox", the definition of the box "ItemInformationBox" with the four-letter code "iinf" is modified, for example, by introducing a new version of this box: Current version:

[0164]

number

[0165] teeth, Next:

[0166]

number

[0167] will be changed to.

[0168] Alternatively, to signal whether a group is in use, the current version uses

[0169]

number

[0170] will be changed to.

[0171] The "group_entry_count" parameter defines the number of information item groups in the media file. For each group of information items, the number of information items is indicated, starting with item_ID=0. Unlike samples, information items have no time constraints or time relationships, so the encapsulation module can assign information item identifiers in any order. By assigning increasing numbers to identifiers according to successive item groups, the list of information groups can be represented more efficiently using the parameter item_run, which identifies a run of consecutive information item identifiers in a group.

[0172] Related information items have an index called, for example, "property_index". This "property_index" parameter, associated with the "grouping_type" parameter, allows a file format parser (or reader) to identify a reference to the descriptive metadata or the descriptive metadata itself. Figure 7 shows two exemplary embodiments.

[0173] The group function in the box "SingleItemTypeReferenceBox" 701 can be used with a group identification "group_ID" instead of the information item identification (item_ID) that is normally used for the value of the from_item_ID parameter. By design, the box "SingleItemTypeReferenceBox" makes it easy to find all references of a particular type or from a particular item. Using it with "group_ID" instead of "item_ID" makes it easy to identify all references of a particular type to find a group of items. Advantageously, there is at most one box "ItemInformationBox" per encapsulation file, so there is no need to define a group identification. The encapsulation module (during encoding) and the parsing module (during decoding) can advance their respective counters for the list of information item groups (as the "g" variable in the box "ItemInformationBox") as groups are created or read. Alternatively, the parser can be told whether to maintain a group identification counter or not using a flag "group_used_flag".

[0174] Returning to the example of using one group of information items corresponding to a tile picture, where one group may contain four entries, reference 700 "SingleItemTypeReference" may indicate a list 704 of information items on which the four tile picture information items depend, and also for a particular reference type 703.

[0175] According to another exemplary embodiment, information items are used in a new kind of box, "ItemReferenceBox," which allows listing multiple reference types 723 from one item 722 to various other information items 724, as will be described below.

[0176] For the latter case, the specific box "ItemReferenceBox" 721 may be implemented as follows:

[0177]

number

[0178] For the standard box "ItemInformationBox", a list of item entries is written, but in a different order depending on the grouping. In our tile example, this leads to an initial group of four information items corresponding to tile pictures collected into groups using a parameter that can be named "tile", followed by composition information, information items for the overall picture, and optionally non-grouped information items for EXIF metadata.

[0179] Thus, a box is modified to create a box that is a specific kind of ItemReferenceBox. Below we describe this new kind of ItemReferenceBox.

[0180] This box, "ItemReferenceBox", can also be extended by distinguishing between different kinds of ItemReferenceBox using flag parameters in the box "FullBox" that is part of the ItemReferenceBox, as follows:

[0181]

number

[0182] Using the box "MultipleItemTypeReferenceBox" 721, one picture with four tiles can be described as follows:

[0183]

number

[0184] This design makes it extremely easy to find all references of any kind from a particular item.

[0185] A description support 711 for a list of items 712 that refer to the same item 714 of a given type 713 can be written as follows:

[0186]

number

[0187] In the example of a picture with four tiles, it is possible to do the following:

[0188]

number

[0189] The present design of the box 'SharedItemTypeReferenceBox' makes it easier to find all references of a particular type that point to a particular item. This is in contrast to the box 'SingleItemTypeReferenceBox'. However, because most of the 'reference_types' defined for track references are not bidirectional, the box 'SingleItemTypeReferenceBox' may not be usable with some unidirectional reference types to signal to all nodes that have this reference type to other items. Alternatively, there could be a flag in 'SingleItemTypeReference' to indicate whether it is a direct or reverse reference, thereby alleviating the need for a new SharedItemTypeReferenceBox.

[0190] In view of the above, it is advisable to associate an information item with tiling information, a description of which will now be presented.

[0191] For example, each tile can be described using a tile descriptor such as the "iref_parameter" 605 of the extended type "ItemInfoEntry" 601. A particular descriptor could be as follows:

[0192]

number

[0193] According to embodiments, the descriptors can be used for a grid of tiles to apply to one or more pictures to be stored.

[0194] Such a descriptor could look like this:

[0195]

number

[0196] This descriptor "TileInfoDataItem" allows to describe a tiling grid (regular or irregular). The grid is described row by row starting from the top left.

[0197] This descriptor must be stored as an item of type "tile". When another item references this item, it must use a reference of type "tile" to this description, and it must specify a parameter "iref_parameter", whose value is the 0-based index of the cell in the grid defined by this descriptor, where 0 is the top-left item, 1 is the cell immediately to the right of cell 0, and so on.

[0198] In this descriptor: - "version" indicates the syntax version for the TileInfoDataItem. Only the value 0 is defined. - "regular_spacing" indicates whether all tiles in the grid have the same width and height. - "reference_width", "reference_height" indicate the units the grid is displayed in. These units may or may not match the pixel resolution level of the image that references this item. If the grid is regular, "reference_width" (respectively "reference_height") must be a multiple of "nb_cell_horiz" (respectively "nb_cell_vert"). - "cell_width" indicates the horizontal division starting from the left of the grid in non-regular tiles. - "cell_height" indicates the vertical division starting from the top of the grid in non-regular tiles.

[0199] The above approach allows for sharing of tiling information for all tiles.

[0200] Furthermore, if there are multiple pictures sharing the same tiling, they can share even more descriptions by simply referencing cells in the tile grid.

[0201] This tiling configuration can be placed in a media data box or a dedicated box that is shared (by reference) between tile information items.

[0202] The above descriptors are purely spatial descriptors in the sense that they provide only the spatial position and size of the sub-image(s) within a larger image. For example, in some use cases involving image acquisition or image composition, spatial placement is not sufficient to describe an image, typically when the images overlap. This is one limitation of the above TileInfoDataBlock descriptor. To enable image composition, regardless of the images—tiles or independent / complete images—it may be useful to define a descriptor that encompasses the image's position and size (spatial relationship) on the one hand and display information for the picture (color, cropping, etc.) on the other. For example, color information may be provided to convert the sub-image from one color space to another for display. This type of information can be conveyed in the "colr" of the ISOBMFF ColorInformationBox. For technical sophistication, it may be useful to have the same data corresponding to different types of display simply by providing the transformation parameters to apply, rather than conveying two different transformed pictures of interest. Similarly, a pixel aspect ratio, such as the "pasp" box in the PixelAspectRatio defined in ISOBMFF Part 12, can be placed in this descriptor to redefine the width and height of each picture, which may differ from the coded width and height. This would indicate the scale ratio to be applied by the display after decoding of the picture. This would be followed by the coded size stored in the video sample entry (e.g., the "stsd" box) and the size of the display derived from the "pasp" box. Another possible information for the display could be the clean aperture information box "clap", also defined in ISOBMFF. According to the SMPTE 274M standard, clean aperture defines an area inside which picture information is not perceptibly contaminated by any edge transient distortions (ringing effects that may be present at the borders of an image after analog-to-digital conversion).This list of parameters useful for display is not limiting and could be placed as optional components in any other descriptive metadata box of the sub-image descriptor. These parameters can be explicitly described, as they are already part of the standard, and provide a general-purpose tool for indicating image cropping, sample aspect ratio correction, and color adjustment. Unfortunately, their use was only possible for media tracks and not for image file formats that rely on the "meta" box. Therefore, we propose a new descriptor, called, for example, "SimpleImageMetaData," to support spatial description of image items, along with other properties such as clean aperture or sample aspect ratio. This applies to any sub-image (tile or independent image) intended for composition into a larger image or back-extracted from a larger image:

[0203]

number

[0204] Or its variant when considering extra parameters to aid in the display process (e.g. via extra_boxes):

[0205]

number

[0206] The ImageSpatialRelationBox in the above syntax is an extension of TileInfoDataBlock, as explained below. Another useful parameter to consider is the possibility of structuring images as layers. We therefore propose to insert a parameter to indicate the level associated with the image in this layered composition. This is generally useful when images overlap. This could be named, for example, "layer" to match the layer information indication. An example syntax for such a description is given: Definition: Box type: "isre" Container: Simple Image Metadata Item ("isre") Mandatory: No Count: Zero or one per item Syntax:

[0207]

number

[0208] Related semantic definitions: horizontal_display_offset specifies the horizontal offset of the image. vertical_display_offset specifies the vertical offset of the image. display_width specifies the width of the image. display_height specifies the height of the image. Layer specifies the front-to-back ordering of images; lower numbered images are closer to the viewer. 0 is the normal value, -1 is before layer 0, and so on.

[0209] This new "isre" box type provides the ability to describe the position of one image relative to other images in a collection of images. It provides a subset of the functionality of the transformation matrices typically found in a media file's movie or track header box. The coordinates in an ImageSpatialRelationBox are expressed on a square grid that gives the program author's intended display size of the collection; these units may or may not coincide with the encoded size of the images. The intended display size is - Horizontally: the maximum of (horizontal_display_offset+display_width) for all "isre" boxes - Vertically: The maximum of (vertical_display_offset+display_height) for all "isre" boxes is defined by

[0210] If some images do not have any "isre" associated with them while other images in the file do have "isre" associated with them, the default images without any "isre" shall be treated as if their horizontal and vertical offsets were 0, their display size was the intended display size and their layer was 0.

[0211] This ImageSpatialRelationBox indicates the relative spatial position of the images after any cropping or sample aspect ratio has been applied to them. This means that when 'isre' is combined with 'pasp' etc. in SimpleImageMetaData, the image is decoded, 'pasp', 'clap', and 'colr' are applied if present, and then the image is moved and scaled to the offset and size declared in the 'isre' box.

[0212] This new descriptor can be used as a description of an image (tile or single image) by defining an association between the item information representing the image and the item information representing the descriptor (given the type 'simd' for the SimpleImageMetadata definition, any reserved 4-character code would be acceptable to the mp4 parser to easily identify the type of metadata it is currently processing). This association is made using a new reference type 'simr' to indicate 'spatial image relationship' using an ItemReferenceBox. The example description below shows an example of a composition of four images, where the composition itself has no associated items. Each image item is associated to a SimpleImageMetadata item via an item reference of type 'simr', and shares the DecoderConfigurationRecord information in a dedicated 'hvcC' item.

[0213]

number

[0214] The above data organization is presented as an example. The image and metadata could be combined in a metadata box, for example, to make the image plus its metadata addressable as a single byte range. Upon receiving this description, the parser would be informed by parsing the information in the "simd" item whether the sub-image was cropped from the whole picture, or conversely, whether the whole picture was composited from the sub-image. In the case of a crop, the whole picture item and the cropped image would share the same data range and the same decoder setting information, as in the example below. The sub-image would then be associated with a "simd" item with no position, in this case no "isre", and only "clap" information.

[0215] Compositing: In this case the overall picture would be associated with an item, a "simd" item containing only the "isre" information, and the sub-images would be associated with "simd" items reflecting their position in the overall picture.

[0216] The example below shows four images being composited into a larger image. All images, including the composite, are visualized as playable items using the proposed descriptors.

[0217]

number

[0218] Another example of this is when the whole picture is actually a tiled HEVC picture (4 tiles).

[0219]

number

[0220] Depending on the use case, it may be possible to have several image items share the same metadata, for example if the same cropping is applied to all images, or it may be possible for one image item to have multiple "simr" references to different SimpleImageMetaData, for example if cropping is shared between images but spatial information is not.

[0221] Another embodiment for the new version of ItemInfoEntry (as shown in Figure 6) is to define more than one parameter (605) per information item entry and reference. In the embodiment of Figure 6, the iref_parameter is a 4-byte code that is useful when the tile index references a cell in a tiling grid. However, to have a richer description and allow for the incorporation of linked descriptions within the item information entry itself rather than using the data (in the mdat box), the following extensions may be useful:

[0222]

number

[0223] In the above expansion, - item_iref_parameter_count specifies the number of reference types to which parameters are assigned. This is unchanged, in contrast to item 605 in Figure 6. - iref_type is shown in the "iref" box and specifies the reference type to which the parameter applies to this item. This is unchanged, in contrast to item 605 in Figure 6. - The parameter above is different from the iref_parameter (item 605 in Figure 6) because it provides a means of extension via a new box, ItemReferenceParameterEntry. By specializing this new box (as was done above with TileIndexItemReferenceParameterEntry in the configuration for tiled tile indexes), any kind of additional metadata can be associated with the information item entry, provided that the encapsulation and parsing modules know the structure of this specialized box. This can be achieved by a standard type for ItemReferenceParameterEntry, or by providing the structure of the parameter entry by construction or in a negotiation step. The semantic definition of this parameter is given by the semantic definition of the item with type iref_type.

[0224] In the following, exemplary descriptive metadata for an information item describing a picture having four tiles and EXIF metadata for the whole picture are presented.

[0225] As shown below, in the prior art, tile pictures were listed as information items without any corresponding description. Furthermore, setup information represented as an "hvcC" type was not described as an item. This allows for decomposition of common data related to HEVC parameters and SEI messages that apply to all tile pictures and the entire picture.

[0226]

number

[0227] According to embodiments using the version 3 extension of the ItemInfoEntry box (601) (see 602, 603 in Figure 6), tile picture information is listed with associated references to parts of the tiling configuration, which information is also described as an information item (ID=8).

[0228]

number

[0229] 8 illustrates the implementation context of embodiments of the present invention. First, various media are recorded, e.g., audio during step 800a, video during step 800b, and one or more pictures during step 800c. Each media is compressed during steps 801a, 801b, and 801c, respectively. During these compression steps, elementary streams 802a, 802b, and 802c are generated. Next, at the application level (e.g., user selection from a graphical user interface, configuration of a multimedia production system), an encapsulation mode is selected to determine whether all these elementary streams should be merged. If the "merge" mode is enabled (decision 803 returns "yes"), during step 806c, the data for audio, video, and still images are encapsulated in the same file as described above. If the "fusion" mode is not enabled (decision 803 is "no"), two encapsulation files are generated in succession or in parallel during steps 806a and 806b, resulting in the generation of one file for the synchronized-time media data during step 807a and an additional file 907b containing only still images, respectively. To set the tile descriptions and region-of-interest characteristics as described above, the audio and video elementary streams are encapsulated according to the ISOBMFF standard during step 806a, and the still images are encapsulated during step 806b. Finally, a media representation 807 is obtained, which can be fed to a DASH generator to prepare it for streaming (step 820a), stored in memory (step 820b), rendered on a display unit (step 820c), or transmitted to a remote entity (step 820d), either in its entirety or after parsing the descriptive metadata to extract parts (e.g., tiles).

[0230] It is noted that in the preceding description of the embodiments, descriptive metadata such as, for example, the SimpleImageMetadata ("simd") box (also called ISOBMFFMetaData in the latest version of the still image file format specification) is described as a mature item. Additional descriptive or normative metadata is also defined by the still image file format specification as set forth in Committee Study Draft Document W14878 of ISO / IEC 23008-12:2013 First Edition, "Information technology - MPEG systems technologies - Part 12: Image File Format," MPEG 110, Strasbourg, October 2014. Examples of descriptive or prescriptive metadata are CleanApertureBox ("clap"), ImageRotation ("irot"), ExifDataBlock ("exif"), or ImageOverlay ("iovl"). More generally, descriptive metadata is metadata that provides additional information or description for an item such as an image or sub-image (e.g., exif metadata), while prescriptive metadata is an operation or transformation that is applied to an item (e.g., a rotation, crop, or the combination of several items that form a transformation operator).

[0231] However, storing such descriptive or normative metadata as complete items in a specification would be extremely cumbersome; these are merely pseudo-items, and the descriptive or normative metadata would need to be stored in the mdat box (110) along with the encoded data, requiring entries to be defined in the itemLocationBox (iloc) (109), itemInfoBox (iinf), and itemProtectionBox (ipro). The need for these entries in iloc, iinf, and ipro imposes a significant overhead. For example, an entry in itemInfoBox requires the use of the entire box with a minimum 12-byte header, plus the definition of item_protection_index (16 bits) plus an empty item_name (8 bits), for a total additional cost of 15 bytes per entry in itemInfoBox(iinf). An entry in itemLocationBox(iloc) also requires a minimum of 9 bytes in the best case (base_offset_size = offset_size = length_size = 1, 1 extent). In practice, itemLocationBox entries are used with base_offset_size = offset_size = length_size = 2 or 4, i.e., at an additional cost of 12 or 18 bytes. Furthermore, this metadata is usually small, allowing for efficient reading of other items. Storing these as dedicated items can make file parsing cumbersome, especially for partial retrieval of files (e.g., increased HTTP requests).

[0232] In another embodiment, all descriptive and normative metadata can be defined as embedded items that can be stored in the meta box (100) as part of other boxes rather than in the mdat box (110), thus avoiding the additional cost of defining itemInfoBox and itemLocationBox entries.

[0233] To store descriptive and normative metadata in the meta box, a virtual item box called "VirtualItemBox" is defined. According to this embodiment, all descriptive and normative metadata boxes are inherited from this virtual item class.

[0234] A virtual item has an item_ID and an item_type assigned to it along with a set of boxes. A virtual item is additional data commonly used to describe metadata that will be associated with other items. For example, it allows associating an entry in an itemInfoBox that identifies an item (image or subimage) with an operation or transformation that will be applied to that item. Typically, this association can be described by defining an entry of type "simr" in an itemReferenceBox from the item_ID of the image to the item_ID of a metadata operation or transformation description box. Virtual items can only be referenced in item reference boxes and primary item boxes and must not be declared or referenced in any other boxes (e.g., itemLocationBox (iloc), itemInfoBox (iinf), itemProtectionBox (ipro)). This "VirtualItemBox" is defined as follows:

[0235]

number

[0236] The semantics for the above parameters are as follows: item_ID: The ID (or identifier) of this item. It is illegal to have entries in iinf, iloc, or ipro with the same item_ID value. item_type is a 32-bit value, usually 4 printable characters, that is a defined valid item type indicator, such as "mime".

[0237] Optionally, in one variant, the "VirtualItemBox" can include an additional parameter called "descriptor_family", which indicates whether the metadata box is descriptive or prescriptive metadata. In one variant, the description family indicates the type of the metadata box from a predefined list of values, for example, transform_operator, composed_image, descriptive_metadata, etc.

[0238] By inheriting this virtual item box, all descriptive and normative metadata boxes can be stored in the meta box without the need to define associated entries in itemInfoBox (iinf) and itemLocationBox (iloc), yet they still retain the advantage of being addressable by the item reference box.

[0239] According to this embodiment, ImageOverlay (iovl), SubSampleItemData (subs), AuxiliaryConfiguration (auxC), ExifDataBlock (exif), SimpleImageMetadata (simd) and derived image items are inherited from the virtual item class.

[0240] Further according to this embodiment, a single generic item type called "dimg" is introduced with item references of type "simr" to items of type "simd". This approach allows property reuse where appropriate and reduces the number of items and item references. An ImageRotationBox is added in SimpleImageMetadata(simd). The "simr" reference type defines a link from image items to "simd" items to provide direct access to image description metadata.

[0241] Additionally, the ImageOverlay (iovl) metadata box is redesigned as follows so that it is no longer dependent on reference order:

[0242]

number

[0243] To clearly identify the item being composed, an explicit item_ID is added to each entry in the loop.

[0244] In another embodiment, all boxes contained within SimpleImageMetadata (simd) are defined as independent metadata boxes inherited from the Virtual Item box.

[0245] In another embodiment, simple image rotation can be declared by embedding the rotation operation directly in the image metadata descriptor SimpleImageMetadata ("simd") box (also called ISOBMFFMetaData in the latest version of the Still Image File Format Specification), as follows:

[0246]

number

[0247] The rotated box is slightly larger than the 'irot' item (12 bytes), but the benefit of using this approach is apparent when combining transformations, such as rotation and CleanApperture, as only one 'simd' is needed instead of a cascade of derived items.

[0248] In such a case, the generic derived item "dimg" (described above) can be used to refer to both the image item and the metadata description. Such an item could then be listed as the primary item in the PrimaryItemBox ("pitm").

[0249] Another benefit of this approach is that the program author can clearly indicate that they want only rotated items to be displayed.

[0250] The following paragraphs propose an alternative to the previously described embodiment that is advantageously simpler regarding how transformations (or "effects") can be applied to images in the ISO still image file format. Specifically, the following problems are solved using this alternative embodiment: - Numerous item references, - Increased number of items when cascading effects, and - Impossibility of reciprocity of effects on a given set of items, i.e., a set of images or parts of images, such as areas of interest.

[0251] Existing solutions have proposed to mutualize the effect as various extents (i.e. byte offsets in the data part) of an item. More precisely, extents means that the derived image will be described in the itemLocationBox ("iloc") as a list of extents, each identifying a fragment of the data part ("mdat"), and each fragment corresponding to one or more descriptive, normative or transformation metadata.

[0252] However, this solution has some inherent drawbacks: - Authoring encapsulated image files is fairly tedious, and modifying one effect in one derived image item means inspecting all derived images to check if they share the same extent, and potentially rewriting parts of them. - Parsing is also not very straightforward, as the image file reader would need to figure out if the chain of transformations / effects is the same for different items in the file (there is no direct signaling). - For each transform / effect, a new extent will be required in the itemLocationBox("iloc") unless the new transform / effect is stored contiguously with the transforms / effects in the chain to which it is applied. Furthermore, combining or cascading effects can be costly if they are not stored on contiguous extents in the data portion.

[0253] Furthermore, these solutions required implementing storage to determine the type of the effect (until now the type of the effect was given by item_type), which meant creating a box to store the effect in. A simpler solution would be to define the effect separately from the item by defining a new box for the effect, and have a direct mapping between item and effect without any additional cost.

[0254] This alternative embodiment proposes to simplify the handling of effects by having a clear separation in the file format: - canonical items (images or parts of images) (e.g., hvc1, …) linked to descriptive metadata (the "init" or "simr" reference types as proposed above, or any reference type representing descriptive metadata); - a "Derived Image", which is a collection of effects (or transformations) applied to one or more source items (images or portions of images) identified via "dimg" item references from the "Derived Image" item to the source items; and - A structure that represents a transformation / effect, containing a collection of several different effects.

[0255] The advantages of this alternative embodiment are: - Effect reusability: declared once and potentially referenced multiple times, - A more compact description by defining a collection of effects (discussed further below), - Overall readability, including the unnecessary new extent of the itemLocationBox, and - Keep the number of item references low, is.

[0256] According to this alternative embodiment, a new single derived item is defined using the item type "dimg". This single derived item is specifically expressed by the following syntax:

[0257]

number

[0258] In the above syntax, nb_effects represents the number of effects that will be applied to the source image to composite the derived image, and effect_id is a unique identifier within the encapsulation file of the effect to be applied. The effects are applied in the reverse order that they appear in the list of effects.

[0259] A derived image or transformation item named "DerivedImage" defines an image, for example, as a set of effects to be applied to a source image before it is presented to the user or a display screen. The source image is identified by an item reference of type "dimg" (or any reserved reference type) from the derived item to the source image. The source image itself may be any image item defined in the ISO Still Image File Format Specification (image or image portion, image overlay, derived image). There may be no more than one "dimg" item reference from the same item (although there may be multiple for the same item if this item is reused multiple times for various compositions).

[0260] Derived items are stored in the data portion of the file.

[0261] If the encapsulation file is edited, for example to remove an effect from an image file, all references to this effect must be removed from the derived images.

[0262] Effects can be applied to an image, part of an image, a composite image, or a derived image via a DerivedImage item. Each effect is described by a box that derives from the BaseEffectBox structure shown below.

[0263]

number

[0264] The semantics of the above are as follows: effect_type is the box type for effects derived from this class, a unique four-character code that identifies this kind of box. effect_id is a unique identifier for a given effect or transformation. This identifier must be unique within a "meta" box. nb_bits_effect is derived from the version value and indicates the number of bits used to represent the effect_id.

[0265] Effects may be declared in an optional EffectDeclarationBox contained within a "meta" box.

[0266] Box type: "effd" Container: meta Mandatory: No Number: Zero or One

[0267]

number

[0268] For example, you can define the following effects (without restricting the list): - Rotation Effect: This rotation effect translates the source image in 90 degree increments counterclockwise.

[0269] Box type: "erot" Container: effd Mandatory: No Number: Zero or more

[0270]

number

[0271] The above semantic definition is angle*90: This specifies the angle in degrees (counterclockwise). - Clean Aperture Effect: The clean aperture effect modifies the visible part of the source image.

[0272] Box type: "ecla" Container: effd Mandatory: No Number: Zero or more

[0273]

number

[0274] The above semantic definition is nb_bits_effect is derived from the parent class BaseEffectBox and indicates the number of bits used to represent the various fields of the CleanApertureEffectBox. hSpacing, vSpacing: Defines the relative width and height in pixels. cleanApertureWidthN, cleanApertureWidthD: Fractions that define the exact clean aperture width of the image in counted pixels. cleanApertureHeightN, cleanApertureHeightD: Fractions that define the exact clean aperture height of the image in counted pixels. horizOffN, horizOffD: Fractions defining the horizontal offset of the center of the clean aperture minus (width-1) / 2 (usually 0). vertOffN, vertOffD: Fractions defining the vertical offset of the center of the clean aperture minus (height-1) / 2 (usually 0).

[0275] Effect Collection: The effect collection box allows you to define a set of several effects as a single effect in order to reuse it for several images, thus reducing the writing cost in terms of bytes.

[0276] Box type: "ecol" Container: effd Mandatory: No Number: Zero or more

[0277]

number

[0278] The above semantic definition is nb_bits_effect is derived from the parent class BaseEffectBox and indicates the number of bits used to represent the various fields of the EffectCollectionBox. apply_effect_id: Indicates the ID of the effect to apply to the source image.

[0279] Effects in the effects collection are applied in the same order as the effects in the DerivedImaged item, i.e., effects shall be applied to the input in the reverse order of their appearance in the list of effects.

[0280] The OverlayEffectBox declares the composition of an image as an overlay. For this particular effect, the resulting derived image does not have a reference to any source images, since this effect declares the list of source images that are part of the composition.

[0281]

number

[0282] The semantics of the above are as follows: nb_bits_effect is derived from the parent class BaseEffectBox and indicates the number of bits used to represent the various fields of the OverlayEffectBox. fill_required indicates whether there are any holes in the resulting composite image that should be filled with background values. canvas_fill_value indicates the per-channel pixel value to be used if any input image pixel is not located at a particular pixel location. If the input image contains fewer than three channels, the semantics of canvas_fill_value corresponding to channels not present in the input image are unspecified.

[0283] nb_images indicates the number of images to be composited, each identified by their item_ID indicated by the image_item_ID parameter.

[0284] output_width, output_height: Specifies the width and height, respectively, of the output image into which the input image is interleaved.

[0285] The picture area of the output image is called the canvas.

[0286] horizontal_offset, vertical_offset: Specifies the offset from the top left of the canvas where the input image is placed. Pixel locations with negative offset values are not included in the output image. Horizontal pixel locations that are larger than or equal to output_width are not included in the output image. Vertical pixel locations that are larger than or equal to output_height are not included in the output image.

[0287] 9 is a simplified block diagram of a computing device 900 for implementing one or more embodiments of the present invention. Computing device 900 may be a microcomputer, a workstation, or a lightweight handheld device. Computing device 900 includes a communication bus connected to: - a central processing unit 901, called CPU, such as a microprocessor: a random access memory 902, called RAM, for storing the executable code of the method of the present invention and registers adapted to record the variables and parameters necessary to implement the method of reading and writing manifests and / or to encode the video and / or to read or generate data in a given file format, the capacity of which can be expanded, for example, by an optional RAM connected to an expansion port: a read-only memory 903, called ROM, for storing a computer program for implementing an embodiment of the invention: - The network interface 904 is typically connected to a communications network through which digital data to be processed is transmitted or received. The network interface 904 can be a single network interface or a set of various network interfaces (e.g., a wired and a wireless interface, or a set of various wired or wireless interfaces). Data is written to the network interface for transmission or read from the network interface for reception under the control of software applications running in the CPU 901; - a user interface 9805 for receiving input from a user or displaying information to a user; - Hard Disk 906 (also called HD): - An I / O module 907 for receiving / sending data to / from external devices such as video sources or displays.

[0288] The executable code may be stored either in the read-only memory 903, on the hard disk 906 or on a removable digital medium, such as a disk. According to a variant, the executable code of the program may be received by means of a communication network, via the network interface 904, to be stored in one of the storage means of the communication device 900, such as the hard disk 906, before being executed.

[0289] The central processing unit 901 is adapted to control and direct the execution of instructions or parts of software code of a program or programs according to embodiments of the present invention, which instructions are stored in one of the aforementioned storage means. After power-on, the CPU 901 is able to execute instructions from the main RAM memory 902 relating to a software application after said instructions have been loaded, for example, from a program ROM 903 or a hard-disk (HD) 906. Such software application, when executed by the CPU 901, causes the CPU 901 to perform steps of methods according to embodiments.

[0290] Alternatively, the present invention can be implemented in hardware (for example, in the form of an Application Specific Integrated Circuit or ASIC).

[0291] The present invention can be incorporated into devices such as cameras, smartphones, or tablets that function as remote controllers for TVs, for example, to zoom in on specific areas of interest. The present invention can also be used in the same devices to have a personalized browsing experience of TV programs by selecting specific areas of interest. Another use of these devices by users is to share selected subportions of their favorite videos with other connected devices. The present invention can also be used in smartphones or tablets to monitor what is happening in specific areas of a building under surveillance, assuming the surveillance cameras support the generation portion of the present invention.

[0292] While the invention has been shown and described in detail in the drawings and foregoing description, such showing and description is to be considered illustrative or exemplary and not restrictive, and the invention is not limited to the disclosed embodiments. In practicing the claimed invention, those skilled in the art will be able to deduce and produce other variations to the disclosed embodiments, from a study of the drawings, the disclosure, and the appended claims.

[0293] In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that different features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope of the invention.

Claims

1. 1. A method for generating a file containing an encoded bitstream, comprising: generating a file in which a data portion stores an encoded bitstream including data corresponding to one or more items of one or more types, and a metadata portion stores identification information for identifying each of the one or more items and one or more properties; Output the generated file, The method, characterized in that the metadata portion includes association information indicating an association between one or more properties and identification information of an item corresponding to the one or more properties, and is included in a "meta" box in ISOBMFF.

2. 1. A method for generating a file containing an encoded bitstream, comprising: generating a file in which a data portion stores an encoded bitstream including data corresponding to one or more items of one or more types, and a metadata portion stores identification information for identifying each of the one or more items and one or more properties; Output the generated file, The method, wherein the metadata portion includes association information indicating an association between one or more properties and identification information for each of a plurality of items corresponding to the one or more properties.

3. 1. A method for generating a file containing an encoded bitstream, comprising: generating a file in which a data portion stores an encoded bitstream including data corresponding to one or more items of one or more types, and a metadata portion stores identification information for identifying each of the one or more items and one or more properties; Output the generated file, the metadata portion includes association information indicating an association between one or more properties and identification information of an item corresponding to the one or more properties; The method, wherein the one or more items of one or more types include at least two of a still image, text, and audio.

4. 1. A method for generating a file containing an encoded bitstream, comprising: generating a file in which a data portion stores an encoded bitstream including data corresponding to one or more items of one or more types, and a metadata portion stores identification information for identifying each of the one or more items and one or more properties; Output the generated file, the metadata portion includes association information indicating an association between one or more properties and identification information of an item corresponding to the one or more properties; The method, wherein the one or more items of one or more types include one type of item and a plurality of sub-items corresponding to the item.

5. 5. The method according to claim 2, wherein the metadata part is contained in a "meta" box in ISOBMFF.

6. 5. The method of claim 1, 2 or 4, wherein the one or more items of one or more types include at least one of a still image, text, and audio.

7. 7. The method according to claim 1, wherein the association information includes information indicating an association between a plurality of properties and identification information of items corresponding to the plurality of properties.

8. 8. The method according to claim 1, wherein the coded bitstream is contained in an "mdat" box in ISOBMFF.

9. The method according to any one of claims 1 to 8, wherein the association information is contained in a predetermined box in the metadata portion.

10. 10. The method according to any one of claims 1 to 9, wherein the file has a format according to the ISO / IEC 23008-12 standard.

11. 1. A method for processing a file containing an encoded bitstream, comprising: obtaining a file in which a coded bitstream including data corresponding to one or more items of one or more types is stored in a data portion, and identification information for identifying each of the one or more items and one or more properties is stored in a metadata portion; outputting data corresponding to the one or more items; the metadata portion includes association information indicating an association between one or more properties and identification information of an item corresponding to the one or more properties; The method, wherein the metadata portion is contained in a "meta" box in ISOBMFF.

12. 1. A method for processing a file containing an encoded bitstream, comprising: obtaining a file in which a coded bitstream including data corresponding to one or more items of one or more types is stored in a data portion, and identification information for identifying each of the one or more items and one or more properties is stored in a metadata portion; outputting data corresponding to the one or more items; The method, wherein the metadata portion includes association information indicating an association between one or more properties and identification information for each of a plurality of items corresponding to the one or more properties.

13. 1. A method for processing a file containing an encoded bitstream, comprising: obtaining a file in which a coded bitstream including data corresponding to one or more items of one or more types is stored in a data portion, and identification information for identifying each of the one or more items and one or more properties is stored in a metadata portion; outputting data corresponding to the one or more items; The method, wherein the one or more items of one or more types include at least two of a still image, text, and audio.

14. 1. A method for processing a file containing an encoded bitstream, comprising: obtaining a file in which a coded bitstream including data corresponding to one or more items of one or more types is stored in a data portion, and identification information for identifying each of the one or more items and one or more properties is stored in a metadata portion; outputting data corresponding to the one or more items; The method, wherein the one or more items of one or more types include one type of item and a plurality of sub-items corresponding to the item.

15. 15. The method according to any one of claims 12 to 14, wherein the metadata part is contained in a "meta" box in ISOBMFF.

16. 15. The method of claim 11, 12 or 14, wherein the one or more items of one or more types include at least one of a still image, text, and audio.

17. 13. The method according to claim 11, wherein the association information includes information indicating association between a plurality of properties and identification information of items corresponding to the plurality of properties.

18. 18. The method according to any one of claims 11 to 17, wherein the coded bitstream is contained in an "mdat" box in ISOBMFF.

19. 13. The method according to claim 11 or 12, wherein the association information is contained in a predetermined box in the metadata portion.

20. A method as described in claim 13 or 14, characterized in that association information indicating an association between one or more properties and identification information of items corresponding to the one or more properties is included in a specified box of the metadata portion.

21. 21. The method according to any one of claims 11 to 20, wherein the file has a format according to the ISO / IEC 23008-12 standard.

22. A program causing a computer to execute the method according to any one of claims 1 to 10.

23. A program causing a computer to carry out the method according to any one of claims 11 to 21.

24. 1. A device for generating a file containing an encoded bitstream, comprising: means for generating a file in which a coded bitstream including data corresponding to one or more items of one or more types is stored in a data portion, and identification information for identifying each of the one or more items and one or more properties are stored in a metadata portion; and means for outputting the generated file; A device characterized in that the metadata portion includes association information indicating an association between one or more properties and identification information of an item corresponding to the one or more properties, and is included in a "meta" box in ISOBMFF.

25. 1. A device for generating a file containing an encoded bitstream, comprising: means for generating a file in which a coded bitstream including data corresponding to one or more items of one or more types is stored in a data portion, and identification information for identifying each of the one or more items and one or more properties are stored in a metadata portion; and means for outputting the generated file; The metadata portion includes association information indicating an association between one or more properties and identification information of each of a plurality of items corresponding to the one or more properties.

26. 1. A device for generating a file containing an encoded bitstream, comprising: means for generating a file in which a coded bitstream including data corresponding to one or more items of one or more types is stored in a data portion, and identification information for identifying each of the one or more items and one or more properties are stored in a metadata portion; and means for outputting the generated file; the metadata portion includes association information indicating an association between one or more properties and identification information of an item corresponding to the one or more properties; The one or more items of one or more types include at least two of a still image, text, and audio.

27. 1. A device for generating a file containing an encoded bitstream, comprising: means for generating a file in which a coded bitstream including data corresponding to one or more items of one or more types is stored in a data portion, and identification information for identifying each of the one or more items and one or more properties are stored in a metadata portion; and means for outputting the generated file; the metadata portion includes association information indicating an association between one or more properties and identification information of an item corresponding to the one or more properties; A device characterized in that the one or more items of one or more types include one type of item and multiple sub-items corresponding to the item.

28. A device for processing a file containing an encoded bitstream, comprising: means for obtaining a file in which a coded bitstream containing data corresponding to one or more items of one or more types is stored in a data portion, and identification information for identifying each of the one or more items and one or more properties are stored in a metadata portion; means for outputting data corresponding to the one or more items; the metadata portion includes association information indicating an association between one or more properties and identification information of an item corresponding to the one or more properties; A device characterized in that the metadata portion is included in a "meta" box in ISOBMFF.

29. A device for processing a file containing an encoded bitstream, comprising: means for obtaining a file in which a coded bitstream containing data corresponding to one or more items of one or more types is stored in a data portion, and identification information for identifying each of the one or more items and one or more properties are stored in a metadata portion; means for outputting data corresponding to the one or more items; The device, wherein the metadata portion includes association information indicating an association between one or more properties and identification information of each of a plurality of items corresponding to the one or more properties.

30. A device for processing a file containing an encoded bitstream, comprising: means for obtaining a file in which a coded bitstream containing data corresponding to one or more items of one or more types is stored in a data portion, and identification information for identifying each of the one or more items and one or more properties are stored in a metadata portion; means for outputting data corresponding to the one or more items; the metadata portion includes association information indicating an association between one or more properties and identification information of an item corresponding to the one or more properties; The one or more items of one or more types include at least two of a still image, text, and audio.

31. A device for processing a file containing an encoded bitstream, comprising: means for obtaining a file in which a coded bitstream containing data corresponding to one or more items of one or more types is stored in a data portion, and identification information for identifying each of the one or more items and one or more properties are stored in a metadata portion; means for outputting data corresponding to the one or more items; the metadata portion includes association information indicating an association between one or more properties and identification information of an item corresponding to the one or more properties; A device characterized in that the one or more items of one or more types include one type of item and multiple sub-items corresponding to the item.

Images