Method and apparatus for encapsulating region descriptive information of an image in a media file

Abstract

The present disclosure concerns a method of encapsulating images in an ISOBMFF based media file, the method comprising, if a region of a first image corresponds to an approximate, merge or summary of at least one region of at least one second image: generating an approximate region item describing the region of the first image, generating an indication that the approximate region item describes a region of the first image that corresponds to an approximate, merge or summary of the at least one region of the at least one second image, and encapsulating the approximate region item and the indication in the media file.

Description

[0001] METHOD AND APPARATUS FOR ENCAPSULATING REGION DESCRIPTIVE INFORMATION OF AN IMAGE IN A MEDIA FILE

[0002] FIELD OF THE DISCLOSURE

[0003] The present disclosure concerns a method and a device for encapsulating region descriptive information of an image in a media file.

[0004] BACKGROUND OF DISCLOSURE

[0005] Modern cameras and image analysis services enable to generate localized metadata for images. Localized metadata are metadata related to a region, a portion, of a media content and not to the whole media content. The media content is typically an image, but it may also be a frame of a video content, or a set or collection of images. For example, a camera may generate the focusing region for a photo or detect faces while taking a picture. As another example, a deep-learning system may identify objects inside an image. These localized metadata may be seen as region annotations.

[0006] Modern cameras also enable to generate very large images or video frames of which only a spatial area may need to be displayed on a given display size (e.g. a smartphone device or an HD or 4K display), either to avoid downloading the complete image before displaying content or for panning across the images. Images or video frames captured by a camera are stored on a storage device like a memory card, for example. The images or video frames are typically either encoded to reduce the size of data on the storage device or the pixels of the images are arranged according to an uncompressed image format (RGB or YUV formats are typical examples of such uncompressed image formats). Many encoding standards may be used, like JPEG, AOMedia Video 1 (AV1), High Efficiency Video Coding (HEVC) or the more recent Versatile Video Coding (VVC) standard. Such uncompressed or encoded image formats allow partitioning an image or video frame, also denoted picture, into image or picture portions e.g. subpictures, slices, or tiles. A subpicture, a slice or a tile may represent a rectangular region of the image or picture.

[0007] The International Standard Organization Base Media File Format (ISOBMFF, ISO / IEC 14496-12) is a well-known flexible and extensible format that describes the encapsulation of timed or untimed media data or bit-streams either for local storage or transmission via a network or via another bit-stream delivery mechanism. An ISO Base media file is object-oriented and structured into “boxes” that are sequentially or hierarchically organized. Boxes are data structures provided to describe the data in the files. Boxes (also denoted objects, atoms, structure-data, or data structures) are building blocks starting with a header which gives both size and a unique type identifier (typically a four-character code (32-bit), also noted FourCC or 4CC). Some boxes, called ‘FullBox’, also contain a version number and flags field.

[0008] All data in a file (media data and metadata describing the media data) is contained in boxes. There is no other data within the file. File-level boxes are boxes that are not contained in other boxes.

[0009] This file format has several extensions developed by the Moving Picture Experts Group (MPEG), for instance:

[0010] - HEIF (for High Efficiency Image File Format or Image File Format - ISO / IEC 23008-12) is a generic standard for storage and sharing of images and image sequences, as well as their associated metadata. It also specifies the format for encapsulating coded images or image sequences with the coding format JPEG, HEVC or VVC for instance.

[0011] - MIAF (for Multi-Image Application Format - ISO / IEC 23000-22) is a standard specifying a multimedia application format which enables precise interoperability points for the creation, reading, parsing and decoding of images embedded in the HEIF format. The MIAF specification fully conforms to the HEIF format and only defines additional constraints to ensure higher interoperability.

[0012] - Carriage of uncompressed video and images in ISO Base Media File Format (ISO / IEC 23001-17) is a standard specifying how uncompressed 2D image and video data is carried in files in the family of standards based on the ISO base media file format (ISO / IEC 14496-12).

[0013] While these file formats provide the ability to store, access and exchange coded and uncompressed images or image sequences, there is a continuous need to improve the storage of data and the description of this data.

[0014] In particular, the HEIF file format provides a mechanism adapted to describe regions of an image, and their associated annotations, relying on region items. However, the number of region items applying to an image may become very large and existing region items do not allow differentiating different types of region items allowing a player or parser to adapt its processing or to reduce the processing cost of the region items. SUMMARY OF THE DISCLOSURE

[0015] The present disclosure has been devised to address one or more of the foregoing concerns.

[0016] According to a first aspect, there is provided a method of encapsulating images in an ISOBMFF based media file, the method comprising, if a region of a first image corresponds to an approximate, merge or summary of at least one region of at least one second image: generating an approximate region item (also called merge region item herein) describing the region of the first image -notably the geometry of the region of the first image-, generating an indication that the approximate region item describes a region of the first image that corresponds to an approximate, merge or summary of the at least one region of the at least one second image, and encapsulating the images, the approximate region item and the indication in the media file.

[0017] The indication allows a content creator generating a media file or a parser reading a media file to differentiate a region item indicating a real or singular object or meaningful region from a region item indicating a region that represents an approximate or summary of one or more real or singular objects or meaningful regions. In this way, it is possible to adapt the parsing of the media file, according to whether a region item is an approximate region item.

[0018] In an embodiment, the method comprises: determining that the region of the first image corresponds to an approximate, merge or summary of at least one region of the at least one second image, then generating the approximate region item describing the region of the first image. In an embodiment, generating the approximate region item comprises: generating a geometry data structure describing the geometry of the region of the first image associated with the approximate region item.

[0019] In an embodiment, generating the approximate region item also comprises generating at least one annotation data structure associated with the approximate region item.

[0020] In an embodiment, the indication comprises a predefined flag value in flags parameter of the approximate region item.

[0021] In an embodiment, the indication comprises a predefined item type value for the approximate region item. In an embodiment, the indication comprises a dedicated item property associated with the approximate region item.

[0022] In an embodiment, the dedicated item property comprises item identifiers of at least one region item describing the at least one region of the at least one second image that is approximated, merged or summarized by the region described by the approximate region item.

[0023] In an embodiment, the indication comprises an entity group comprising a dedicated grouping type, item identifier of the approximate region item and item identifiers of at least one region item describing the at least one region of the at least one second image that is approximated, merged or summarized by the region described by the approximate region item.

[0024] In an embodiment, the indication comprises an item reference of a dedicated type from the approximate region item to at least one region item describing the at least one region of the at least one second image that is approximated, merged or summarized by the region described by the approximate region item.

[0025] In an embodiment, the method comprises: generating at least one region item describing the at least one region of the at least one second image that is approximated, merged or summarized by the region described by the approximate region item; associating the at least one region item to the approximate region item.

[0026] In an embodiment, the approximate region item inherits of annotations from the at least one region item.

[0027] In an embodiment, the method comprises generating an entity group -for example a pyramid entity group, an alternate entity group, a progressive rendering entity group, a bracketing set entity group- comprising image items whose reconstructed images represent the same visual content or part thereof, possibly at different resolutions, the entity group comprising an image item describing the at least one second image associated with the at least one region item and an image item describing the first image associated with the approximate region item.

[0028] According to another aspect, there is provided a method of parsing an ISOBMFF based media file, the method comprising: obtaining an item describing a region of a first image from the media file based on an indication indicating that the item is an approximate region item, the region of the first image being an approximate, merge or summary of at least one region of at least one second image; processing the obtained approximate region item.

[0029] In an embodiment, the indication comprises an item reference of a dedicated type from the approximate region item to at least one region item describing the at least one region of the at least one second image that is approximated, merged or summarized.

[0030] In an embodiment, the processing of the approximate region item includes rendering the geometry or shape of the region described by the approximate region item with an indication that the rendered region is an approximate, merge or summary of the at least one region.

[0031] In an embodiment, if the approximate region item is associated with an item describing the at least one region of the at least one second image, the processing of the approximate region item includes retrieving and processing the annotations associated with the item describing the at least one region of the at least one second image.

[0032] In an embodiment, the first image inherits from at least one item describing a region of the at least one second image. In this case, if the approximate region item is associated with a region item describing the at least one region of the at least one second image, the method comprises filtering the region item associated with the approximate region item from inherited items of the first image.

[0033] Such method may allow a user to differentiate annotations associated with a real object in an image from annotations associated with a region representing an approximate of multiple real objects

[0034] Such method may also allow a user to render the annotation of associated region items that also apply to an approximate region item.

[0035] Such method may also allow a user to first render the approximate region items and ignore the associated region items.

[0036] Filtering the region items associated with the approximate region item allows avoiding these region items to be processed because these region items are already summarized or approximated by the approximate region item. In a variant, Filtering the region items allows avoiding processing at least the geometry of the region items associated with the approximate region, but the region annotations associated with the region items may still be processed as inherited region annotations for the approximate region item.. Therefore, filtering the region items associated with the approximate region item allows reducing the processing cost of handling region items applying to an image (e.g., depending on a zoom / scale level).

[0037] In an embodiment, the first image inherits from the at least one second image if the at least one second image corresponds to an alternate image of the first image or to an image comprising same visual content with higher resolution than the first image.

[0038] According to another aspect, there is provided a computer program product for a programmable apparatus, the computer program product comprising a sequence of instructions for implementing a method of encapsulating images in an ISOBMFF based media file or a method of parsing an ISOBMFF based media file according to the aforementioned embodiments, when loaded into and executed by the programmable apparatus.

[0039] According to another aspect, there is provided a computer-readable storage medium storing instructions of a computer program product according to the aforementioned embodiments.

[0040] According to another aspect, there is provided a device for encapsulating images in an ISOBMFF based media file, the device comprising a processor configured to, if a region of a first image corresponds to an approximate, merge or summary of at least one region of at least one second image: generate an approximate region item describing the region of the first image-notably the geometry of the region of the first image-, generate an indication that the approximate region item describes a region of the first image that corresponds to an approximate, merge or summary of the at least one region of the at least one second image, and encapsulate the images, the approximate region item and the indication in the media file.

[0041] According to another aspect, there is provided a device for parsing an ISOBMFF based media file, the device comprising a processor configured to: obtain an item describing a region of a first image -notably the geometry of the region of the first image- from the media file based on an indication indicating that the item is an approximate region item, the region of the first image being an approximate, merge or summary of at least one region of at least one second image; process the obtained approximate region item.

[0042] According to another aspect, there is provided an ISOBMFF based media file comprising: an approximate region item, describing a region of a first image -notably the geometry of the region of the first image- that corresponds to an approximate, merge or summary of at least one region of at least one second image. an indication that the approximate region item describes a region of a first image that corresponds to an approximate, merge or summary of at least one region of at least one second image.

[0043] According to another aspect, there is provided a computer-readable storage medium storing an ISOBMFF based media file according to the aforementioned embodiment.

[0044] At least parts of the methods according to some embodiments of the disclosure may be computer implemented. Accordingly, some embodiments of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit", "module" or "system". Furthermore, some embodiments of the present invention may take the form of a computer program product embodied in any tangible medium of expression having computer usable program code embodied in the medium.

[0045] Since the methods can be implemented in software, some embodiments of the present disclosure can be embodied as computer readable code for provision to a programmable apparatus on any suitable carrier medium. A tangible, non-transitory carrier medium may comprise a storage medium such as a floppy disk, a CD-ROM, a hard disk drive, a magnetic tape device or a solid-state memory device and the like. A transient carrier medium may include a signal such as an electrical signal, an electronic signal, an optical signal, an acoustic signal, a magnetic signal or an electromagnetic signal, e.g. a microwave or RF signal.

[0046] BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Embodiments of the disclosure will now be described, by way of example only, and with reference to the following drawings in which:

[0048] Figure 1 illustrates an example of a media file that contains media data like one or more still images and possibly one or more video and / or one or more sequences of images; Figure 2a illustrates an example of region items describing regions of an image in an ISOBMFF-based media file;

[0049] Figure 2b illustrates an embodiment of region items describing regions of images in an ISOBMFF-based media file;

[0050] Figure 3 illustrates another embodiment of region items describing regions of an image pyramid in an ISOBMFF-based media file.

[0051] Figure 4 illustrates the association of region items and approximate region items with image items.

[0052] Figure 5 illustrates the main steps of an implementation of a method of encapsulating images in a media file;

[0053] Figure 6 illustrates the main steps of an implementation of a method of parsing a media file;

[0054] Figure 7 is a schematic block diagram of a computing device for implementation of one or more embodiments of the disclosure.

[0055] DETAILED DESCRIPTION OF THE DISCLOSURE

[0056] The JPEG, AOMedia Video 1 (AV1), High Efficiency Video Coding (HEVC) standards or the more recent Versatile Video Coding (VVC) standard define a profile for the encoding of images (also denoted pictures) and describes specific tools for compressing single images or sequences of images, including the partitioning of such images into image or picture portions e.g. subpictures, slices, or tiles. An extension of the ISO Base Media File Format (ISOBMFF) used for such kind of image data is described into the ISO / IEC 23008 standard, in Part 12, under the name: “HEIF” or “High Efficiency Image File Format" or “Image File Format”. In addition, a multimedia application format has been defined into the ISO / IEC 23000 standard, in Part 22, under the name “MIAF” or “Multi-Image Application Format” which enables precise interoperability points for the creation, reading, parsing and decoding of images embedded in the HEIF format.

[0057] The HEIF and MIAF standards cover two forms of storage corresponding to different use cases: the storage of image sequences, which can be indicated to be displayed as a timed sequence or by other means, and in which the images may be dependent on other images, and the storage of a single coded image or a collection of independently coded images, possibly with derived images. In the first case, the encapsulation is close to the encapsulation of the video tracks in the ISO Base Media File Format (see document « Information technology — Coding of audiovisual objects — Part 12: ISO base media file format», w23485, ISO / IEC 14496-12, Eighth edition, January 2024), and similar tools and concepts are used, such as the ‘trak’ boxes and the sample grouping for description of groups of samples. The ‘trak’ box is a file format box that contains sub boxes for describing a track, that is to say, a timed sequence of related samples.

[0058] In the second case, a set of ISOBMFF boxes dedicated to the description of untimed data is used. These boxes are typically declared within the ‘meta’ box hierarchy and relate to “information items” or “items” instead of related samples in a video. It is to be noted that the wording ‘box’ and the wording ‘container’ may be both used with the same meaning to refer to data structures that contain metadata describing the organization or / and properties of an image and its data in the file.

[0059] Another extension of the ISO Base Media File Format (ISOBMFF) and of the Image File Format (HEIF) is also described into the ISO / IEC 23001 standard, in Part 17, for the carriage of uncompressed video and images, including but not limited to monochromatic data, colour data, transparency (alpha) information and depth information, and including the description of uncompressed image or picture organized as a set of one or more rectangular, non-overlapping and contiguous (without holes) areas called tiles.

[0060] In the following, a coded image designates the representation of either an image encoded to reduce the size of data according to an encoding standard (e.g. JPEG, AV1 , HEVC, VVC...) or an image for which the pixels are arranged according to an uncompressed image format (e.g. ISO / IEC 23001-17).

[0061] Figure 1 illustrates an example of a media file 101 that contains media data like the data of one or more images and possibly one or more video and / or one or more sequences of images. This file contains a first ‘ftyp’ box (FileTypeBox) 111 that contains an identifier of the type of file (typically a set of four-character codes, also denoted brands). This file contains a second box called ‘meta’ (MetaBox) 102 that is used to contain general untimed metadata including metadata structures describing the one or more images. This 'meta' box 102 contains an ‘iinf’ box (ItemlnfoBox) 121 that describes several single images or untimed data (e.g. Exif or XMP metadata). Each single image or untimed data is described by a metadata structure ItemlnfoEntry also denoted items 1211 and 1212. An item corresponds to data that does not require timed processing, as opposed to sample data. An item corresponding to image data is also denoted image items. An image item is an item whose data is a coded image or a derived image. A derived image is a representation of an image as an operation on other images (also denoted input images). Each item has a unique 16-bit or 32-bit identifier item_ID. The media data corresponding to these items is stored in the container for media data, the MediaDataBox ‘mdat’ 104, or possibly in an identified media data box ‘imda’ (not represented), or possibly in an ItemDataBox ‘idat’ 126 in the MetaBox ‘meta’ 102. An Hoc’ box (I tern Location Box) 122 provides for each item a set of offsets and lengths of its associated media data in the ‘mdat’ box 104 or in the ItemDataBox ‘idat’ 126. An ‘iref’ box (ItemReferenceBox) 123 may also be defined to describe the association of one item with other items via typed references.

[0062] Optionally, for describing the storage of image sequences or video, the media file 101 may contain a third box called ‘moov’ (MovieBox) 103 that describes one or more image sequences or video tracks 131 and 132. Typically, the track 131 may be an image sequence ('pict') track designed to describe a set of images for which the temporal information is not necessarily meaningful and 132 may be a video ('vide') track designed to describe video content. Both tracks describe a series of image samples, an image sample being a set of pixels captured at the same time, for example a frame of a video sequence. The main difference between the two tracks is that in 'pict' tracks the timing information is not necessarily meaningful whereas for 'vide' tracks the timing information is intended to constraint the timing of the display of the samples. The data corresponding to these samples is stored in the container for media data, the ‘mdat’ box (MediaDataBox) 104 or in a ‘imda’ box (IdentifiedMediaDataBox, not represented).

[0063] The 'mdat' container 104 stores the media data (e.g. data or bitstream of a coded or uncompressed image, Exif metadata, etc...) corresponding to items as represented by the data portions 141 and 142 and the timed coded or uncompressed images corresponding to samples as represented by the data portion 143.

[0064] A media file 101 offers different alternatives to store multiple images. For instance, it may store the multiple images either as items or as a track of samples that can be a ‘pict’ track or a ‘vide’ track. The actual choice is typically made by the application or device generating the file according to the type of images and the contemplated usage of the file.

[0065] Several alternatives can be used to group samples or items depending on the container that holds the samples or items to group. These alternatives can be considered as grouping data structures or grouping mechanism, i.e., boxes or data structures providing metadata describing a grouping criterion and / or group properties and / or group entities.

[0066] A grouping mechanism represented by an EntityToGroupBox is adapted for the grouping of items or tracks. In this mechanism, the wording ‘entity’ is used to refer to items or tracks or other EntityToGroupBoxes. This mechanism specifies the grouping of entities. An EntityToGroupBox comprises a grouping_type, a groupjd and a list of entityjds.

[0067] The grouping_type is used to specify the type of the group. The groupjd provides an identifier for the group of entities. The entityjd represents the identifier of entities that compose the group, i.e. , either a trackJD for a track, an item J D for an item or another groupjd for an entity group. In Figure 1 , the groups of entities inheriting from the EntityToGroup box 1241 and 1242 are comprised in the container 124 identified by the four characters code ‘gprl’ for GroupsListBox.

[0068] Entity grouping consists in associating a grouping type which identifies the reason of the grouping of a set of items, tracks or other entity groups. In this document, it is referred to Grouping Information as information in one of the EntityToGroup Boxes which convey information to group a set of images.

[0069] Properties may be associated with items. These properties are called item properties and derived from a data structure ItemProperty or ItemFullProperty characterised by a 4CC also denoted property ype. The ItemPropertiesBox ‘iprp’ 125 enables the association of any item with an ordered set of item properties. The ItemPropertiesBox consists of two parts: an item property container box ‘ipco’ 1251 (“ItemPropertyContainerBox") that contains an implicitly indexed list of item properties 1253, and an item property association box ‘ipma’ 1252 (“ItemPropertyAssociation”) that contains one or more entries. Each entry in the item property association box associates an item with its item properties. When a specific brand ‘unit’ is defined in the media file, it indicates a unified handling of identifiers in the media file, for example across tracks, track groups, entity groups, and file-level MetaBoxes. When this brand is defined, it is possible to associate item properties with items and / or entity groups. Note that in the description, for genericity, we generally use item properties to designate both properties of an item or properties of an entity group. An item property associated with an entity group applies to the entity group as a whole and not individually to each entity within the group.

[0070] The ItemProperty and ItemFullProperty boxes are designed for the description of an item property. ItemFullProperty allows defining several versions of the syntax of the box and may contain one or more parameters whose presence is conditioned by either the version or the flags parameter.

[0071] The ItemPropertyContainerBox ‘ipco’ is designed for describing a set of item properties as an array of ItemProperty boxes or ItemFullProperty boxes.

[0072] The ItemPropertyAssociation box ‘ipma’ is designed to describe the association between items and / or entity groups and their item properties. It provides the description of a list of item identifiers and / or entity group identifiers, each identifier (itemJD) being associated with a list of item property index referring to an item property in the ItemPropertyContainerBox.

[0073] The HEIF and MIAF standards also enable to annotate regions from an image. The description of one or more region is realized in a “region item”. This region item can be associated with an image item through an item reference of type ‘cdsc’. Annotations can be associated with the regions described by a region item by either associating other items with the region item or by associating item properties with the region item.

[0074] In the following, the description targets HEIF or MIAF files only as an example. Other types of files following the ISOBMFF standard or similar structure may also take advantage of the present disclosure.

[0075] Figure 2a illustrates an example of region items describing regions of an image in an ISOBMFF-based media file.

[0076] An image 200 is for example captured by a device such as a camera, received from a communication device or read from a storage device. Objects or meaningful regions can be identified in the image 200, e.g., by a user, an image analysis service or a deep-learning-based system. The objects or meaningful regions correspond to the regions 210 and 220 in the image 200. For instance, a meaningful region can be a road or a building in an image representing a city map, a part of an object, or a person in an image representing a group of persons...

[0077] This image 200 is encapsulated in an ISOBMFF-based media file 101 , for instance compliant with the HEIF standard. An item 230 contained in ItemlnfoBox 121 , also denoted image item in HEIF standard, describes the image 200 with an item identifier (itemJD) and a type (item_type), typically a four-character code, identifying the data format of the image (e.g. ‘hvcT for HEVC, ‘vvcT for WC...). The image 200 may be a coded image, possibly partitioned or tiled comprising coded tiles or subpictures. The data of the image is stored in a MediaDataBox ‘mdat’ 104 as described by an I tern Location Box Hoc’ 122. In a variant, the image 200 may be composed of a plurality of images or spatially split in different images forming a grid of images organised in a given grid order without gap or overlap. According to the variant, the image 200 is described by a specific item 230, called a derived image item, with an item_type value of 'grid', also denoted a derived grid image item, indicating that the reconstructed image of the item is formed from input images of one or more input image items in a given grid order within a larger canvas. The input images are inserted in row-major order, top-row first, left to right, in the order of an item reference of type 'dimg' from this derived image item to the input image items, the item reference being defined within the ItemReferenceBox ‘iref’ 123. The data (also denoted the body of the item) associated with the grid derived image item and located via the item Location Box Hoc’ 122 comprises the definition of the grid layout (e.g. the number of rows and columns and the size (output_width and output_height) of the reconstructed image 200.

[0078] The regions 210 and 220 of the image 200 are described by items 240 and 250, with an item_type value of 'rgan', also denoted region items, indicating that the item defines one or more regions of an image. The data or payload of the region items 240 and 250 is a data structure describing the geometries of the regions 210 and 220 and may be stored preferably in an ‘idat’ box 126. It may also be stored in either an ‘mdat’ 104 or an ‘imda’ box. The geometries define the shape, position and size of the regions inside a reference space (defined by two parameters reference_width and reference_height of the data structure) that is mapped to the image after applying an implicit resampling caused by the difference between the size of the reference space and the size of the image. The geometry of a region can be represented either by a point, a polyline, a rectangle, an ellipse, a polygon, or a mask, the mask being stored in the data structure or stored in another image item referenced by the region item with an item reference of type 'mask'.

[0079] The region items 240 and 250 are associated with the image item 230 inside which the regions are defined using an item reference of type 'cdsc' from the region item to the image item.

[0080] Each region item 240 or 250 can be associated with an annotation, also denoted region annotation (not represented). A region annotation corresponds to metadata or data representing an annotation associated with the region of the image described by the region item.

[0081] A region annotation may be associated with one or more regions of an image described by a region item by associating any or all of the following with the region item: descriptive image properties, using the Item PropertyAssociation Box ‘ipma’; metadata items, using an item reference of type 'cdsc' from the metadata item to the region item; image items or, when the 'unit' brand is used, an entity group, using an item reference of type 'eroi' from the region item to the image item or entity group. The data structure corresponding to the data of a region item can be described as follows: aligned ( 8 ) class Regionitem { unsigned int ( 8 ) version = 0 ; unsigned int ( 8 ) flags ; unsigned int field size = ( ( flags & 1 ) + 1 ) * 16 ; / / this is a temporary, non-parsable variable unsigned int ( field size ) reference width; unsigned int ( field size ) reference height ; unsigned int ( 8 ) region count ; for ( r=0 ; r < region count ; r++ ) {

[0082] RegionGeometryStruct (version, flags , 0 ) region geometry;

[0083] }

[0084] }

[0085] Where version is equal to 0.

[0086] (flags & 1) equal to 0 specifies that the length of fields defined using field_size is 16 bits, (flags & 1) equal to 1 specifies that the length of fields defined using field_size is 32 bits., reference_width, reference_height specify, in pixel units, the width and height, respectively, of the reference space on which the regions are placed, region_count specifies the number of regions defined in the region item, region_geometry specifies the geometry of a region as defined in ISO / IEC 23008- 12 (HEIF).

[0087] A media file 101 corresponding to the encapsulation illustrated by Figure 2a may be represented with the following structure in which the image 200 is illustrated by a coded HEVC ‘hvcT image and each region item ‘rgan’ 240 and 250 is associated with an annotation described by an item property UserDescriptionProperty:

[0088] FileTypeBox ' ftyp ' : maj or-brand= ' mif 1 ' , compatible-brands= ' mif 1 , heic ' MetaBox 'meta' : (container)

[0089] HandlerBox ' hdlr ' : 'pict'

[0090] PrimaryltemBox 'pitm' : item ID=1;

[0091] Itemlnf oBox 'iinf' : entry count=3

[0092] ' inf e ' : item ID=1 item type='hvcl'; / / image item 230

[0093] ' inf e ' : item ID=2 item type='rgan'; / / region item 240

[0094] ' inf e ' : item ID=3 item type='rgan'; / / region item 250

[0095] ItemLocationBox ' iloc ' : item count=3 item ID=1 extent count=l extent of f set=P0 extent length=Q0; item ID=2 extent count=l extent offset=Pl extent length=Ql; item ID=3 extent count=l extent offset=P2 extent length=Q2;

[0096] ItemRef erenceBox ref erenceType= ' cdsc ' from item ID=2 reference count=l to item ID=1; ref erenceType= ' cdsc ' from item ID=3 reference count=l to item ID=1;

[0097] ItemPropertiesBox 'iprp' :

[0098] ItemPropertyContainer Box ' ipco ' :

[0099] HEVCConf igurationBox ' hvcC ' ;

[0100] ImageSpatialExtentsProperty 'ispe' ;

[0101] UserDescriptionProperty 'udes' ;

[0102] UserDescriptionProperty 'udes' ;

[0103] ItemPropertyAssociation ' ipma ' : entry count=3

[0104] 1) item ID=1 association count=2 essential=l property index=l; essential=0 property index=2;

[0105] 2) item ID=2 association count=l essential=0 property index=3;

[0106] 3) item ID=3 association count=l essential=0 property index=4;

[0107] MediaDataBox :

[0108] HEVC Image 200 (at file offset P0, with length Q0) Data ( geometry) of region 210 ( at file offset Pl , with length

[0109] QI )

[0110] Data ( geometry) of region 220 ( at file offset P2 , with length

[0111] Q2 )

[0112] Figure 2b illustrates an embodiment of region items describing regions of images in an ISOBMFF-based media file.

[0113] The image 200, the regions 210 and 220, the image item 230 and region items 240 and 250 are described as illustrated by figure 2a.

[0114] Depending on the zoom or scale factor corresponding to an image, a region in the image may actually correspond to a merge of a plurality of objects or meaningful regions or encompass a plurality of objects or meaningful regions that may be visible in another image corresponding to a different zoom or scale factor.

[0115] It is provided a solution to define an approximate region item that may describe a region of an image that may represents a merge, an approximate or a summary of at least one region of at least one another image.

[0116] This solution allows a content creator generating a media file or a parser reading a media file to differentiate a region item indicating a real or singular object or meaningful region from a region item indicating a region that represents an approximate or summary of one or more real or singular objects or meaningful regions. In this way, it is possible to adapt the parsing of the media file, according to whether a region item is an approximate region item or a regular region item.

[0117] In the present disclosure, a region item that describes a region of an image that represents an approximate, merge or summary of regions of at least one another image is denoted approximate region item. It may also be denoted merge region item or summary region item. A region item that describes a region of an image that does not represent an approximate, merge or summary of regions of another image is denoted regular region item.

[0118] In other words, an approximate region item is a region item, associated with an image item, that represents an approximate, merge or summary of one or more region items associated with one or more other image items.

[0119] For example, the image 260 may represent a same visual content, or part thereof, of the image 200 at a different zoom or scale factor. The image 260 may have a lower resolution than the image 200 or may be an alternate of the image 200. Similarly, to image 200, the image 260 is for example captured by a device such as a camera, received from a communication device or read from a storage device. The image 260 can also be generated from another image by a processing device. A region 270 can be identified in the image 260, e.g., by a user, an image analysis service or a deep-learning- based system. The image 260 may be a coded image, possibly partitioned or tiled comprising coded tiles or subpictures, or may be a grid composed of a plurality of images organised in a given grid order without gap or overlap. The region 270 represents an approximate, merge or summary of real or singular objects or meaningful regions not clearly visible in the image. In particular, in Figure 2b, the region 270 of image 260 represents an approximate, merge or summary of regions 210 and 220 of image 200.

[0120] According to an embodiment, the image 260 is encapsulated in an ISOBMFF- based media file as an image item 280. The region 270 is described by a region item 290.

[0121] In particular the region item 290 is associated with the image item 280. The region item 290 also comprises an indication that the region item is an approximate region item. This indication can be signalled by different methods.

[0122] For example, the indication that the region item is an approximate region item can be signalled as a flag value in the flags parameter of the data structure describing the geometry of the region. In particular, the data structure corresponding to the data of the approximate region item can be the one described previously for the region item, and the flag value for indicating that the region item represents an approximate or summary of singular objects or meaningful regions can be defined such that (flags & 2) equal to 2. In that case, (flags & 2) equal to 0 specifies that the region item represents a singular object or meaningful region. The values of flags greater than 2 are reserved.

[0123] In an alternative, the indication that the region item 290 is an approximate region item is signalled by the item_type value of the item, using a four-character code, different from ‘rgan’, e.g. ‘argn’ or any other four-character code available from the registration authority. An approximate region item ‘argn’ uses the same syntax of data structure for describing the geometries as for a region item ‘rgan’.

[0124] In another alternative, the indication that the region item 290 is an approximate region item is signalled by associating a dedicated item property with the region item 290. Optionally, the item property may comprise the itemJDs of the region items 240, 250 that are approximated by the region item 290. The dedicated item property may be an item property corresponding to a region annotation, e.g., a UserDescriptionProperty. In such case, the indication that the region item 290 is an approximate region item is signalled as a flag value in the flags parameter of the item property.

[0125] According to another embodiment, both the image 200 and the image 260 are encapsulated into an ISOBMFF-based media file 101 as an image item 230 and an image item 280, respectively.

[0126] The regions 210 and 220 of the image 200 are represented by the region items 240 and 250 as illustrated by figure 2a.

[0127] The region 270 of the image 260, representing an approximate, merge or summary of the regions 210 and 220 of the image 200, is described by a region item 290 (with an item_type value of 'rgan'), and the indication, that this region item 290 is an approximate region item, is by defining an item reference of type ‘base’ from the approximate region item 290 to the region items 240 and 250 corresponding to the regions approximated or summarized by the region 270.

[0128] In other words, if a region item associated with an image item represents an approximate or summary of region items associated with other image items then it is linked to those region items by item references of type ' base ' from the region item to region items of other image items it approximates from. A region item including a ' base ' item reference is referred to as an approximate region item.

[0129] This embodiment has the advantage both to indicate that a region item is an approximate region item, and to indicate the regions of another image that are actually approximated or summarized by the approximate region item. In this case, it is possible to verify that a region item 290 is an approximate region item without reading this region item 290. Therefore, the processing of the region items can be accelerated.

[0130] An approximate region item may be associated with region annotations. The region annotations associated with region items referenced by an approximate region item through a 'base' item reference also apply to this approximate region item. For instance, a player can follow the 'base' item references of an approximate region item to retrieve and possibly render the region annotations associated with the referred region items as part of the region annotations associated with this approximate region item.

[0131] In a variant, the embodiments in which the indication that the region item is an approximate region item is signalled using a flag in the data structure, a specific item type (e.g. ‘argn’), a dedicated item property or an item reference of type ‘base’ can be combined.

[0132] An approximate region item that is associated with another region item, e.g. through an item reference of type ‘base’, can inherit from the region annotations associated with the referenced another region item. In other words, the region annotations associated with a given region item applied to the given region item, but also to the approximate region item that references the given region item.

[0133] Therefore, a media file 101 corresponding to the encapsulation illustrated by Figure 2b may be represented with the following structure in which the image 200 and 260 are illustrated by a coded HEVC ‘hvcT image and each region item ‘rgan’ 240, 250 and 290 is associated with an annotation described by an item property UserDescriptionProperty:

[0134] FileTypeBox 'ftyp' : maj or-brand= 'mif 1 ' , compatible-brands= 'mif 1, heic '

[0135] MetaBox 'meta' : (container)

[0136] HandlerBox ' hdlr ' : 'pict'

[0137] PrimaryltemBox 'pitm' : item ID=1;

[0138] ItemlnfoBox 'iinf' : entry count=4

[0139] 1 ) ' inf e ' : item ID=1, item type='hvcl'; / / image item 230

[0140] 2 ) ' inf e ' : item ID=2, item type='rgan'; / / region item 240

[0141] 3 ) ' inf e ' : item ID=3, item type='rgan'; / / region item 250

[0142] 4 ) ' inf e ' : item ID=4, item type='hvcl'; / / image item 280

[0143] 5 ) ' inf e ' : item ID=5, item type='rgan'; / / region item 290

[0144] ItemLocationBox loc' : item count=5 item ID=1, extent count=l, extent offset=P0, extent length=Q0; item ID=2, extent count=l, extent offset=Pl, extent length=Ql; item ID=3, extent count=l, extent offset=P2, extent length=Q2; item ID=4, extent count=l, extent offset=P3, extent length=Q3; item ID=5, extent count=l, extent offset=P4, extent length=Q4;

[0145] ItemRef erenceBox ' iref ref erenceType= ' cds ' , from item ID=2, reference count=l, to item ID=1; ref erenceType= ' cds ' , from item ID=3, reference count=l, to item ID=1; ref erenceType= ' cdsc ' , from item ID=5, reference count=l, to item ID=4; ref erenceType= ' base ' , from item ID=5, reference count=2 , to item ID=2, 3;

[0146] ItemPropertiesBox 'iprp' :

[0147] ItemPropertyContainerBox ' ipco ' :

[0148] HEVCConf igurationBox 'hvcC ;

[0149] ImageSpatialExtentsProperty ' ispe ' ;

[0150] UserDescriptionProperty 'udes';

[0151] UserDescriptionProperty 'udes';

[0152] ImageSpatialExtentsProperty ' ispe ' ;

[0153] ImageSpatialExtentsProperty ' udes ' ;

[0154] ItemPropertyAssociation ' ipma ' : entry count=3

[0155] 1) item ID=1, association count=2 , essential=l, property index=l; essential=0, property index=2;

[0156] 2) item ID=2, association count=l, essential=0, property index=3;

[0157] 3) item ID=3, association count=l, essential=0, property index=4;

[0158] 4) item ID=4, association count=2 , essential=l, property index=l; essential=0, property index=5;

[0159] 5) item ID=5, association count=l, essential=0, property index=6;

[0160] MediaDataBox :

[0161] HEVC Image 200 (at file offset P0, with length Q0)

[0162] Data (geometry) of region 210 (at file offset Pl, with length QI)

[0163] Data (geometry) of region 220 (at file offset P2, with length Q2)

[0164] HEVC Image 260 (at file offset P3, with length Q3 )

[0165] Data (geometry) of region 270 (at file offset P4, with length Q4)

[0166] In an alternative, the approximate region item 290 can be described as a derived region item with a dedicated item_type, e.g. ‘argn’, and with an item reference ‘drgn’ from the approximate region item 290 to the region items 240 and 250 corresponding to the regions approximated or summarized by the region 270. The data or payload of the item 290 provides a description of the geometry of the region 270.

[0167] In another alternative, the items 240, 250 and 290 are described as region items with item_type ‘rgan’ whose data describes the geometry of the regions 210, 220 and 270, respectively. These region items are grouped in an entity group (EntityToGroupBox) comprising a groupjd, a grouping_type and a list of identifiers of entities belonging to the group, the region item 290 being the first entity of the group followed by the other region items 240 et 250 with a grouping_type indicating that the first entity of the group (i.e. region item 290) represents an approximate or summary of the other entities in the group (i.e. region items 240 and 250).

[0168] It can be noted that in all embodiments of this disclosure, an approximate region item can represent an approximate or summary of approximate region items or a set of approximate region items and region items.

[0169] Figure 3 illustrates another embodiment of region items describing regions of an image pyramid in an ISOBMFF-based media file.

[0170] According to this embodiment, an overview image, 301 or 302, is defined as a grid derived image item or a tiled pre-derived coded image item, 311 or 312, whose reconstructed image is formed from generating a lower resolution, ‘binned’ version of the image 303 of a base image item 313. The base image item 313 is also a tiled image item. The tiling may be implemented using a feature of a specific codec, or by using a grid derived image item. When a grid derived image item is used, the input items to the grid define the tiles. Derived image items should not be used as inputs to the image grid, due to the need for in place byte range accessing of content. Individual tiles are written contiguously in memory, thereby allowing access with a single read or write action.

[0171] An overview image, 301 or 302, is tiled using the same tiling scheme as the base image 303, i.e. if tiles in the base image are X by Y pixels, they are X by Y pixels in the tiles of an overview image. In cases where the binned resolution results in a fractional, or incomplete tile at the end of a row (column), the last tile in a row (column) of tiles is padded with the value zero at the end of the row (column) to complete the last tile in the row (column). If necessary, the clean aperture transformative property ('clap') may be applied to crop padded rows and / or columns. The number of tiles in a row (column) of tiles is determined by dividing the width (height) of the overview image by the tile size in X (tile size in Y) and rounding up. It is noted that the exact derivation process (approaches such as the sum, average, median, minimum, or maximum value of a binned region) used to produce an overview from the base image is left to the implementer.

[0172] An image pyramid groups a set of image items, formed as a base image item and a series of progressively binned overview image items. In other words, an image pyramid is generated by stacking a set of image items, formed as a base image item 313 and a series of progressively binned overview image items 311 and 312, which together form an image pyramid and creating an entity group ImagePyramidEntityGroup 300 of grouping_type 'pymd' extending an EntityToGroupBox. This entity group contains entityjd values that point to a base image item and a set of overview image items and contains no entityjd values that point to tracks. The entities are listed in the order of lowest resolution overview image item to the highest resolution overview image item, followed finally by the base image item of the image pyramid. The amount of binning of each overview image is indicated in the ImagePyramidEntityGroup The image format of the overviews is the same as the base image item. i.e. number of bands, bit depth, color format, etc.

[0173] An image pyramid ‘pymd’ may be defined as extending an EntityToGroupBox as follows: aligned ( 8 ) class ImagePyramidEntityGroup extends EntityToGroupBox ( ' pymd ' , version = 0 , flags = 0 ) { unsigned int ( 16 ) tile size x; unsigned int ( 16 ) tile size y; for ( i=0 ; i<num entities in group; i++ ) { unsigned int ( 16 ) layer binning; unsigned int ( 16 ) tiles in layer row minus l ; unsigned int ( 16 ) tiles in layer column minusl ;

[0174] } } Where: num_entities_in_group is as defined for EntityToGroupBox. In addition, it also specifies the number of layers of the image pyramid. tile_size_x, tile_size_y indicate the size in pixels of a tile in the width and height dimension, respectively, for all layers of the image pyramid.

[0175] Iayer_binning indicates for each layer of the pyramid the level of binning between the base image and the overview image. A 2x2 binning is defined to be a layer_binning of 2, a 4x4 binning is defined to be 4, etc. The width and height for an overview image with layer_binning of 2 are half the width and half the height of the base image respectively, etc. A base image has a layer_binning of 1. tiles_in_layer_row_minus1 , tiles_in_layer_column_minus1 indicate the number of tiles minus one in a row and a column, respectively, of a specific layer. If the layer is represented by a grid derived image item, tiles_in_layer_row_minus1 is equal to rows_minus_one and tiles_in_layer_column_minus1 is equal to columns_minus_one. If the layer is represented by a tiled pre-derived coded image item associated with a ConstrainedExtentsGridProperty, then tiles_in_layer_row_minus1 is equal to rows_minus_one and tiles_in_layer_column_minus1 is equal to columns_minus_one.

[0176] In a variant, some parameters may not be present in the ImagePyramidEntityGroup data structure such as layer_binning, tiles_in_layer_row_minus1 or tiles_in_layer_column_minus1. For instance, the grid layout and the binning associated with each image in the pyramid can be deduced as follows: tiles_in_layer_row = round_up(image_width / tile_size_x); tiles_in_layer_column = round_up(image_height / tile_size_y); binning_width = base_image::image_width / overview::image_width binning_height = base_image::image_height / overview::image_height where tiles_in_layer_row: Signals the number of tiles in a row. tiles_in_layer_column: Signals the number of tiles in a column. binning_width and binning_height Specifies the level of binning between the base image and an image overview in the pyramid. tile_size_x and tile_size_y parameters may also not be present in the ImagePyramidEntityGroup data structure. In such case, when a layer of the image pyramid is represented by a grid derived image item, tile_size_x and tile_size_y values can be deduced from the size of the input image items of the grid derived image item. When a layer of the image pyramid is represented by a tiled pre-derived coded image item associated with a ConstrainedExtentsGridProperty, tile_size_x and tile_size_y values can be deduced from the ConstrainedExtentsGridProperty.

[0177] It is noted that all the entities of a same image pyramid entity group, or only some of them, can also be members of a same entity group of type 'prgr' if they are stored in the file for allowing a progressive refinement. They can also be members of a same entity group of type 'altr' if they are proposed by the content creator as alternatives to be displayed for players not supporting the image pyramid entity group.

[0178] Region items may be explicitly associated with each layer of the image pyramid by defining an item reference 'cdsc' between each layer and a same region item. But when the number of region items increases, the number of 'cdsc' item reference also increase. Alternatively, as explained hereafter, a region partition group can be used to lists the region items that are contained inside an area of an image of an image pyramid to reduce the number of region items to process when only an area of an image of the image pyramid is displayed.

[0179] In other words, a region item may be associated with an image item within an ImagePyramidEntityGroup either:

[0180] - through an item reference of type 'cdsc' from the region item to the image item;

[0181] - through a RegionPartitionGroupBox associated with the image item via an item reference of type 'rpds' and referencing the itemJD of the region item.

[0182] A region item associated with a base image or an image overview within a same ImagePyramidEntityGroup may be applied to the output image of any image item within this ImagePyramidEntityGroup by applying the implicit resampling caused by the difference between the reference space of the region item and the size of the image. For instance, a player can use the item reference of type 'base' of an approximate region item to filter the region items that are inherited from other image items in the ImagePyramidEntityGroup.

[0183] Region items 321 , 322, 323 describing regions in the base image 303 and associated with the base image item 313 may also be applied to each individual image overviews by applying an implicit resampling caused by the difference between the size of their reference space and the size of the image overview.

[0184] In other words, in an image pyramid entity group, thanks to the implicit resampling applied to the geometry of a region item caused by the difference between the size of its reference space and the size of the image, the region items associated with a base image could also be applied or rendered for any image overview of the image pyramid even if not directly associated with this image with a 'cdsc' item reference.

[0185] For instance, the images 301 , 302, 303 contains the same complete set of objects even if they may blur together while the resolution of the images in the image pyramid is decreasing. The region item 324 related to the overview image 302 can be seen as an approximate or summary of region items 322 and 323 of the base image 303.

[0186] Therefore, any image item 311 or 312, describing an image overview, can inherit from region items associated with another image item describing an image with higher resolution in the image pyramid, even if there is no direct association, e.g., through an item reference of type ‘cdsc’. In variants, in an entity group comprising alternative images, e.g. in an alternate entity group of type ‘altr’, in a progressive rendering entity group of type ‘prgr’, or in a bracketing set entity group, any image item of the group can inherit from region items associated with another image item describing an image with the same or higher resolution in the group, even if there is no direct association between the region item and the image item.

[0187] An image item representing an image overview can be associated with one or more regular region items and / or with one or more approximate region items as described by reference to figure 2b. The approximate region items can be used to describe a region in an image overview that represents an approximate, merge or summary of one or more region items associated with an item representing a higher resolution image such as the base image item.

[0188] For example, in Figure 3, the image item 312 representing an image overview 302 can be associated with an approximate region item 324

[0189] The approximate region item 324 related to the image overview 302 and associated with the image item 312 with an item reference of type ‘cdsc’ represents an approximate or summary of the region items 322 and 323 describing a singular object or meaningful region of the base image 303. Similarly, the image item 311 representing an image overview 301 can be associated with a regular region item 326 and with an approximate region item 325. The approximate region item 325 related to the image overview 301 represents an approximate or summary of the region items 322 and 323 describing a singular object or meaningful region of the base image 303.

[0190] Such approximate region items can be useful for filtering inherited region items that applies to any image of the pyramid but can become too small or may merge when a user zooms out within the image pyramid.

[0191] Figure 4 illustrates the association of region items and approximate region items with image items.

[0192] A group of items 400 comprises a plurality of image items, e;g. 401 and 402. The group of items 400 may represent an image pyramid entity group ‘pymd’, an alternate entity group ‘altr’, a progressive rendering entity group ‘prgr’, a bracketing set entity group, e.g. ‘aebr’, ‘wbbr’..., or any other type of entity group comprising image items whose reconstructed images represent the same visual content or part thereof, possibly at different resolutions. For instance, the image item 401 may represent the base image of an image pyramid, and the image item 402 may represent on image overview of the image pyramid. Each image item 401 and 402 can be either a coded image item, possibly tiled, or a derived image item such as a grid or overlay derived image item.

[0193] The region items and approximate region items describing regions and associating annotations to image items 401 and 402 are associated with the image item to which they applied through an item reference of type ‘cdsc’. For instance, the region items 410, 411 are associated with the image item 401. The region item 412 is associated with the image item 401 and also with the image item 402. The region item 415 is associated with the image item 402. In an embodiment, the item 413 is an approximate region item associated with the image item 402 and characterized by the item reference of type ‘base’ to region items 410 and 411 .

[0194] Moreover, the regions items and / or approximate region items applying to an image item can be grouped into different region partition groups depending on the area of the image on which they actually apply.

[0195] A region partition groups region items that are contained inside an area of an image. In other words, a region partition group, 421 to 425, is an entity group of type ‘rgpa’ that lists the region items that are contained inside an area of an image (illustrated by the numbers 1 to 4 in the image item 401). This area is the area covered by the region partition group and is defined as a rectangle in the image. For instance, the region partition group 421 describes the rectangle area 1 in the image item 401 , indicating that the region items 410 and 411 describe regions located in the rectangle area 1. The region partition group 422 describes the rectangle area 2 in the image item 401 , and the region partition group 423 describes the rectangle area 3 in the image item 401 . The region partition group 424 indicates that the region item 412 describes a region in the rectangle area 4 or that intersects the rectangle area 4 in the image item 401. The region partition group 425 indicates that the region item and the approximate region item describe regions in an area of the image item 402, or possibly in the whole image item 402.

[0196] The region partition group is associated with the image item containing the region items it groups through an item reference of type 'rpds' from the image item to the region partition group. In other words, the region items can be grouped in a region partition group associated with an image item via a 'rpds' item reference.

[0197] The area covered by a region partition group is determined depending on a flags value of the region partition group and either is defined in the region partition group structure itself or is the area of the image item the region partition group is associated with through an item reference of type 'rpds'. The region partition group can be used by a player or parser to filter the region items to process depending on the area of an image actually displayed.

[0198] A region partition group ‘rgpa’ may be defined as extending an EntityToGroupBox as follows: aligned ( 8 ) class RegionPartitionGroupBox extends EntityToGroupBox ( ' rgpa ' , version, flags ) { unsigned int field size = ( ( flags & 0xl 000 ) »12 + 1 ) * 16 ; if ( ( flags & 0x2000 ) »12 == 2 ) { unsigned int ( field size ) reference width; unsigned int ( field size ) reference height ; unsigned int ( field size ) top; unsigned int ( field size ) left ; unsigned int ( field size ) width; unsigned int ( field size ) height ;

[0199] }

[0200] }

[0201] Where:

[0202] The following flags values are defined to describe the area covered by a group of region items contained in a region partition group:

[0203] - (flags & 0x1000), when set specifies that the length of the fields reference_width, reference_height, top, left, width, and height is 32 bits. When not set, it specifies that the length of the fields reference_width, reference_height, top, left, width, and height is 16 bits.

[0204] - (flags & 0x2000), when set, specifies that the fields reference_width, reference_height, top, left, width, and height are present. When not set, it specifies that the fields reference_width, reference_height, top, left, width, and height are not present. In this case, the area covered by the region partition group is the area of the image it is associated with. reference_width, reference_height specify, in pixel units, the width and height, respectively, of the reference space in which the region partition group is specified. top, left specify the coordinates of the top-left corner of the area covered by the region partition group relatively to the reference space. width, height specify the coordinates of the width and the height of the area covered by the region partition group relatively to the reference space. Still according to this embodiment, only the region items that are explicitly associated with an image item through an item reference of type ‘cdsc’ can apply to the image item, either directly or as part of a region partition group associated with the image item.

[0205] The advantage of this embodiment is that all the associations are explicit in the media file, but the number of associations or item references of type ‘cdsc’ may become very large when the number of region items increases.

[0206] In a variant, when a region item belongs to or is referred by a region partition group associated with an image item, the region item is not associated anymore with the image item through an item reference of type ‘cdsc’. Therefore, the region item or approximate region item is associated with the image item inside which the regions are defined using either an item reference of type 'cdsc' from the region item to the image item, or through a region partition group referring to the region item and associated with the image item via an item reference of type ‘rpds’ from the image item to the region partition group.

[0207] According to this variant, only the region items that are explicitly associated with an image item through an item reference of type ‘cdsc’, or through a region partition group referring to the region item and associated with the image item via an item reference of type ‘rpds’, can apply to or be processed with the image item.

[0208] This variant allows reducing the number of item references of type ‘cdsc’ when the region items or approximate region items belongs or are referred by a region partition group.

[0209] In another variant, rather than explicitly associating the region items that apply to an image item via an item reference of type ‘cdsc’ or through a region partition group, an image item can inherit from the region items associated with another image item of the group of items 400. For instance, the image item 402 can inherit from the region items 410 to 412 associated with the image item 401 by applying an implicit resampling caused by the difference between the size of their reference space and the size of the image item 402.

[0210] Preferably, an image item may inherit from the region items associated with another image item of the group of items 400 if the another image item has a higher or equal resolution. For example, image item 402 can inherit from all region items associated with the image item 401 , including the region items 410, 411 and possibly other region items (not represented), directly associated with the image item 401 via an item reference of type ‘cdsc’, or through the region partition groups 421 , 422, 423 and 424.

[0211] If the image item, that inherits region items from another image item, is associated with an approximate region item, then the region items that are referred by the approximate region item are removed from the inherited region items if present in the inherited region items. For example, image item 402 is associated with approximate region item 413 and inherits all region items associated with the image item 401 , including region items 410 and 411. This approximate region item 413 has item references to region items 410 and 411. Thus, region items 410 and 411 are removed from the inherited region items. This allows avoiding processing the inherited region items 410 and 411 because they are already summarized or approximated by the approximate region item 413. In a variant, this allows avoiding processing at least the geometry of the inherited region items 410 and 411 , but the region annotations associated with the region items 410 and 411 may still be processed as inherited region annotations for the approximate region item 413.

[0212] According to this another variant, the region items that are explicitly associated with an image item through an item reference of type ‘cdsc’, or through a region partition group and the inherited region items after filtering of the approximated region items, can apply to or be processed with the image item.

[0213] Figure 5 illustrates the main steps of a method of encapsulating images in a media file.

[0214] In a step 500 an image is obtained. The image may be uncompressed according to a raw format like, for example, RGB, YUV, or compressed according to a compression format like, for example, JPEG, HEVC or VVC. The image may be partitioned according to a regular, or non regular, grid of rectangular portions, e.g. tiles or subpictures. When the image is compressed, the compressions format may support that the image portions are “extractable” or “independently decodable”.

[0215] In a step 501 , a single image item is generated for encapsulating the image with an item type corresponding to the coding format or uncompressed format of the image. In a variant, a derived image item is generated for describing the grid of rectangular portions, and each rectangular portion encapsulated as an image item associated with the derived image item with an item reference of type ‘dimg’. The rectangular portion may be compressed according to a compression format before being encapsulated. In a step 502, the geometry of a region of the image is obtained. The geometry can be obtained from a user annotating the image, from an image analysis service or a deep-learning-based system.

[0216] In a step 503, it is determined whether the region corresponds to a singular object or meaningful region or possibly corresponds to an approximate, merge or summary of one or more real or singular objects or meaningful regions of another image. For instance, this step can be conducted by comparing the area corresponding to the region with the corresponding area in an image comprising same visual content at higher resolution. In another example, the information that a region is an approximate, merge or summary of a region of another image can be provided by the user, the device that captures or generates the image or by the image analysis service.

[0217] When the region corresponds to a singular object or meaningful region, a region item describing the geometry of the region is generated at step 504.

[0218] When the region corresponds to an approximate, merge or summary of one or more real or singular objects or meaningful regions, an approximate region item is generated at step 505 according to an embodiment of this disclosure illustrated by reference to figures 2b to 4. In particular, an indication that this region item is an approximate region item is embedded in or associated with the approximate region item.

[0219] Steps 504 and 505 are followed by a step 506 where the generated region item or approximate region item is associated with the generated image item, either with an item reference of type ‘cdsc’ from the region item to the image item, or through an entity group referring to the region item, the entity group being associated with the image item via an item reference, e.g. of type ‘cdsc’, ‘rpds’ or others.

[0220] In a step 507, an annotation or a plurality of annotations can be optionally associated with the generated region item or approximate region item.

[0221] An annotation, also denoted region annotation, can be associated with one or more regions described by a region item, or approximate region item, by associating any or all of the following with the region item or approximate region item: descriptive image properties, using the Item PropertyAssociation Box ‘ipma’; metadata items, using an item reference of type 'cdsc' from the metadata item to the region item or approximate region item; image items or, when the 'unit' brand is used, an entity group, using an item reference of type 'eroi' from the region item or approximate region item to the image item or entity group. Then at step 508, a loop on the regions to be described is executed corresponding to steps 502 to 507.

[0222] When steps 502 to 507 have been executed for all the regions to be described, the process includes a step 509 in which a media file, e.g. ISOBMFF-based media file, is generated encapsulating the image, the generated image item or image items, the generated region items or approximate region items, and possibly their annotations. More particularly, when the generated media file encapsulates approximate region items, the media file also encapsulates the indications embedded in or associated with the approximate region items.

[0223] Figure 6 illustrates the main steps of a method of parsing a media file comprising an image and a region item and outputting the image and the related region.

[0224] In a step 600, the image item representing the image is selected based on the metadata from the media file and the image corresponding to the image item is obtained based on the ItemLocationBox and ItemlnfoEntry from the media file.

[0225] In a step 601 , an item describing a region of the image and associated with the image item is obtained based on the item Reference Box and the item reference of type ‘cdsc’ between this item and the image item, or through an entity group associated with the image item, the entity group referring to this item. The item describing the region may be a region item, an approximate region item or any item describing a region of an image according to an embodiment of this disclosure. In particular, an indication is embedded in or associated with the item describing the region for determining whether the item is an approximate region item.

[0226] In a step 602, it is checked whether the selected item describing a region of the image corresponds to an approximate region item or to a region item or another type of item describing a region of an image. This check is performed by identifying whether the item describing the region embeds or is associated with an indication that this item is an approximate region item.

[0227] When the region corresponds to a regular region item or another type of item describing a region of an image, this item is processed at step 604. The processing may comprise, e.g., retrieving a geometry of the region described by the item and an annotation or a plurality of annotations associated with the item, providing an indication to a user that a region with an annotation is available in the image, rendering the geometry or shape of the region and possibly the associated annotations over the image. When the region corresponds to an approximate region item, this approximate region item is processed at step 605. The processing may comprise, e.g., retrieving the geometry of the approximate region item and possibly an annotation or a plurality of annotations associated with the approximate region item, providing an indication to a user that an approximate region with an annotation is available in the image, rendering the geometry of at least one region of the approximate region item with an indication that the at least one rendered region is an approximate, merge or summary region of another region, rendering the geometry or shape of the approximate region and possibly the associated annotations over the image. The processing may also comprise filtering the region items, and possibly the approximate region items, associated with the approximate region item, notably from the region items inherited by the image item associated with the approximate region item from another image item corresponding to an alternate image or to an image comprising same visual content with higher resolution as illustrated by reference to figure 4. Filtering the region items associated with the approximate region item from the inherited region items allows avoiding processing these region items, e.g. their associated geometries, that are already approximated or summarized by the approximate region item. Therefore, filtering these region items allows reducing the processing cost of handling region items applying to an image (e.g., depending on a zoom / scale level). The processing may also comprise retrieving the annotations inherited from the region items associated with the approximate region item. The retrieved annotations inherited from the region items associated with the approximate region item may then be rendered.

[0228] A player can use the item reference of type 'base' of an approximate region item to filter the region items that are inherited from other image items in an image pyramid.

[0229] It is useful for a player to distinguish a region item representing a summary or approximate of region items from other images, for instance to differentiate the rendering of the region items and to easily retrieve the annotations of region items that are summarized by an approximated region item.

[0230] Then at step 606, a loop on the items describing a region associated with the selected image item is executed corresponding to steps 601 to 605.

[0231] In a step 607, the selected image is decoded, if needed, and is outputted or processed by a player or parser device, for instance, for rendering.

[0232] Such method may allow a user to differentiate the annotations associated with a real object in an image from the annotations associated with a region representing an approximate of multiple real objects. Such method may also allow a user to render the annotation of referred region items that also apply to an approximate region item.

[0233] Such method may also allow a user to first render the approximate region items and ignore the associated referred region items.

[0234] Figure 7 is a schematic block diagram of a computing device 90 for implementation of one or more embodiments of the present disclosure. The computing device 90 may be a device such as a microcomputer, a workstation or a light portable device. The computing device 90 comprises a communication bus connected to: a central processing unit 91 , such as a microprocessor, denoted CPU; a random-access memory 92, denoted RAM, for storing the executable code of the method of encapsulating images in a media file and / or parsing a media file according to embodiments of the present disclosure as well as the registers adapted to record variables and parameters necessary for implementing the method according to embodiments of the disclosure, the memory capacity thereof can be expanded by an optional RAM connected to an expansion port, for example; a read-only memory 93, denoted ROM, for storing computer programs for implementing the method of encapsulating images in a media file and / or parsing a media file according to embodiments of the disclosure; a network interface 94 is typically connected to a communication network over which digital data to be processed are transmitted or received. The network interface 94 can be a single network interface, or composed of a set of different network interfaces (for instance wired and wireless interfaces, or different kinds of wired or wireless interfaces). Data packets are written to the network interface for transmission or are read from the network interface for reception under the control of the software application running in the CPU 91 ; a user interface 95 may be used for receiving inputs from a user or to display information to a user; a hard disk 96 denoted HD may be provided as a mass storage device; an I / O module 97 may be used for receiving / sending data from / to external devices such as a video source or display.

[0235] The executable code may be stored either in read only memory 93, on the hard disk 96 or on a removable digital medium such as for example a disk. According to a variant, the executable code of the programs can be received by devices of a communication network, via the network interface 94, in order to be stored in one of the storage devices of the communication device 90, such as the hard disk 96, before being executed.

[0236] The central processing unit 91 is adapted to control and direct the execution of the instructions or portions of software code of the program or programs according to embodiments of the disclosure, which instructions are stored in one of the aforementioned storage devices. After powering on, the CPU 91 is capable of executing instructions from main RAM memory 92 relating to a software application after those instructions have been loaded from the program ROM 93 or the hard disk (HD) 96 for example. Such a software application, when executed by the CPU 91 , causes the steps of the flowcharts of the aforementioned methods to be performed.

[0237] Any step of the algorithms of the aforementioned methods may be implemented in software by execution of a set of instructions or program by a programmable computing machine, such as a PC (“Personal Computer”), a DSP (“Digital Signal Processor”) or a microcontroller; or else implemented in hardware by a machine or a dedicated component, such as an FPGA (“Field-Programmable Gate Array”) or an ASIC (“Application-Specific Integrated Circuit”).

[0238] Although the present disclosure has been described hereinabove with reference to specific embodiments, the present disclosure is not limited to the specific embodiments, and modifications will be apparent to a skilled person in the art which lie within the scope of the present disclosure.

[0239] Many further modifications and variations will suggest themselves to those versed in the art upon making reference to the foregoing illustrative embodiments, which are given by way of example only and which are not intended to limit the scope of the present disclosure, that being determined solely by the appended claims. In particular the different features from different embodiments may be interchanged, where appropriate.

[0240] Each of the embodiments of the disclosure described above can be implemented solely or as a combination of a plurality of the embodiments. Also, features from different embodiments can be combined where necessary or where the combination of elements or features from individual embodiments in a single embodiment is beneficial.

[0241] In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. The mere fact that different features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be advantageously used.

Claims

CLAIMS1. A method of encapsulating images in an ISOBMFF based media file, the method comprising, if a region of a first image corresponds to an approximate, merge or summary of at least one region of at least one second image: generating a merge region item describing the geometry of the region of the first image, generating an indication that the merge region item describes a region of the first image that corresponds to an approximate, merge or summary of the at least one region of the at least one second image, and encapsulating the images, the merge region item and the indication in the media file.

2. The method according to claim 1 , comprising: determining that the region of the first image corresponds to an approximate, merge or summary of at least one region of the at least one second image, then generating the merge region item describing the region of the first image.

3. The method according to claim 1 or 2, wherein generating the merge region item comprises: generating a geometry data structure describing the geometry of the region of the first image associated with the merge region item.

4. The method according to claim 3, wherein generating the merge region item also comprises generating at least one annotation data structure associated with the merge region item.

5. The method according to any one of claims 1 to 4, wherein the indication comprises a predefined flag value in flags parameter of the merge region item.

6. The method according to any one of claims 1 to 5, wherein the indication comprises a predefined item type value for the merge region item.

7. The method according to any one of claims 1 to 6, wherein the indication comprises a dedicated item property associated with the merge region item.

8. The method according to claim 7, wherein the dedicated item property comprises item identifiers of at least one region item describing the at least one region of the at least one second image that is approximated, merged or summarized by the region described by the merge region item.

9. The method according to any one of claims 1 to 7, wherein the indication comprises an entity group comprising a dedicated grouping type, item identifier of the merge region item and item identifiers of at least one region item describing the at least one region of the at least one second image that is approximated, merged or summarized by the region described by the merge region item.

10. The method according to any one of claims 1 to 7, wherein the indication comprises an item reference of a dedicated type from the merge region item to at least one region item describing the at least one region of the at least one second image that is approximated, merged or summarized by the region described by the merge region item.

11. The method according to any one of claims 1 to 10, comprising: generating at least one region item describing the at least one region of the at least one second image that is approximated, merged or summarized by the region described by the merge region item; associating the at least one region item to the merge region item.

12. The method according to claim 11 , wherein the merge region item inherits of annotations from the at least one region item.

13. The method according to claim 11 or 12, comprising generating an entity group comprising image items whose reconstructed images represent the same visual content or part thereof, possibly at different resolutions, the entity group comprising an image item describing the at least one second image associated with the at least one region item and an image item describing the first image associated with the merge region item.

14. A method of parsing an ISOBMFF based media file, the method comprising: obtaining an item describing the geometry of a region of a first image from the media file based on an indication indicating that the item is a merge region item, the region of the first image being an approximate, merge or summary of at least one region of at least one second image; processing the obtained merge region item.

15. The method according to claim 14, wherein the indication comprises an item reference of a dedicated type from the merge region item to at least one region item describing the at least one region of the at least one second image that is approximated, merged or summarized.

16. The method according to claim 14 or 15, wherein the processing of the merge region item includes rendering the geometry or shape of the region described by the merge region item with an indication that the rendered region is an approximate, merge or summary of the at least one region.

17. The method according to any one of claims claim 14 to 16, wherein if the merge region item is associated with an item describing the at least one region of the at least one second image, the processing of the merge region item includes retrieving and processing the annotations associated with the item describing the at least one region of the at least one second image.

18. The method according to any one of claims claim 14 to 17, wherein the first image inherits from at least one item describing a region of the at least one second image; and, if the merge region item is associated with a region item describing the at least one region of the at least one second image, the method comprises filtering the region item associated with the merge region item from inherited items of the first image.

19. The method of claim 18, wherein the first image inherits from the at least one second image if the at least one second image corresponds to an alternate image of the first image or to an image comprising same visual content with higher resolution than the first image.

20. A computer program product for a programmable apparatus, the computer program product comprising a sequence of instructions for implementing a method according to any one of claims 1 to 19, when loaded into and executed by the programmable apparatus.

21. A computer-readable storage medium storing instructions of a computer program product according to claim 20.

22. A device for encapsulating images in an ISOBMFF based media file, the device comprising a processor configured to, if a region of a first image corresponds to an approximate, merge or summary of at least one region of at least one second image: generate a merge region item describing the geometry of the region of the first image, generate an indication that the merge region item describes a region of the first image that corresponds to an approximate, merge or summary of the at least one region of the at least one second image, and encapsulate the images, the merge region item and the indication in the media file.

23. A device for parsing an ISOBMFF based media file, the device comprising a processor configured to: obtain an item describing the geometry of a region of a first image from the media file based on an indication indicating that the item is a merge region item, the region of the first image being an approximate, merge or summary of at least one region of at least one second image; process the obtained merge region item.

24. An ISOBMFF based media file comprising: a merge region item, describing the geometry of a region of a first image that corresponds to an approximate, merge or summary of at least one region of at least one second image. an indication that the merge region item describes a region of a first image that corresponds to an approximate, merge or summary of at least one region of at least one second image.

Images