Method and apparatus for encapsulating region-related annotations in image files

Abstract

The present invention relates to a method for encapsulating an image item in a file, the method comprising performing the following operations on the image item: generating an item portion annotation data structure related to a portion of the image item, the item portion annotation data structure being associated with the image item; generating a geometric data structure, the geometric data structure describing the geometry of the portion of the image item associated with the item portion annotation data structure; generating at least one annotation data structure associated with the item portion annotation data structure; and embedding the image item, the item portion annotation data structure, the geometric data structure and the at least one annotation data structure into a file.

Description

Technical Field

[0001] The present disclosure relates to methods and apparatus for encapsulating region-related information in an image file. Background Art

[0002] Modern cameras and image analysis services enable the generation of localized metadata for images. Localized metadata is metadata related to a region or part of a media content, rather than metadata related to the entire media content. The media content is usually an image, but it can also be video content or a collection of images. For example, a camera can generate a focus area of ​​a photo or detect a face when taking a picture. As another example, a deep learning system can recognize objects within an image. These localized metadata can be considered as region annotations.

[0003] For example, images captured by a camera or processed by an image analysis service are stored on a storage device such as a memory card. Images are usually encoded to reduce the size of the data on the storage device. Many encoding standards can be used, such as JPEG, AV1, or the newer HEVC standard.

[0004] The HEVC standard defines a profile for encoding still images and describes specific tools for compressing a single still image or a sequence of still images. An extension to the ISO Base Media File Format (ISOBMFF) for such image data has been proposed for inclusion in Part 12 of the ISO / IEC 23008 standard, under the name "HEIF" or "High Efficiency Image File Format".

[0005] HEIF (High Efficiency Image File Format) is a standard developed by the Moving Picture Experts Group (MPEG) for storing and sharing images and image sequences.

[0006] MIAF (Multiple Image Application Format) is a standard developed by MPEG as part of ISO / IEC 23000 standard part 22. The MIAF specification specifies a multimedia application format, Multi-Image Application Format (MIAF), which enables precise interoperability points for creating, reading, parsing and decoding images embedded in the High Efficiency Image File (HEIF) format. The MIAF specification is fully compliant with the HEIF format and only defines additional constraints to ensure higher interoperability.

[0007] While providing the ability to store documents containing metadata such as EXIF ​​or XMP documents, the HEIF and MIAF file formats do not provide a mechanism suitable for linking annotations to image regions. Summary of the invention

[0008] The present invention aims to solve one or more of the above problems.

[0009] According to a first aspect of the present invention, there is provided a method for encapsulating an image item in a file, the method comprising performing the following operations on the image item:

[0010] generating an item portion annotation data structure related to a portion of the image item, the item portion annotation data structure being associated with the image item;

[0011] generating a geometry data structure describing a geometry of the portion of the image item associated with the item portion annotation data structure;

[0012] generating at least one annotation data structure associated with the item portion annotation data structure;

[0013] The image item, the item part annotation data structure, the geometry data structure and the at least one annotation data structure are embedded in the file.

[0014] In the examples:

[0015] The item part annotation data structure is an item attribute;

[0016] The item portion annotation data structure includes the geometry data structure;

[0017] The item part annotation data structure includes reference information related to the at least one annotation data structure generated as an item attribute.

[0018] In the examples:

[0019] The item part annotation data structure is an item attribute;

[0020] The item portion annotation data structure includes the geometry data structure;

[0021] The item part annotation data structure is associated with the image item through association information in an association container, and the item part annotation data structure is associated with the at least one annotation data structure generated as an item attribute through another association information in the association container.

[0022] In the examples:

[0023] The item part annotation data structure is an item attribute;

[0024] The item portion annotation data structure includes the geometry data structure;

[0025] The item part annotation data structure is associated with the image item through association information in an association container, and the item part annotation data structure is associated with the at least one annotation data structure generated as an item attribute through the association information in the association container.

[0026] In the examples:

[0027] The item part annotation data structure is an item attribute;

[0028] The item portion annotation data structure includes the geometry data structure;

[0029] The item part annotation data structure includes the at least one annotation data structure.

[0030] In the examples:

[0031] The item part annotation data structure is an item;

[0032] The item portion annotation data structure includes the geometry data structure;

[0033] The item part annotation data structure is associated with the image item via reference information;

[0034] The item part annotation data structure is associated with the at least one annotation data structure generated as an item attribute through association information in an association container.

[0035] In the examples:

[0036] The item part annotation data structure is an item;

[0037] The item portion annotation data structure includes the geometry data structure;

[0038] grouping at least one item part annotation data structure associated with the image item, the group being associated with the image item by reference information;

[0039] The item part annotation data structure is associated with the at least one annotation data structure generated as an item attribute through association information in an association container.

[0040] In the examples:

[0041] The item part annotation data structure is an item;

[0042] The item part annotation data structure is associated with the image item via reference information;

[0043] The item part annotation data structure is associated with the at least one annotation data structure generated as an item attribute through association information in an association container;

[0044] The item part annotation data structure is associated with the geometric data structure generated as an item attribute through the association information in the association container.

[0045] In an embodiment, the method further comprises generating a supplementary item part annotation data structure, the supplementary item part annotation data structure being associated with the item part annotation data structure.

[0046] According to another aspect of the present invention, a method for reading a file including an image item is provided, the method comprising performing the following operations on the image item:

[0047] reading an item portion annotation data structure related to a portion of the image item, the item portion annotation data structure being associated with the image item;

[0048] reading a geometry data structure describing a geometry of the portion of the image item associated with the item portion annotation data structure;

[0049] reading at least one annotation data structure associated with the item portion annotation data structure;

[0050] Read the associated image item.

[0051] According to another aspect of the invention, there is provided a computer program product for a programmable device, the computer program product comprising a sequence of instructions which, when loaded into and executed by the programmable device, implements the method according to the invention.

[0052] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores instructions of a computer program for implementing the method according to the present invention.

[0053] According to another aspect of the invention, a computer program is provided which, when executed, causes the method according to the invention to be performed.

[0054] According to another aspect of the present invention, there is provided an apparatus for encapsulating an image item in a file, the apparatus comprising a processor, wherein the processor is configured to perform the following operations on the image item:

[0055] generating an item portion annotation data structure related to a portion of the image item, the item portion annotation data structure being associated with the image item;

[0056] generating a geometry data structure describing a geometry of the portion of the image item associated with the item portion annotation data structure;

[0057] generating at least one annotation data structure associated with the item portion annotation data structure;

[0058] The image item, the item part annotation data structure, the geometry data structure and the at least one annotation data structure are embedded in the file.

[0059] According to another aspect of the present invention, there is provided an apparatus for reading a file including an image item, the apparatus comprising a processor, wherein the processor is configured to perform the following operations on the image item:

[0060] reading an item portion annotation data structure related to a portion of the image item, the item portion annotation data structure being associated with the image item;

[0061] reading a geometry data structure describing a geometry of the portion of the image item associated with the item portion annotation data structure;

[0062] reading at least one annotation data structure associated with the item portion annotation data structure;

[0063] Read the associated image item.

[0064] At least part of the method according to the invention may be computer-implemented. Thus, the invention may take the form of a fully hardware embodiment, a fully software embodiment (including firmware, resident software, microcode, etc.), or a combination of software and hardware embodiments, which may all be generally referred to herein as "circuits," "modules," or "systems." Furthermore, the invention may take the form of a computer program product embodied in any tangible medium of expression having a computer-usable program code embodied in the medium.

[0065] Since the present invention can be implemented in software, the present invention can be embodied as computer readable code for providing to a programmable device on any suitable carrier medium. Tangible non-transitory carrier media may include storage media such as floppy disks, CD-ROMs, hard disk drives, magnetic tape devices, or solid-state memory devices. Transient carrier media may include signals such as electric signals, electronic signals, optical signals, acoustic signals, magnetic signals, or electromagnetic signals (e.g., microwave or RF signals). BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Embodiments of the present invention will now be described, by way of example only, and with reference to the following drawings, in which:

[0067] Figure 1 An example of a HEIF file 101 is shown, which contains media data, such as one or more still images and possibly a video or image sequence;

[0068] Figure 2a A high level view showing the present invention for associating annotations with regions of an image;

[0069] Figure 2b An example of region annotation is shown;

[0070] Figure 3 A first embodiment of the present invention is shown;

[0071] Figure 4 A second embodiment of the present invention is shown;

[0072] Figure 5 A third embodiment of the present invention is shown;

[0073] Figure 6 shows the main steps of a process for adding a new region annotation associated with an image item stored in a HEIF file when the region annotation is described by item attributes according to an embodiment of the present invention;

[0074] Figure 7 The main steps of a process for reading a HEIF file containing region annotations when the region annotations are described by item attributes according to an embodiment of the present invention are shown;

[0075] Figure 8 A fifth embodiment of the present invention is shown;

[0076] Fig. 9 A sixth embodiment of the present invention is shown;

[0077] Fig.10 A seventh embodiment of the present invention is shown;

[0078] Fig.11 shows the main steps of a process for adding a new region annotation to an image item stored in a HEIF file when the region annotation is described by the item according to an embodiment of the present invention;

[0079] Fig.12 The main steps of a process for reading a HEIF file containing region annotations when the region annotations are described by items according to an embodiment of the present invention are shown;

[0080] Fig.13 Showing main steps of a process for reading a HEIF file containing a region comment when the region comment is described by an item according to a sixth embodiment;

[0081] Fig.14 A process for processing a HEIF file containing an image and one or more region annotation items associated with the image according to an embodiment of the present invention is shown;

[0082] Fig.15 is a schematic block diagram of a computing device for implementing one or more embodiments of the present invention. DETAILED DESCRIPTION

[0083] The HEVC standard defines a profile for encoding still images and describes specific tools for compressing a single still image or a sequence of still images. An extension to the ISO Base Media File Format (ISOBMFF) for such image data has been proposed for inclusion in Part 12 of the ISO / IEC 23008 standard, under the name "HEIF" or "High Efficiency Image File Format".

[0084] The HEIF and MIAF standards cover two forms of storage corresponding to different use cases:

[0085] - storage of sequences of images, which may be indicated to be displayed as a timed sequence or by other means, and in which images may be dependent on other images, and

[0086] - Storage of a single coded picture or a set of independent coded pictures, possibly with derived pictures.

[0087] In the first case, the encapsulation is close to that of video tracks in the ISO base media file format (see document Information technology—Coding of audio-visual objects—Part 12: ISO base media file format, w18855, ISO / IEC 14496-12, sixth edition, October 2019), and uses similar tools and concepts, such as the "trak" box and sample grouping for describing groups of samples, etc. The "trak" box is a file format box that contains sub-boxes for describing a track (i.e., a timed sequence of related samples).

[0088] A box (also called a container) is a metadata structure provided to describe the data in a file. A box is an object-oriented building block defined by a unique type identifier (usually a four-character code, also called FourCC or 4CC) and a length. All the data in a file (media data and metadata describing the media data) is contained in a box. There is no other data within a file. A file-level box is a box that is not contained in other boxes.

[0089] In the second case, a collection of ISOBMFF boxes, "meta" boxes, are used. These boxes and their hierarchy provide fewer descriptive tools than "track-related" boxes ("trak" box hierarchy), and refer to "information items" or "items" instead of related samples. It should be noted that the word "box" and the word "container" may be used in the same meaning to refer to a data structure containing metadata describing the organization and / or properties of the image data in the file.

[0090] Figure 1 An example of a HEIF file 101 is shown, which contains media data, such as one or more still images and possibly one or more videos or one or more image sequences. The file contains a first "ftyp" box (FileTypeBox) 111, which contains an identifier of the file type (usually a set of four-character codes). The file contains a second box (MetaBox) 102 called "meta", which is used to contain general non-timed metadata including a metadata structure describing one or more still images. The "meta" box 102 contains an "iinf" box (IteminfoBox) 121 describing several individual images. Each individual image is described by a metadata structure ItemInfoEntry, also represented as items 1211 and 1212. Each item has a unique 16-bit or 32-bit identifier item_ID. The media data corresponding to these items are stored in a container for media data ("mdat" box 104). An "iloc" box (ItemLocationBox) 122 provides for each item the offset and length of its associated media data in the "mdat" box 104. An "iref" box (ItemReferenceBox) 123 may also be defined to describe the association of an item with other items via typed references.

[0091] Optionally, in order to describe the storage of an image sequence or video, the HEIF file 101 may contain a third box (MovieBox) 103 called "moov", which describes one or more image sequence or video tracks 131 and 132. Typically, track 131 may be an image sequence ("pict") track designed to describe a collection of images for which temporal information is not necessarily meaningful, and track 132 is a video ("vide") track designed to describe video content. These two tracks describe a series of image samples, which are sets of pixels taken at the same time, such as frames of a video sequence. The main difference between the two tracks is that in the "pict" track, the timing information is not necessarily meaningful, while for the "vide" track, the timing information is intended to constrain the timing of the display of the samples. The data corresponding to these samples are stored in a container ("mdat" box) 104 for media data.

[0092] The “mdat” container 104 stores untimed coded images corresponding to the items represented by data portions 141 and 142 and timed coded images corresponding to the samples represented by data portion 143 .

[0093] HEIF file 101 provides different alternatives to store multiple images. For example, multiple images can be stored as items or as tracks that can be samples of a "pict" track or a "vide" track. The actual choice is usually made by the application or device generating the file based on the type of image and the intended use of the file.

[0094] The ISO base media file format specifies several alternatives for grouping samples or items according to the containers that hold the samples or items to be grouped. These alternatives can be considered as grouping data structures or grouping mechanisms (i.e., boxes or data structures that provide metadata describing grouping criteria and / or grouping attributes and / or grouping entities).

[0095] The first grouping mechanism, represented by EntityToGroupBox, applies to the grouping of items or tracks. In this mechanism, the word "entity" is used to refer to an item or track or another EntityToGroupBox. This mechanism specifies the grouping of entities. EntityToGroupBox is defined according to the following syntax:

[0096]

[0097]

[0098] grouping_type is used to specify the type of group. Several values ​​for grouping_type are specified in HEIF. group_id provides an identifier for the group of entities. entity_id represents the identifier of the entity that makes up the group, i.e., track_ID for a track, item_ID for an item, or other group_id for an entity group. Figure 1 , groups of entities inherited from EntityToGroup boxes 1241 and 1242 are included in the container 124 identified by the four-character code "gprl" of GroupsListBox.

[0099] Entity grouping consists in associating a grouping type that identifies the reason for the grouping of a collection of items, tracks, or other entity groups. In this document, the grouping information is referred to as the information in one of the EntityToGroup boxes that conveys information for grouping a collection of images.

[0100] ISOBMFF provides a mechanism to describe properties and associate properties with items. These properties are called item properties. The ItemPropertiesBox "iprp" 125 enables any item to be associated with an ordered set of item properties. The ItemPropertiesBox consists of two parts: an item property container box "ipco" 1251 containing an implicit index list of item properties 1253, and an item property association box "ipma" 1252 containing one or more entries. Each entry in the item property association box associates an item with its item properties. The HEIF standard extends this mechanism to enable item properties to be associated with items and / or entity groups. Note that in the description, for generality, we generally use item properties to specify both properties of an item or properties of an entity group. Item properties associated with an entity group apply to the entire entity group, rather than to each entity within the group individually.

[0101] The associated syntax is as follows:

[0102]

[0103]

[0104] The ItemProperty and ItemFullProperty boxes are designed for the description of item properties. ItemFullProperty allows several versions of the syntax of the definition box and may contain one or more parameters, the presence of which is regulated by a version or flag parameter.

[0105] The ItemPropertyContainerBox is designed to describe a collection of item properties as an array of ItemProperty boxes or ItemFullProperty boxes.

[0106] The ItemPropertyAssociation box is designed to describe associations between items and / or entity groups and their item properties. A description of a list of item identifiers and / or entity group identifiers is provided, each identifier (item_ID) being associated with a list of item property indexes that reference item properties in the ItemPropertyContainerBox.

[0107] None of these mechanisms allow describing some properties associated with a part or region of an image. The present invention aims to solve this problem by providing a way to describe region annotations and associate these region annotations with actual regions of the image.

[0108] Figure 2aA high-level view of the present invention for associating annotations with regions of an image is shown. A region annotation 210 is associated with an entity 200. The entity is, for example, an image item (e.g., an encoded image item or a derived image item) that describes an image. It can also be any type of item or entity group. The association means that the region annotation 210 includes information related to a region (meaning a portion) of the media content, image, or entity group described by the entity 200.

[0109] When the entity is an image item, the reconstructed image is an image obtained by decoding the encoded image item or by applying the operation of the derived image item to a set of input images (which are themselves encoded image items or derived image items). The output image is an image obtained by applying the potential transform item attributes of the image item to the reconstructed image. For example, the output image can be an image obtained after decoding the data associated with the item and applying the transform item attributes to the decoded image. The transform item attributes include, for example, an "irot" transform item attribute for rotating an image, a "clap" transform item attribute for cropping an image, or an "imir" transform item attribute for mirroring an image. The reconstructed image of the entity 200 is an annotated image to which the region annotation 210 is applied, which means that the region is part of the reconstructed image.

[0110] The region annotation 210 can be defined by its geometry 220. Geometry can be defined as a position, a position and a shape, a set of points, or a mask. For example, the geometry of the region can be a rectangle defined by the upper left corner and the width and height. For example, a set of points can represent a polygon defined by the coordinates of all vertices. Geometry can be defined in other ways. Geometry can be split into positions and optionally shapes. Several shapes can be defined: for example, points (in this case, the geometry is only a position), rectangles, or circles. The position is where the reference point of the shape is on the annotated image. For example, the reference point can be the upper left corner of the rectangle, the center of the circle... The geometry can also be defined as a mask, selecting individual pixels in the image corresponding to the entity 200. Other geometries, such as a 3D box projected onto the image plane, can also be used. The geometry can be described with a general identifier plus parameters providing its type (point, rectangle, circle...), or described as a specific identifier for each type. For images encapsulated in HEIF files, the identifier can be a four-character code.

[0111] Region annotation 210 may be linked to one or more annotations 230, 231, and 232. These annotations are used to store different annotations corresponding to region 210 of entity 200. An annotation may be, for example, a focus position of a picture, a face detected in an image, an object detected in an image, a GPS location corresponding to an object in an image, a description of a portion of an image, text contained in a region of an image, user annotations of a portion of an image, or any kind of information associated with a region.

[0112] Figure 2b An example of region annotation is shown. Image 250 is captured, for example, by a camera. The camera adds information about two detected faces at the time of capture. These faces correspond to regions 260 and 261 in image 250. Both regions are rectangular regions. The captured image can be stored in a HEIF file using the present invention.

[0113] Later, for example, when image 250 is transferred to an online-based photo storage device, further processing is applied to the image to enhance the information describing the image. First, the recognition algorithm finds the name of the person whose face is in area 261. Second, the object detection algorithm detects and identifies the building in area 270 of image 250. The area is a circular area. These processing steps produce an annotated image that can be stored in a HEIF file using the present invention. For example, the HEIF file containing image 250 is edited to store a description of the new area annotation 270 and / or the name of the detected person in area 261.

[0114] Finally, the user edits image 250, for example keeping only the portion containing known people in area 261 and buildings in area 270. Perspective distortion is also corrected. As a result, only portion 280 of image 250 is kept. Due to the perspective correction, portion 280 is not rectangular. These editing steps produce an edited image that can be stored in a HEIF file using the present invention. The HEIF file thus edited can then contain the original image 250 plus instructions for obtaining the cropped image 280, or can contain only the image description and data of area 280.

[0115] In various embodiments, the following structure may be used to represent the geometry 220 of the region annotation:

[0116]

[0117]

[0118] The semantics of this structure can be:

[0119] -type is an integer value that specifies the type of geometry.

[0120] -field_size is an integer value that specifies the size in bits (e.g., 16 or 32 bits) of the field used to describe the geometry.

[0121] - If the geometry is a point, then x and y are the coordinates of the point in the image.

[0122] - If the geometry is a rectangle, then x and y are the coordinates of the upper-left corner of the rectangle, and width and height are the width and height of the rectangle.

[0123] - If the geometry is a circle, then x and y are the coordinates of the center of the circle, and radius is the radius of the circle.

[0124] - If the geometry is a polygon, point_count is the number of points in the polygon, and for each point, x and y are the coordinates of that point in the image. polygon is a closed polygon.

[0125] - If the geometry is a 3d box, then x_bottom_0, y_bottom_0, ..., x_bottom_3, y_bottom_3 are the coordinates of the bottom rectangle of the 3d box. x_top_0, y_top_0, ..., x_top_3, y_top_3 are the coordinates of the top rectangle of the 3d box.

[0126] - If the geometry is a bitmask or a colormask, ox and oy define the position of the top left pixel of the image mask within the annotation image. mask_item_ID is the identifier of an image item that describes the mask image that defines the region. The output image of this image item is the mask image of the region. Preferably, the size of the mask image is the same as or smaller than the size of the annotation image.

[0127] - If the geometry is a bitmask, its mask image is preferably a black and white image (i.e. an image with 1-bit color components). A region is defined as all pixels in the mask image that have a black value and whose upper left corner is located at the place defined by ox and oy.

[0128] - If the geometry is a color mask, mask_color defines the color used to define the region in the mask image. The region is defined as all pixels in the mask image whose color value is the value of mask_color. The upper left corner of the region is located at the place defined by ox and oy.

[0129] In a different example, the geometry of a region can also be split into two structures corresponding to its position and its shape:

[0130]

[0131]

[0132]

[0133] In this example, the semantics of the location structure could be:

[0134] - field_size is an integer value that specifies the size in bits (e.g., 16 or 32 bits) of the field used to describe the location.

[0135] - x and y are the coordinates of the region in the image.

[0136] The semantics of shape structures are similar to those of geometry structures, with the following changes:

[0137] -For all shapes, the reference point of the shape is specified by the location on the annotation image.

[0138] - If the shape is a point, then the reference point is the point itself.

[0139] - If the shape is a rectangle, the reference point is its upper-left corner.

[0140] - If the shape is a circle, the reference point is its center.

[0141] - If the shape is a polygon, the first point is the reference point. For the other points, dx and dy are the relative coordinates of the point with respect to the reference point.

[0142] - If the geometry is a 3d box, the first point of the bottom rectangle is the reference point. All other points are specified using relative coordinates with respect to the reference point.

[0143] - If geometry is a bitmask or colormask, then position specifies where the top left pixel of the image mask is within the annotation image.

[0144] In some embodiments, points of the polygonal shape and / or 3d box shape are specified not using relative coordinates but using absolute coordinates.

[0145] The units used to define coordinates, width, height, radius in geometry can be in pixels or luma samples as integer values ​​or fixed point 16.16 values. Different units are possible. Possibly, the units used can be specified in the geometry, position and / or shape structures.

[0146] Figures 3 to 7 Some embodiments of the present invention are shown for the HEIF or MIAF standard, where an item attribute 330 is used to represent the region annotation 210.

[0147] Figure 3A first embodiment of the present invention is shown. A "meta" box 300 includes, for example, an "item" 310 corresponding to an image item. The "meta" box 300 also includes an "iprp" box 320 which itself includes two boxes. First, an "ipco" box 323 includes item attributes of different entities present in the "meta" box. Second, an "ipma" box 321 includes associations between item attributes and entities (e.g., image items or entity groups).

[0148] The "ipco" box 323 includes a first property, the region annotation 330. The region annotation includes two parts: the first part is a description of its geometry 331, the second part is a link to the region's annotation 332. These links associate the property items 340 and 341 with the region.

[0149] The “ipma” box 321 associates a zone note 330 with the item 310 via its entry 322 .

[0150] In this embodiment, the region annotation is an item property that can be defined as follows:

[0151] -Box Type: "rgan"

[0152] - Attribute Type: Descriptive Item Attribute

[0153] -Container: ItemPropertyContainerBox

[0154] -Mandatory (for each item): No

[0155] -Quantity (per item): zero or more

[0156] The syntax of the region annotation item attribute can be:

[0157]

[0158]

[0159] The semantics of the region annotation item attributes can be:

[0160] A geometry is a geometric structure of an area. Variants The geometry 331 may be described as a position structure and optionally as a shape structure.

[0161] -entry_count is the number of entry attributes (340, 341, ...) associated with the region.

[0162] -property_index is either 0 indicating no property is associated, or a 1-based index of the associated property box in the ItemPropertyContainerBox.

[0163] In this embodiment, all item properties associated with a region within the region annotation item property 330 by property_index are applied to the region of the annotation image defined by the geometry 331 .

[0164] In this embodiment, the Figure 2b The annotation image 250 may be stored in a HEIF file having the following structure:

[0165]

[0166]

[0167] It should be noted that in this description of the structure of this HEIF file, as well as in the description of the structure of other HEIF files below, parts that are not necessary for understanding the present invention are omitted. For example, the version field of the "meta" box and the "iinf" box is omitted, and the item_name field of the "infe" box is also omitted.

[0168] In this HEIF file, image item 250 is represented by an item with an item_ID value of 1. Region 260 is represented by the item attribute at index 3 and is associated with the item attribute at index 6 corresponding to a face annotation. Region 261 is represented by the item attribute at index 4 and is associated with the item attributes at indexes 6 and 7, which correspond to a face annotation and an annotation describing a person named "Jean", respectively. Region 270 is represented by the item attribute at index 5 and is associated with the item attribute at index 8, which corresponds to an annotation describing "Notre-Dame de Paris" as a building.

[0169] In this HEIF file, the association between a region and its item attributes is described in an "rgan" item attribute representing a region.

[0170] In a first variation of this first embodiment, the syntax of the region note item attribute 330 may be:

[0171]

[0172] This first variant enables several regions with respective associated item properties to be defined in a single region annotation 330 item property.

[0173] In a second variation of this embodiment, the syntax of the region annotation item attribute 330 may be:

[0174]

[0175] This second variation enables defining in a single region annotation item property 330 several regions that share an associated set of item properties.

[0176] The first and second variants may be combined to enable multiple region sets to be defined in a single region annotation item property that share an associated set of item properties.

[0177] Figure 4 A second embodiment of the present invention is shown. This embodiment is similar to the first embodiment except for the association between the region note item attribute 330 and its item attributes (340, 341). In this embodiment, the association is achieved through the "ipma" box 321. The "ipma" box contains a new entry 423 that associates the region note 330 with the item attributes 340 and 341.

[0178] In this embodiment, the syntax of the region annotation item attribute 330 may be:

[0179]

[0180] The syntax for region comments can also be:

[0181]

[0182] As in the first embodiment, the geometric figures may be described as position structures and optionally as shape structures.

[0183] To associate an item property with a region annotation item property, the syntax of the "ipma" box can be:

[0184]

[0185]

[0186] The semantics of the associated loop of the area annotation of the "ipma" box can be:

[0187] - region_count is the number of region annotation item attributes defining the association in the "ipma" box. Possibly, this number can be encoded on 8 or 16 bits. Possibly, a limit on the number of region annotation item attributes present in a HEIF file can be specified.

[0188] - from_property_index is the 1-based index of the area annotation item property box associated with the item property.

[0189] -association_count is the number of item attributes associated with the region annotation item attribute box.

[0190] - to_property_index is a 1-based index of the item property box associated with the region annotation item property box. The value of the to_property_index field may be constrained to be different from the value of the from_property_index field.

[0191] To maintain backward compatibility with existing HEIF files and implementations, two new versions of the "ipma" box may be defined. These new versions (numbered 2 and 3) both contain an association between a region and its item attributes. The size of the item_ID field differs between these versions.

[0192] The index of the zone note item property box should appear only once as the value of the from_property_index field in the 'ipma' box. The index in the ipco corresponding to the zone note item property box should not appear as the value of the to_property_index field in the 'ipma' box.

[0193] The second bit of the flags field of the box is used to define the size of the from_property_index and to_property_index fields in the region annotation association loop. Possibly, the first bit of the flags field can be used instead to have the same size as the from_property_index field, the to_property_index field, and the property_index field. Different sizes of the from_property_index and to_property_index fields can be achieved using possibly different bits.

[0194] In this second embodiment, the Figure 2b The annotated images can be stored in a HEIF file with the following structure:

[0195]

[0196]

[0197] Except for the association between the region annotation item attribute and its item attribute, the HEIF file is similar to the HEIF file corresponding to the first embodiment. In the HEIF file, the association between the region annotation item attribute 330 and its item attributes 340, 341 is described in the new version of the "ipma" box 321.

[0198] In a variation of the second embodiment, the entity association loop and the region annotation association loop are merged together. In this alternative embodiment, the item attribute index is used as the identifier of the region annotation item attribute to be used as the value of the item_ID field. Therefore, the identifier of the item, entity group and / or track must not correspond to the index of the item attribute. Possibly, the identifier of the item, entity group and / or track takes precedence over the index of the item attribute. Possibly, a new version of the "ipma" box is defined to signal the use of this alternative embodiment.

[0199] In another variation of the second embodiment, the entity association loop and the region annotation association loop are merged together. In this other variation of the second embodiment, the first bit of the item_ID field is used to signal whether the item_ID field corresponds to an item, an entity group or a track, or whether it corresponds to an item attribute index. Possibly, a new version of the "ipma" box is defined to signal the use of this variation of the second embodiment.

[0200] Figure 5 A third embodiment of the present invention is shown. This embodiment is similar to the first and second embodiments except for the association between the region annotation and its attributes. In this embodiment, the association is achieved through the "ipma" box 321 (as part of the association between the item 310 and its attributes). The association is specified as a list of attributes in the entry 522. The attributes 340 and 341 corresponding to the region annotation 330 are directly associated with the item 310 in the entry 522.

[0201] In this third embodiment, the ordering of the attributes within the entry 522 indicates to which portion of the item 310 each item attribute applies. The descriptive attributes associated with the entire item 310 appear first, before any regional annotation item attributes. Then, the individual regional annotation item attributes are listed, followed by the descriptive attributes associated therewith. Finally, the transformability attributes associated with the entire item 310 are listed.

[0202] In this third embodiment, the descriptive item attribute is then associated with the previous region annotation item attribute in the list (if any), or if no previous region annotation item attribute exists, then the descriptive item attribute is associated with the entire image.

[0203] In this third embodiment, specific handling of region annotation attributes may be indicated by defining a new item attribute type "region item attribute" therefor.

[0204] In this third embodiment, the region annotation attributes may be indicated as required to inform the reader that these attributes should not be ignored.

[0205] In this third embodiment, the Figure 2b The annotated images can be stored in a HEIF file with the following structure:

[0206]

[0207]

[0208] Except for the association between the region and its attributes, the HEIF file is similar to the HEIF files corresponding to the first and second embodiments. In the HEIF file, the association between the region and its attributes is described in the "ipma" box in the list of attributes associated with the image item to which the region annotation is applied.

[0209] The image item with item_ID value 1, corresponding to image item 250, has 9 attributes associated with it. First, the attributes of index 1 and 2 apply to the entire image. They describe the encoding parameters used for the image and the size of the image. Then, the item attribute of index 3 describes the first region corresponding to region 260. This is followed by the item attribute of index 6 that describes the facial annotation and is associated with this region. Next, the item attribute of index 4 describes the second region corresponding to region 261. This is followed by two attributes of index 6 and 7 and are associated with this region. These attributes describe the facial annotation and the annotation of a person named "Jean". Finally, the item attribute of index 5 describes the third region corresponding to region 270. This item attribute is followed by the item attribute of index 8, which corresponds to the annotation of the "Notre-Dame de Paris" building.

[0210] A fourth embodiment of the invention not shown in the drawings is now described. In this embodiment, the geometry of a region and the annotations associated therewith are described within a single (aggregate) item attribute. Preferably, new item attributes are defined for various types of annotations that can be used for entity regions.

[0211] The syntax of the region annotation item attribute can be:

[0212]

[0213] This syntax combines geometries with fields from user-described item attributes. Their semantics are not modified.

[0214] The variants can be:

[0215]

[0216]

[0217] have the same semantics.

[0218] Figure 6The main steps of a process for adding a new region annotation associated with an image item stored in a HEIF file according to an embodiment of the invention are shown. It corresponds to an embodiment in which the region annotation is described by an item attribute. These steps can be applied to HEIF files stored on disk, in a memory, or in a memory with an adapted representation. The new region annotation includes the geometry of the region and the annotation itself. Possibly, these steps can be modified to add several annotations to a region of an image item at the same time. Possibly, these steps can also be modified to add annotations to several regions of an image item at the same time.

[0219] This process can be used when creating a new HEIF file, or when modifying an existing HEIF file.

[0220] In a first step 600, it is determined whether the region already exists in the HEIF file. This determination is achieved by comparing the geometry of the region with the geometry of the existing regions.

[0221] If the area already exists, the next step is step 610 , otherwise, the next step is step 620 .

[0222] In step 610 , a RegionAnnotationProperty structure corresponding to an existing region is selected. The next step is step 640 .

[0223] A new RegionAnnotationProperty structure is created to represent the region at step 620. The geometry of the region is stored within the new RegionAnnotationProperty structure.

[0224] Then, at step 630 , the new RegionAnnotationProperty structure is associated with the image item by storing the association in the “ipma” box. The next step is step 640 .

[0225] At step 640, it is determined whether the item attribute corresponding to the annotation already exists in the HEIF file. This can be done by parsing the item attribute in the "ipco" container box. If the item attribute already exists, the next step is step 650, otherwise, the next step is step 660.

[0226] In step 650, an existing item property is selected.

[0227] In step 660, a new item attribute is created to represent the annotation within the "ipco" container box. The type of the item attribute may depend on the content of the annotation. The information contained in the annotation is stored in the item attribute.

[0228] After step 650 or step 660, at step 670, the item attributes are associated with the regions.

[0229] In the context of the first embodiment, an index to an item property is added to the RegionAnnotationProperty structure.

[0230] In the case of the second embodiment, it is determined whether there is an entry for the RegionAnnotationProperty structure in the "ipma" box. If this is not the case, an entry is created for the RegionAnnotationProperty structure in the "ipma" box. The index of the item property is added to the entry.

[0231] In the case of the third embodiment, it is determined whether there is an entry corresponding to the image item in the "ipma" box. If this is not the case, an entry is created for the image item in the "ipma" box. Then, it is determined whether there is an index to a RegionAnnotationProperty structure in the "ipma" box. If this is not the case, the index is inserted into the entry after the index of the descriptive attribute associated with the image item and before the index of the transform attribute associated with the image item. Finally, the index of the item attribute is inserted after the index of the RegionAnnotationProperty structure and before the index of any other RegionAnnotationProperty structure or any transform item attribute. Preferably, the index is inserted before the index of the next RegionAnnotationProperty structure, or before the index of the first transform item attribute, or at the end of the entry.

[0232] In the case of the fourth embodiment, the RegionAnnotationProperty structure contains both the geometry and the annotation, and no association is required. In this case, only steps 620, 630 and 660 are implemented.

[0233] Possibly, if a RegionAnnotationProperty structure enables several regions to be defined with their respective associated properties, then at step 620, it is determined whether such a structure is already associated with the image item in the HEIF file. If this is the case, a new entry is added to the RegionAnnotationProperty structure and the geometry of the region is stored in the entry. Otherwise, a new RegionAnnotationProperty structure is created and the geometry of the region is stored in the structure.

[0234] Possibly, if the RegionAnnotationProperty structure enables the associating of a set of properties with several regions, several steps may be modified. At step 600, it is determined whether the region exists in a RegionAnnotationProperty structure that does not include any other regions. At step 620, it is determined whether one or more regions are already associated with a set of properties corresponding to the annotation of the new region annotation in the RegionAnnotationProperty structure. If this is the case, the region is created in the RegionAnnotationProperty structure and steps 640 to 670 are not performed. If this is not the case, a new RegionAnnotationProperty structure is created.

[0235] Figure 7 Main steps of a process for reading a HEIF file containing region annotations when the region annotations are described by item attributes according to an embodiment of the present invention are shown.

[0236] In a first step 700, an image item is extracted from a HEIF file. Possibly, only a portion of the metadata describing the image item is extracted.

[0237] Next, in step 710, the first attribute associated with the image item is extracted. The index of the attribute is the first non-zero index in the entry of the "ipma" box corresponding to the image item. This index is used to extract the attribute from the "ipco" box. If there is no entry corresponding to the image item in the "ipma" box, or if the entry does not contain an attribute index or contains only an attribute index with a value of 0, the process ends directly at step 790.

[0238] In step 720 , it is determined whether the extracted item attribute is a regional annotation item attribute. If it is a regional annotation item attribute, the next step is step 730 , otherwise, the next step is step 770 .

[0239] In step 730, a description of the region is extracted from the region annotation item attributes. The description of the region may include the type of geometric figure and / or parameters defining the geometric figure. The region may be associated with the image item extracted in step 700.

[0240] Next, in step 740, the index of the first item attribute associated with the region is extracted. If there is no item attribute associated with the region, the process continues directly at step 770.

[0241] In a first embodiment of the present invention, the index of the first attribute item associated with a region is the first one present in the region annotation item attribute structure.

[0242] In a second embodiment of the present invention, the index of the first attribute associated with a region is the first one of the entries corresponding to the region annotation attribute in the "ipma" box.

[0243] In a third embodiment of the present invention, the index of the first attribute associated with a region is the first one of the entries of the image item in the "ipma" box following the index of the region annotation attribute.

[0244] The index of the item property is used to extract the item property from the "ipco" box.

[0245] The item attributes may then be associated with the region at step 750. The information contained in the item attributes may be extracted and associated with the geometry of the region extracted at step 730.

[0246] At step 760, it is determined whether there are other item attributes associated with the region.

[0247] In the first embodiment of the present invention, it is determined whether there are any unprocessed indexes in the region annotation item attribute structure.

[0248] In the second embodiment of the present invention, it is determined whether there is any unprocessed index in the entry corresponding to the area annotation item attribute in the "ipma" box.

[0249] In a third embodiment of the present invention, it is determined whether the next index in the entry of the image item in the "ipma" box corresponds to a descriptive item attribute that is not an area annotation item attribute.

[0250] If there are other item attributes associated with the region, the next step is step 765. Otherwise, the next step is step 770.

[0251] In step 765, an index of another attribute associated with the region is retrieved. The first, second and third embodiments of the present invention are processed in a similar manner as in step 740. The next step is step 750.

[0252] In a fourth embodiment of the present invention, there is no item attribute associated with the region, and steps 740, 760 and 765 are skipped. The annotation of the region is directly extracted from the region annotation item attribute and can be associated with the region at step 750.

[0253] In step 770, it is determined whether there is another attribute associated with the image item. This determination is achieved by checking whether there is any unprocessed index in the entry corresponding to the image item in the "ipma" box. If there is another attribute associated with the image item, the next step is step 780. Otherwise, the next step is step 790.

[0254] At step 780, another attribute associated with the image item is extracted. This step is similar to step 710. The next step is step 720.

[0255] At step 790, the process ends.

[0256] Figures 8 to 13 Some embodiments of the present invention are shown for the HEIF or MIAF standard, where terms are used to represent region annotations 210.

[0257] Figure 8 A fifth embodiment of the present invention is shown. The "meta" box 800 contains, for example, an item 810 corresponding to an image item. The "meta" box 800 also contains two other items 820 and 830 corresponding to region annotations. The geometry of these two region annotation items is described in their respective contents, represented as geometry 821 and geometry 826. These contents are identified in the "iloc" box. The contents of the region annotation items can preferably be stored in the "idat" box. It can also be stored in the "mdat" or "imda" boxes.

[0258] The “iref” box 830 contains two entries 831 and 832 , which associate the region annotation items ( 820 and 825 , respectively) with the image item 810 .

[0259] The "iprp" box 840 contains an "ipco" box 860 and an "ipma" box 850. The "ipco" box 860 contains item attributes 861 and 862 corresponding to the annotations of the two area annotation items 820 and 825. The "ipma" box 850 associates the area annotation items 820 and 825 with their respective item attributes 861 and 862 through two entries (851 and 852, respectively).

[0260] In this embodiment, the region annotation item may be defined with an item_type of "rgan". The region annotation item may be associated with the image item through an item reference box of type "cdsc".

[0261] The syntax of the content of a region annotation item can be:

[0262] aligned(8)class RegionAnnotationItem{

[0263] Geometry geometry;

[0264] }

[0265] The semantics of a region annotation item can be:

[0266] -geometry is the geometry of the region.

[0267] Possibly, a version field could be added to the Region Annotation Item for future evolution. Extension of the Region Annotation Item could also be achieved by defining new geometry types.

[0268] In this embodiment, all item properties associated with a region annotation item through an entry of the "ipma" box apply to the region of the annotated image defined by the geometry of the region annotation item.

[0269] In this embodiment, the Figure 2b The annotated images can be stored in a HEIF file with the following structure:

[0270]

[0271]

[0272]

[0273] In this HEIF file, image item 250 is represented by an item with an item_ID value of 1. Region 260 is represented by an item with an item_ID value of 2. Region 260 is associated with the item attribute at index 3 via the second entry of the "ipma" box. This attribute corresponds to a face annotation. The geometry of the region annotation item is described in the first region annotation section of the MediaDataBox.

[0274] Region 261 is represented by the item with an item_ID value of 3. Region 261 is associated with the item attributes at indexes 3 and 4 through the third entry of the "ipma" box. These attributes correspond to a facial annotation and an annotation describing a person named "Jean", respectively. The geometry of the region annotation item is described in the second region annotation section of the MediaDataBox.

[0275] Region 270 is represented by the item with item_ID value 4. Region 270 is associated with the item attribute at index 5 through the fourth entry of the "ipma" box. This attribute corresponds to the annotation describing "Notre-Dame de Paris" as a building. The geometry of the region annotation item is described in the third region annotation part of the MediaDataBox.

[0276] Each region annotation item is associated with the annotated image via an item reference of type "cdsc".

[0277] All region annotation items may have a hidden property set to true, indicating to the reader that the item is not intended to be displayed.

[0278] In a first variation of the fifth embodiment, the syntax of the region annotation item may be:

[0279]

[0280] This first variation of the fifth embodiment enables the definition of several regions sharing the same set of associated item properties in a single region-annotated item.

[0281] Fig. 9 A sixth embodiment of the present invention is shown. This embodiment is similar to the fifth embodiment except for the association between the region annotation item and the image item.

[0282] In this embodiment, the region annotations 820 and 825 are grouped together by a region group 970. The region group 970 may be an EntityToGroupBox with a grouping_type of “rgan” or “rcog.” The region group 970 is associated with the image item 810 by an entry 933 of the “iref” box 830.

[0283] In this embodiment, the Figure 2b The annotated images can be stored in a HEIF file with the following structure:

[0284]

[0285]

[0286] Except for the association between the region annotation item and the image item, the HEIF file is similar to the HEIF file corresponding to the fifth embodiment. In the HEIF file, the region annotation items with item_IDs 2, 3, and 4 are grouped in a group of grouping_type "rgan" and have group_id 100. The group is associated with the image item through an item reference with referenceType "cdsc".

[0287] In a variation of this embodiment, item attributes 861 and 862 are associated with region group 970 .

[0288] In another variation of this embodiment, the item attributes 861 and 862 are associated with the region group 970, the region group 970 is not associated with the image item 810, and the region annotation items 820 and 825 are associated with the image item 810 through the item reference.

[0289] Fig.10 A seventh embodiment of the present invention is shown. This embodiment is similar to the fifth embodiment except for the description of the geometry of the regions. In this embodiment, the two region annotation items 1020 and 1025 do not have their geometry within their respective contents. Instead, these region annotation items are associated with geometry item attributes 1063 and 1064, respectively, via entries 851 and 852, respectively, within the "ipma" box 850.

[0290] In this embodiment, the geometry item attributes may be defined as follows:

[0291] -box type: "geom"

[0292] - Attribute Type: Descriptive Item Attribute

[0293] -Container: ItemPropertyContainerBox

[0294] - Mandatory (for each item): Yes for items of type 'rgan'

[0295] -Quantity (per item): zero or more

[0296] The syntax for geometry attributes can be:

[0297]

[0298] The semantics of geometry item attributes can be:

[0299] -geometry is the geometry of the region.

[0300] In this embodiment, the Figure 2b The annotated images can be stored in a HEIF file with the following structure:

[0301]

[0302]

[0303] Except for the description of the geometry of the region annotation item, the HEIF file is similar to the HEIF file corresponding to the fifth embodiment. The region annotation item is associated with the geometry item attribute to specify its geometry. Since the region annotation item has no content in the HEIF file, there may be no associated entry in the ItemLocationBox, or its associated entry may have no content.

[0304] In a first variation of the seventh embodiment, the syntax of the geometry item attributes may be:

[0305]

[0306]

[0307] This first alternative enables several geometries to be defined within a single geometry item attribute.A region annotation item associated with a geometry item attribute comprises one region for each geometry defined in the geometry item attribute.All item attributes associated with a region annotation item apply to all regions of the region annotation item.

[0308] In a second variation of this seventh embodiment, the geometry of the region is defined by two item attributes: a position item attribute and a shape item attribute.

[0309] In this second variation, the location item attributes may be defined as follows:

[0310] - Frame type: "relo"

[0311] - Attribute Type: Descriptive Item Attribute

[0312] -Container: ItemPropertyContainerBox

[0313] - Mandatory (for each item): Yes for items of type 'rgan'

[0314] -Quantity (per item): zero or more

[0315] The syntax for a position attribute can be:

[0316] aligned(8)class LocationProperty

[0317] extends ItemFullProperty('relo',version=0,flags=0){

[0318] Location location;

[0319] }

[0320] The semantics of position item attributes can be:

[0321] -Location is the location of the area.

[0322] In this second variation, the shape item attributes may be defined as follows:

[0323] -Box Type: "resh"

[0324] - Attribute Type: Descriptive Item Attribute

[0325] -Container: ItemPropertyContainerBox

[0326] - Mandatory (for each item): Yes for items of type 'rgan'

[0327] -Quantity (per item): zero or more

[0328] The syntax for shape attributes can be:

[0329] aligned(8)class ShapeProperty

[0330] extends ItemFullProperty('resh',version=0,flags=0){

[0331] Shape shape;

[0332] }

[0333] The semantics of shape item attributes can be:

[0334] -Shape is the shape of the region.

[0335] Alternatively, several location item attributes can be associated with the same region annotation item. In this alternative, the region annotation item includes one region for each location item attribute. All these regions share the same shape.

[0336] Alternatively, several position item attributes and / or several shape item attributes can be associated with the same region annotation item. In this alternative, the region annotation item includes one region for each combination of the position item attributes and the shape item attributes. For example, if the region annotation item is associated with two position item attributes and three shape item attributes, six regions are included.

[0337] In another alternative of this second variant, the position item attribute and / or the shape item attribute can contain several positions or several shapes respectively. In this alternative, the region annotation item includes a region for each combination of a position contained in one of the position item attributes and a shape contained in one of the item attributes.

[0338] In another alternative, several geometry item attributes may be associated with the same region annotation item. In this alternative, the region annotation item includes one region for each geometry item attribute.

[0339] For all alternatives where several regions are included in a region note item, all item properties associated with the region note item apply to all regions included in the region note item.

[0340] Fig.11 The main steps of a process for adding a new region annotation to an image item stored in a HEIF file when the region annotation is described by the item according to an embodiment of the present invention are shown. Figure 6 There are similarities. These steps can be applied to HEIF files stored on disk, in memory, or in memory in an adapted representation. The new region annotation includes the geometry of the region and the annotation itself. Possibly, these steps can be modified to add several annotations to a region of an image item at the same time. Possibly, these steps can also be modified to add annotations to several regions of an image item at the same time.

[0341] This process can be used when creating a new HEIF file, or when modifying an existing HEIF file.

[0342] In the first step 1100, it is determined whether a region annotation item with the same geometry already exists in the HEIF file.

[0343] If there is already a region annotation item with the same geometry, the next step is step 1110 , otherwise, the next step is step 1120 .

[0344] In step 1110 , the item_ID value corresponding to an existing region annotation item is selected. The next step is step 1140 .

[0345] At step 1120, a new region annotation item is created to represent the region. An "infe" box describing the region annotation item may be created within the "iinf" box of the HEIF file. An entry within the "iloc" box may be added to indicate the location of the content of the region annotation item. An item_ID value is associated with the new region annotation item.

[0346] In case of the fifth or sixth embodiment, the geometry of the region is stored within the content of the region annotation item.

[0347] In the case of the seventh embodiment, a new geometry item attribute may be created to store the geometry of the region. The index of the new item attribute is stored. Possibly, if the same geometry item attribute exists in the HEIF file, its index is stored and no new item attribute is created.

[0348] In the case of the seventh embodiment, a new position item attribute and a new shape item attribute may be created to store the geometry of the region. The indexes of these new item attributes are stored. Possibly, if one or both of these item attributes exist in the HEIF file, their indexes are stored and the corresponding item attributes are not created.

[0349] Then, at step 1130, the new region annotation item is associated with the image item.

[0350] In the case of the fifth or seventh embodiment and for some variants of the sixth embodiment, a new reference of type "cdsc" is created in the "iref" box of the HEIF file. This reference associates the region annotation item with the image item.

[0351] In the case of some variants of the sixth embodiment, it is first determined whether a region group, which may be an EntityToGroupBox of grouping_type "rgan", is associated with an image item. If this is the case, the region annotation item is added to the region group. If this is not the case, a new region group of type "rgan" is created containing the new region annotation item. Then, a new reference of type "cdsc" is created in the "iref" box of the HEIF file. This reference associates the region group with the image item.

[0352] In the case where the sixth embodiment is combined with the fifth embodiment or the seventh embodiment, any of these processes can be applied. Possibly, first determine whether the regional group of the EntityToGroupBox which is grouping_type "rgan" is associated with the image item. If this is the case, add the regional annotation item to the regional group. If this is not the case, determine whether another regional annotation item is associated with the image item. If this is the case, create a new regional group of type "rgan" that contains both the other regional annotation items and the new regional annotation item. Remove the reference between the other regional annotation items and the image item, and create a new reference of type "cdsc" between the regional group and the image item. If this is not the case, create a new reference of type "cdsc" between the new regional annotation item and the image item.

[0353] The next step is step 1140 .

[0354] At step 1140 , it is determined whether the item attribute corresponding to the annotation already exists in the HEIF file. If the item attribute already exists, the next step is step 1150 , otherwise, the next step is step 1160 .

[0355] In step 1150, an existing item property is selected.

[0356] In step 1160, a new item attribute is created to represent the annotation. The type of the item attribute depends on the content of the annotation. The information contained in the annotation is stored in the item attribute.

[0357] After step 1150 or step 1160 , the next step is step 1170 .

[0358] At step 1170, item attributes are associated with the region annotation item.

[0359] If the zone note item already has an associated entry in the "ipma" box, the index of the item attribute is added to that entry.

[0360] If the zone annotation item does not have an associated entry in the "ipma" box, a new entry is created for the item and the index of the item attribute is added to the entry.

[0361] In the case of the seventh embodiment, the geometry item attributes, the index of the position item attributes, and / or the shape item attributes stored at step 1120 are also added to the entry.

[0362] Possibly, if the region annotation item may include several regions, several steps may be modified. At step 1100, it is determined whether there is a region annotation item including a single region with the same geometry. At step 1120, it is determined whether the existing region annotation item is associated with an image item having a set of attributes corresponding to the annotation of the new region annotation. If this is the case, the geometry of the new region is added to the existing region annotation item, and steps 1140 to 1170 are not performed. If this is not the case, a new region annotation item is created.

[0363] Fig.12 Main steps of a process for reading a HEIF file containing region annotations when the region annotations are described by items according to an embodiment of the present invention are shown.

[0364] In a first step 1200, an image item is extracted from a HEIF file. Possibly, only a portion of the metadata describing the image item is extracted.

[0365] In step 1210, the first item different from the image item is extracted from the HEIF file. If no other items exist in the HEIF file, the algorithm continues directly at step 1270.

[0366] Then, in step 1220 , it is determined whether the other item is a regional annotation item. If it is a regional annotation item, the next step is step 1230 , otherwise, the next step is step 1250 .

[0367] At step 1230 , it is determined whether the region annotation item is associated with the image item via a reference of type “cdsc” in the “iref” box. If this is the case, the next step is step 1240 , otherwise, the next step is step 1250 .

[0368] At step 1240, item attributes associated with the zone annotation item via the entry of the "ipma" box are extracted.

[0369] Possibly, in the context of the sixth embodiment, it is determined whether the region annotation item is contained in a region group which is an EntityToGroupBox with a grouping_type of type “rgan.” If this is the case, the item attributes associated with the region group through the entry of the “ipma” box are extracted.

[0370] The geometry of the region annotation items is then extracted.

[0371] In the context of some variations of the fifth embodiment or the sixth embodiment, the geometry of the area annotation item is extracted from the content of the area annotation item.

[0372] In the context of the seventh embodiment, the geometry of the area annotation item is derived from geometry item attributes, position item attributes and / or shape item attributes associated with the area annotation item.

[0373] A region linked to an extracted item attribute is associated with the image item. Information contained in an item attribute can be extracted and associated with the geometry of a region of an image item.

[0374] If the region annotation item comprises several regions, each of these regions linked with the extracted item attributes is associated with the image item.The information contained in the item attributes may be extracted and associated with the geometry of each of these regions of the image item.

[0375] The next step is step 1250 .

[0376] At step 1250 , it is determined whether there are other items in the HEIF file that are different from the image item. If this is the case, the next step is step 1260 , otherwise, the next step is step 1270 .

[0377] At step 1260 , another item is extracted from the HEIF file. The next step is step 1220 .

[0378] At step 1270, the process ends.

[0379] Fig.13 The main steps of a process for reading a HEIF file containing region annotations when the region annotations are described by items according to some variations of the sixth embodiment are shown. The process is similar to Fig.12 Processing shown.

[0380] In a first step 1300, an image item is extracted from the HEIF file. Possibly, only a portion of the metadata describing the image item is extracted.

[0381] In step 1310 , the first group is extracted from the HEIF file. If the group does not exist in the HEIF file, the algorithm continues directly at step 1370 .

[0382] Then, in step 1320 , it is determined whether the group is a regional group of EntityToGroupBox with a grouping type of “rgan.” If it is a regional group, the next step is step 1330 , otherwise, the next step is step 1350 .

[0383] At step 1330 , it is determined whether the region group is associated with an image item by a reference of type “cdsc” in the “iref” box. If this is the case, the next step is step 1340 , otherwise, the next step is step 1350 .

[0384] At step 1340 , the region annotation items contained in the group are retrieved. For each region annotation item, its region and its item attributes are extracted and associated with the image item, as described in step 1240 .

[0385] If the region annotation item includes several regions, these regions linked with the extracted item attributes are each associated with the image item, as described in step 1240 .

[0386] Possibly, item attributes associated with the region group are extracted. For each region annotation item contained in the group, its region is extracted. Each extracted region combined with all extracted item attributes is then associated with the image item, as described in step 1240.

[0387] The next step is step 1350 .

[0388] At step 1350 , it is determined whether there are other groups in the HEIF file. If this is the case, the next step is step 1360 , otherwise, the next step is step 1370 .

[0389] At step 1360 , another group is extracted from the HEIF file. The next step is step 1320 .

[0390] At step 1370, the process ends.

[0391] Fig.14 A process for processing a HEIF file containing an image and one or more area annotation items associated with the image according to an embodiment of the present invention is shown. The process may be an image editing process. The process may be a process of removing a portion of an image, such as cropping. The process may be a process of transforming the geometry of an image, such as perspective correction, rotation, or affine transformation. The process may be a process of changing the content of an image, such as applying a filter or drawing on an image. The process may be a metadata editing process. The process may be a process for removing private metadata. The process may be a process for filtering metadata. The process may be a process for translating metadata.

[0392] In a first step 1400, the process is applied to an image item. Possibly, an image associated with the image item may be modified.

[0393] Possibly, the result of the processing may be stored in another image item as a derived image item. In this case, the region annotation item associated with the original image item is also associated with the derived image item. In the following steps, the processed image item is a derived image item.

[0394] In step 1410 , a first region annotation associated with the processed image item is retrieved. If no region annotation is associated with the processed image item, the next step is step 1470 .

[0395] In step 1420, it is determined whether the region annotation should be removed. Depending on the process, different criteria may be used. The process may remove all region annotations. The process may remove any region annotations of a particular type. For example, a privacy protection filter may remove any region annotations represented by a user-defined item attribute. The process may remove region annotations based on positioning. For example, cropping may remove any region annotations that do not intersect with the remaining image.

[0396] As an example, in Figure 2b In the example of FIG. 2 , the region annotation 260 may be removed from the edited image because the region to which the region annotation 260 refers is cropped from the edited image.

[0397] If it is determined that the region annotation is to be removed, the next step is step 1425. Otherwise, the next step is step 1430.

[0398] In step 1425 , the region annotation is removed. The next step is step 1450 .

[0399] In step 1430, it is determined whether the geometry of the region annotation should be modified. Any processing that transforms the geometry of the image should also modify the geometry of the region annotation. Such processing includes rotating the image, scaling the image, applying an affine transformation to the image, and correcting for perspective distortion of the image.

[0400] If it is determined that the geometry of the region annotation should be modified, the next step is step 1435. Otherwise, the next step is step 1440.

[0401] In step 1435, the geometry of the region annotation is modified according to the process. Possibly, the modified geometry is the exact result of applying the geometric transformation to the geometry of the region annotation. Possibly, the modified geometry is the approximate result of applying the geometric transformation to the geometry of the region annotation.

[0402] For example, in Figure 2b, region 261 is initially a rectangle. The perspective correction applied to the image transforms the region from a rectangle to a trapezoid. The modified geometry may be a polygon that accurately represents the trapezoid. The modified geometry may be a polygon with integer coordinates that approximate the rational coordinates of the trapezoid. The modified geometry may be a rectangle that approximates the shape of the trapezoid.

[0403] The next step is step 1440 .

[0404] In step 1440, it is determined whether the annotation of the region annotation should be modified. Depending on the process, different criteria may be used. The process of translating the text annotation may modify the text representing the annotation. The process of filtering the image, for example by applying blur, may modify the annotation to remove a precise portion therefrom.

[0405] If it is determined that the annotation of the area annotation should be modified, the next step is step 1445. Otherwise, the next step is step 1450.

[0406] In step 1445, the annotation of the region annotation is modified according to the process.

[0407] For example, in Figure 2b , region 261 has a region annotation corresponding to a description of a person. The privacy protection process may maintain the indication that the region annotation corresponds to a person, but remove the name of the person.

[0408] The next step is step 1450 .

[0409] In step 1450 , it is determined whether there are other region annotations to be processed. If it is determined that there are other region annotations to be processed, the next step is step 1460 . Otherwise, the next step is step 1470 .

[0410] In step 1460 , another region annotation associated with the image item is retrieved. The next step is step 1420 .

[0411] In step 1470, the process ends.

[0412] Note Description

[0413] An annotation may be a focus position within a captured picture. An annotation may be represented using user-descriptive item attributes, for example using a specific value of a name field and / or a tag field, such as "focus". An annotation may be represented by a new item attribute.

[0414] An annotation may be a detected face within an image. An annotation may be represented using user-descriptive item attributes, for example using a specific value of a name field and / or a tag field, such as "face". An annotation may be represented by a new item attribute.

[0415] An annotation may be an object detected within an image by an object detection tool. An annotation may be represented using user-descriptive item attributes, such as using specific values ​​for the name field and / or the tag field, and / or using a more descriptive value for the description field. For example, the name field may be "building" and the description field may be "House, building or monument". An annotation may be represented by a new item attribute.

[0416] An annotation may be a specific object instance detected within an image by an object detection tool. An annotation may be represented using user-descriptive item attributes, such as using a specific value in the tag field to describe the general type of the object, using a more precise value corresponding to the object instance in the name field, and / or using a descriptive value for the object instance in the description field. For example, the tag field may be "church", the name field may be "Notre Dame", and the description field may be "Notre-Dame de Paris". An annotation may be represented by a new item attribute.

[0417] An annotation can be the GPS location of an object in an image. An annotation can be represented by a new item attribute.

[0418] The annotation can be text extracted from the image by an OCR tool. The annotation can be represented by a user-described item attribute, for example using a specific value indicating that the annotation corresponds to the OCR result in the name field and / or the tag field and represents the OCR output in the description field. For example, the tag field can be "ocr" and the description field can be "Notre-Damede Paris". The annotation can be represented by a new item attribute. The new item attribute can include information about the font used, such as the family, style, weight and / or size of the font. Possibly, the results of the OCR tool are split into several area annotations corresponding to different detected text areas and / or different detected text styles.

[0419] An annotation can describe an edit or modification applied to an area of ​​an image. An annotation can be represented by a user description item property. An annotation can be represented by a new item property.

[0420] The annotation may be stored in an item. The item may be associated with a region annotation item by a reference of type "cdsc". The item may be associated with a region annotation item attribute by a new item attribute that is associated with and references the region annotation item attribute. The box type of the new item attribute may be "rgcd". For example, the annotation may be stored in an item of "Exif". As another example, the annotation may be stored in an XMP document contained in an item of type "mime" and having a content type of "application / rdf+xml".

[0421] In other words, a metadata item may be associated with a region item by an item reference of type "cdsc" from the metadata item to the region item. A metadata item may be an item with an item_type value of "Exif". A metadata item may be an item with an item_type value of "mime" and a content type of "application / rdf+xml". A metadata item may be an item with an item_type value of "mime" and a content type set equal to the MIME type specified for MPEG-7 metadata.

[0422] Possibly, when a user describes an item attribute, the language field is set to an appropriate value. Possibly, when a user describes an item attribute, several instances of the item attribute are used with different language field values.

[0423] It is possible that a region annotation associated with an entity has no associated annotations. This may be the case when a region annotation item is associated with an image item by reference without having an associated attribute item. This may be useful when the annotations are predefined or known by external means. For example, an application aimed at counting people in an image may identify individual people in an image by regions that do not have any associated annotations. When storing the image, the application may also store in an external file the indications that the regions in the image correspond to people.

[0424] Geometric Alternatives

[0425] Possibly, the geometry of the region annotation may include other types. The geometry may include a type for representing an ellipse. The geometry may include a type for representing a 3d box expressed in world coordinates and an associated world-to-image projection. The geometry may include a type for representing a specific polygon. The geometry may include a type for representing a sphere.

[0426] Possibly, the geometry of the region annotation may be a segment.Possibly, the geometry of the region annotation may be a line, for example defined as a spline or a Bézier curve.Possibly, the geometry of the region annotation may be a collection of connected and / or unconnected segments and / or lines.

[0427] The geometry of a collection of connected segments can be defined by the following structure:

[0428]

[0429] Possibly, the geometries of region annotations can be restricted to fewer types. Geometries can be restricted to rectangular regions.

[0430] Possibly, the type field of the geometry structure and / or shape structure is a four-character code.

[0431] Possibly, the location within the geometry of the region annotation may correspond to the center of the shape.

[0432] Possibly, a location without an associated shape represents a point-type area.

[0433] Possibly, the geometry of a region can be represented by one or more than one equation and / or inequality. For example, a line can be represented by an equation. As another example, the interior of a triangle can be represented by three inequalities.

[0434] Possibly, some or all of the coordinates of a region annotation's geometry can be represented as rational numbers instead of integers. For example, a point can be represented by the following fields:

[0435]

[0436] in

[0437] -xN and xD are the numerator and denominator of the horizontal coordinate of the point respectively.

[0438] -yN and yD are the numerator and denominator of the ordinate of the point respectively.

[0439] Possibly, some or all of the coordinates of the region annotation's geometry can be represented as signed integers instead of unsigned integers. For example, the region annotation's geometry can be represented using the following structure:

[0440]

[0441]

[0442] Negative values ​​of the x or y fields enable specifying a point outside the image, the top left corner, and / or the center. This may be useful for updating region annotations during editing of HEIF files. A region may be empty if it falls completely outside the image. Such empty regions should be ignored by a parser or reader.

[0443] Possibly, the field_size of the geometry structure, position structure and / or shape structure can be fixed, for example using 32 bits. In this case, the field_size may not be specified in the structure. When several of these structures are included in a box, the definition of field_size may be shared by all structures. When one or more than one structure is included in a complete box, the definition of field_size may be based on a flag field or a version field of a complete box. When one or more than one structure is included in a box or a complete box, the field_size may depend on the box type. For example, two different geometry item attribute boxes each having its own specific field_size may be defined.

[0444] Possibly, in some embodiments, a geometry structure, a position structure and / or a shape structure may be used together to represent the geometry of the region annotation.

[0445] Possibly, several geometries sharing the same type can be represented in a compressed form. For example, a list of rectangles can be represented by representing each field as a difference with the corresponding field of the previous rectangle.

[0446] Possibly, the mask is always located at the top left of the image and no place is specified for it.

[0447] Possibly, the mask always has the same size as the annotated image.

[0448] Possibly, the mask has a different pixel density than the pixel density of the annotation image. For example, one pixel of the mask may correspond to 4 pixels of the annotation image. As another example, 4 pixels of the mask may correspond to 1 pixel of the annotation image. This difference in pixel density may be described in a geometry or shape structure. The difference may be described using a pixel aspect ratio attribute.

[0449] Possibly, a color mask may combine masks corresponding to several region annotations into another mask for another region annotation. For example, a mask for a region annotation describing a first specific person may use pixel color values ​​0x1. A mask for another region annotation describing a second specific person may use pixel color values ​​0x2. A mask for region annotations describing people in a generic way and combining these two regions may use pixel color values ​​0x3.

[0450] Possibly, the type of the geometry structure or shape structure may be specified by the type of the region annotation item attribute, the type of the region annotation item, the type of the geometry item attribute and / or the type of the shape item attribute.

[0451] For example, point area annotation item attributes and rectangle area annotation item attributes can be defined by the following syntax:

[0452]

[0453]

[0454] Possibly, a specific region annotation item may be defined for representing a mask. The content of the specific region annotation item may be an image corresponding to the mask. Other information used to define the region represented by the mask, such as the position of the mask or the color used by the mask, etc., may be represented in a specific item attribute associated with the mask region annotation item.

[0455] Possibly, the syntax of the content of such an item could be:

[0456] aligned(8)class MaskItem{

[0457] bit(8)data[];

[0458] }

[0459] Possibly, the content of such an item may include the location of the region using the following syntax:

[0460]

[0461] Possibly, the mask can be stored directly within the geometry (payload) of the RegionItem. Optionally, the MaskItem can have a parameter indicating the number of bits_per_pixel. Possible values ​​can be 1, 8, 16, 24. Specifying a mask with more than 1 bit per pixel enables the use of an alpha plane layer as a mask.

[0462] Possibly, the mask may be stored as an image item and may be associated with a region item using an item reference of type "mask" from the region item to the image item. This image item is hereinafter referred to as the "mask image item". As previously described, the region is defined by the image resulting from applying any transform item attributes to the mask image item. Possibly, the location of the region may be specified by associating an "rloc" item attribute with the mask image item. Possibly, the geometry of a region annotation as a bitmask or color mask may be represented in a region item payload using the following structure:

[0463]

[0464] Possibly, when using a bit mask, a region can be defined as all pixels in the mask image having a value of 0. Possibly, a region can be defined as all pixels in the mask image having a maximum value. Possibly, a region can be defined based on another pixel value. Possibly, a region can be defined based on a predefined color of a pixel. Possibly, when the number of bits per pixel is greater than 1, the minimum value corresponds to pixels belonging to the region, the maximum value corresponds to pixels not belonging to the region, and the intermediate values ​​correspond to pixels that are partially in the region, thereby enabling the definition of regions with fuzzy boundaries. The use of item references between region items and mask images advantageously simplifies the editing process of HEIF files containing region items by locating all links between items and / or entities within a single box.

[0465] Possibly, an item reference may be from a mask image item to a region item.

[0466] Possibly, the item reference content could use a different type, such as "rmsk".

[0467] Possibly, an item reference may be from a region item to several mask image items. A region is defined as the union of sub-regions defined by the mask image items. In a variant, there may be several item references from a region item to a mask image item.

[0468] Possibly, a region item may contain a description of the geometry of several regions acting as a mask. In this case, an item reference of type "mask" may be used to associate a region item with a mask image item. There is a reference to each such item in the item reference of type "mask". If the same mask stored as an image item is used for multiple regions declared in a region item, the image item will be referenced several times in the item reference of type "mask". The mask image items are listed in the item reference in the same order as their corresponding regions in the region item. In other words, an image that defines a mask for the Nth region with geometry of type "mask" in the order of declaration and is stored as an image item is identified by the Nth reference in the item reference of type "mask".

[0469] In a variation, region items may be associated with mask image items using one item reference of type "mask" for each mask image item. The "mask" item references appear in the "iref" box in the same order as their corresponding regions in the region item.

[0470] In another variation, the mask image items are listed in an arbitrary order within the "mask" item reference. The definition of each mask region contains an index to the corresponding mask image item within the "mask" item reference.

[0471] For this variant, the geometry of the mask region can be defined using the following syntax:

[0472]

[0473]

[0474] Possibly, the location of the region can be specified by associating an "rloc" item attribute with the mask image item.

[0475] Possibly, a mask region may be associated with several image items by including a list of mask item indices in the definition of the mask region.

[0476] In a variant, region items may be associated with mask image items using one item reference of type "mask" for the respective mask image item.

[0477] Possibly, two mask regions specified for an image item may overlap each other. This means that some pixels of the image belong to both regions.

[0478] Possibly, the location of a region annotation item may be represented in the content of an item and / or in an associated item attribute, and the shape of a region annotation item may be represented in the content of an item and / or in an associated item attribute. For example, the shape of a region may be represented in the content of an item, and the location of a region may be represented in an associated item attribute. In this example, several locations may be associated with a region annotation item. A region annotation item includes one region for each location, all of which share the same shape.

[0479] Possibly, the fields defining the number of items contained in the list can be represented with different sizes. For example, the region_count field can be represented as a 16-bit unsigned integer.

[0480] Item property alternatives

[0481] Possibly, the region annotation item properties could extend the ItemProperty box instead of extending the ItemFullProperty box.

[0482] Possibly, a Region Note Item Attribute associated with an Item has a required field set to a value of 1 to ensure that the Reader understands the Region Note Item Attribute.

[0483] Possibly, in addition to the descriptive attribute and the transformative attribute types, a new attribute type, the regional attribute, is defined. The processing specific to the regional annotation item attribute can be associated with the regional attribute type, rather than with the regional annotation item attribute itself. For example, the processing of the item attribute according to the third embodiment of the present invention can be defined based on the regional attribute type.

[0484] Alternatives

[0485] Possibly, two region annotation items may share the same content to represent the same region but have different annotations.

[0486] Possibly, the region annotation item can be associated with the image item using a different reference type. The reference type used can be "tbas" or "dimg". New reference types can be defined, such as "rgan", for describing such an association.

[0487] Possibly, depending on the type of annotation, possibly different types of references may be used. For example, a region annotation item corresponding to a focus region may be associated with its annotated image item by a reference of type "rfoc" indicating that this is a focus region. In this example, the region annotation item may not be associated with an item attribute describing its annotation.

[0488] Possibly, a region annotation item is of type "hvc1" and its content is an annotation image.

[0489] Possibly, a region annotation item is a derived image item of type "iden" without any content.

[0490] Possibly, a region annotation item is of type "hvt1" and its content is a tile of the annotated image. The geometry of a region annotation item may be the area covered by the tile. Possibly, the content of a region annotation item is a set of tiles of the annotated image and the geometry of its region may be the area covered by the tile.

[0491] Possibly, the region annotation item corresponds to a sub-picture of the annotated image. The geometry of the region annotation item may be the area covered by the sub-picture. Possibly, the region annotation item corresponds to a set of sub-pictures of the annotated image. The geometry of the region annotation item may be the area covered by the sub-picture.

[0492] Possibly, the box type of the geometry item attribute is different. For example, the box type could be "rcgr".

[0493] Possibly, use the "rloc" item attribute to describe the location of the region annotation item.

[0494] Possibly, the shape of the area annotation item is described as a rectangle using the "ispe" item attribute.

[0495] Possibly, use the "clap" item attribute to describe the geometry of the region annotation item.

[0496] For example, in Figure 2bIn the example, region 261 can be represented by a region annotation item "rgan" without any content. The region annotation item is associated with the image item through a reference of type "rgan". The location of region 261 is represented by the "rloc" item attribute. The shape of region 261 is represented by the "ispe" item attribute.

[0497] Group Alternatives

[0498] The sixth embodiment of the present invention may be combined with the fifth embodiment, so that a region annotation item can be associated with an image item directly and / or through a region group.

[0499] The sixth embodiment of the present invention may be combined with the seventh embodiment, so that a region annotation item can be associated with an image item directly and / or through a region group.

[0500] Possibly, in the sixth embodiment, an item attribute may be associated with a region group to indicate that the attribute is applicable to all region annotation items included in the region group. For example, a shape attribute may be associated with a region group to describe the shapes of all region annotation items of the group. As another example, a user description item attribute may be associated with a region group to associate the user description item attribute with all region annotation items of the group.

[0501] In the sixth embodiment of the present invention, the grouping_type used by the regional group may be "rcog".

[0502] Possibly, a region group can be associated with an image item using a different reference type. A new reference type (eg "rcob") can be defined to describe such an association.

[0503] Using some of these alternatives, as examples, from Figure 2b An image can be stored in a HEIF file with the following structure:

[0504]

[0505]

[0506]

[0507] Associating other items with a region

[0508] In an embodiment, an item (regardless of its type) may be associated with a region of the first image. This association may be made directly from the item to the region, or indirectly by associating an entity group containing the item with the region. The region may be described by a region annotation item or by region annotation item attributes. An item or entity group may be associated with a region annotation item by an entry of the "iref" box.

[0509] For example, an image item or entity group may be associated with a region annotation item by an item reference frame of type "eroi" (for "encoded region of interest"). The reference type "eroi" is an example, and other reference types may be used, such as "rgim" etc.

[0510] Associations may be directed from a region annotation item to an image item (or to an entity group), or from an image item (or to an entity group) to a region annotation item.

[0511] Alternatively, an item (or entity group) may be associated with a region note item attribute via a new item attribute box that is associated with the region note item attribute and references the item (or entity group). The new item attribute box may be an entry of an "iref" box. The new item attribute box may contain a reference type, a reference count, and a list of referenced item identifiers.

[0512] In a variation, an image item named "ROI image item" may be associated with a region annotation item or a region annotation item attribute to provide alternative or supplementary content for the region. The ROI image item may have a higher resolution than the first image, or may have a higher quality than the first image. Alternatively, the ROI image item may also correspond to a field of view different from the field of view of the first image.

[0513] For example, if the first image is a photograph of Notre Dame de Paris and its surroundings during the 2019 fire, the region corresponds to Notre Dame de Paris itself, and the ROI image item may be a photograph of Notre Dame de Paris before the fire.

[0514] In another variation, an entity group named "ROI entity group" may be associated with a region annotation item or a region annotation item attribute to provide an alternative content set for the region. The corresponding entity group box may be of type "altr".

[0515] For example, the ROI entity group may include a first image having a higher resolution representation of the region, a second image having a higher quality representation of the region, and a third image having an HDR representation of the region.

[0516] Other examples of items that may be associated with a region, either directly or as part of an entity group, through a region annotation item or region annotation item attributes, are: a depth map for the region, a thumbnail for the region, and / or a quality or resolution refinement layer for the region.

[0517] When an image (hereinafter named "second image") is associated with a region of a first image, either directly or through a group, the precise positioning of the second image relative to the first image may depend on the geometry of the region. If the region is a point, the center of the second image is located at the point of the first image. If the region is a rectangle, the second image may fill the rectangle within the first image. Otherwise, the second image may fill the bounding box of the region.

[0518] In an embodiment, the precise positioning of the second image relative to the first image can be refined by using item attributes associated with the region. If the region is a point, the offset of the center of the second image and the width and height of the second image can be specified. If the region is a rectangle, the offset relative to the upper left corner of the rectangle and the width and height of the second image can be specified. Otherwise, if the region is neither a point nor a rectangle, the offset relative to the upper left corner of the bounding box and the width and height of the second image can be specified. These offsets, widths, and heights can be specified using a new item attribute or several new item attributes. Alternatively, these offsets, widths, and heights can also be specified using the "rloc" and / or "iscl" item attributes.

[0519] In an embodiment, the second image may be larger than the area of ​​the first image, and then only a portion of the second image may correspond to the area. This configuration may be indicated with a specific item attribute (e.g., "clap") associated with the image item or the entity group containing the image item. This configuration may be indicated with a new item attribute associated with the region annotation item or with a region annotation item attribute.

[0520] Misc alternatives

[0521] Possibly, in Figure 7 After step 790 of the illustrated process, the transformability attributes associated with the image item are extracted. The transformability attributes are then applied to the geometry of the annotated image item and its associated region annotation item attributes according to their ordering within the "ipma" entry. This application may be based on Fig.14 The processing shown is implemented.

[0522] Possibly, in Figure 7 After step 1370 of the illustrated process, the transformability attributes associated with the image item are extracted. The transformability attributes are then applied to the geometry of the annotated image item and its associated region annotation item according to their ordering within the "ipma" entry. This application may be based on Fig.14 The processing shown is implemented.

[0523] Possibly, a region annotation is applied to the output image of the image item.

[0524] Possibly, a region annotation may be associated with several image items. For example, the same region annotation may be applied to several image items corresponding to consecutive images.

[0525] Possibly, a region annotation may be associated with several image items. Additionally, geometry modifier attributes are associated with the respective associations to adapt the geometry of the region to the respective image item. Geometry modifier attributes may be associated with regions and / or region annotations and may contain an indication of the image item to which they apply. Geometry modifier attributes may be associated with image items and may contain an indication of the region and / or region annotation to which they apply. For example, the same region annotation may be associated with several image items corresponding to consecutive images containing moving objects. Geometry modifier attributes change the geometry of the region for fitting the moving objects in the respective image item.

[0526] Possibly, a region annotation can be associated with a group of items. This association means that the region annotation applies to each item included in the group.

[0527] Possibly, the reference between a region annotation item or region group and an image item may be reversed. For example, an association between one or more region annotation items and an image item may use a reference of type "rgan" from an image item to one or more region annotation items.

[0528] Possibly, region annotations may be described by new types of boxes. The structure of these boxes may be similar to the structure described for geometry item attributes, position item attributes, and / or shape item attributes. These boxes may have identifiers associated with them. These boxes may be associated with items in the "iref" box or in a new box. These boxes may be associated with attributes in the "ipma" box or in a new box.

[0529] Finally, although embodiments have been described with reference to region annotation items, they may be more generally applicable to region items. The purpose of a region item is to characterize the geometry of a region, not necessarily with an associated annotation, but may be used to associate any kind of object (such as an image, video, or sound, etc.) with a region of an image item.

[0530] Fig.15 is a schematic block diagram of a computing device 150 for implementing one or more embodiments of the present invention. The computing device 150 may be a device such as a microcomputer, a workstation, or a light portable device. The computing device 150 includes a communication bus connected to:

[0531] - a central processing unit 151 denoted as CPU, such as a microprocessor or the like;

[0532] - a random access memory 152 marked as RAM, used to store executable code of the method according to the embodiment of the present invention and registers suitable for recording variables and parameters, which are necessary for implementing the method according to the embodiment of the present invention, and the memory capacity of the random access memory can be expanded, for example, by connecting to an optional RAM of the expansion port;

[0533] - a read-only memory 153 marked as ROM, for storing a computer program for implementing an embodiment of the present invention;

[0534] - A network interface 154, which is usually connected to a communication network through which digital data to be processed is sent or received. The network interface 154 may be a single network interface, or may be composed of a collection of different network interfaces (e.g., wired and wireless interfaces, or different kinds of wired or wireless interfaces). Under the control of a software application running in the CPU 151, data packets are written to the network interface for sending, or data are read from the network interface for receiving;

[0535] - User interface 155 may be used to receive input from a user or display information to a user;

[0536] - A hard disk 156 labeled as HD can be provided as a mass storage device;

[0537] - The I / O module 157 may be used to receive / send data from / to an external device (such as a video source or a display, etc.).

[0538] The executable code may be stored in the read-only memory 153, may be stored on the hard disk 156, or may be stored on a removable digital medium (e.g., a disk). According to a variant, the executable code of the program may be received via the network interface 154 by means of a communication network so that the executable code of the program is stored in one of the storage components of the communication device 600 (e.g., the hard disk 156, etc.) before being executed.

[0539] The central processing unit 151 is adapted to control and direct the execution of instructions or parts of software code of one or more programs according to embodiments of the present invention, which instructions are stored in one of the aforementioned storage components. After being powered on, the CPU 151 is able to execute instructions from the main RAM memory 152 related to the software application after loading these instructions, for example, from the program ROM 153 or the hard disk (HD) 156. Such software applications, when executed by the CPU 151, cause the steps of the flowchart of the present invention to be performed.

[0540] Any step of the algorithm of the present invention may be implemented in software by executing instructions or a set of programs by a programmable computing machine (such as a PC ("personal computer"), a DSP ("digital signal processor") or a microcontroller, etc.); or may be implemented in hardware by a machine or a dedicated component (such as an FPGA ("field programmable gate array") or an ASIC ("application-specific integrated circuit"), etc.).

[0541] Although the present invention has been described above with reference to specific embodiments, the present invention is not limited to the specific embodiments, and modifications within the scope of the present invention will be apparent to those skilled in the art.

[0542] Many further modifications and variations will occur to those skilled in the art when referring to the foregoing illustrative embodiments, which are given as examples only and are not intended to limit the scope of the invention, which is determined solely by the appended claims. In particular, different features from different embodiments may be interchanged where appropriate.

[0543] The various embodiments of the present invention described above may be implemented individually or as a combination of multiple embodiments. In addition, features from different embodiments may be combined when necessary or when a combination of elements or features from separate embodiments is beneficial in a single embodiment.

[0544] In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. The mere fact that different features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be used to advantage.

Claims

1. A method for encapsulating an image (250) in an ISOBMFF-based media file (800), the method comprising: Get Image (250); generating an image item (810) representing the image (250); generating geometric descriptive data (821, 826), the geometric descriptive data describing the geometry of a region (260, 261, 270, 280) of the image (250); generating a region item (820, 825) corresponding to the geometric figure described by the geometric descriptive data (821, 826), wherein the region item (820, 825) is associated with the image item (810) via item reference information (831, 832); generating at least one annotation data structure (861, 862) including data associated with the region of the image, the at least one annotation data structure being associated (851, 852) with the region entry (820, 825); generating the media file (800), the media file (800) comprising the image (250), the image item (810), the region item (820, 825) and the at least one annotation data structure (861, 862), The region item is associated with at least one generated annotation data structure through association information in an association container.

2. The method according to claim 1, wherein: The association container is an item attribute association box; and The region item is associated with the at least one annotation data structure generated as an item attribute.

3. The method according to claim 1, wherein: The associated container is an item reference frame; and The region item is associated with the at least one annotation data structure generated as an item.

4. The method according to claim 1, wherein: The associated container is an item reference frame; and The region item is associated with a grouping data structure that references the at least one annotation data structure generated as an item.

5. The method according to claim 1, wherein: The region items are related to a plurality of regions of the image; The geometric descriptive data describes the geometry of each of the plurality of regions of the image; as well as The at least one annotation data structure is associated with at least one region of the plurality of regions of the image.

6. The method according to claim 5, wherein: The at least one annotation data structure is associated with all regions of the plurality of regions of the image.

7. The method according to claim 1, wherein: The geometry of the area of ​​the image is a point, a 3d box or a rectangle or an ellipse or a sphere or a polygon or a polyline.

8. The method according to claim 1, wherein: The geometry of the region of the image is a mask.

9. The method according to claim 8, wherein: The data of the mask is included in the image item associated with the region item through item reference information.

10. The method of claim 1, wherein: At least one region item associated with the image item is grouped, and the group is associated with the image item through item reference information.

11. The method according to claim 6, wherein: The region item includes count information indicating the number of the plurality of regions in the image.

12. The method of claim 1, wherein: The at least one annotation data structure is an item property box included in an item property container box, namely "ipco", and wherein the area item is associated with the at least one annotation data structure (861, 862) in the item property container box using an item property association box, namely "ipma".

13. A method for reading a media file based on ISOBMFF, the method comprising: Obtaining the media file, the media file comprising: an image item (810) representing an image (250); geometric descriptive data (821, 826) describing a geometry of a region (260, 261, 270, 280) of the image (250); a region item (820, 825) corresponding to the geometry described by the geometric descriptive data (821, 826), the region item (820, 825) being associated with the image item (810) via item reference information (831, 832); and at least one annotation data structure (861, 862) comprising data associated with the region of the image, the at least one annotation data structure being associated (851, 852) with the region item (820, 825); Reading the region item from the media file; reading the at least one annotation data structure associated with the region item in the media file; reading the image item from the media file, The region item is associated with at least one generated annotation data structure through association information in an association container.

14. The method according to claim 13, wherein: The association container is an item attribute association box; and The region item is associated with the at least one annotation data structure generated as an item attribute.

15. The method of claim 13, wherein: The associated container is an item reference frame; and The region item is associated with the at least one annotation data structure generated as an item.

16. A computer program product for a programmable device, the computer program product comprising a sequence of instructions which, when loaded into and executed by the programmable device, implements the method according to any one of claims 1 to 15.

17. A computer-readable storage medium storing instructions of a computer program for implementing the method according to any one of claims 1 to 15.

18. An apparatus for encapsulating an image (250) in an ISOBMFF-based media file (800), the apparatus comprising a processor configured to: Get Image (250); generating an image item (810) representing the image (250); generating geometric descriptive data (821, 826), the geometric descriptive data describing the geometry of a region (260, 261, 270, 280) of the image (250); generating a region item (820, 825) corresponding to the geometric figure described by the geometric descriptive data (821, 826), wherein the region item (820, 825) is associated with the image item (810) via item reference information (831, 832); generating at least one annotation data structure (861, 862) including data associated with the region of the image, the at least one annotation data structure being associated (851, 852) with the region entry (820, 825); generating the media file (800), the media file (800) comprising the image (250), the image item (810), the region item (820, 825) and the at least one annotation data structure (861, 862), The region item is associated with at least one generated annotation data structure through association information in an association container.

19. An apparatus for reading a media file based on ISOBMFF, the apparatus comprising a processor, the processor being configured to perform the following operations: Obtain the media file, where the media file includes: an image item (810) representing an image (250); geometric descriptive data (821, 826) describing the geometry of a region (260, 261, 270, 280) of the image (250); a region item (820, 825) corresponding to the geometry described by the geometry descriptive data (821, 826), the region item (820, 825) being associated with the image item (810) via item reference information (831, 832); and at least one annotation data structure (861, 862) comprising data associated with the region of the image, the at least one annotation data structure being associated (851, 852) with the region item (820, 825); Reading the region item from the media file; reading the at least one annotation data structure associated with the region item in the media file; reading the image item from the media file, The region item is associated with at least one generated annotation data structure through association information in an association container.

Images