Method and apparatus for storing information relating to regions of an image in an image file - Patents.com

Abstract

To provide a method, program and device for storing image data in a media file compliant with the ISOBMFF standard.SOLUTION: A method includes: acquiring image data; generating image item information corresponding to the image data; generating geometry description data describing a geometry of a region in a part of an image based on the image data and region item information corresponding to the geometry description data; generating first association information for associating the region item information with the image item information; and generating a media file storing the image data, the geometry description data, the image item information, the region item information and the first association information.SELECTED DRAWING: Figure 2a

Description

[Technical field]

[0001] The present disclosure relates to a method and apparatus for storing information relating to regions of an image within an image file. [Background technology]

[0002] Modern cameras and image analysis services make it possible to generate local metadata for images. Local metadata is metadata that pertains to a region (part) of media content, rather than the entire media content. The media content is usually an image, but it could also be video content or a collection of images. For example, a camera can generate focus regions for a photo or detect faces while taking a photo. Or a deep learning system might identify objects in an image. These local metadata can be considered as region annotations.

[0003] Images captured by a camera or processed by an image analysis service are stored on a storage device, for example a memory card. Images are usually encoded to reduce the data size on the storage device. Many encoding standards can be used, such as JPEG, AV1, or more recently HEVC.

[0004] The HEVC standard defines profiles for the coding of still images and describes specific tools for compressing single still images or still image bursts. An extension to the ISO Base Media File Format (ISOBMFF) for such image data has been proposed in Part 12 of the ISO / IEC 23008 standard under the name "HEIF" (High Efficiency Image File Format).

[0005] HEIF (High Efficiency Image File Format) is a standard developed by MPEG (Movie Experts Group) for storing and sharing images and image sequences.

[0006] MIAF (Multi-Image Application Format) is a standard established by MPEG in Part 22 of the ISO / IEC 23000 standard. The MIAF specification defines the Multi-Image Application Format (MIAF), a multimedia application format, and precisely realizes interoperability points for creating, reading, parsing, and decoding images embedded in the High Efficiency Image File Format (HEIF). The MIAF specification is fully compliant with the HEIF format, and only defines additional constraints to ensure higher interoperability.

[0007] While HEIF and MIAF can store documents containing metadata like EXIF ​​and XMP, they do not provide a mechanism for associating annotations with regions of an image. Summary of the Invention

[0008] The present invention has been made to address one or more of the above-mentioned concerns.

[0009] According to a first aspect of the present invention, a method for storing image data in a media file conforming to the ISOBMFF standard comprises: acquiring image data; generating image item information corresponding to the image data; generating geometry description data describing a geometry of a partial region of an image based on the image data, and region item information corresponding to the geometry description data; generating first association information for associating the area item information with the image item information; generating the media file storing the image data, the geometry description data, the image item information, the region item information, and the first association information; Includes.

[0010] In one embodiment, the image data and the geometry description data are stored in a data portion of the media file; The image item information, the region item information and the first association information are stored in a metadata portion of the media file.

[0011] In one embodiment, The image item information and the area item information are described in an ItemInfoBox.

[0012] In one embodiment, The first association information is described in an ItemReferenceBox.

[0013] In one embodiment, The geometry description data includes information describing one of a point, a 3D box, a rectangle, an ellipse, a sphere, a polygon, a line, and a mask.

[0014] In one embodiment, When the geometry description data includes information indicating a mask, data of the mask of a partial region of the image is associated with the region item information by an item reference.

[0015] In one embodiment, The geometry description data includes information indicating a plurality of geometry types of the partial region.

[0016] In one embodiment, generating at least one annotation data structure corresponding to the domain item information; generating second association information for associating the domain item information with the at least one annotation data structure; storing the at least one annotation data structure and the second association information in the media file; Further includes:

[0017] In one embodiment, The at least one annotation data structure is described in an ItemProprtiesBox, The second association information is described in ItemPropertyAssociation 'ipma'.

[0018] In one embodiment, The at least one annotation data structure is described in an ItemInfoBox, The second association information is described in an ItemReferenceBox.

[0019] In one embodiment, The at least one annotation data structure includes information regarding focus positions, faces, and objects.

[0020] According to another aspect of the present invention, a method for processing a media file compliant with the ISOBMFF standard comprises the steps of: Obtaining a media file including image data, image item information corresponding to the image data, geometry description data describing a geometry of a partial region of an image based on the image data, region item information corresponding to the geometry description data, and first association information for associating the region item information with the image item information; processing the media file; Includes.

[0021] According to another aspect of the invention there is provided a computer readable storage medium having stored thereon computer program instructions for carrying out a method according to the invention. According to another aspect of the present invention, there is provided a program for causing a computer to carry out the method according to the present invention.

[0022] According to another aspect of the present invention, an apparatus for storing image data in a media file conforming to the ISOBMFF standard comprises: A means for acquiring image data; means for generating image item information corresponding to the image data; means for generating geometry description data describing a geometry of a partial region of an image based on the image data, and region item information corresponding to the geometry description data; means for generating first association information for associating the area item information with the image item information; means for generating the media file, the media file storing the image data, the geometry description data, the image item information, the area item information, and the first association information; Equipped with.

[0023] According to another aspect of the present invention, an apparatus for processing a media file compliant with the ISOBMFF standard comprises: A means for acquiring a media file including image data, image item information corresponding to the image data, geometry description data describing a geometry of a partial area of ​​an image based on the image data, area item information corresponding to the geometry description data, and first association information for associating the area item information with the image item information; means for processing the media file; Equipped with.

[0024] At least some of the methods according to the present invention may be computer-implemented. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be referred to generally herein as a "circuit," "module," or "system." Furthermore, the present invention may take the form of a computer program product embodied in any tangible medium of expression having computer usable program code embodied in the medium.

[0025] Since the present invention can be implemented in software, the present invention can be embodied as computer readable code for provision to a programmable device on any suitable carrier medium. The tangible non-transitory carrier medium may consist of a storage medium such as a floppy disk, a CD-ROM, a hard disk drive, a magnetic tape device or a solid-state memory device. The transitory carrier medium may include a signal such as an electric signal, an electronic signal, an optical signal, an acoustic signal, a magnetic signal or an electromagnetic signal, e.g., a microwave or RF signal. [Brief description of the drawings]

[0026] Embodiments of the present invention will now be described, by way of example only, with reference to the following drawings, in which:

[0027] [Figure 1] FIG. 1 shows an example of a HEIF file 101 that contains one or more still images and possibly media data such as a video or sequence of images. [Figure 2a] FIG. 2a shows a high-level view of the present invention for associating annotations with regions of an image. [Figure 2b] Figure 2b shows an example of region annotation. [Diagram 3] FIG. 3 is a diagram showing a first embodiment of the present invention. [Figure 4] FIG. 4 is a diagram showing a second embodiment of the present invention. [Diagram 5] FIG. 5 is a diagram showing a third embodiment of the present invention. [Figure 6] FIG. 6 illustrates the main steps of a process for adding a new region annotation associated with an image item stored in a HEIF file, when the region annotation is described by an item property, according to an embodiment of the invention. [Figure 7] FIG. 7 illustrates the main steps of a process for reading a HEIF file containing region annotations, where the region annotations are described by item properties, according to an embodiment of the invention. [Figure 8]FIG. 8 is a diagram showing a fifth embodiment of the present invention. [Figure 9] FIG. 9 is a diagram showing a sixth embodiment of the present invention. [Figure 10] FIG. 10 is a diagram showing a seventh embodiment of the present invention. [Figure 11] FIG. 11 illustrates the main steps of a process for adding a new region annotation to an image item stored in a HEIF file when the item describes the region annotation, according to an embodiment of the invention. [Figure 12] FIG. 12 illustrates the main steps of a process for reading a HEIF file containing region annotations when the region annotations are described by an item, according to an embodiment of the invention. [Figure 13] FIG. 13 shows the main steps of a process for reading a HEIF file containing region annotations when the region annotations are described by an item according to a sixth embodiment. [Figure 14] FIG. 14 illustrates a process for processing a HEIF file that includes an image and one or more region annotation items associated with the image, according to an embodiment of the invention. [Figure 15] FIG. 15 is a schematic block diagram of a computing device for implementing one or more embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] The HEVC standard defines profiles for the coding of still images and describes specific tools for compressing single still images or still image bursts. An extension to the ISO Base Media File Format (ISOBMFF) for such image data has been proposed in Part 12 of the ISO / IEC 23008 standard under the name "HEIF" (High Efficiency Image File Format).

[0029] The HEIF and MIAF standards cover two types of storage that serve different use cases. - storing image sequences that can be ordered to be displayed chronologically or by other means, where images may depend on other images; -Storage of a single coded image or a collection of independent coded images (possibly including derived images).

[0030] In the first case, the encapsulation is done using the ISO Base Media File Format (see ref.<Information technology - Coding of audiovisual objects - Part 12: ISO base media file format> It is close to the encapsulation of video tracks in ISO / IEC 14496-12, 6th edition, October 2019 (see ISO / IEC 14496-12, 6th edition, October 2019) and uses similar tools and concepts such as "trak" boxes and sample groupings for describing sample groups. A "trak" box is a box in the file format that contains sub-boxes for describing tracks (i.e. temporal sequences of related samples).

[0031] A box, also called a container, is a data structure provided to describe the data within a file. A box is an object-oriented building block defined by a unique type identifier (usually a four-letter code, also written as FourCC or 4CC) and a length. All data within a file (media data and metadata describing the media data) is contained in a box. No other data exists within the file. A file-level box is a box that is not contained within another box.

[0032] In the second case, a set of ISOBMFF boxes is used: the "meta" boxes. These boxes and their hierarchy offer fewer descriptive tools than the "track-related" boxes (the "trak" box hierarchy) and relate to "information items" or "items" instead of related samples. Note that the expressions "box" and "container" are sometimes used interchangeably to refer to data structures that contain metadata describing the organization and / or characteristics of the image data in a file.

[0033] FIG. 1 shows an example of a HEIF file 101 that contains media data such as one or more still images and possibly one or more videos and / or one or more image sequences. The file contains a first "ftyp" box (FileTypeBox) 111 that contains an identifier of the type of the file (typically a set of four character codes). The file contains a second box called "meta" (MetaBox) 102 that is used to contain general non-timed metadata including metadata structures describing one or more still images. This "meta" box 102 contains an "iinf" box (ItemInfoBox) 121 that describes several single images. Each single image is described by a metadata structure ItemInfoEntry, also denoted as items 1211 and 1212. Each item has a unique 16-bit or 32-bit identifier item_ID. The media data corresponding to these items is stored in an "mdat" box 104, which is a container for media data. An "iloc" box (ItemLocationBox) 122 provides for each item the offset and length of its associated media data in the "mdat" box 104. Additionally, an "iref" box (ItemReferenceBox) 123 may be defined to describe the relationship of one item to other items by typed reference.

[0034] Optionally, to describe the storage of image sequences or videos, the HEIF file 101 can contain a third box called "moov" (MovieBox) 103, which describes one or more image sequence or video tracks 131 and 132. Typically, track 131 may be an image sequence ("pict") track designed to describe a collection of images where time information is not necessarily meaningful, and 132 may be a video ("vide") track designed to describe video content. Both tracks describe a series of image samples, where an image sample is a set of pixels captured at the same time, for example frames of a video sequence. The main difference between the two tracks is that in the "pict" track, timing information is not necessarily meaningful, whereas in the "vide" track, timing information is intended to constrain the display timing of the samples. The data corresponding to these samples is stored in the "mdat" box 104, which is a container for media data.

[0035] The “mdat” container 104 stores non-time-specified encoded images corresponding to items represented by data portions 141 and 142, and a time-specified encoded image corresponding to a sample represented by a data portion 143.

[0036] HEIF files 101 offer different options for storing multiple images: for example, multiple images can be stored as items or as tracks of samples that can be "pict" tracks or "vide" tracks. The actual choice is typically made by the application or device generating the file, according to the type of image and the expected use of the file.

[0037] The ISO Base Media File Format specifies several options for grouping, depending on the container that stores the samples or items to be grouped. These options can be viewed as grouping data structures or mechanisms (i.e., boxes or data structures that provide grouping criteria, group properties, and metadata describing the group entity).

[0038] The first grouping mechanism, represented by EntityToGroupBox, is adapted for grouping of items or tracks. In this mechanism, the expression "entity" is used to refer to items or tracks or other EntityToGroupBoxes. This mechanism specifies a grouping of entities. EntityToGroupBox is defined with the following syntax: TIFF0007654130000001.tif23157TIFF0007654130000002.tif14157

[0039] grouping_type is used to specify the type of group. In HEIF, several values ​​are specified for grouping_type. group_id indicates the identifier of a group of entities. entity_id indicates the identifier of an entity that composes a group. That is, track_ID is specified for a track, item_ID for an item, and another group_id for an entity group. In FIG. 1, the entity groups inherited from the EntityToGroup boxes 1241 and 1242 are configured in a container 124 identified by the four-character code "gprl" of GroupsListBox.

[0040] Entity grouping is the association of a grouping type that indicates the reason for grouping a set of entity groups such as items and tracks. In this document, the EntityToGroupBox that conveys information for grouping a set of images is called grouping information.

[0041] ISOBMFF provides a mechanism to describe and associate properties with items. These properties are called item properties. The ItemPropertiesBox "iprp" 125 allows the association of any item with an ordered set of item properties. The ItemPropertiesBox consists of two parts: an item properties container box "ipco" 1251 that contains an implicit indexed list 1253 of item properties, and an item properties association box "ipma" 1252 that contains one or more entries. Each item in the item properties association box associates an item with its item properties. The HEIF standard extends this mechanism to allow the association of item properties with items and / or entity groups. In this description, we note that item properties are generally used to specify both properties of items and properties of entity groups. Item properties associated with an entity group apply to the entity group as a whole and not to each entity in the group individually.

[0042] The relevant syntax is: TIFF0007654130000003.tif39157TIFF0007654130000004.tif134157

[0043] The ItemProperty and ItemFullProperty boxes are designed to describe item properties. ItemFullProperty can define multiple versions of the box's syntax and can contain one or more parameters whose presence is conditioned by the version or flags parameters.

[0044] The ItemPropertyContainerBox is designed to describe a collection of item properties as an array of ItemProperty or ItemFullProperty boxes.

[0045] ItemPropertyAssociation is a box for describing the association between items or entity groups and their item properties. It associates an item identifier (item_ID) and a list of item property indexes that refer to the item properties of an ItemPropertyContainerBox, and provides a description of the list of item identifiers or entity group identifiers.

[0046] None of these mechanisms make it possible to describe some properties associated with parts, or regions, of an image. The present invention aims to solve this problem by providing a way to describe region annotations and to associate these region annotations with actual regions of an image.

[0047] 2a shows a high-level view of the invention for associating annotations with regions of an image. A region annotation 210 is associated with an entity 200. This entity may be, for example, an image item (e.g., an encoded image item or a derived image item) that describes an image, or it may be any type of item or entity group. This association means that the region annotation 210 constitutes information related to a region that represents part of the media content, image, or entity group described by the entity 200.

[0048] If the entity is an image item, the reconstructed image is the image that results from decoding an encoded image item or applying a derived image item operation to a set of input images (which are themselves encoded image items or derived image items). The output image is the image that results from applying potential transformation item properties of the image item to the reconstructed image. For example, the output image may be the image that results after decoding data related to the item and applying transformation item properties to the decoded image. The transformation item properties include, for example, an "irot" transformation item property to rotate an image, a "clap" transformation item property to crop an image, or an "imir" transformation item property to mirror an image. The reconstructed image of the entity 200 is an annotation image with region annotations 210 applied, meaning that the regions are part of the reconstructed image.

[0049] This region annotation 210 may be defined by its geometry 220. The geometry may be defined as a position, a position and a shape, a set of points, or a mask. For example, the geometry of a region may be a rectangle defined by its top left corner and its width and height. For example, a set of points may represent a polygon defined by the coordinates of all its vertices. The geometry may be defined by other means. The geometry may be divided into a position and, optionally, a shape, e.g., a point (in this case the shape is only a position), a rectangle, a circle, etc. The position is the location on the annotated image of the reference point of that shape. For example, the reference point may be the top left corner of a rectangle, the center of a circle.... The shape may also be defined as a mask that selects individual pixels in the image that correspond to the entity 200. It is also possible to use other geometries, such as a 3D box projected onto the image plane. The geometry may be described with parameters that provide one generic identifier and its type (point, rectangle, circle...) or as a specific identifier for each type. For images encapsulated within a HEIF file, this identifier is a four-letter code.

[0050] Region annotation 210 may be linked to one or more annotations 230, 231, and 232. These annotations are used to store different annotations corresponding to region 210 of entity 200. The annotations may be, for example, a focus position of an image, a face detected in the image, an object detected in the image, a GPS location corresponding to an object in the image, a description for a portion of an image, text contained in a region of an image, a user annotation for a portion of an image, or any kind of information associated with a region.

[0051] Fig. 2b shows an example of region annotation. An image 250 is captured, for example, by a camera. The camera adds information about two detected human faces at the time of capture. The faces correspond to regions 260, 261 in the image 250. Both regions are rectangular regions. This captured image can be stored in a HEIF file using the present invention.

[0052] The image is then further processed to enhance the information describing the image, for example during the transfer of image 250 to an online-based photo storage. First, a recognition algorithm finds the name of the person whose face is in region 261. Second, an object detection algorithm detects and identifies the building in region 270 of image 250, which is a circular region. These processing steps result in an annotated image that can be stored in a HEIF file using the present invention. For example, the HEIF file containing image 250 is edited to store a description of the new region annotation 270 and / or the name of the detected person in region 261.

[0053] Finally, the user edits the image 250, for example to leave only the part containing a known person in region 261 and the part containing a building in region 270. She also corrects the perspective distortion. As a result, only part 280 of the image 250 is kept. Due to the perspective correction, this part 280 is not rectangular. These editing steps result in an edited image that can be stored in a HEIF file using the present invention. And such an edited HEIF file may contain instructions to obtain the cropped image 280 in addition to the original image 250, or it may contain only the image description and the data of region 280.

[0054] In different embodiments, the geometry 220 of a region annotation may be represented using the following structure: TIFF0007654130000005.tif238157TIFF0007654130000006.tif30157

[0055] The semantics of this structure may be as follows: -type is an integer value that specifies the type of geometry. -field_size is an integer value that specifies the size in bits of the field used to describe the geometry (for example, 16 or 32 bits). -If the geometry is a point, x and y are the coordinates of this point in the image. -If the geometry is a rectangle, x and y are the coordinates of the top-left corner of the rectangle, width and height are the width and height of the rectangle. -If the geometry is a circle, x and y are the coordinates of the center of the circle, and radius is the radius of the circle. -If the geometry is a polygon, point_count is the number of points of the polygon, and for each point, x and y are the coordinates of this point in the image. The polygon is a closed polygon. -If the geometry is a 3D box, x_bottom_0, y_bottom_0, ..., x_bottom_3, y_bottom_3 are the coordinates of the bottom rectangle of the 3D box. x_top_0, y_top_0, ... x_top_3, y_top_3 are the coordinates of the top rectangle of the 3D box. -If the geometry is a bitmask or color mask, ox and oy define the position of the top-left pixel of the image mask in the annotation image. mask_item_ID is the identifier of the image item describing the mask image that defines the region. The output image of this image item becomes the mask image of the region. Preferably, the size of the mask image is the same as or smaller than the size of the annotation image. -If the geometry is a bitmask, the mask image should be a monochrome image (i.e. an image with 1-bit color components). The region is defined as all pixels in the mask image with black value, whose upper left corner is at the location defined by ox and oy. -If the geometry is a color mask, mask_color defines the color used to define a region in the mask image. A region is defined as all pixels in the mask image whose color value is mask_color. The upper left corner of the region is at the position defined by ox and oy.

[0056] Or, in another example, we can split the geometry of a region into two structures that correspond to its position and shape: TIFF0007654130000007.tif205157TIFF0007654130000008.tif47157

[0057] The semantics of the Location structure in this example are as follows: -field_size is an integer value that specifies the size in bits of the field used to describe the Location (eg 16 bits, 32 bits). -x,y are the coordinates of the area in the image.

[0058] The semantics of the Shape structure are similar to those of the Geometry structure, with the following changes: - For every shape, the location on the annotation image of the shape's reference point is specified by position. -If the shape is a point, the reference point is the point itself. - If the shape is a rectangle, its top left corner is used as the reference point. -If the shape is a circle, the center is used as the reference point. -If the shape is a polygon, the first point is the reference point. For the other points, dx and dy are the coordinates of the points relative to the reference point. -If the geometry is a 3D box, the first point of the base rectangle is the reference point, and all other points are specified in coordinates relative to the reference point. - If the geometry is a bitmask or color mask, the position specifies the location of the top-left pixel of the image mask within the annotation image.

[0059] In some embodiments, points for polygon shapes and / or 3d box shapes are specified using absolute coordinates rather than relative coordinates.

[0060] The units used to define the coordinates, width, height, and radius of a geometry may be integer or fixed-point 16.16 values ​​of pixels or luma samples, or they may be other units. It is also possible to specify units in geometry, position, and shape structures.

[0061] 3-7 show some embodiments of the present invention for the HEIF or MIAF standard, where region annotations 210 are expressed using item properties 330. In the embodiment shown in FIG.

[0062] Figure 3 illustrates a first embodiment of the invention. The "meta" box 300 contains an "item" 310, which corresponds for example to an image item. The "meta" box 300 also contains an "iprp" box 320, which itself contains two boxes. First, the "ipco" box 323 contains item properties of the different entities present in the "meta" box. Secondly, the "ipma" box 321 contains associations between item properties and entities (for example image items or entity groups).

[0063] The "ipco" box 323 contains a first item property, a region annotation 330. This region annotation contains two parts: first, a description of its geometry 331, and second, links 332 to annotations for the region. These links associate item properties 340 and 341 with the region.

[0064] The “ipma” box 321 associates, through its entry 322 , a region annotation 330 with the item 310 .

[0065] In this embodiment, the region annotation is an item property and can be defined as follows: -Box Type: "rgan" -Property type: Descriptive item property -Container: ItemPropertyContainerBox -Required (per item): No -Quantity (per item): 0 or more

[0066] The syntax for the area annotation item property can be as follows: TIFF0007654130000009.tif51157

[0067] The semantics of the area annotation item property can be as follows: -geometry is the geometry structure of the region. In a variant, the geometry 331 can be described as a Location structure, and optionally a Shape structure. -entry_count is the number of item properties (340, 341...) associated with the region. -property_index is either 0, indicating that no property is associated, or the 1-based index of the associated property box within the ItemPropertyContainerBox.

[0068] In this embodiment, all item properties associated with a region within region annotation item properties 330 by its property_index apply to the region of the annotated image defined by geometry 331.

[0069] In this embodiment, the annotation image 250 from FIG. 2b may be stored in a HEIF file with the following structure: TIFF0007654130000010.tif163157

[0070] Note that in this description of the HEIF file structure, as well as in subsequent descriptions of the HEIF file structure, portions that are not necessary for understanding the present invention are omitted. For example, the version fields in the "meta" and "iinf" boxes are omitted, and the item_name field in the "infe" box is also omitted.

[0071] In this HEIF file, image item 250 is represented by an item with an item_ID value of 1. Region 260 is represented by an item property with index 3 and is associated with an item property with index 6, which corresponds to a face annotation. Region 261 is represented by an item property with index 4 and is associated with item properties with indexes 6 and 7, which correspond to a face annotation and an annotation describing a person named "Jean", respectively. Region 270 is represented by an item property with index 5 and is associated with an item property with index 8, which corresponds to an annotation describing "Notre-Dame de Paris" as a building.

[0072] In this HEIF file, the association between a region and its item properties is described in the "rgan" item property that represents the region.

[0073] In a first variation of the first embodiment, the syntax of the region annotation item property 330 may be as follows: TIFF0007654130000011.tif63157

[0074] In this first modification, a plurality of regions and their associated item properties can be defined in the item properties of one region annotation 330.

[0075] In a second variation to this embodiment, the syntax of the region annotation item property 330 may be as follows: TIFF0007654130000012.tif22157TIFF0007654130000013.tif43157

[0076] This second variation allows one region annotation item property 330 to define multiple regions that share a related set of item properties.

[0077] By combining the first and second modified examples, it becomes possible to define a plurality of region sets that share related item properties for one region annotation item property.

[0078] 4 illustrates a second embodiment of the present invention. This embodiment is similar to the first embodiment, except for the association of the region annotation item property 330 with its item properties (340, 341). In this embodiment, the association is achieved through an "ipma" box 321. This "ipma" box contains a new item 423 that associates the region annotation 330 with the item properties 340, 341.

[0079] In this embodiment, the syntax of the region annotation item property 330 may be as follows: TIFF0007654130000014.tif17157

[0080] And the syntax for region annotations can be: TIFF0007654130000015.tif30157

[0081] As in the first embodiment, the geometry can be described as a Location structure and, optionally, a Shape structure.

[0082] To allow the association of item properties with area annotation item properties, the syntax of the "ipma" box can be as follows: TIFF0007654130000016.tif156157

[0083] The semantics of the region annotation association loop for the "ipma" box can be as follows: -region_count is the number of region annotation item properties whose association is defined in the "ipma" box. This number may be encoded in 8 or 16 bits. Also, an upper limit on the number of region annotation item properties present in a HEIF file can be specified. -from_property_index is the 1-based index of the area annotation item property box with which the item property is associated. -association_count is the number of item properties associated with the area annotation item property box. -to_property_index is the 1-based index of the item property box that is related to the region annotation item property box. The value of the to_property_index field may be constrained to be different from the value of the from_property_index field.

[0084] To maintain backward compatibility with existing HEIF files and implementations, two new versions of the "ipma" box can be defined. These new versions (numbered 2 and 3) contain an association between a region and its item properties. These versions differ in the size of the item_ID field.

[0085] An index of an area annotation item property box MUST appear exactly once as a value of the from_property_index field of an ipma box. An index of an ipco corresponding to an area annotation item property box MUST NOT appear as a value of the to_property_index field of an "ipma" box.

[0086] The second bit of the box's flags field is used to define the size of the from_property_index and to_property_index fields in the region annotation association loop. Alternatively, the first bit of the flags field can be used to make the from_property_index, to_property_index, and property_index fields the same size. Different bits can be used to enable different sizes for the from_property_index and to_property_index fields.

[0087] In this second embodiment, the annotated image from FIG. 2b may be stored in a HEIF file with the following structure: TIFF0007654130000017.tif77157TIFF0007654130000018.tif122157

[0088] This HEIF file is similar to the corresponding one in the first embodiment, except for the association of the region annotation item property with its item property. In this HEIF file, the association of the region annotation item property 330 with its item properties 340, 341 is described inside the new version of the "ipma" box 321.

[0089] In a variant of the second embodiment, the entity association loop and the region annotation association loop are merged together. In this alternative embodiment, the item property index is used as an identifier for the region annotation item property that is used as the value of the item_ID field. Therefore, the item, entity group, track identifiers must not correspond to the item property index. The item, entity group, track identifiers may take precedence over the item property index. It is also possible to define a new version of the "ipma" box to inform the use of this alternative embodiment.

[0090] In another variation of the second embodiment, the entity association loop and the region annotation association loop are merged together. In this variation of the second embodiment, the first bit of the item_ID field is used to signal whether the item_ID field corresponds to an item, an entity group, or a track, or to an item property index. Possibly, a new version of the "ipma" box is defined to signal the use of this variation of the second embodiment.

[0091] 5 illustrates a third embodiment of the present invention. This embodiment is similar to the first and second embodiments, except for the association of a region annotation with its properties. In this embodiment, this association is achieved through an "ipma" box 321 as part of the association between an item 310 and its properties. This association is specified within entry 522 as a list of properties. Properties 340 and 341 corresponding to region annotation 330 are directly associated with item 310 within entry 522.

[0092] In this third embodiment, the order of properties within entry 522 indicates which portion of item 310 each item property applies to. Descriptive properties related to the entire item 310 come first, before any region annotation item properties. Next, each region annotation item property is listed, followed by its associated descriptive properties. Finally, transformation properties related to the entire item 310 are listed.

[0093] In this third embodiment, the descriptive item property is associated with the previous region annotation item property in the list, if there is one, or with the entire image if there is no previous region annotation item property.

[0094] In this third embodiment, the specific handling of region annotation properties can be demonstrated by defining a new item property type for this purpose, in this case "region item property".

[0095] In this third embodiment, the region annotation property may be marked as required to inform the reader that it should not be ignored.

[0096] In this third embodiment, the annotated image from FIG. 2b may be stored in a HEIF file with the following structure: TIFF0007654130000019.tif63157TIFF0007654130000020.tif113157

[0097] This HEIF file is similar to the corresponding ones of the first and second embodiments, except for the association of regions and their properties, which are described inside an "ipma" box, inside the property list associated with the image item to which the region annotation applies.

[0098] The image item with item_ID value 1, corresponding to image item 250, has nine properties associated with it. First, the properties with indexes 1 and 2 apply to the whole image. They describe the coding parameters used for the image and the size of the image. Next, the item property with index 3 describes a first region, corresponding to region 260. It is followed by an item property with index 6, which describes a face annotation, and is associated with this region. Next, the item property with index 4 describes a second region, corresponding to region 261. It is followed by two properties with indexes 6 and 7, which are associated with this region. These properties describe the face annotation, and the annotation of the person "Jean". Finally, the item property with index 5 describes a third region, corresponding to region 270. This item property is followed by an item property with index 8, which corresponds to the annotation of the building "Notre-Dame de Paris".

[0099] We now describe a fourth embodiment of the invention, not shown in the drawings. In this embodiment, the geometry of a region and its associated annotations are described inside a single (aggregated) item property. Preferably, a new item property is defined for each type of annotation that can be used for an entity's region.

[0100] The syntax for the area annotation item property can be as follows: TIFF0007654130000021.tif34157

[0101] This syntax combines fields for geometry and user-written item properties; their semantics do not change.

[0102] Here is a variant with the same semantics: TIFF0007654130000022.tif38157

[0103] Fig. 6 illustrates the main steps of a process for adding a new region annotation associated with an image item stored in a HEIF file according to an embodiment of the invention, corresponding to an embodiment in which the region annotation is described by item properties. These steps can be applied to HEIF files stored on disk, in memory, or in a representation adapted to memory. A new region annotation consists of the geometry of the region and the annotation itself. These steps can be modified to add multiple annotations to one region of an image item at the same time. Also, these steps can be modified to add annotations to several regions of an image item at the same time.

[0104] This process can be used to create new HEIF files or to modify existing HEIF files.

[0105] In a first step 600, it is determined whether the region already exists in the HEIF file. This is accomplished by comparing the geometry of the region to the geometry of the existing region.

[0106] If the region already exists, proceed to the next step 610; otherwise, proceed to the next step 620.

[0107] In step 610, a RegionAnnotationProperty structure that corresponds to an existing region is selected. The next step is step 640.

[0108] A new RegionAnnotationProperty structure is created to represent the region in step 620. The geometry of the region is stored inside this new RegionAnnotationProperty structure.

[0109] The new RegionAnnotationProperty structure is then associated with the image item by storing the association in the "ipma" box in step 630. The next step is step 640.

[0110] In step 640, it is determined whether an item property corresponding to the annotation already exists in the HEIF file. This can be done by parsing the item properties in the "ipco" container box. If the item property already exists, the next step is step 650, otherwise the next step is step 660.

[0111] In step 650, an existing item property is selected.

[0112] In step 660, a new item property is created to represent the annotation in the "ipco" container box. The type of the item property may depend on the content of the annotation. The information contained in the annotation is stored inside the item property.

[0113] After either step 650 or step 660, step 670 associates item properties with the region.

[0114] In the context of the first embodiment, the index of the item property is added to the RegionAnnotationProperty structure.

[0115] For the second embodiment, it is determined whether an entry exists in the "ipma" box of the RegionAnnotationProperty structure. If not, an entry is created in the "ipma" box of the RegionAnnotationProperty structure. The index of the item property is added to this entry.

[0116] For the third embodiment, it is determined whether there is an entry in the "ipma" box corresponding to the image item. If not, an entry is created in the "ipma" box corresponding to the image item. Then, it is determined whether an index of a RegionAnnotationProperty structure is present in this "ipma" box. If not, this index is inserted in the entry after the index of the descriptive property associated with the image item and before the index of the transformation property associated with the image item. Finally, the index of the item property is inserted after the index of the RegionAnnotationProperty structure and before the index of another RegionAnnotationProperty structure or a transformation item property. Preferably, the index is inserted immediately before the index of the next RegionAnnotationProperty structure, or immediately before the index of the first transformation item property, or at the end of the entry.

[0117] For the fourth embodiment, the RegionAnnotationProperty structure contains both the geometry and the annotation, and no association is required. In this case, only steps 620, 630, and 660 are implemented.

[0118] Possibly, if the RegionAnnotationProperty structure allows for defining multiple regions, each with its associated properties, then in step 620 it is determined whether such a structure is already associated with the image item in the HEIF file. If so, a new entry is added to the RegionAnnotationProperty structure and the geometry of the region is stored within this entry. Otherwise, a new RegionAnnotationProperty structure is created and the geometry of the region is stored within this structure.

[0119] If the RegionAnnotationProperty structure allows associating a set of properties to some regions, some steps may be modified. In step 600, it is determined whether the region exists in a RegionAnnotationProperty structure that does not constitute another region. In step 620, it is determined whether one or more regions are already associated inside a RegionAnnotationProperty structure with a set of properties that correspond to the annotation of the new region. In this case, the region is created inside this RegionAnnotationProperty structure and steps 640 to 670 are not executed. Otherwise, a new RegionAnnotationProperty structure is created.

[0120] FIG. 7 illustrates the main steps of a process for reading a HEIF file containing region annotations, where the region annotations are described by item properties, according to an embodiment of the invention.

[0121] In a first step 700, a pictorial item is extracted from a HEIF file. It is possible that only a part of the metadata describing the pictorial item is extracted.

[0122] Then, in step 710, the first item property associated with the image item is extracted. The index of this item property is the first non-zero index in the "ipma" box entry corresponding to the image item. This index is used to retrieve the item property from the "ipco" box. If there is no item in the "ipma" box corresponding to the image item, or if the item does not have an item property index or contains only property indexes with value 0, the process ends directly in step 790.

[0123] In step 720, it is determined whether the extracted item property is a region annotation item property. If it is a region annotation item property, the next step is step 730; if not, the next step is step 770.

[0124] In step 730, a region description is extracted from the region annotation item properties. The region description may include the type of geometry and / or parameters that define this geometry. This region may be associated with the image item extracted in step 700.

[0125] Next, the index of the first item property associated with the region is extracted in step 740. If there is no item property associated with the region, processing continues directly with step 770.

[0126] In a first embodiment of the present invention, the index of the first item property associated with a region is the one that occurs first in the region annotation item property structure.

[0127] In a second embodiment of the present invention, the index of the first item property associated with a region is the first of the entries corresponding to the region annotation item property in the "ipma" box.

[0128] In a third embodiment of the invention, the index of the first item property associated with a region is the first one in the image item's entry in the "ipma" box, following the index of the region annotation item property.

[0129] Used to extract the item property index from the "ipco" box.

[0130] The item properties may then be associated with the region in step 750. The information contained in the item properties may be extracted and associated with the geometry of the region extracted in step 730.

[0131] In step 760, a determination is made whether there are any other item properties associated with the region.

[0132] In a first embodiment of the present invention, it is determined whether there is an unprocessed index in the region annotation item property structure.

[0133] In a second embodiment of the present invention, it is determined whether there is an unprocessed index in the entry corresponding to the region annotation item property in the "ipma" box.

[0134] In a third embodiment of the present invention, it is determined whether the next index of the image item's entry in the "ipma" box corresponds to a description item property that is not a region annotation item property.

[0135] If there are other item properties associated with the region, the next step is step 765. Otherwise, the next step is step 770.

[0136] In step 765, an index of another item property associated with the region is obtained. The first, second and third embodiments of the present invention proceed similarly to step 740. The next step is step 750.

[0137] In a fourth embodiment of the invention, there are no item properties associated with a region and steps 740, 760 and 765 are skipped. Annotations for a region may be extracted directly from the region annotation item properties and associated with the region in step 750.

[0138] In step 770, it is determined whether there are other item properties associated with the image item. This is accomplished by checking whether there is a non-processing index in the entry corresponding to the image item in the "ipma" box. If there are other item properties associated with the image item, the next step is step 780. If not, the next step is step 790.

[0139] In step 780, further item properties associated with the image item are extracted. This step is similar to step 710. The next step is step 720.

[0140] In step 790, the process ends.

[0141] 8 to 13 are diagrams illustrating some embodiments of the present invention for the HEIF or MIAF standard, in which region annotations 210 are represented using items.

[0142] Figure 8 illustrates a fifth embodiment of the invention. The "meta" box 800 contains an item 810, which corresponds for example to an image item. The "meta" box 800 also contains two other items 820 and 830, which correspond to region annotations. The geometry of these two region annotation items is described in their respective contents shown as geometry 821 and geometry 826. These contents are identified in "iloc" boxes. The content of the region annotation items may preferably be stored in an "idat" box, and may also be stored in either an "mdat" box or an "imda" box.

[0143] The "iref" box 830 contains two entries, 831 and 832, that associate region annotation items 820 and 825, respectively, with image item 810.

[0144] The "iprp" box 840 includes an "ipco" box 860 and an "ipma" box 850. The "ipco" box 860 stores item properties 861, 862 corresponding to annotations of two region annotation items 820 and 825. The "ipma" box 850 associates the region annotation items 820 and 825 with their respective item properties 861 and 862 via two items 851 and 852, respectively.

[0145] In this embodiment, a region annotation item may be defined as having an item_type of "rgan." A region annotation item may be associated with its image item via an item reference box of type "cdsc."

[0146] The content syntax of an area annotation item can be as follows: TIFF0007654130000023.tif13157

[0147] The semantics of a region annotation item can be as follows: -geometry is the geometry of the region.

[0148] We may also add a version field to the region annotation item to allow for future evolution, and we will enable the extension of the region annotation item by defining new geometry types.

[0149] In this embodiment, all item properties associated through an "ipma" box entry to a region annotation item are applied to the region of the annotated image defined by the geometry of the region annotation item.

[0150] In this embodiment, the annotation image from FIG. 2b may be stored in a HEIF file with the following structure: TIFF0007654130000024.tif89157TIFF0007654130000025.tif151157

[0151] In this HEIF file, image item 250 is represented by the item with item_ID value 1, and region 260 is represented by the item with item_ID value 2, and is associated through the second entry in the ipma box with the item property at index 3, which corresponds to the face annotation. The geometry of the region annotation item is described in the first region annotation part of the MediaDataBox.

[0152] Region 261 is represented by an item with item_ID value 3, and is associated through the third entry in the "ipma" box with item properties at indexes 3 and 4. These properties correspond to a face annotation and an annotation describing a person named "Jean", respectively. The geometry of the region annotation item is described in the second region annotation part of the MediaDataBox.

[0153] Region 270 is represented by the item with item_ID value 4, and is associated through the fourth entry of the "ipma" box with the item property at index 5. This property corresponds to an annotation describing "Notre-Dame de Paris" as a building. The geometry of the region annotation item is described in the third region annotation part of the MediaDataBox.

[0154] Each region annotation item is associated with an annotation image through an item reference of type "cdsc".

[0155] All area annotation items can have the hidden property set to true to indicate to the reader that this item is not intended to be displayed.

[0156] In a first variant of this fifth embodiment, the syntax of the region annotation item may be as follows: TIFF0007654130000026.tif26157

[0157] In a first variation of this fifth embodiment, one region annotation item can define multiple regions that share a related set of item properties.

[0158] 9 is a diagram showing a sixth embodiment of the present invention, which is similar to the fifth embodiment except for the association between the region annotation item and the image item.

[0159] In this embodiment, the region annotations 820 and 825 are grouped by a region group 970. The region group 970 may be an EntityToGroupBox with a grouping_type of "rgan" or a grouping_type of "rcog". This region group 970 is associated with the image item 810 via an entry 933 in the "iref" box 830.

[0160] In this embodiment, the annotation image from FIG. 2b may be stored in a HEIF file with the following structure: TIFF0007654130000027.tif51157TIFF0007654130000028.tif184157

[0161] This HEIF file is similar to the corresponding one in the fifth embodiment, except for the association of the region annotation items with the image items. In this HEIF file, region annotation items with item_ID 2, 3, 4 are grouped in a group with group_id 100 and grouping_type "rgan". This group is associated with the image item through an item reference with reference type "cdsc".

[0162] In a variation of this embodiment, the item properties 861 , 862 are associated with a region group 970 .

[0163] In another variation of this embodiment, item properties 861 and 862 are associated with region group 970, region group 970 is not associated with image item 810, and region annotation items 820 and 825 are associated with image item 810 through item references.

[0164] 10 illustrates a seventh embodiment of the present invention. This embodiment is similar to the fifth embodiment, except for the description of the geometry of the regions. In this embodiment, two region annotation items 1020 and 1025 do not have their geometry inside their content. Instead, these region annotation items are associated with geometry item properties 1063 and 1064 through entries 851 and 852, respectively, in the "ipma" box 850.

[0165] In this embodiment, the geometry item properties can be defined as follows: -Box Type: "geom" -Property type: Descriptive item property -Container: ItemPropertyContainerBox -Required (per item): Yes for items of type 'rgan' -Quantity (per item): 0 or 1

[0166] The syntax for a geometry item property can be as follows: TIFF0007654130000029.tif18157

[0167] The semantics of geometry item properties can be as follows: -geometry is the geometry of the region.

[0168] In this embodiment, the annotation image from FIG. 2b may be stored in a HEIF file with the following structure: TIFF0007654130000030.tif51157TIFF0007654130000031.tif184157

[0169] This HEIF file is similar to the one corresponding to the fifth embodiment, except for the description of the geometry of the area annotation item. The area annotation item has an associated geometry item property to specify its geometry. In this HEIF file, the area annotation item has no content, so it may not have an associated item in the ItemLocationBox, or the associated item may not have a range.

[0170] In a first variation on this seventh embodiment, the syntax of the geometry item property can be as follows: TIFF0007654130000032.tif30157

[0171] This first variant allows for multiple geometries to be defined within one geometry item property. The area annotation item associated with the geometry item property consists of one area for each geometry defined in the geometry item property. All item properties associated with an area annotation item apply to all areas of the area annotation item.

[0172] In a second variation on this seventh embodiment, the geometry of the region is defined by two item properties: a position item property and a shape item property.

[0173] In this first variant, the properties of the location information item may be defined as follows: -Box Type: "relo" -Property type: Descriptive item property -Container: ItemPropertyContainerBox -Required (per item): Yes for items of type 'rgan' -Quantity (per item): 0 or 1

[0174] The syntax for the location property can be as follows: TIFF0007654130000033.tif18157

[0175] The semantics of properties on location items can be as follows: -location is the location of the region.

[0176] In this second variant, the shape item property can be defined as follows: -Box Type: "resh" -Property type: Descriptive item property -Container: ItemPropertyContainerBox -Required (per item): Yes for items of type 'rgan' -Quantity (per item): 0 or 1

[0177] The syntax for the shape property can be: TIFF0007654130000034.tif18157

[0178] The semantics of shape item properties can be as follows: -shape represents the shape of the region.

[0179] Alternatively, multiple location information properties may be associated with the same area annotation item, in which case the area annotation item consists of one area per location item property, all of which have the same shape.

[0180] Also, multiple position item properties and multiple shape item properties may be associated with the same area annotation item. In this case, the area annotation item consists of one area for each combination of position item property and shape item property. For example, if an area annotation item is associated with two position item properties and three shape item properties, it consists of six areas.

[0181] In yet another alternative of this second variant, the location item property and / or the shape item property may contain multiple locations or multiple shapes, respectively, in which the area annotation item is composed of one area for each combination of a location included in one of the location item properties and a shape included in one of the item properties.

[0182] In another alternative, multiple geometry item properties may be associated with the same region annotation item, where the region annotation item consists of one region per geometry item property.

[0183] When multiple regions exist within a single region annotation item, all items associated with that region annotation item apply to all regions present within that region annotation item.

[0184] Fig. 11 illustrates the main steps of a process for adding a new region annotation to an image item stored in a HEIF file when the region annotation is described by the item according to an embodiment of the invention. This diagram is similar to Fig. 6. These steps can be applied to HEIF files stored on disk, in memory, or in a representation adapted to memory. A new region annotation consists of the geometry of the region and the annotation itself. These steps can be modified to add multiple annotations to one region of an image item at the same time. Also, these steps can be modified to add annotations to several regions of an image item at the same time.

[0185] This process can be used to create new HEIF files or to modify existing HEIF files.

[0186] In a first step 1100, it is determined whether a region annotation item with the same geometry already exists in the HEIF file.

[0187] If an area annotation item with the same geometry already exists, the next step is step 1110, otherwise the next step is step 1120.

[0188] In step 1110, an item_ID value that corresponds to an existing region annotation item is selected. The next step is step 1140.

[0189] At step 1120, a new region annotation item is created to represent the region. An "infe" box describing the region annotation item may be created inside the "iinf" box of the HEIF file. Also, an entry may be added in the "iloc" box to indicate the location of the content of the region annotation item. An item_ID value is associated with this new region annotation item.

[0190] In the case of the fifth or sixth embodiment, the geometry of the region is stored inside the content of the region annotation item.

[0191] In the seventh embodiment, a new geometry item property may be created to store the geometry of the region. The index of this new item property is stored. Also, if an identical geometry item property exists in the HEIF file, its index may be stored and no new item property may be created.

[0192] For the seventh embodiment, a new position item property and a new shape item property can be created to store the geometry of the region. The indexes of these new item properties are stored. Also, if either or both of these item properties exist in the HEIF file, their indexes may be stored and the corresponding item properties may not be created.

[0193] Next, in step 1130, the new region annotation item is associated with the image item.

[0194] In the case of the fifth or seventh embodiment, and in some variations of the sixth embodiment, a new reference of type "cdsc" is created in the "iref box" of the HEIF file, which associates the region annotation item with the image item.

[0195] For some variations of the sixth embodiment, it is first determined whether a region group, which may be an EntityToGroupBox of grouping_type "rgan", is associated with the image item. If so, the region annotation item is added to the region group. If not, a new region group of type "rgan" is created containing the new region annotation item. Then a new reference of type "cdsc" is created in the "iref" box of the HEIF file. This reference associates the region group with the image item.

[0196] Note that, when the sixth embodiment is combined with either the fifth or seventh embodiment, any of these processes may be applied. Possibly, first, determine whether the image item is associated with a region group that is an EntityToGroupBox of grouping_type "rgan". In this case, the region annotation item is added to the region group. Determine whether the image item is associated with another region annotation item, and if not, add a new region annotation item. In this case, a new region group (type "rgan") is created that contains both the other region annotation item and the new region annotation item, and this group contains the other region annotation item and the new region annotation item. The reference between the other region annotation item and the image item is deleted, and a new reference (type "cdsc") between the region group and the image item is created. Otherwise, a new reference (type "cdsc") between the new region annotation item and the image item is created.

[0197] The next step is step 1140.

[0198] In step 1140, it is determined whether an item property corresponding to the annotation already exists in the HEIF file. If the item property already exists, the next step is step 1150; otherwise, the next step is step 1160.

[0199] In step 1150, an existing item property is selected.

[0200] In step 1160, a new item property is created to represent the annotation. The type of the item property depends on the content of the annotation. The information contained in the annotation is stored inside the item property.

[0201] After either step 1150 or step 1160 , the next step is step 1170 .

[0202] At step 1170, item properties are associated with the region annotation item.

[0203] If the area annotation item already has an item associated with the "ipma" box, the item property index is added to this item.

[0204] If the area annotation item does not have an item associated with the "ipma" box, a new item is created for this item and the item property index is added to this item.

[0205] For the seventh embodiment, the geometry item property, the index of the position item property, and / or the shape item property stored in step 1120 are also added to this entry.

[0206] Possibly, some steps can be modified if a region annotation item may consist of multiple regions. In step 1100, it is determined whether a region annotation item exists that consists of a single region with the same geometry. In step 1120, it is determined whether an existing region annotation item is associated with an image item that has a set of properties that correspond to the annotations of the new region annotation. If so, the geometry of the new region is added to the existing region annotation item and steps 1140-1170 are not performed. If not, a new region annotation item is created.

[0207] FIG. 12 illustrates the main steps of a process for reading a HEIF file containing region annotations when the region annotations are described by an item according to an embodiment of the invention.

[0208] In a first step 1200, an image item is extracted from a HEIF file. It is possible that only a portion of the metadata describing the image item is extracted.

[0209] In step 1210, the first item different from the image item is extracted from the HEIF file. If there are no other items in the HEIF file, the algorithm continues directly with step 1270.

[0210] Next, in step 1220, it is determined whether the other item is a region annotation item. If it is a region annotation item, the next step is step 1230; if not, the next step is step 1250.

[0211] In step 1230 it is determined whether the region annotation item is associated with the image item by a reference of type "cdsc" in the "iref" box. If so, the next step is step 1240, if not, the next step is step 1250.

[0212] In step 1240, item properties associated with the region annotation item through the entry in the "ipma" box are extracted.

[0213] In the context of the sixth embodiment, it is considered whether the region annotation item is included in a region group with grouping_type "rgan" in the EntityToGroupBox. In this case, the item properties associated with the region group through the entry in the "ipma" box are extracted.

[0214] Then, the geometry of the region annotation item is extracted.

[0215] In the context of the fifth embodiment, or some variants of the sixth embodiment, the geometry of a region annotation item is extracted from the content of the region annotation item.

[0216] In the context of the seventh embodiment, the geometry of a region annotation item is from geometry item properties, position item properties, and / or shape item properties associated with the region annotation item.

[0217] The region linked to the extracted item property is associated with a pictorial item. Information contained in the item property can be extracted to associate the shape of the region with the pictorial item.

[0218] If the region annotation item consists of multiple regions, each of these regions linked with the extracted item properties is associated with an image item. Information contained in the item properties may be extracted to associate the geometry of each of these regions with the image item.

[0219] The next step is step 1250.

[0220] Step 1250 determines whether there are other items in the HEIF file that are different from the image item. If so, the next step is step 1260; if not, the next step is step 1270.

[0221] In step 1260, another item is extracted from the HEIF file.

[0222] In step 1270, the process ends.

[0223] 13 is a diagram showing the main steps of a process for reading a HEIF file containing region annotations when the region annotations are described by items according to some variants of the sixth embodiment, which is similar to the process illustrated in FIG.

[0224] In a first step 1300, an image item is extracted from a HEIF file. In some cases, only a portion of the metadata describing the image item is extracted.

[0225] In step 1310, a first group is extracted from the HEIF file. If no groups are present in the HEIF file, the algorithm continues directly with step 1370.

[0226] Then, in step 1320, it is determined whether the region group is an EntityToGroupBox with a grouping type of “rgan.” If it is a region group, the next step is step 1330;

[0227] In step 1330 it is determined whether the region group is associated with the pictorial item by a reference of type "cdsc" in the "iref" box. If so, the next step is step 1340; if not, the next step is step 1350.

[0228] The region annotation items contained in the group are retrieved in step 1340. For each region annotation item, its region and its item properties are extracted and associated with the image item as described in step 1240.

[0229] If the region annotation item contains multiple regions, each of these regions that are linked with the extracted item properties is associated with the image item as described in step 1240 .

[0230] Also, item properties related to the region group may be extracted. For each region annotation item contained in the group, its region is extracted. Each extracted region is then combined with all the extracted item properties and associated with the image item as described in step 1240.

[0231] The next step is step 1350.

[0232] Step 1350 determines whether there are more groups in the HEIF file. If so, the next step is step 1360; if not, the next step is step 1370.

[0233] In step 1360, another group is extracted from the HEIF file.

[0234] In step 1370, the process ends.

[0235] FIG. 14 illustrates a process for processing a HEIF file that includes an image and one or more region annotation items associated with the image, according to an embodiment of the present invention. The process may be an image editing process; a process of removing a portion of the image, such as cropping; a process of transforming the geometry of the image, such as perspective correction, rotation, affine transformation, etc.; a process of changing the content of the image, such as applying a filter or drawing on the image; a metadata editing process; a process of removing private metadata; a process of filtering metadata; or a process of translating metadata.

[0236] In a first step 1400, an operation is applied to a graphical item, potentially altering the image associated with the graphical item.

[0237] Possibly, the result of the processing may be stored in another image item as a derived image item, in which case the region annotation items associated with the original image item are also associated with this derived image item. In the following steps, the processed image item becomes a derived image item.

[0238] In step 1410, a first region annotation associated with the processed graphical item is retrieved. If no region annotation is associated with the processed graphical item, the next step is step 1470.

[0239] In step 1420, it is determined whether a region annotation should be removed. Different processes may use different criteria. Some processes may remove all region annotations. Some processes may remove any region annotations that have a particular type. For example, a privacy preservation filter may remove region annotations that are represented by user-defined item properties. Region annotations may be removed depending on their locality. For example, a crop may remove region annotations that do not intersect with the rest of the image.

[0240] As an example, in FIG. 2b, region annotation 260 may be removed from the edited image because its region of interest has been cropped out of the edited image.

[0241] If it is decided to remove the region annotation, the next step is step 1425. Otherwise, the next step is step 1430.

[0242] In step 1425, the region annotations are removed. The next step is step 1450.

[0243] In step 1430, it is determined whether the geometry of the region annotation should be modified. Any operation that transforms the geometry of the image needs to modify the geometry of the region annotation. Such operations consist of rotating the image, scaling the image, applying an affine transformation to the image, correcting the perspective distortion of the image...

[0244] If it is determined that the geometry of the region annotation should be modified, the next step is step 1435. Otherwise, the next step is step 1440.

[0245] In step 1435, the geometry of the region annotation is modified according to the process. The modified geometry may be an exact result of applying a geometric transformation to the geometry of the region annotation. Possibly, the modified geometry is an approximate result of applying a geometric transformation to the geometry of the region annotation.

[0246] For example, in Figure 2b, region 261 is initially a rectangle. A perspective correction applied to the image transforms this region from a rectangle to a trapezoid. The corrected geometry may be a polygon that represents exactly this trapezoid, or it may be a polygon with integer coordinates that approximate the rational coordinates of the trapezoid, or it may be a rectangle that approximates the shape of the trapezoid.

[0247] The next step is step 1440.

[0248] In step 1440, it is determined whether the annotation of the region should be modified. Depending on the process, different criteria may be used. A process that translates a text annotation may modify the text representing the annotation. A process that filters an image, for example by applying a blur, may modify the annotation to remove a precise portion from the annotation.

[0249] If it is determined that the annotation of the region should be revised, the next step is step 1445. Otherwise, the next step is step 1450.

[0250] In step 1445, the annotation of the region annotation is changed according to the processing content.

[0251] For example, in Figure 2b, region 261 has a region annotation that corresponds to a description of a person. A privacy-preserving process may leave an indication that the region annotation corresponds to a person, but remove the person's name.

[0252] The next step is step 1450.

[0253] In step 1450, it is determined whether there are more region annotations to process. If it is determined that there are more region annotations to process, the next step is step 1460. If not, the next step is step 1470.

[0254] In step 1460, another region annotation associated with the graphical item is searched for. The next step is step 1420.

[0255] In step 1470, the process ends.

[0256] Annotation Description An annotation may be the focus position in the captured image. This may be expressed using user-written item properties, e.g. specific values ​​of the name and / or tag fields (e.g. "focus"), or it may be expressed with new item properties.

[0257] Annotations can be faces detected from an image, and can be expressed as user-written item properties, such as "face" in the name and tags fields, or as new item properties.

[0258] An annotation may be an object detected in an image by an object detection tool. This object may be represented using user-written item properties, e.g., specific values ​​in the name and / or tag fields, and / or a descriptive value in the description field. For example, the name field may be "Building" and the description field may be "House, Building, Monument". It may also be represented by new item properties.

[0259] Annotations can be specific object instances detected in an image by an object detection tool. They can be represented using user description item properties, for example using a specific value in a tag field to describe the general type of object, a more precise value corresponding to the object instance in a name field, and / or a descriptive value of the object instance in a description field. For example, the tags field can be "church", the name field can be "Notre Dame", the description field can be "Notre-Dame de Paris", or they can be represented by new item properties.

[0260] An annotation can be the GPS location of an object in an image, which may also be represented by a new item property.

[0261] An annotation may be a text extracted from an image by an OCR tool. It may be represented by a user description item property, for example using a specific value to indicate that the annotation corresponds to an OCR result in the name and / or tag fields, and representing the OCR output in the description field. For example, the tag field may be "ocr" and the description field "Notre-Dame de Paris". It may also be represented by a new item property. This new item property may contain information about the font used, such as its family, style, weight, and / or size. The result of the OCR tool may be split into several area annotations corresponding to the different detected text areas and / or the different detected text styles.

[0262] Annotations can describe edits or modifications applied to a region of an image. They can be represented by user-written item properties or by new item properties.

[0263] Annotations may be stored in an item. This item may be associated with a region annotation item by a reference of type "cdsc". This item may be associated with a region annotation item property by a new item property that references this item. The box type of this new item property may be "rgcd". For example, annotations may be stored in an "Exif" item. As another example, annotations may be stored in an XMP document contained in an item of type "mime" and content type "application / rdf+xml".

[0264] That is, a metadata item may be associated with a domain item through an item reference from the metadata item to the domain item with item_type value "cdsc". The metadata item may be an item with item_type value "Exif". The metadata item may be an item with item_type "mime" and content type "application / rdf+xml". The metadata may be an item with item_type mime and equal to the MIME type of MPEG-7 metadata.

[0265] When a user-written item property is used, the language field may be set to an appropriate value. When a user-written item property is used, multiple instances of this item property may be used with different language field values.

[0266] A region annotation associated with an entity may have no associated annotations. This may be the case when a region annotation item associated with an image item through an item reference has no associated properties. This may be useful when the annotations are predefined or known by external means. For example, an application that counts people in an image may identify each person in the image by the regions that have no associated annotations. Also, when the application saves the image, it may save in an external file that the regions shown in the image correspond to people.

[0267] Geometry Substitution The geometry of a region annotation may include other types: It may include a type that represents an ellipse, It may include a type that represents a 3D box expressed in world coordinates and the associated world-to-image projection, It may include a type that represents a particular polygon, It may include a type that represents a sphere.

[0268] The geometry of a region annotation can be a segment. The geometry of a region annotation can be a line, defined for example as a spline or a Bézier curve. The geometry of a region annotation can be a collection of connected or disconnected segments and / or lines.

[0269] The geometry of a collection of connected segments can be defined with the following structure: TIFF0007654130000035.tif38157

[0270] The geometry of an area annotation may be restricted to fewer types: It may be restricted to rectangular areas.

[0271] The type field of the Geometry and Shape structures can be a four-character code.

[0272] The location of the region annotation in the geometry may correspond to the center of the shape.

[0273] Locations that do not have a shape associated with them may represent areas of point type.

[0274] The geometry of an area may be represented by one or more equations and / or inequalities. For example, a line may be represented by an equation. As another example, the interior of a triangle may be represented by three inequalities.

[0275] Some or all of the geometric coordinates of an area annotation may be expressed as rational numbers rather than integers. For example, a point may be represented by a field such as: TIFF0007654130000036.tif22157, where -xN and xD are the numerator and denominator of the horizontal axis of the point, respectively. -yN, yD are the numerator and denominator of the vertical axis of the point, respectively.

[0276] Some or all of the coordinates of an area annotation's geometry may be expressed as signed integers instead of unsigned integers. For example, an area annotation's geometry may be represented by the following structure: TIFF0007654130000037.tif143157TIFF0007654130000038.tif125157

[0277] Negative values ​​in the x or y fields can be used to specify a point outside the image, the top left corner, or the center. This is useful for updating region annotations while editing a HEIF file. If a region is completely outside the image, it may be empty. Such empty regions should be ignored by parsers and readers.

[0278] The field_size of Geometry, Location, and Shape structures can be a fixed value, such as 32 bits. In this case, the field_size inside the structure is not specified. If multiple of these structures are contained in a box, the field_size definition is shared by all structures. If one or more structures are contained in a full box, the field_size definition can be based on the flags or version fields of the full box. If one or more structures are contained in a box or full box, the field_size may depend on the type of box. For example, you can define two different geometry item property boxes, each with its own field_size.

[0279] Possibly, in some embodiments, the Geometry structure, the Location structure, and / or the Shape structure may be used together to represent the geometry of a region annotation.

[0280] Multiple geometries sharing the same type may be represented in a compressed form, for example a list of rectangles, where each field can be represented as the difference between the corresponding field of the previous rectangle.

[0281] The mask is always placed in the top left of the image, and its position may not be specified.

[0282] The mask can always be the same size as the annotated image.

[0283] The pixel density of the mask may differ from the pixel density of the annotation image. For example, one pixel of the mask may correspond to four pixels of the annotation image. Or, four pixels of the mask may correspond to one pixel of the annotation image. This difference in pixel density may be described in a Geometry or Shape structure. This difference may be described using the pixel aspect ratio property.

[0284] Color masks may allow a mask corresponding to multiple region annotations to be combined into another mask for a different region annotation. For example, a mask for a region annotation representing one specific person may use a pixel color value of 0x1, and a mask for a region annotation representing another specific person may use a pixel color value of 0x2. A mask for a region annotation that combines these two regions, describing people in a generic way, may use a pixel color value of 0x3.

[0285] Also, the type of the Geometry structure or Shape structure may be specified by the type of the area annotation item property, area annotation item, Geometry item property, or Shape item property.

[0286] For example, a point area annotation item property and a rectangular area annotation item property can be defined with the following syntax: TIFF0007654130000039.tif76157TIFF0007654130000040.tif63157

[0287] Also, a specific region annotation item may be defined to represent a mask. The content of this specific region annotation item may be an image corresponding to the mask. Other information used to define the region that the mask represents, such as the mask position and mask color, may be represented in specific item properties associated with the mask region annotation item.

[0288] A possible syntax for expressing the contents of such an item might be: TIFF0007654130000041.tif14157

[0289] The content of such an item could indicate the location of the region using the following syntax: TIFF0007654130000042.tif34157

[0290] The mask may be stored directly within the geometry (payload) of the area item. Optionally, a MaskItem may have a parameter indicating the number of bits_per_pixel. Possible values ​​are 1, 8, 16, 24. Specifying a mask with more than 1 bit per pixel allows the use of an alpha plane layer as the mask.

[0291] A mask is stored as an image item and may be associated with a region item using an item reference of type "mask" from the region item to the image item. In the following, this image item is called the "mask image item". A region is defined by the image that results from applying any transformation item properties to the mask image item, as described above. It is also possible to specify the location of the region by associating the "rloc" item property with the mask image item. The geometry of a region annotation, which may be a bitmask or color mask, may also be represented in the region item payload using the following structure: TIFF0007654130000043.tif47157

[0292] When using a bit mask, it is possible for a region to be all pixels in the mask image, including 0. It is also possible for a region to be the pixel in the mask image with the maximum value. It is also possible to define regions based on the values ​​of other pixels. It is also possible to define regions based on a predefined color for the pixel. It is also possible to define regions with a fuzzy boundary, where the minimum value corresponds to pixels that belong to the region, the maximum value to pixels that do not belong to the region, and intermediate values ​​to pixels that are part of the region, when the number of bits per pixel is greater than 1. Using item references between region items and mask images is advantageous in simplifying the editing process of HEIF files containing region items by placing all links between items and / or entities in a single box.

[0293] It can also be a reference from a mask image item to a region item.

[0294] Item references can also use different types, for example "rmsk".

[0295] An item reference may be a reference from a region item to multiple mask image items. A region is defined as the union of sub-regions defined by mask image items. In a variant, there may be multiple item references from a region item to a mask image item.

[0296] An area item may contain the description of the geometry of several areas as masks. In this case the area item is associated with mask image items using an item reference of type "mask", one reference for each in the item reference of type "mask". If the same mask, stored as an image item, is used by several areas declared in the area item, this image item is referenced multiple times in the item reference of type "mask". The mask image items are listed in the item reference in the same order as the corresponding areas in the area item. That is, the images defining the mask for the Nth area are stored as image items, in declaration order, and are identified by the Nth reference in the item reference of type "mask".

[0297] In a variant, region items may be associated with mask image items using one item reference of type "mask" for each mask image item. The "mask" item references appear in the "iref" boxes in the same order as the corresponding regions in the region item.

[0298] In another variation, the mask image items are listed in any order inside the "mask" item reference, and each mask region definition includes the index of the corresponding mask image item inside the "mask" item reference.

[0299] In this variant, the geometry of the mask region can be defined with the following syntax: TIFF0007654130000044.tif61157

[0300] It is also possible to specify the location of the region by associating the "rloc" property with the mask image.

[0301] A mask region can potentially be associated with multiple image items by including a list of mask item indices in the mask region definition.

[0302] In a variant, the region items may be associated with the mask image items using one item reference of type "mask" for each mask image item.

[0303] Two mask regions specified for an image item may overlap each other, which means that some pixels of the image belong to both regions.

[0304] The location of a region annotation item may be represented by the item's content and / or associated item properties, and the shape of a region annotation item may be represented by the item's content and / or associated item properties. For example, the shape of a region may be represented by the item's content and the location of the region by the associated item properties. In this example, multiple locations may be associated with a region annotation item. The region annotation item consists of one region for each location, and all these regions share the same shape.

[0305] Fields that define the number of items contained in a list may be represented in different sizes, for example the region_count field may be represented as a 16-bit unsigned integer.

[0306] Item property substitution Item properties of an area annotation may be able to extend the ItemProperty box rather than extending the ItemFullProperty box.

[0307] Possibly, any area annotation item properties associated with the item have the required field set to a value of 1 to ensure that readers understand the area annotation item properties.

[0308] It is also conceivable to define a new property, called a region property, in addition to the descriptive property and the transformation property. Processing specific to a region annotation item property can be associated with a region property type, rather than with the region annotation item property itself. For example, processing of item properties according to the third embodiment of the present invention can be defined based on the region property type.

[0309] Item Substitution Two region annotations share the same content because they represent the same region, but may have different annotations.

[0310] Possibly, a region annotation item may be associated with an image item using different reference types. The reference types used may be "tbas" or "dimg". A new reference type, for example "rgan", may be defined to describe such an association.

[0311] Depending on the type of annotation, different types of references may be used. For example, a region annotation item corresponding to a focus region may be associated with its annotation image item by a reference of type "rfoc" indicating that this is a focus region. In this example, the region annotation item may not be associated with any item properties to describe the annotation.

[0312] Chances are that the region annotation item is of type "hvc1" and its content is the annotation image.

[0313] A region annotation item may be a derived image item of type 'iden' that has no content.

[0314] Possibly, the region annotation item is of type "hvt1" and its content is a tile of the annotated image. The geometry of the region annotation item may be the area covered by the tiles. Possibly, the content of the region annotation item is a set of tiles of the annotation image and its geometry may be the area covered by the tiles.

[0315] Possibly, a region annotation item corresponds to a sub-picture of the annotated image. The geometry of the region annotation item may be the area covered by the sub-picture. Possibly, a region annotation item corresponds to a collection of sub-pictures of the annotated image. The geometry of the region annotation item may be the area covered by the sub-picture.

[0316] The box type of a geometry item property may be different, for example it may be "rcgr".

[0317] Potentially, the location of the region annotation item is described using the "rloc" item property.

[0318] Potentially, the shape of an area annotation item is described as a rectangle in the "ispe" item property.

[0319] Potentially, the geometry of the area annotation item is described using the "clap" item property.

[0320] For example, in Fig. 2b, region 261 may be represented by a contentless region annotation item "rgan". This region annotation item is associated with the image item by a reference of type "rgan". The location of region 261 is represented by the item property "rloc". The shape of region 261 is represented by the item property "ispe".

[0321] Group Substitution The sixth embodiment of the present invention can be combined with the fifth embodiment to associate region annotation items with image items directly and / or via region groups.

[0322] The sixth embodiment of the present invention can be combined with the seventh embodiment to associate a region annotation item with an image item directly and / or via a region group.

[0323] Possibly, in the sixth embodiment, an item property may be associated with a region group to indicate that it applies to all region annotation items configured in the region group. For example, a shape property may be associated with a region group to describe the shape of all region annotation items in that group. As another example, a user-written item property can be associated with a region group to associate the user-written item property with all region annotation items in the group.

[0324] In the sixth embodiment of the present invention, the grouping_type used by the region group may be "rcog".

[0325] Region groups may be associated with pictorial items using different reference types: a new reference type, for example "rcob", may be defined to describe such associations.

[0326] Using some of these options, as an example, the image in Figure 2b might be stored in a HEIF file with the following structure: TIFF0007654130000045.tif151157TIFF0007654130000046.tif88157

[0327] Associating areas with other items In one embodiment, an item (regardless of type) may be associated with a region of the first image. This association may be achieved directly from the item to the region, or indirectly by associating an entity group that contains the item with the region. A region may be described by a region annotation item or a region annotation item property. An item or entity group may be associated with a region annotation item by an entry in the "iref" box.

[0328] For example, an image item or entity group may be associated with a region annotation item via an item reference box of type "eroi" (for "encoded region of interest"). The reference type "eroi" is an example, other reference types such as "rgim" may also be used.

[0329] The association can be in the following direction: from the region annotation item to the image item (or to the entity group), or from the image item (or from the entity group) to the region annotation item.

[0330] Alternatively, an item (or entity group) can be associated with the region annotation item property via a new item property box that is associated with the region annotation item property and references this item (or entity group). This new item property box can be an entry in the "iref" box. This new item property box can contain a reference type, a reference count, and a list of the referenced item identifiers.

[0331] In a variant, an image item named "ROI image item" may be associated with a region annotation item or region annotation item properties to provide alternative or complementary content for that region. The ROI image item may have a higher resolution than the first image, or may have a higher quality than the first image. Alternatively, the ROI image item may correspond to a different field of view than that of the first image.

[0332] For example, if the first image is a photograph of Notre Dame Paris and its surroundings during the 2019 fire, the region may correspond to Notre Dame Paris itself, and the ROI image item may be a photograph of Notre Dame Paris before the fire.

[0333] In another variant, an entity group named "ROI entity group" may be associated with a region annotation item or region annotation item properties to provide a set of alternative content for a region. The corresponding entity group box may be of type "altr".

[0334] For example, an ROI entity group may include a first image having a higher resolution representation of the region, a second image having a higher quality representation of the region, and a third image having an HDR representation of the region.

[0335] Other examples of items that may be associated with a region, either directly or as part of an entity group, through a region annotation item or region annotation item properties include a depth map of the region, a thumbnail of the region, and / or a quality or resolution refinement layer for the region.

[0336] When an image (hereafter referred to as the "second image") is associated with a region of a first image, either directly or through a group, the exact location of the second image relative to the first image may depend on the geometry of the region. If the region is a point, the center of the second image is located at this point in the first image. If the region is a rectangle, the second image may fill this rectangle within the first image. Otherwise, it is displayed to fill the bounding box of the region.

[0337] In embodiments, the exact positioning of the second image relative to the first image may be refined using region-related item properties. If the region is a point, an offset relative to the center of the second image, and a width and height of the second image may be specified. If the region is a rectangle, an offset from the top left corner of the rectangle, and a width and height of the second image may be specified. Alternatively, if the region is neither a point nor a rectangle, an offset from the top left corner of the bounding box, and a width and height of the second image may be specified. These offsets, widths, and heights may be specified with a new item property, or multiple new item properties, or may be specified using the "rloc" and / or "iscl" item properties.

[0338] In an embodiment, the second image may be larger than the region of the first image, and only a portion of the second image may correspond to the region. This configuration may be indicated with a specific item property (e.g., "clap") associated with the image item or an entity group that contains the image item. This configuration may be indicated with a new item property associated with the region annotation item or region annotation item property.

[0339] Other alternatives Possibly, after step 790 of the process illustrated by Fig. 7, the transformation properties related to the image items are extracted. They are then applied to the geometry of the annotated image item and its associated region annotation item properties according to their order inside the "ipma" entry. This application can be achieved according to the process shown by Fig. 14.

[0340] Possibly, after step 1370 of the process illustrated by Fig. 7, the transformation properties related to the image items are extracted. They are then applied to the geometry of the annotated image item and its associated region annotation items according to their order inside the "ipma" entry. This application may be achieved according to the process shown by Fig. 14.

[0341] Region annotations may be applied to the output image of an image item.

[0342] A region annotation may be associated with multiple image items - for example, the same region annotation may apply to multiple image items corresponding to a single image burst.

[0343] One region annotation may be associated with multiple image items. Furthermore, a geometry modification item property is associated with each association to adapt the geometry of the region to each image item. The geometry modification item property may include an indication of the region and / or region annotation associated with the image item to which it applies. The geometry modification item property may include an indication of the region and / or region annotation associated with the image item to which it applies. For example, the same region annotation may be associated with multiple image items corresponding to a burst of images containing moving objects. The geometry modification item property modifies the geometry of the region to adapt the moving object in each image item.

[0344] A region annotation may be associated with a group of items, meaning that the region annotation applies to each item comprised within the group.

[0345] References between region annotation items or region groups and image items can be reversed: for example, an association between one or more region annotation items and an image item can use a reference of type "rgan" from the image item to one or more region annotation items.

[0346] Area annotations may be described by new types of boxes. The structure of these boxes may be similar to the structures described for geometry item properties, position item properties, and shape item properties. These boxes may have identifiers associated with them. These boxes may be associated with items inside "iref" boxes or inside new boxes. These boxes may be associated with properties inside "ipma" boxes or inside new boxes.

[0347] Finally, although the embodiments have been described with reference to region annotation items, they may be applied more generally to region items: the purpose of a region item is to characterize the geometry of a region, not necessarily with associated annotations, but may be used to associate any kind of object, such as an image, video, or sound, with a region of an image item.

[0348] 15 is a schematic block diagram of a computing device 150 for implementing one or more embodiments of the present invention. The computing device 150 may be a device such as a microcomputer, a workstation, or a lightweight handheld device. The computing device 150 is configured with the following connected to a communication bus: - a central processing unit 151 (denoted as CPU), such as a microprocessor; a random access memory 152 (denoted RAM) for storing the executable code of the methods according to embodiments of the invention, as well as registers adapted to record variables and parameters necessary for the implementation of the methods according to embodiments of the invention, the capacity of which can be expanded, for example, by an optional RAM connected to an expansion port; - a read-only memory 153 (denoted ROM) for storing computer programs for implementing embodiments of the invention; - a network interface 154, typically connected to a communications network over which digital data to be processed is sent and received. The network interface 154 may be a single network interface or may consist of a collection of different network interfaces (e.g. wired and wireless interfaces, or different types of wired and wireless interfaces). Data packets are written to the network interface for transmission, or read from the network interface for reception, under the control of software applications running on the CPU 151; - a user interface 155 that may be used to receive input from a user or display information to a user; - a hard disk 156 (denoted HD) which may be provided as mass storage device; - An I / O module 157 that can be used to receive / send data to / from external devices such as video sources and displays.

[0349] The executable code can be stored either in the read-only memory 153, in the hard disk 156 or on a removable digital medium such as a disk. According to a variant, the executable code of the program can be received by the communication network means, via the network interface 154, to be stored in one of the storage means of the communication device 150, such as the hard disk 156, before being executed.

[0350] The central processing unit 151 is adapted to control and direct the execution of instructions or parts of a program or software code of a program according to an embodiment of the present invention, the instructions of which are stored in one of the aforementioned storage means. After power-on, the CPU 151 is capable of executing instructions from the main RAM memory 152 associated with a software application, after those instructions have been loaded, for example, from a program ROM 153 or a hard disk (HD) 156. Such a software application, when executed by the CPU 151, causes the steps of the flowcharts of the present invention to be performed.

[0351] Any step of an algorithm of the present invention may be implemented in software by execution of an instruction set or program by a programmable computing device such as a PC ("Personal Computer"), a DSP ("Digital Signal Processor"), a microcontroller, etc., or in hardware by a machine or dedicated component such as an FPGA ("Field Programmable Gate Array"), an ASIC ("Application Specific Integrated Circuit"), etc.

[0352] Although the present invention has been described with reference to specific embodiments, it is understood that the present invention is not limited to those embodiments, and modifications within the scope of the present invention will be apparent to those skilled in the art.

[0353] Many further modifications and variations will be suggested to those skilled in the art by reference to the foregoing exemplary embodiments, which are given by way of example only and are not intended to limit the scope of the invention, which is determined solely by the appended claims. In particular, different features from different embodiments may be interchanged where appropriate.

[0354] Each of the above-described embodiments of the present invention can be implemented alone or in combination in multiple embodiments. Also, features from different embodiments can be combined where appropriate or where a combination of elements or features from the individual embodiments in a single embodiment is beneficial.

[0355] In the claims, the term "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. The mere fact that different features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be used to advantage.

Claims

1. 1. A method for storing image data in a media file compliant with the ISOBMFF standard, comprising: Obtaining image data; generating image item information corresponding to the image data; generating geometry description data describing a geometry of a partial region of an image based on the image data, and region item information corresponding to the geometry description data; generating first association information for associating the area item information with the image item information; generating the media file storing the image data, the geometry description data, the image item information, the region item information, and the first association information; The method includes:

2. the image data and the geometry description data are stored in a data portion of the media file; The image item information, the region item information, and the first association information are stored in a metadata portion of the media file. The method of claim 1.

3. The image item information and the area item information are described in ItemInfoBox. The method according to claim 1 or 2.

4. The first association information is described in an ItemReferenceBox.

4. The method according to any one of claims 1 to 3.

5. The geometry description data includes information describing one of a point, a 3D box, a rectangle, an ellipse, a sphere, a polygon, a line, and a mask.

5. The method according to any one of claims 1 to 4.

6. When the geometry description data includes information indicating a mask, data of the mask of a partial region of the image is associated with the region item information by an item reference. The method according to claim 5.

7. The geometry description data includes information indicating a plurality of geometry types of the partial region.

7. The method according to any one of claims 1 to 6.

8. generating at least one annotation data structure corresponding to the domain item information; generating second association information for associating the domain item information with the at least one annotation data structure; storing the at least one annotation data structure and the second association information in the media file; The method of any one of claims 1 to 7, further comprising:

9. The at least one annotation data structure is described in an ItemPropertiesBox, The second association information is described in ItemPropertyAssociation 'ipma'. The method according to claim 8.

10. The at least one annotation data structure is described in an ItemInfoBox; The second association information is described in an ItemReferenceBox. The method according to claim 8.

11. The at least one annotation data structure includes information about focus positions, faces, and objects.

11. The method according to any one of claims 8 to 10.

12. 1. A method for processing a media file compliant with the ISOBMFF standard, comprising: Obtaining a media file including image data, image item information corresponding to the image data, geometry description data describing a geometry of a partial region of an image based on the image data, region item information corresponding to the geometry description data, and first association information for associating the region item information with the image item information; processing the media file; The method includes:

13. the image data and the geometry description data are stored in a data portion of the media file; The image item information, the area item information, and the first association information are stored in a metadata portion of the media file. The method of claim 12.

14. The image item information and the area item information are described in ItemInfoBox.

14. The method according to claim 12 or 13.

15. The first association information is described in an ItemReferenceBox.

15. The method according to any one of claims 12 to 14.

16. The geometry description data includes information describing one of a point, a 3D box, a rectangle, an ellipse, a sphere, a polygon, a line, and a mask.

16. The method according to any one of claims 12 to 15.

17. When the geometry description data includes information indicating a mask, data of the mask of a partial region of the image is associated with the region item information by an item reference.

17. The method of claim 16.

18. The geometry description data includes information indicating a plurality of geometry types of the partial region.

18. The method according to any one of claims 12 to 17.

19. generating at least one annotation data structure corresponding to the domain item information; generating second association information for associating the domain item information with the at least one annotation data structure; storing the at least one annotation data structure and the second association information in the media file; 19. The method of any one of claims 12 to 18, further comprising:

20. The at least one annotation data structure is described in an ItemPropertiesBox, The second association information is described in ItemPropertyAssociation 'ipma'.

20. The method of claim 19.

21. The at least one annotation data structure is described in an ItemInfoBox; The second association information is described in an ItemReferenceBox.

20. The method of claim 19.

22. The at least one annotation data structure includes information about focus positions, faces, and objects.

22. The method according to any one of claims 19 to 21.

23. A computer readable storage medium having stored thereon computer program instructions for carrying out the method according to any one of claims 1 to 11.

24. A computer readable storage medium having stored thereon computer program instructions for carrying out the method according to any one of claims 12 to 22.

25. A program causing a computer to execute the method according to any one of claims 1 to 11.

26. A program causing a computer to execute the method according to any one of claims 12 to 22.

27. An apparatus for storing image data in a media file conforming to the ISOBMFF standard, comprising: A means for acquiring image data; means for generating image item information corresponding to the image data; means for generating geometry description data describing a geometry of a partial region of an image based on the image data, and region item information corresponding to the geometry description data; means for generating first association information for associating the area item information with the image item information; means for generating the media file, the media file storing the image data, the geometry description data, the image item information, the area item information, and the first association information; An apparatus comprising:

28. An apparatus for processing media files conforming to the ISOBMFF standard, comprising: A means for acquiring a media file including image data, image item information corresponding to the image data, geometry description data describing a geometry of a partial area of ​​an image based on the image data, area item information corresponding to the geometry description data, and first association information for associating the area item information with the image item information; means for processing the media file; An apparatus comprising:

Images