Method and apparatus for encapsulating image data in file for progressive rendering

Abstract

To provide a method and apparatus for encapsulating image data in a file for progressive rendering.SOLUTION: There is provided a method of encapsulating image data in a media file, the image data being related to a main image to be generated on the basis of a plurality of sub images, the method comprising: obtaining the plurality of sub images each of which is provided in at least one version; generating descriptive metadata for describing the main image and the plurality of sub images; encapsulating the plurality of sub images and the descriptive metadata in the media file; and generating progressive information defining a set of consecutive progressive steps for generating a version of the main image, each progressive step being associated with a set of sub image versions required to generate the version of the main image. The method further comprises embedding the progressive information in the descriptive metadata.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates to methods and apparatus for encapsulating image data within a file, and more particularly to a method of encapsulation that enables progressive rendering of images. [Background technology]

[0002] The image displayed to the user can be loaded from a local disk, a network disk, or a remote server. To be loaded, the image is typically encapsulated in a file that contains image data and metadata. This metadata is used to describe the organization of the image data within the file, the type of content, and any useful information for the rendering or image data.

[0003] This image is typically encoded to reduce the size of the data on storage: many encoding standards can be used, such as JPEG, AV1, or the more recent HEVC or VVC standards.

[0004] The HEVC and VVC standards define profiles for the coding of still images and describe specific tools for compressing single still images or bursts of still images. An extension to the ISO Base Media File Format (ISOBMFF) used for this type of image data has been proposed for inclusion in Part 12 of the ISO / IEC 23008 standard under the name "HEIF or High Efficiency Image File Format".

[0005] High Efficiency Image File Format (HEIF) is a standard developed by the Moving Picture Experts Group (MPEG) for storing and sharing images and image sequences.

[0006] MIAF (Multi-Image Application Format) is a standard developed by MPEG in part 22 of the ISO / IEC 23000 standard. The MIAF specification specifies the Multi-Image Application Format (MIAF), a multimedia application format that enables precise interoperability points for creating, reading, parsing, and decoding images embedded in the High Efficiency Image File (HEIF) format. The MIAF specification is fully compliant with the HEIF format and only defines additional constraints to ensure higher interoperability.

[0007] Depending on the location of the image, loading an image from a local or remote storage system may require a delay long enough to be noticeable to the user. Some formats, such as JPEG or JPEG-2000, allow for progressive rendering of images, where a complete but lower-quality version of the image is included at the beginning of the file. This lower-quality version of the image is possibly improved in multiple passes through the rest of the file to provide a full-quality version throughout the file. Progressive formats allow a full-sized preview version of the image to be displayed quickly without waiting for the complete image to be received. Thus, it increases the responsiveness of the display of slowly loading images and improves the user experience.

[0008] The HEIF format does not explicitly support progressive rendering.

[0009] HEIF defines derived images, which are representations of images constructed by applying operations to other images present in the same file. In particular, two types of derived images are grid and overlay types. A grid is an array of smaller images, all of the same size. An overlay defines a derived image by overlaying one or more images in a given layered order within a larger canvas.

[0010] The HEIF and MIAF file formats do not provide mechanisms to enable progressive rendering of grid and overlay derived image items: in particular, there is no description of how to progressively build up from individual components and, as a whole, progressively render a grid or overlay type derived image, or any type of derived image based on multiple input images.

[0011] The HEIF and MIAF file formats also do not provide a mechanism to enable progressive rendering of multi-layer images. Summary of the Invention

[0012] The present invention has been devised to address one or more of the problems set forth above. The present invention relates to a method and apparatus for encapsulating image data within a file to allow for progressive rendering of the image. The present invention more particularly relates to images constructed based on multiple input images. The present invention has particular application to grid or overlay type derived images, although it is not limited to these types of images.

[0013] According to a first aspect of the present invention, there is provided a method of encapsulating image data within a media file, the image data relating to a main image generated based on a number of sub-images, the method comprising: obtaining a plurality of sub-images, each of which is provided in at least one version; generating descriptive metadata for describing the main image and a plurality of sub-images; Encapsulating a plurality of sub-images and descriptive metadata within a media file, comprising: This method further generating progressive information defining a set of successive progressive steps for generating a version of a main image, each progressive step being associated with a set of sub-image versions required to generate said version of the main image; Embedding progressive information in descriptive metadata; A method is provided, comprising:

[0014] In one embodiment, each sub-image represents a different subset of the layers of the main image.

[0015] In one embodiment, the sub-images are input images, and at least one of the input images is associated with a different version of the input image.

[0016] In one embodiment, the progressive information includes, for each progressive step, the location within the file of the sub-image version data associated with the progressive step.

[0017] In one embodiment, this location indicates the last byte of data for the sub-image version in the file associated with the progressive step.

[0018] In one embodiment, the location includes an offset and a length to indicate the sub-image version data associated with the progressive step.

[0019] In one embodiment, this location includes a list of the extents of data of the sub-image version associated with the progressive step.

[0020] In one embodiment, the sub-image data is organized into one or more extents, each extent consisting of successive image data associated with one version of the sub-image, and the progressive information includes information in descriptive metadata describing the extent indicating that the last byte of the extent corresponds to a progressive step.

[0021] In one embodiment, the sub-image versions are described as image items and the progressive information comprises, for each progressive step, a list of image item identifiers that identify the image item associated with the progressive step.

[0022] In one embodiment, the progressive information includes, for each progressive step, the number of input image versions that are included in the progressive step.

[0023] In one embodiment, at least one input image is composed of multiple layers, each layer associated with a version of the input image, and the progressive information further comprises a layer identifier associated with an identifier of an image item of the input image.

[0024] In one embodiment, generating the progressive information includes generating a progressive refinement data structure comprising data for determining the location of reconstruction points within the media file, each reconstruction point indicating that a main image can be reconstructed using image data associated with a previously received sub-image version.

[0025] In one embodiment, the progressive refinement data structure further comprises a number of reconstruction points.

[0026] In one embodiment, the progressive information characterizes the composition of the main image such that its quality improves gradually.

[0027] In one embodiment, the progressive information is associated with the main image.

[0028] In one embodiment, the main image is part of an entity group and the progressive information is associated with the entity group.

[0029] According to another aspect of the present invention, there is provided a method for generating a main image based on a plurality of sub-images from an image data file, the method comprising the steps of: obtaining, from the image data file, descriptive metadata describing the main image and a plurality of sub-images, each sub-image being provided in at least one version; - obtaining from the descriptive metadata progressive information defining a set of successive progressive steps for generating a version of the main image, each progressive step being associated with a set of sub-image versions required to generate a version of the main image; obtaining image data corresponding to a sub-image from an image data file; generating at least two versions of a main image corresponding to each progressive step, each version of the main image being generated when a set of sub-image versions associated with each progressive step is obtained from an image data file; A method is provided, comprising:

[0030] According to another aspect of the invention there is provided a computer program product for a programmable device comprising sequences of instructions for implementing the method according to the invention when loaded and executed on a programmable device.

[0031] According to another aspect of the invention there is provided a computer readable storage medium storing computer program instructions for implementing the method according to the invention.

[0032] According to another aspect of the invention there is provided a computer program which, when executed, causes the computer to perform the method of the invention.

[0033] According to another aspect of the present invention, there is provided a device for encapsulating image data within a media file, the image data relating to a main image generated based on a plurality of sub-images, the device comprising: obtaining a plurality of sub-images, each of which is provided in at least one version; generating descriptive metadata for describing the main image and a plurality of sub-images; Encapsulating a plurality of sub-images and descriptive metadata within a media file, comprising: This method further generating progressive information defining a set of successive progressive steps for generating a version of the main image, each progressive step being associated with a set of sub-image versions required to generate the version of the main image; Embedding progressive information in descriptive metadata; A device is provided that includes a processor configured for:

[0034] According to another aspect of the present invention, there is provided a device for generating a main image generated based on a plurality of sub-images from an image data file, comprising: obtaining, from the image data file, descriptive metadata describing the main image and a plurality of sub-images, each sub-image being provided in at least one version; - obtaining from the descriptive metadata progressive information defining a set of successive progressive steps for generating a version of the main image, each progressive step being associated with a set of sub-image versions required to generate a version of the main image; obtaining image data corresponding to a sub-image from an image data file; generating at least two versions of a main image corresponding to each progressive step, each version of the main image being generated when a set of sub-image versions associated with each progressive step is obtained from an image data file; A device is provided that includes a processor configured for:

[0035] At least some of the methods according to the invention may be computer-implemented. Accordingly, the invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be referred to generally in this specification as a "circuit," "module," or "system." Furthermore, the invention may take the form of a computer program product embodied in any tangible medium of expression having computer-usable program code embodied in the medium.

[0036] Since the present invention can be realized in software, the present invention can be embodied as computer readable code for providing to a programmable device on any suitable carrier medium. Tangible non-transitory carrier medium may include storage media such as floppy disks, CD-ROMs, hard disk drives, magnetic tape devices or solid state memory devices. Transient carrier medium may include signals such as electric, electronic, optical, acoustic, magnetic or electromagnetic signals, e.g. microwave or RE signals. [Brief description of the drawings]

[0037] Embodiments of the present invention will now be described, by way of example only, with reference to the following drawings, in which:

[0038] [Figure 1] 1 shows an example of a HEIF file that contains media data such as one or more still images and possibly one or more videos and / or one or more image sequences.

[0039] [Diagram 2] 1 shows a first example of a file structure to enable progressive rendering of grid items.

[0040] [Diagram 3]1 shows an alternative file structure to enable progressive rendering of similar grid items.

[0041] [Figure 4a] , [Figure 4b] , and [Figure 4c] 1 shows image data within an image file in different orders;

[0042] [Figure 4d] FIG. 4b shows another example where the same ordering is used in conjunction with four different progressive steps.

[0043] [Figure 4e] FIG. 4c represents another example in which a different ordering is used in conjunction with four different progressive steps.

[0044] [Figure 4f] 4c shows the rendering of the grid image item after a second progressive step corresponding to FIG. 4d.

[0045] [Figure 4g] 4c shows the rendering of the grid image item after a second progressive step corresponding to FIG. 4e.

[0046] [Diagram 5] 1 illustrates the main steps for generating a HEIF file according to an embodiment of the present invention.

[0047] [Figure 6] 1 illustrates the main steps for progressive rendering of a HEIF file generated according to an embodiment of the present invention.

[0048] [Figure 7] 1 illustrates the main steps for progressive rendering of a HEIF file generated according to another embodiment of the present invention.

[0049] [Figure 8] 1 is a schematic block diagram of a computing device for implementing one or more embodiments of the present invention.

[0050] [Figure 9] 1 illustrates an example of associating thumbnails and previews with image items to provide progressive rendering for the image items.

[0051] [Figure 10] An example is given of using item properties to describe progressive rendering steps based on the coding layers of an image item.

[0052] [Figure 11] 13 shows an example of using item properties to describe a progressive rendering step of a derived image item based on input image items.

[0053] [Figure 12] 2 illustrates an example of a progressive rendering step of a derived image item based on the coding layers of its input image items. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0054] The HEVC or VVC standard defines a profile for the coding of still images and describes specific tools for compressing a single still image or a burst of still images. An extension to the ISO Base Media File Format (ISOBMFF) used for this type of image data has been proposed for inclusion in Part 12 of the ISO / IEC 23008 standard under the name "HEIF or High Efficiency Image File Format".

[0055] The HEIF and MIAF standards address different use cases: The storage of image sequences that can be directed to be displayed as a time-series sequence or by other means, where images may depend on other images; and Storage of a single encoded image or a set of independently encoded images, possibly with derived images This covers two forms of storage:

[0056] In this first case, the encapsulation is close to the encapsulation of video tracks in the ISO Base Media File Format (see document "Information technology - Coding of audiovisual objects - Part 12: ISO Base Media File Format", w18855, ISO / IEC 14496-12:2015, 6th Edition, October 2019) and similar tools and concepts are used, such as "trak" boxes, as well as sample groupings for the description of groups of samples. A "trak" box is a file format box that contains sub-boxes for describing a track, i.e. a chronological sequence of related samples.

[0057] A box, also called a container, is a hierarchical data structure provided to describe the data in a file. A box is an object-oriented building block defined by a unique type identifier (usually a four-letter code, also written as FourCC or 4CC) and a length. All data in a file (media data and metadata describing that media data) is contained in a box. There is no other data in the file. A file-level box is a box that is not contained in another box.

[0058] In this second case, a set of ISOBMFF boxes, the "meta" boxes, are used. These boxes and their hierarchy offer fewer descriptive tools than the "track related" boxes (the "trak" box hierarchy) and relate to "information items" or "items" instead of related samples. It should be noted that the expressions "box" and "container" may both be used interchangeably to refer to data structures that contain metadata describing the organization and / or properties of image data within a file.

[0059] FIG. 1 shows an example of a HEIF file that contains media data such as one or more still images and possibly one or more videos and / or one or more image sequences. The file contains an "ftyp" box (FileTypeBox) 111 that contains an identifier for the type of file (usually a set of four character codes). The file contains a box called "meta" (MetaBox) 102 that is used to contain general untimed metadata that contains metadata structures that describe one or more still images. This "meta" box 102 contains an "iinf" box (ItemInfoBox) 121 that describes several single images. Each single image is described by a metadata structure ItemInfoEntry, also shown as items 1211 and 1212. Each item has a unique 16-bit or 32-bit identifier item_ID. The media data corresponding to these items is stored in a container for media data, for example an "mdat" box 104. This media data may be stored in an "idat" or "mdat" box, an "imda" box, or in a separate file. An 'iloc' box (ItemLocationBox) 122 provides, for each item, the offset and length of the associated media data in the 'mdat', 'idat', or 'imda' box 104. The media data of an item may be fragmented into extents. In this case, the 'iloc' box 122 provides the number of extents for the item and, for each extent, its offset and length in the 'mdat', 'idat', or 'imda' box 104. An 'iref' box (ItemReferenceBox) 123 may also be defined to describe the association of one item with other items via typed references.

[0060] Optionally, to describe the storage of an image sequence or video, the HEIF file 101 may contain a box called "moov" (MovieBox) 103 that describes one or more image sequence or video tracks 131 and 132. Typically, track 131 may be a picture sequence ("pict") track designed to describe a set of images where time information is not necessarily meaningful, and track 132 may be a video ("vide") track designed to describe video content. Both tracks describe a series of image samples, where an image sample is a set of pixels captured at the same time, e.g. frames of a video sequence. The main difference between the two tracks is that in the "pict" track, timing information is not necessarily meaningful, whereas for the "vide" track, timing information is intended to constrain the timing of the display of the samples. The data corresponding to these samples is stored in an "mdat" box 104, which is a container for media data.

[0061] The "mdat" container 104 stores untimed encoded images corresponding to the items represented by data portions 141 and 142, and optionally, timed encoded images corresponding to the samples represented by data portion 143 when a video track is also present in the HEIF file.

[0062] HEIF files 101 offer different alternatives for storing multiple images. For example, multiple images may be stored either as an item or as a track of samples, which may be a "pict" track or a "vide" track. The actual choice is typically made by the application or device generating the file according to the type of image and the expected use of the file.

[0063] A HEIF item is a derived image item when it has a "dimg" item reference to one or more other image items that are input to the derivation.

[0064] An item with an item_type value of "iovl" defines a derived image item by overlaying one or more input images in a given layered order within a larger canvas. The input images are listed in layered order on a "dimg" item basis for this derived image item. The overlay image item's data specifies the position of each input image within the larger canvas.

[0065] An item with an item_type value of "grid" defines a derived image item whose reconstructed image is formed from one or more input images in a given grid order within a larger canvas. The input images are listed in grid order based on the "dimg" item for this derived image item. The data for this grid image item specifies the number of rows and columns in the grid, as well as the size of the larger canvas.

[0066] The ISO Base Media File Format specifies several alternatives for grouping samples or items depending on the container that holds the samples or items to be grouped. These alternatives can be viewed as grouping data structures or mechanisms, i.e. boxes or data structures that provide metadata describing grouping criteria and / or group properties and / or group entities.

[0067] The first grouping mechanism, represented by EntityToGroupBox, applies to grouping of items or tracks. In this mechanism, the expression "entity" is used to refer to items or tracks or other EntityToGroupBoxes. This mechanism specifies a grouping of entities. EntityToGroupBox is defined according to the following syntax: aligned(8) class EntityToGroupBox(grouping_type, version, flags) extends FullBox(grouping_type, version, flags) { unsigned int(32) group_id; unsigned int(32) num_entities_in_group for (i = 0; i < num_entities_in_group; i++) { unsigned int (32) entity_id; / / The remaining data may be specified for a particular grouping_type }

[0068] grouping_type is used to specify the type of group. Several values ​​of grouping_type are specified in HEIF. group_id provides an identifier for a group of entities. entity_id represents the identifiers of the entities that make up the group, i.e. either a track_ID for a track, an item_ID for an item, or another group_id for an entity group. In FIG. 1, a group of entities inheriting from EntityToGroup boxes 1241 and 1242 is contained in a container 124 identified by the four-letter code "gprl" of the GroupsListBox.

[0069] An entity grouping consists of associating a grouping type that identifies the reason for grouping a set of items, tracks, or other entity groups. In this document, grouping information is referred to as information in one of the EntityToGroups that conveys information for grouping a set of images.

[0070] A grouping_type value of "altr" indicates that the items and tracks mapped to this grouping are alternatives to each other and only one of them should be rendered. The player should choose the first entity in the group that it can handle and that fits the application's needs.

[0071] ISOBMFF provides a mechanism for describing and associating properties with items. These properties are called item properties. The ItemPropertiesBox 125 allows the association of any item with an ordered set of item properties. The ItemPropertiesBox consists of two parts: an item property container box "ipco" 1251 that contains an implicitly indexed list of item properties 1253, and an item property association box "ipma" 1252 that contains one or more entries. Each entry in the item property association box associates an item with its item properties. The HEIF standard extends this mechanism to allow the association of item properties with items and / or entity groups. Note that in this description, we use item properties generically to specify both properties of items or properties of entity groups in a generic way. Item properties associated with an entity group apply to the entity group as a whole, not to each entity in the group individually.

[0072] The associated syntax is: aligned(8) class ItemProperty(property_type) extends Box('property_type') { } aligned(8) class ItemFullProperty(property_type, version, flags) extends FullBox(property_type, version, flags) { } aligned(8) class ItemPropertyContainerBox extends Box('ipco') { properties ItemProperty() []; / / Box is derived from / / ItemProperty or ItemFullProperty, fill the box } aligned(8) class ItemPropertyAssociation extends FullBox('ipma', version, flags) { unsigned int(32) emtry_count; for(i = 0; i < entry_count; i++) { if (version < 1) unsigned int(16) item_ID; else unsigned int(32) item_ID; unsigned int(8) association_count; for (i=0; i <association_count; i++) { bit(1) essential; if (flags & 1) unsigned int(15) property_index; else unsigned int(7) property_index; } } } aligned(8) class ItemPropertyBox extends Box('iprp') { ItemPropertyContainerBox property_container; ItemPropertyAssociation association[]; }

[0073] The ItemProperty and ItemFullProperty boxes are designed for the description of item properties. ItemFullProperty allows to define several versions of the box syntax and may contain one or more parameters whose presence is conditioned by either the version or flag parameters.

[0074] The ItemPropertyContainerBox is designed to describe a set of item properties as an array of ItemProperty boxes or ItemFullProperty boxes.

[0075] The ItemPropertyAssociation box is designed to describe the association between items and / or entity groups and their item properties. It provides a description of a list of item identifiers and / or entity group identifiers, where each identifier (item_ID) is associated with a list of item property indexes that refer to the item properties in the ItemPropertyContainerBox.

[0076] The object of the invention is to provide fine control of the granularity of progressive rendering in HEIF of image items built from several image components. Such image items may be grid derived image items (also denoted grid items), overlay derived image items (also denoted overlay items), or any other type of derived image based on several input images. They are built from several input image items. Such input image items may also be coded image items coded using HEVC tiles or layers, or using VVC sub-pictures or layers. For clarity, the remainder of the description is based on grid items, but can also be applied to other image items built from several image components (also denoted input images).

[0077] In the following description, the target image of progressive drawing is called the main image. The constituent or input images used to construct this main image are called sub-images. A progressive step refers to a reconstructed version of this main image. The initial progressive step corresponds to an empty image and the final progressive step corresponds to the main image. Intermediate progressive steps may have a lower resolution and / or lower quality than the main image. Note that in the following description, for simplicity, the initial progressive step may be ignored.

[0078] In some cases, the progressive rendering feature may be used for purposes other than displaying images: for example, a process for detecting people in an image may use a quick first check against a lower quality version of the image to quickly eliminate images that do not contain people.

[0079] The progressive rendering of the main image is realized by using different coded versions of the sub-images. The reconstructed version of the main image may consist of decoding all input images and assembling these decoded sub-images according to the type of main image item (e.g. grid, overlay, sub-image configuration...). The sub-image versions may be the sub-image itself, a thumbnail of the sub-image, a lower quality and / or lower resolution coded version of the sub-image. The different coded versions may also be included in one or more layers of a multi-layer image. When this sub-image is coded at several quality or resolution levels, the progressive rendering of the main image may use a subset of these qualities or resolution levels. The content of a progressive step refers to the set of sub-image versions used during the reconstruction of the version of the main image corresponding to this progressive step. The refinements of a progressive step refer to the subset of sub-image versions that are part of the content of the progressive step and were not part of the content of the previous progressive step.

[0080] Each progressive step may be associated with a reconstruction point, which is a position in the file corresponding to a point in the file where all the versions of the input images necessary to generate the version of the main image corresponding to the progressive step have been performed. When reading a file, the reconstruction point means the point in the file where all the image data needed for a given progressive step has been read.

[0081] If a sub-image is a multi-layer image, i.e. encoded with several quality or resolution layers, the sub-image may be represented by a single item whose media data includes all encoded layers. In some cases, media data corresponding to different layers may be fragmented and described in the item location box "iloc" using extents. For example, media data for a sub-image encoded as a multi-layer image with two layers may be located using two extents, each indicating the location of the data corresponding to a layer when this sub-image is described by one image item.

[0082] This multi-layer image may also be represented by several image items corresponding to different layers, or different subsets of layers. These image items may share a common portion of media data. For example, two different image items may correspond to a sub-image coded as a multi-layer image with two layers. The first image item corresponds to the first layer, and its media data will be located using an extent indicating the location of the data corresponding to the first layer. The second image item corresponds to a combination of both layers, and its media data is located using two extents, each indicating the location of the data corresponding to the two layers. This description of the layers may also be available in the image properties to indicate which layers are included in the image item. Thus, each image item representing a sub-image coded as a multi-layer image may be considered as a sub-image version, like a different coded version of a non-multi-layer image item.

[0083] If the main image is a tile-coded HEVC image or a subpicture-coded VVC image, the subpictures may be image items corresponding to tiles (e.g., "hvt1" item) or subpictures (e.g., "vvs1" or "vvc1" item). This main image may be an HEVC item, also referred to herein as an HEVC base item, reconstructed from its associated tile subpictures using item references of type "tbas" from each tile subpicture item to the base item, with the location of the tile subpictures being provided by the "rloc" property. This main image may be a VVC base item reconstructed from its associated subpicture subpictures using item references of type "subp" from the base item to the subpicture subpicture items. A subpicture version may be a tile or subpicture coded at a given quality or resolution level. A tile or subpicture may also be coded as a multi-layer image, and such a subpicture version may be obtained by decoding a subset of its quality or resolution layers.

[0084] FIG. 2 shows a first example of a file structure for enabling progressive rendering of grid items (the same applies to overlays or HEVC or VVC base items, as follows: overlays also reference sub-images through the "dimg" item reference type, while HEVC base items are referenced by sub-images through the "tbas" item reference type and VVC base items reference sub-images through the "subp" item reference type). The main or primary image item is a grid item 202 "high quality grid", which is built using nine sub-images "h1" referenced at 220 to "h9" referenced at 228. These sub-images are coded at high quality. This main image is included in an entity group of type "altr" 200, which also contains grid item 201, and a "low quality grid", which is built using nine sub-images "l1" referenced at 210 to "l9" referenced at 218. These sub-images are coded at low quality. Progressive rendering of the main image 202 may be achieved by first rendering grid item 201 once all of the lower quality sub-images 210-218 are available, and then rendering grid item 202 once all of the higher quality sub-images 220-228 are available. However, this file structure only provides for two refinement steps. Especially when the main image 202 is of high quality and large size, the amount of data corresponding to the sub-images 220-228 may be large and take a long time to load. Furthermore, there is no guarantee that the file will organize the data for the lower and higher quality versions of the sub-images in a way that allows for progressive refinement.

[0085] 3 shows another file structure for enabling progressive rendering of similar grid items. The main or primary image item is a grid item 300 composed of nine sub-images "h1", referenced at 330 through "h9", referenced at 338, encoded at high quality. Each sub-image is associated with a lower quality encoded alternative via an "altr" entity group, with each of the nine sub-images "h1" through "h9" being associated with nine sub-images "l1", referenced at 320 through "l9", referenced at 328. For example, the "altr" entity group relating "h1" to "l1" is referenced at 310. This grid item may therefore be reconstructed using either a lower quality or a higher quality version of each sub-image, depending on which version is available for each sub-image. This file structure can provide up to 10 different progressive steps: the first step consists of all low quality sub-images, the second step replaces one of these low quality sub-images with a high quality sub-image, and so on up to the tenth step where only high quality sub-images are used.

[0086] However, the actual progressive steps depend on the order of the sub-image data in the HEIF file. Figure 4a represents a first order where almost no progressive rendering is possible, where low- and high-quality sub-images are interleaved in the HEIF file. Similarly, in the case of multi-layer sub-images, the base and enhancement layers may not be organized to have all the base layers first, followed by all the enhancement layers. As a result, a complete reconstruction of the grid items is not possible until the final sub-image "l9" is received. The main image can be reconstructed immediately after the last sub-image "h9" is received.

[0087] Figure 4b represents a second ordering, more suitable for progressive rendering, where all low-quality sub-images occur first in the HEIF file, followed by all high-quality sub-images. With this ordering, 10 progressive steps are possible, from using only low-quality sub-images to using only high-quality sub-images. After loading each high-quality image, a new progressive step is possible. The same data ordering may be used for multi-layer images, to provide first the base layer and then the enhancement layers. In this case, sub-images used by the main image have dependencies on other sub-image items, and the data dependent on the sub-image items should be placed before the data of the sub-image items.

[0088] Fig. 4c represents a third ordering that is also suitable for progressive rendering with 10 possible progressive steps, albeit using a different order for the high quality sub-images: the ordering in Fig. 4b provides row-wise progressive rendering, while the ordering in Fig. 4c provides column-wise progressive rendering.

[0089] The file structure described in relation to FIG. 3 allows for more progressive steps than the structure described in relation to FIG. 2, but still has some drawbacks. First, the typical use of grid items may use a much larger number of input image items, resulting in many possible progressive steps. A large number of progressive steps also requires more processing power, while allowing for smoother refinement of the image, since the grid image must be reconstructed at each step. Second, there is no guarantee that the ordering of the sub-image versions in the HEIF file effectively allows for progressive rendering of the grid image. Furthermore, there is no indication of how this progressive rendering will be organized, whether row-by-row, column-by-column, or using another refinement pattern. Any refinement may result in a poor user experience when viewing the image.

[0090] The present invention proposes to extend HEIF by adding information about the organization of the file with respect to progressive rendering. This information, hereafter referred to as progressive information, may take several forms, detailed in the progressive embodiments described below. This progressive information may include a progressive rendering method or a progressive method. Examples of progressive rendering methods are the top-bottom ordering from FIG. 4b or the left-right ordering from FIG. 4c. This progressive information may include a description of the progressive steps proposed by the file.

[0091] Figure 4d shows another example in which the same ordering as in Figure 4b is used in conjunction with four different progressive steps, namely an initial progressive step including only a low-quality sub-image, followed by three subsequent progressive steps, each adding a row of high-quality sub-images.

[0092] Fig. 4f shows the rendering of the grid image item after the second progressive step, when three high quality sub-images corresponding to the first row of the grid have been received. In this example, the content of the second progressive step is three high quality sub-images h1 to h3 and six low quality sub-images l4 to l9. The refinement of this progressive step is the three high quality sub-images h1 to h3.

[0093] Figure 4e represents another example in which the same ordering as in Figure 4c is used in connection with four different progressive steps, namely an initial progressive step including only low quality sub-images, followed by three subsequent progressive steps, each adding a row of high quality sub-images.

[0094] Fig. 4g shows the rendering of the grid image item after the second progressive step, when three high quality sub-images corresponding to the first column of the grid have been received. In this example, the content of the second progressive step is three high quality sub-images h1, h4 and h7, and six low quality sub-images l2, l3, l5, l6, l8 and l9. The refinement of this progressive step is the three high quality sub-images h1, h4 and h7.

[0095] FIG. 5 illustrates the main steps for generating a HEIF file according to an embodiment of the present invention.

[0096] In a first step 500, image data to be included in the HEIF file is obtained. In the example of FIG. 3, this image data consists of nine low quality sub-images and nine high quality sub-images. Furthermore, information about the structure of the HEIF file is obtained, including for example the type of main image, its relationship to the sub-images, and the relationship between the sub-images. In the example of FIG. 3, this information is that the main image is a grid item consisting of nine sub-images h1 to h9, each of which has associated low quality sub-images ranging from l1 to l9.

[0097] In a second step 510, progressive specifications for organizing file contents for progressive rendering of the main image are obtained. These progressive specifications depend on the selected progressive implementation. These progressive specifications may be obtained directly from user input or from a configuration file, or may be determined by some process. For example, a software component may determine the type of content represented by the main image and select the most appropriate progressive rendering method. If the main image corresponds to a landscape image, the main image may select the top-bottom progressive method, whereas if the main image corresponds to a person whose face is in the center of the image, the main image may select the center-to-boundary progressive method. As another example, when an object or region of interest is present, the progressive rendering method may begin by first refining the area surrounding the object or region of interest and then proceeding towards the image border.

[0098] The next step 520 is to generate progressive information to be included in the HEIF file from the progressive specification obtained in step 510. This generation depends on the progressive implementation chosen, which are described in more detail below.

[0099] In the next step 530, the image data is ordered according to the progressive specification. For example, if the progressive specification indicates a top-bottom progressive rendering method as in the example of Figure 4b, the image data will be ordered as shown in Figure 4b.

[0100] For this ordering step, the image data is organized into progressive data blocks, which are ordered according to the progressive specification.

[0101] If the main image is a grid item, there is a progressive data block containing data for the grid item itself. There is also one progressive data block for each version of each sub-image that makes up the grid. In the example of Figure 4b, there are 19 progressive data blocks: the grid, 9 low quality image data, and 9 high quality image data.

[0102] If the main image is an overlay item, there is a progressive data block containing data for the overlay item itself, and one progressive data block for each version of each sub-image that makes up the overlay.

[0103] This may be similarly transposed for any kind of main image based on the input sub-image.

[0104] When the main image is a tile-coded HEVC image or a sub-picture-coded VVC image, there may be one or more progressive data blocks for the image itself, and one progressive data block for each version of each tile or sub-picture. If the sub-image is coded as a multi-layer image with several quality layers, e.g. a base layer and one or several enhancement layers, each layer corresponds to a progressive data block.

[0105] In some cases, the progressive data block may be further divided into several sub-blocks.

[0106] Preferably, the data block containing data relating to the main image structure is placed first, for example the data for the grid item (or overlay) itself is preferably placed first.

[0107] This ordering may also cover data that is not directly related to the main image: for example there may be a progressive data block that corresponds to a thumbnail of the main image, which is preferably ordered before the progressive data block of the main image.

[0108] There may be progressive data blocks corresponding to metadata associated with the main image. Preferably, these data blocks are ordered after all progressive data blocks for the main image. In some cases, these data blocks may be inserted between two progressive steps, for example if it is useful to process or render some metadata information before loading the main image. Some progressive specifications may indicate where to place these progressive data blocks.

[0109] There may be progressive data blocks corresponding to region items associated with the main image. Preferably, these data blocks are ordered after all progressive data blocks for the main image. In some cases, these data blocks may be inserted between two progressive steps, for example, if it is useful to process or render some region related information before loading the main image. Some progressive specifications may indicate where to place these progressive data blocks.

[0110] In the next step 540, metadata is written for the HEIF file, including the progressive information generated in step 520.

[0111] In a final step 550, all progressive data blocks are written, in the order determined in step 530, to the media portion of the file, for example the "mdat" box.

[0112] In some cases, the order of steps 520 and 530 may be switched. This may be useful if the generation of the progressive information depends on the order of the progressive data blocks.

[0113] 6 shows the main steps for progressive rendering of a HEIF file generated according to an example of the invention. These steps are intended to be performed during the loading of a HEIF file, which may for example be received from a local storage, from a network storage or from a network server using the HTTP protocol.

[0114] In a first step 600, the first part of the HEIF file is loaded, which contains all the metadata that makes up the "meta" box of the HEIF file.

[0115] In step 610, the "meta" box is parsed to obtain the structure of the HEIF file, including, for example, the number of items, the types of these items, their relationships, properties associated with these items, etc. This "meta" box may include an "idat" box that provides data defining the overlay item or grid item structure. The "meta" box may include a thumbnail.

[0116] In the next step 620, progressive information is extracted from the result of the analysis of step 610. The refinement of the progressive information depends on the selected progressive implementation. As part of this extraction, the progressive steps contained in the HEIF file are obtained. This retrieval depends on the selected progressive implementation.

[0117] In the next step 630, some of the data present in the file is loaded.

[0118] In step 640 it is checked whether the data corresponding to the new progressive step has been loaded. This check depends on the progressive implementation selected.

[0119] If the result of the check is positive, the next step is step 650, where a new rendering of the main image version corresponding to this progressive step is realized. This rendering may include computing a reconstructed image from the sub-image versions corresponding to the progressive step reached. This rendering may also include some post-processing computed on the reconstructed image version, for example rescaling to fit the display area or applying a filter selected by the user. This rendering may also include applying some transformation properties to the reconstructed image version, such as rotation, scaling, mirroring or cropping.

[0120] It should be noted that in some embodiments, a further check is realised based on the progressive information and the progressive step reached to decide whether a reconstructed image version should be rendered: for example, if only a part of the main image version is displayed, this check may further verify whether the progressive step reached provides a refinement to the displayed part of the reconstructed image version.

[0121] If the result of the check in step 640 is negative, and after step 650, the next step is step 660, which checks whether the file has been completely received.

[0122] If the result of the check is negative, the next step is step 630, which waits for additional portions of the file to be received.

[0123] If the result of the check is positive, the next step is step 670, where the algorithm ends.

[0124] It should be noted that in step 630, data corresponding to several refinement steps may be received, in which case the check in step 640 will be positive and in step 650 the reconstructed image will be rendered using the sub-image corresponding to the last progressive step reached.

[0125] Note that these steps are all for receiving a HEIF file in sequence. However, these steps can be easily adapted to receive a HEIF file as chunks in any order. This adaptation mainly concerns steps 640 and 660. In these steps, checks must take into account that parts of a HEIF file may be received in any order, and do not utilize sequential reception of the HEIF file, explicitly verifying that all data corresponding to progressive steps or the entire file has been received.

[0126] Various embodiments for encoding progressive information in HEIF files are now described.

[0127] [Embodiment of refinement pattern] In a first embodiment (hereafter referred to as refinement pattern embodiment), the progressive information included in the HEIF file is based on a progressive refinement pattern that defines the progressive refinement method used in the HEIF file. The progressive refinement pattern may for example be the top-bottom progressive refinement of Figs. 4d and 4f, or the left-right progressive refinement of Figs. 4e and 4g. The progressive refinement pattern indicates the scan order of the input image. A progressive step is defined when a set of versions of the input image corresponding to a step of the scan order of the input image is loaded and allows the rendering of a version of the main image. In other words, the progressive refinement pattern indicates the ordering of refinement for different spatial regions of the main image. For example, in the case of a grid item, the progressive refinement pattern indicates the order of refinement for the different sub-images that make up the grid.

[0128] This first embodiment advantageously provides compact signaling for progressive information within a HEIF file.

[0129] The progressive refinement pattern used within a HEIF file may be exposed at the application level, allowing an application to select a HEIF file among several HEIF files based on its progressive refinement pattern. For example, for the same main image, a first HEIF file may use top-bottom progressive refinement and a second HEIF file may use left-right progressive refinement. An application may select among these two files the one that best suits its needs.

[0130] [Prescribed refinement variant] In a first variant of this first embodiment, hereinafter referred to as the predefined refinement pattern variant, a list of progressive refinement patterns is predefined. The progressive information includes an indication of one of the predefined progressive refinement patterns.

[0131] This variant advantageously allows simple signaling of the progressive refinement method used in the HEIF file.

[0132] A progressive refinement pattern may be expressed as an item property associated with the image item to which it is applied. This progressive refinement pattern item property may be described by the following structure: aligned(8) class ProgressivePatternProperty extends ItemFullProperty('ppat', version = 0, flags = 0){ unsigned int(8) pattern_type; unsigned int(1) reverse_flags; unsigned int(1) bidirectional_flag; unsigned int(1) parameter_flags; unsigned int(5) reserved; unsigned int(8) scale; if (parameter_flag == 1) { unsigned int (32) parameter; } }

[0133] This progressive refinement pattern item property may be identified, for example, by the "ppat" 4cc. Alternative 4ccs may also be used.

[0134] In this structure, pattern_type specifies the type of refinement pattern to be used. Several refinement pattern types may be defined, such as top-bottom, left-right, top-left to bottom-right diagonal, bottom-left to top-right diagonal, center to border, clockwise, etc. In addition, specific refinement pattern types for overlays may be defined, such as front-to-back, or maximum-to-minimum.

[0135] As an example, the following values ​​may be defined for pattern_type: When equal to 0, refinement is done row-based, top-to-bottom, which ensures that the data for the input image is row-based ordered. When equal to 1, refinement is column-based and left-to-right. If it is equal to 2, refinement is performed diagonally from the top left to the bottom right. When equal to 3, refinement is done from the center to the boundary. If it is equal to 4, refinement is done on an input image item basis in counterclockwise order. When equal to ·5, refinement is done on an input image item basis in a front-to-back order (i.e. according to the overlay layering order). If it is equal to 6, refinement is done based on the input image items, from the largest to the smallest item.

[0136] The reverse_flag indicates that the order defined by the refinement pattern is reversed. For example, a top-bottom refinement pattern becomes a bottom-to-top refinement pattern with the reverse flag. In some cases, this flag may be removed from the ProgressivePatternProperty and replaced with a new refinement pattern type. For example, a top-bottom refinement pattern type could be added.

[0137] The bidirectional_flag indicates that the ordering starts from both ends defined by the refinement pattern. For example, a top-bottom refinement pattern results in a refinement pattern where progressive refinement starts from both the top and bottom rows and progresses towards the middle row. This flag may be combined with the reverse_flag, which reverses the ordering obtained after applying the reordering associated with the bidirectional_flag. In some cases, this bidirectional_flag may be removed from the ProgressivePatternProperty and replaced with a new refinement pattern type. For example, a centered refinement pattern type from above and below could be added.

[0138] parameter_flag indicates whether a parameter value exists for the refinement pattern. If the value of this flag is 1, then the parameter value is present in the item properties. The meaning of this parameter value depends on the type of refinement pattern. For example, a clockwise refinement pattern may by default start at a position corresponding to noon on a clock continuing in a clockwise order. A parameter value for this refinement pattern may indicate a different start position by indicating a different start time or by giving the start angle in degrees. A parameter value may be present for all refinement pattern types. A parameter value may also not be present for all refinement pattern types. The presence of a parameter value may depend on the pattern type. This parameter value may be a set of values ​​or a more complex structure.

[0139] This scale value indicates that the progressive steps defined by the refinement pattern type are grouped into a single coarser step (called the scaled progressive step). In such a case, only the scaled progressive steps should be rendered by the reader, and each single progressive step is not rendered. The scale value may also indicate how many progressive steps are grouped together. For example, a scale value of 2 for a left-right refinement pattern type indicates that each scaled progressive step refinement contains two columns of refinements at once. This scale value may also be a fractional value. In this case, the scaled progressive step n for the refinement pattern specified by the item property corresponds to the number of progressive steps defined by the refinement pattern type [n × scale], i.e., the rounded value of the progressive step index n multiplied by the scale value. In this formula, the initial step corresponding to the lower quality version of the subimage is numbered 0. For example, with a scale value of 1.5, the top-bottom ordering alternates between scaled progressive steps, adding one or two rows of high quality imagery. In some cases, this scale value may be removed from the ProgressivePatternProperty and replaced with a new refinement pattern type. For example, there may be a refinement pattern type with two rows, one above the other.

[0140] In a variant, this scale value may be replaced by a list of numbers indicating how many progressive steps are grouped into each scaled progressive step. For example, if this list has the values ​​[2,1,3] for the up and down refinement pattern, the refinement of the first scaled progressive step will contain 2 rows of high quality sub-images, the next 1 row, and the last 3 rows. If there are more progressive steps than the sum of the values ​​contained in the list, this list may be repeated to continue grouping the progressive steps, the remaining progressive steps may be grouped using a default group size, or the remaining progressive steps may be kept alone.

[0141] In this embodiment, in step 510 of FIG. 5, the obtained progressive specification includes a pattern type, and may include a flag indicating whether reverse ordering is used, a flag indicating whether bidirectional ordering is used, scaling values ​​for the progressive steps, and parameter values ​​associated with the pattern type.

[0142] In step 520 of Figure 5, an item property structure is generated to represent all these specifications. This item property structure is written as part of the "meta" box of the HEIF file in step 540 and associated with the image item corresponding to the main image in the HEIF file. For example, this item property structure is written as part of the item property container box "ipco" and associated with the image item corresponding to the main image via the item property association box "ipma".

[0143] An ordering defined by the pattern type is derived and applied to the progressive data blocks in step 530 of Figure 5. This ordering may be modified by the reverse and bidirectional flags.

[0144] In this embodiment, in step 620 of FIG. 6, progressive information of an image item is extracted by parsing the progressive refinement pattern property item associated with this image item. Using the pattern type specified in the progressive refinement pattern property item and the different flags and values ​​contained in this property item, a list of progressive steps is calculated. Each progressive step is specified by a list of sub-image versions that are added to the previous progressive step. For example, in FIG. 4f, the pattern is an up-down pattern. There are three progressive steps, one for each row of the grid item. Each progressive step contains three sub-image versions corresponding to the three columns of the grid item.

[0145] In step 640, it is checked whether all sub-image versions contained in the list associated with the progressive step (or the scaled progressive step, if a scale value is present and not equal to 1) have already been loaded.

[0146] In some cases, no progressive steps are calculated in step 620. In this case, the checks in step 640 may be based on the number of sub-image versions received. For example, the checks may be positive every time a new predefined number of new sub-image versions are received, or when the number of new sub-image versions received is a predefined percentage of the total number of sub-image versions. These checks may not take into account sub-image versions of possibly low quality.

[0147] The check may also be based on the number of bytes received. For example, the check may be positive each time a new predetermined number of bytes is received, or when the number of new bytes received is a predetermined percentage of the total number of bytes in the HEIF file. In some cases, bytes corresponding to content of lower quality sub-image versions and / or content of the HEIF file structure, such as "meta" boxes, may not be considered in these checks.

[0148] In a variant, this scale value may be replaced by a scale_fraction value indicating how many progressive steps are grouped into a scaled progressive step as a fraction of the total number of progressive steps.

[0149] In another variant, this scale value may be replaced by a scale_number value indicating that the progressive steps are grouped into a number of scaled progressive steps equal to this scale_number value.

[0150] [Analytical refinement patterns] In a second variant of the first embodiment, hereafter referred to as analytical refinement pattern, the progressive refinement pattern is defined in an analytical way. In this embodiment, the progressive information comprises an indication of the scan order of the input image based on an equation and related parameters.

[0151] This variant advantageously allows for a richer expressiveness in the possible progressive refinement patterns used in HEIF files.

[0152] Analytical refinement patterns may be defined inside a progressive refinement pattern item property, the structure of which may be as follows: aligned(8) class ProgressivePatternProperty extends ItemFullProperty('ppat', version = 0, flags = 0){ unsigned int(2) pattern_type; unsigned int(1) bidirectional_flag; unsigned int(5) reserved; / / Linear if (pattern_type == 0) { unsigned int(8) start_x; unsigned int(8) start_y; unsigned int(8) end_x; unsigned int(8) end_y; unsigned int(16) step_numerator; unsigned int(16) step_denominator; } / / Radial else if (pattern_type == 1) { unsigned int(8) center_x; unsigned int(8) center_y; int (16) step_numerator; unsigned int(16) step_denominator; } / / Circular else if (pattern_type == 2) { unsigned int(8) center_x; unsigned int(8) center_y; unsigned int(16) startin_angle; int (16) step_numerator; unsigned int(16) step_denominator; } }

[0153] In this structure, pattern_type indicates the general type of the refinement pattern, for example whether it is linear, radial, or circular.

[0154] The bidirectional_flag indicates that the ordering starts from both ends defined by the refinement pattern.

[0155] If the pattern_type value is 0, the refinement pattern is a linear refinement pattern. A linear refinement pattern may be, for example, an up-down refinement pattern or a left-right refinement pattern.

[0156] The start_x and start_y values ​​indicate the start point of the refinement pattern, which is the centre of the grid item whose coordinates within the grid are given by these values.

[0157] Similarly, the end_x and end_y values ​​indicate the end point of the refinement pattern, which is the centre of the grid item whose coordinates within the grid are given by these values.

[0158] Finally, the values ​​of step_numerator and step_denominator indicate the size of each progressive refinement step (equal to step_numerator divided by step_denominator).

[0159] To determine which progressive step the high quality version of the sub-image of a grid item belongs to, the following operations are realized using the grid coordinates: First, the center of the grid item is projected orthogonally onto the line defined by the start and end points. The distance of this projected point from the start point is calculated. This distance is considered positive if the projected point belongs to a half-line that also contains the end point, and negative otherwise. This distance is then divided by the size of the progressive refinement step, rounded up, and increased by one to obtain the index of the progressive step to which this grid item belongs. If the calculated index is smaller than one, the grid item belongs to the first progressive step. If the calculated index is greater than the index of the end point, the grid item belongs to the last progressive step.

[0160] The calculation of the progressive step index of a grid item can be summarized by the following formula:

number

[0161] Here, P i is the starting point of the refinement pattern, and P f is the end point of the refinement pattern and G is the center point of the grid item.

[0162] If the pattern_type value is 1, the refinement pattern is a radial refinement pattern. The radial refinement pattern may be, for example, a center to boundary refinement pattern.

[0163] The center_x and center_y values ​​indicate the starting point of the radial refinement pattern as the center of the grid item at a given position in the grid.

[0164] Finally, the values ​​of step_numerator and step_denominator indicate the size of each progressive refinement step. step_numerator is a signed integer, allowing the size to be either positive or negative. A positive size indicates that the progressive steps go outwards from the centre of the refinement pattern, while a negative size indicates that the progressive steps go inwards towards the centre of the refinement pattern.

[0165] In some cases, step_numerator is an unsigned integer, and a flag is added to signal whether the progressive step is outward or inward.

[0166] To determine which progressive step a grid item belongs to, the following calculation is realized using the grid coordinates: The distance between the center of the grid item and the center of the start point of the radial refinement pattern is calculated. This distance is divided by its size value, rounded up and incremented by 1 to get the relative index of the progressive step this grid item belongs to. Once all relative indexes have been calculated for all grid item progressive steps, the absolute index is calculated by subtracting the value of the smallest relative index from the relative index.

[0167] The calculation of the progressive step index of a grid item can be summarized by the following formula:

number

[0168] where C is the starting point of the radial refinement pattern and G is the center point of the grid item.

[0169] If the pattern_type value is 2, the refinement pattern is a circular refinement pattern. The circular refinement pattern may be, for example, a clockwise refinement pattern.

[0170] The center_x and center_y values ​​indicate the reference point of the circular refinement pattern as the center of the grid item at a given position in the grid.

[0171] The starting_angle value indicates the starting direction of the circular refinement pattern, with a value of 0 corresponding to a direction towards the top of the image and the starting_angle value being expressed in degrees in a clockwise direction.

[0172] Finally, the values ​​of step_numerator and step_denominator indicate the size of each progressive refinement step. step_numerator is a signed integer, allowing the size to be either positive or negative. A positive size indicates that the progressive steps proceed clockwise around the reference point, while a negative size indicates that the progressive steps proceed counterclockwise around the reference point.

[0173] In some cases, step_numerator is an unsigned integer and a flag is added to signal whether the progressive steps are clockwise or counterclockwise.

[0174] To determine which progressive step a grid item belongs to, the following calculation is realized using the grid coordinates: The angle between the start direction and the vector from the reference point to the center point of the grid item is calculated. This angle is divided by its size value, rounded up, and incremented by 1 to get the relative index of the progressive step this grid item belongs to. Once all relative indexes have been calculated for all grid item progressive steps, the absolute index is calculated by subtracting the value of the smallest relative index from the relative index.

[0175] The calculation of the progressive step index of a grid item can be summarized by the following formula:

number

[0176] where d is the vector corresponding to the starting direction, R is the reference point of the circular refinement pattern, and G is the center point of the grid item.

[0177] Instead of grid coordinates, other coordinate systems may be used, for example pixel coordinates, coordinates linked to coding blocks, or coordinates corresponding to a predefined number of pixels. In some cases, the horizontal and vertical units may correspond to different numbers of pixels.

[0178] In some cases, the values ​​contained in the progressive refinement pattern item properties may be encoded on a larger number of bits. This may be useful, for example, when using pixel coordinates. For example, the start_x and start_y values ​​may be encoded on 16 or 32 bits.

[0179] In some cases, the start_x, start_y, end_x, end_y, center_x, and center_y values ​​may be encoded as signed integers to allow these points to fall outside the grid itself.

[0180] Optionally, other refinement pattern types may be defined, for example a pattern to provide a Venetian blind effect.

[0181] In some cases, flags may be added to the refinement pattern to more precisely define the positions of the different points used. For example, flags may be added to specify whether a point is in the center of the corresponding grid item or in one of its corners. In some cases, a two-bit flag may be used to allow each point to be located in one of four positions of a grid item: the top left corner, the center, the center of the left border, and the center of the top border.

[0182] [Multiple levels of variation] In variations on these aforementioned embodiments, the sub-images have more than two different versions, for example each sub-image may have a low quality version, a medium quality version and a high quality version, or even more intermediate versions.

[0183] In this variant, the same refinement pattern may be used repeatedly to describe all the transitions between the different quality levels, for example to define the progressive steps between a low quality sub-image and a medium quality sub-image and also to define the progressive steps between a medium quality sub-image and a high quality sub-image.

[0184] For example, in Fig. 4f, using the top-bottom progressive refinement pattern, there are three progressive steps corresponding to the three rows of a grid item. Using three versions for each subimage, this would result in seven progressive steps in the successive progressive refinement of a row: an initial progressive step that uses only a low quality subimage, followed by three progressive steps that each add a medium quality subimage for one row, followed by three other progressive steps that each add a high quality subimage for one row.

[0185] Possibly different refinement patterns may be used to describe the transition between different versions of a sub-image, for example a first refinement pattern may be used to define a progressive step between a low quality sub-image and a medium quality sub-image, and a second refinement pattern may be used to define a progressive step between a medium quality sub-image and a high quality sub-image.

[0186] To allow for several refinement patterns, the progressive refinement pattern item property may include the number of refinement patterns it contains and a loop for describing each of these patterns.

[0187] To allow mixing different versions of sub-images, specific multi-level refinement patterns may possibly be defined, for example in a center-to-border refinement pattern, where a high quality version of a center sub-image can be used in a progressive step before a medium quality version of a border sub-image.

[0188] In a variation of the predefined refinement patterns, new multi-level refinement patterns may be defined.

[0189] In some cases, a step_offset parameter may be added to the progressive refinement pattern description to indicate in which progressive step a new quality level for a version of a sub-image is introduced. In the example of Fig. 4f, with a step_offset parameter having a value of 2, the high quality sub-image of the first row is included in the same progressive step as the medium quality sub-image of the third row, and the refinement of the first row with the high quality sub-image is achieved two progressive steps later by refining this first row with the medium quality sub-image.

[0190] In some cases, a version_split_flag parameter may be added to the progressive refinement pattern description, preferably in combination with the step_offset parameter. If the value of this flag is true, the version of the subimage corresponding to the new quality level is included in a separate progressive step refinement before the progressive step whose refinement includes the version of the subimage corresponding to the current quality level. In the example of Fig. 4f, with the step_offset parameter having a value of 2, a progressive step whose refinement includes the medium quality version of the second row is followed by another progressive step whose refinement includes only the high quality version of the first row, followed by another progressive step whose refinement includes the medium quality version of the third row.

[0191] In some cases, refinement patterns may be defined for the lowest quality versions of the subimages: in the example of Fig. 4f, upper and lower refinement patterns may be applied to lower quality versions of the subimages, and the first progressive step may only include lower quality versions of the subimages in the first row of the grid.

[0192] [Incomplete Level Variation] In a variant, some versions of the sub-images may be missing: in the example of Fig. 4f, the high quality version h9 of the bottom right sub-image may be missing, or the low quality version l7 of the bottom left sub-image may be missing.

[0193] In some cases, these missing sub-images are simply ignored: if the refinement of a progressive step only involves a missing sub-image, then that missing sub-image is simply skipped during the progressive rendering of the main image.

[0194] In some cases, these missing subimages are indicated by adding a list of the missing subimages to a progressive refinement pattern item property.

[0195] In some cases, a progressive step is considered to be still loaded in step 640 if the number of missing sub-image versions included in the refinement of the progressive step is below a predefined threshold. This threshold may be defined as a number of sub-image versions. This threshold may also be defined as a percentage of the total number of sub-image versions included in the refinement of the progressive step (e.g. 5% or 10% of the sub-image versions). In some cases, a progressive step may be considered to be loaded even though some sub-image versions included in its refinement are missing only if the time elapsed since the previous progressive step was loaded is equal to or greater than a predefined threshold.

[0196] When generating a HEIF file using the steps of FIG. 5, the progressive specification may include a list of sub-images that should not be included in the resulting HEIF file.

[0197] [Multi-level pattern] In a variant, the progressive refinement order within the progressive steps may be further specified by another progressive refinement pattern: for example, an up-down progressive refinement pattern may indicate that progressive refinement of the main image is realized row by row, and inside each row, a second progressive refinement pattern may be used to indicate that progressive refinement starts from the center of the row and moves towards its boundary.

[0198] This second progressive refinement pattern may be specified inside a progressive refinement pattern item property, for example using the following structure: aligned(8) class ProgressivePatternProperty extends ItemFullProperty('ppat', version = 0, flags = 0){ unsigned int(8) main_pattern_type; unsigned int(1) main_reverse_flag; unsigned int(1) main_bidirectional_flag; unsigned int(1) main_parameter_flag; unsigned int(5) main_reserved; unsigned int(8) main_scale; if (parameter_flag == 1) { unsigned int (32) main_parameter; } unsigned int(8) secondary_pattern_type; unsigned int(1) secondary_reverse_flag; unsigned int(1) secondary_bidirectional_flag; unsigned int(1) secondary_parameter_flag; unsigned int(5) secondary_reserved; unsigned int(8) secondary_scale; if (parameter_flag == 1) { unsigned int(32) secondary_parameter; } }

[0199] Several second progressive refinement patterns may be specified for different progressive steps.

[0200] The second progressive refinement pattern may have a different structure than the main progressive refinement pattern.

[0201] [Offset embodiment] In a second embodiment of the invention (hereafter referred to as the offset embodiment), the progressive steps are indicated using positions within the HEIF file.

[0202] Advantageously, this second embodiment provides an explicit and simple indication of the progressive steps in a HEIF file.

[0203] Some information about these progressive steps may be exposed at the application level, allowing an application to choose between HEIF files based on their number of progressive steps.

[0204] [Step position variation] In a first variant of this second embodiment, the step position variant is such that, for each progressive step, the position of the last byte in the HEIF file that is part of the last sub-image version included in the refinement of this progressive step is specified in the progressive information stored in that HEIF file.

[0205] The position of a progressive step relative to an image item in a HEIF file may be indicated in an item property associated with this image item. This progressive step position item property may be described by the following structure: aligned(8) class ProgressiveStepLocationProperty extends ItemFullProperty('ploc', version = 0, flags = 0) { unsigned int(16) step_count; for (i=0; i < step_count; i++) { if (flags & 1) { unsigned int(32) item_index; } else { unsigned int(16) item_index; } unsigned int(16) extent_index; } }

[0206] Different sizes may be used for the fields of this progressive step position item property, and possibly several flags may be used to specify the sizes of the different fields, either using the flags parameter on the ItemFullProperty, or flag values ​​directly encoded in the progressive step position item property.

[0207] In this structure, step_count indicates the number of progressive steps described in the progressive step position item property.

[0208] For each progressive step, the item_index value indicates the index of the resource in the array in the item position box "iloc". The extent_index value indicates the index of an extent of this resource. The position of the progressive step is the last byte of this specified extent.

[0209] In some cases, extent_index is an optional field. If extent_index is not present, the position of the progressive step is the last byte of the last extent of the item indicated by the item_index value.

[0210] In some cases, the progressive step position item property does not include an extent_index field.

[0211] In this embodiment, in step 510 of Fig. 5, the progressive specification includes some information for defining the progressive steps. This information may include, for example, a list of the sub-image versions included in each progressive step refinement. In this embodiment, the order of steps 520 and 530 is switched.

[0212] First, the progressive step definition is used to implement an ordering of the progressive data blocks in step 530. This ordering is used to create the structure of the HEIF file and generate the content of the "iloc" box.

[0213] Step 520 is then executed to generate a progressive step position item property based on the contents of the "iloc" box and the progressive step definition. This item property is written as part of the "meta" box of the HEIF file in step 540 and associated with the image item corresponding to the main image in the HEIF file. For example, this item property structure is written as part of the item property container box "ipco" and associated with the image item corresponding to the main image via the item property association box "ipma".

[0214] In this embodiment, in step 620 of Figure 6, progressive information is extracted by parsing the Progressive Step Position Item property. This progressive information is used to generate a list of progressive steps for the main image. Each progressive step is specified by its position, which is the last byte of the extent indicated in the Progressive Step Position Item property.

[0215] In some cases, if the extent_index field is not present, the progressive step position is the last byte of the last extent of the item indicated in the progressive step position item property.

[0216] In step 640, the position of the progressive step is compared to the last loaded byte of the HEIF file. If the position of the progressive step is before the position of the last loaded byte, the progressive step has been loaded.

[0217] If the progressive steps are ordered according to their position, then in step 640 only the first progressive step that has not yet been loaded needs to be checked.

[0218] Preferably, the progressive steps are ordered in the Progressive Step Position Item property according to their position order in the HEIF file, otherwise the progressive steps can be reordered according to their position after being extracted from the Progressive Step Position Item property.

[0219] Optionally, in this variation, an item may be identified by its identifier instead of its index in the array in the Item Position box.

[0220] [Step refinement variation] In a second variant, the step refinement variant, each progressive step is specified by a list of extents containing the data of the sub-image versions included in the refinement of the progressive step.

[0221] The list of extents that constitute the refinements of the different progressive steps of an image item may be indicated with an item property associated with this image item. This progressive step refinement item property may be described by the following structure: aligned(8) class ProgressiveStepRefinementProperty extends ItemFullProperty('prco', version = 0, flags = 0) { unsigned int(16) step_count; for (i=0; i < step_count; i++) { unsigned int(16) item_count; for (j=0; j < item_count; j++) { if (flags & 1) { unsigned int(32) item_index; } else { unsigned int(16) item_index; } unsigned int(16) extent_count; for (k=0; k < extent_count; k++) { unsigned int(16) extent_index; } } } }

[0222] Different sizes may be used for the fields of this progressive step refinement item property, and possibly several flags may be used to specify the sizes of the different fields, either using the flags parameter on the ItemFullProperty, or using flag values ​​directly encoded in the progressive step refinement item property.

[0223] In this structure, step_count indicates the number of progressive steps described in the progressive step refinement item property.

[0224] For each progressive step, the item_count value indicates the number of items included in the refinement of the progressive step. For each item, the item_index value indicates the index of the resource in the array within the item position box "iloc".

[0225] For each item, the extent_count value indicates the number of extents included in the refinement for this item in the progressive step. For each extent, the extent_index value indicates the index of the extent for this resource.

[0226] Generating a HEIF file using this variant is similar to the "iloc" standard.

[0227] In this variant, in step 620 of FIG. 6, each progressive step is specified by a list of extents that are part of its refinement.

[0228] In step 640 it is checked whether all extents belonging to the refinement of the progressive step have been loaded.

[0229] This variant may relate to image items that are divided into spatial portions such as tiles or subpictures or even layers, where each spatial portion has its data identified by an extent within an 'iloc' box.

[0230] Optionally, in this variation, an item may be identified by its identifier instead of its index in the array in the Item Position box.

[0231] In some cases, a progressive step refinement item property may be described by the following structure: aligned(8) class ProgressiveStepRefinementProperty extends ItemFullProperty('prco', version = 0, flags = 0) { unsigned int(16) step_count; for (i=0; i < step_count; i++) { unsigned int(16) item_count; for (j=0; j < item_count; j++) { if (flags & 1) { unsigned int(32) item_index; } else { unsigned int(16) item_index; } unsigned int(16) extent_count; } } }

[0232] In this variant, the extent_count value indicates the number of extents of an item that are included in the refinement of the progressive step or in a refinement of a previous progressive step.In this variant, for an item, the extents are organized in decoding order.

[0233] In some cases, the extents that are part of a refinement of a progressive step are calculated by comparing the list of extents specified for this progressive step in the refinement item property of the progressive step with the extents present in the refinement of the previous progressive step. These extents are used in step 640 to check whether the progressive step has been loaded.

[0234] Optionally, the check in step 640 is modified to check whether all extents specified for this progressive step in the progressive step refinement item properties have been loaded.

[0235] In some cases, the extent_count field is optional in this variation: if extent_count is not present, all extents for the item are included in the refinement of the progressive step or in the refinement of the previous progressive step.

[0236] In a variant, the specification of the progressive steps may use a pointer to a HEIF file. This variant may be used together with the step position variant or together with the step refinement variant.

[0237] In some cases, these pointers may be absolute offsets into the HEIF file.

[0238] For example, in the step position variant, for each progressive step, the position of the last byte that is part of the sub-image version contained in this progressive step is specified directly as an offset into the HEIF file.

[0239] As another example, for each progressive step in the step refinement variant, the locations of the sub-image versions included in the progressive step refinement are specified as a list of offsets into the HEIF file, with each offset associated with a length.

[0240] In some cases, these pointers may be specified using a mechanism similar to that used by item position boxes. In some cases, only a subset of the mechanisms used by item position boxes may be used. For example, these pointers may be specified as offsets into an ItemDataBox 'idat' or an IdentifiedMediaDataBox 'imda'.

[0241] ["iloc"-based variant] In a variant, the specification of progressive steps is realized in a new version of the item position box.

[0242] In this variant, a flag is added to each extent to indicate whether the last byte of this extent is the last byte of the sub-image version included in the progressive step refinement. In this way, the progressive step may be reconstructed from the information contained in the item position box. This flag can be named progressive_step_end.

[0243] In some cases, this flag may be added to each item instead of to each extent.

[0244] In some cases, a progressive step number field is added to each extent to indicate which progressive step refinement this extent is included in. The value 0 may be reserved to indicate that the extent is not included in a progressive step refinement. This progressive step number field may be named progressive_step_number. In some cases, a flag may be used to indicate the presence of this field.

[0245] Optionally, a progressive step number end field is added to each extent to indicate that the last byte of this extent is the last byte of the sub-picture version included in the corresponding progressive step refinement. The value 0 may be reserved to indicate that the last byte of the extent is not the last byte of the sub-picture version included in the progressive step refinement. This progressive step number field may be named progressive_step_number_end. Optionally, a flag may be used to indicate the presence of this field.

[0246] [Item embodiment] In a third embodiment of the invention (hereafter referred to as the item embodiment), the progressive steps are indicated using representations of the sub-image versions they contain in their refinements.

[0247] Advantageously, this third embodiment provides a simple link between the sub-images that make up a main image and the progressive steps for this main image.

[0248] Advantageously, some variations of this third embodiment require minimal signaling in the HEIF file.

[0249] In a first variant of this third embodiment (hereinafter referred to as the version list variant), for each progressive step, a list of sub-image versions included in the refinement of this progressive step is specified in the progressive information stored in the HEIF file.

[0250] The sub-image versions included in the refinement of a progressive step for an image item in a HEIF file may be indicated in an item property associated with this image item. This Progressive Version List Item property may be described by the following structure: aligned(8) class ProgressiveVersionListProperty extends ItemFullProperty('pvli', version = 0, flags = 0) { unsigned int(16) step_count; for (i=0; i < step_count; i++) { unsigned int(16) item_count; for (j=0; j < item_count; j++) { if (flags & 1) { unsigned int(32) item_ID;} else { unsigned int(16) item_ID; } } } }

[0251] In this structure, step_count indicates the number of progressive steps described in the progressive version list item property.

[0252] For each progressive step, the item_count value indicates the number of sub-image versions included in the refinement of this progressive step.

[0253] For each sub-image version, the item_ID value indicates the identifier of the image item that corresponds to this sub-image version.

[0254] In this embodiment, in step 510 of Fig. 5, the progressive specification includes some information for defining the progressive steps, which may include, for example, a list of the sub-image versions included in each progressive step refinement.

[0255] In step 520 of Figure 5, progressive version list item properties are generated based on the progressive step definition. The item properties are written as part of the "meta" box of the HEIF file in step 540 and associated with the image item corresponding to the main image in the HEIF file. For example, the item property structure is written as part of the item property container box "ipco" and associated with the image item corresponding to the main image via the item property association box "ipma".

[0256] In this embodiment, in step 620 of Fig. 6, progressive information is extracted by parsing the progressive version list item property. This progressive information is used to generate a list of progressive steps for the main image. Each progressive step is specified by the list of sub-image versions that it contains in its refinements.

[0257] In step 640 it is checked whether all data for the image items specified in the list of sub-image versions included in the refinement of the progressive step has already been loaded.

[0258] In some cases, a sub-image may be coded as a multi-layer image described by a single image item. Different versions of the sub-image are coded as different layers of this image item. In this case, all versions of the sub-image may be described by a single image item. The ProgressiveVersionListItem property may be modified to indicate the coding layers that correspond to the versions of the sub-image. This modified ProgressiveVersionListItem property may be described by the following structure: aligned(8) class ProgressiveVersionListProperty extends ItemFullProperty('pvli', version = 0, flags = 0) { unsigned int(16) step_count; for (i=0; i < step_count; i++) { unsigned int(16) item_count; for (j=0; j < item_count; j++) { if (flags & 1) { unsigned int(32) item_ID; } else { unsigned int(16) item_ID; } unsigned int(1) has_layer; if (has_layer == 1) { unsigned int(15) layer_id; } else { unsigned int(7) reserved; } } } }

[0259] In this structure, the has_layer value is a flag indicating whether the subimage version corresponds to a particular coding layer of the image item with a given item_ID. If the value of the has_layer flag is true, the layer_id value indicates the identifier of the coding layer that corresponds to the subimage version.

[0260] In some cases, the different layers included in the image item corresponding to the sub-image may be described by an item property associated with this image item. This progressive layer item property may be described by the following structure: aligned(8) class ProgressiveLayerProperty extends ItemFullProperty('play', version = 0, flags = 0) { unsigned int(8) layer_count; for (i=0; i < layer_count; i++) { unsigned int(16) layer_id; } }

[0261] In this structure, the layer_count value indicates the number of layers used for progressive rendering. The total number of layers present in an image item may be more than the number of displayed layers.

[0262] For each layer, the layer_id value indicates the identifier of this layer.

[0263] The progressive version list item property structure may be modified to take advantage of this progressive layer item property. aligned(8) class ProgressiveVersionListProperty extends ItemFullProperty('pvli', version = 0, flags = 0) { unsigned int(16) step_count; for (i=0; i < step_count; i++) { unsigned int(16) item_count; for (j=0; j < item_count; j++) { if (flags & 1) { unsigned int(32) item_ID; } else { unsigned int(16) item_ID; } unsigned int(8) layer_index; } } }

[0264] In this structure, the layer_index value indicates the 1-based index of the layer within the progressive layers item property that corresponds to the sub-image version and that is associated with the image item specified by the value of item_ID. The value 0 is reserved to indicate that the sub-image version corresponds to the complete content of the image item.

[0265] In some cases, the progressive layer item property may be replaced by several "lsel" layer selection item properties, each of which indicates one of the layers contained in the image item. In this case, the layer_index value indicates the 1-based index of the "lsel" item property associated with the image item corresponding to the subimage and specified by the value of item_ID, which indicates the layer corresponding to the subimage version.

[0266] In these two variants, in step 640 of figure 6, for each image item specified in the list of sub-image versions included in the refinement of the progressive step, it is checked whether the corresponding sub-image version has been loaded. If the specified layer identifier for this image item does not exist and all data for this image item has been loaded, the corresponding sub-image version has been loaded. If the specified layer identifier for this image item exists and data for this layer and all previous layers have been loaded, the corresponding sub-image version has been loaded.

[0267] In some cases, identifiers for items of sub-image versions are assigned in progressive refinement order. The Progressive Version List Item property may be described by the following structure: aligned(8) class ProgressiveVersionListProperty extends ItemFullProperty('pvli', version = 0, flags = 0) { unsigned int(16) step_count; for (i=0; i < step_count; i++) { if (flags & 1) { unsigned int(32) item_ID;} else { unsigned int(16) item_ID; } } }

[0268] In this structure, the item_ID value indicates the identifier of the last image item that corresponds to the sub-image version included in the progressive step refinement.

[0269] [Item list modification example] In a second variant of this third embodiment, an item list variant is provided where for each progressive step, the list of subimages whose versions are included in the refinement of this progressive step is specified in the progressive information stored in the HEIF file.

[0270] Whereas the first variant explicitly indicates the subimage versions included in the refinement of the progressive step, this second variant only indicates the subimages whose versions are included in the refinement of the progressive step, which means that the renderer must determine, for each subimage indicated in relation to a progressive step, which of the subimage versions are included in the refinement of this progressive step.

[0271] Sub-images whose versions are included in a progressive step refinement for an image item in a HEIF file may be indicated with item properties associated with this image item. The progressive item list item properties may be described by the following structure: aligned(8) class ProgressiveItemListProperty extends ItemFullProperty('pili', version = 0, flags = 0) { unsigned int(16) step_count; for (i=0; i < step_count; i++) { unsigned int(16) item_count; for (j=0; j < item_count; j++) { if (flags & 1) { unsigned int(32) item_ID; } else { unsigned int(16) item_ID; } } } }

[0272] In this structure, step_count indicates the number of progressive steps described in the progressive item list item property.

[0273] For each progressive step, the item_count value indicates how many subimages whose versions are included in the refinement of this progressive step.

[0274] For each subimage, the item_ID value indicates the identifier of the image item whose version is included in the refinement of this progressive step.

[0275] In this embodiment, in step 510 of Fig. 5, the progressive specification includes some information for defining the progressive steps, which may include, for example, a list of the sub-image versions included in each progressive step refinement.

[0276] In step 520 of Figure 5, progressive item list item properties are generated based on the progressive step definition. The item properties are written as part of the "meta" box of the HEIF file in step 540 and associated with the image item corresponding to the main image in the HEIF file. For example, the item property structure is written as part of the item property container box "ipco" and associated with the image item corresponding to the main image via the item property association box "ipma".

[0277] Preferably, for each sub-image that is a component of the main image, a list of its versions is explicitly included as part of the "meta" box in the HEIF file. If the versions are based on the coding layer of the sub-image, the set of progressive layer item properties or layer selection item properties "lsel" may be used to list the versions of this sub-image. Otherwise, the "altr" entity group may be used to list all image items that correspond to versions of this sub-image. The highest quality version of the sub-image should be listed first in this "altr" entity group.

[0278] In this embodiment, in step 620 of Fig. 6, progressive information is extracted by parsing the progressive item list item properties. This progressive information is used to generate a list of progressive steps for the main image. Each progressive step is specified by a list of sub-images whose versions are included in the refinements of the progressive step.

[0279] In step 640, it is checked whether new versions are available for all sub-images included in the list associated with the progressive step.

[0280] As a variant, in step 620, a list of versions of each sub-image may be calculated. Preferably, this list of versions is extracted from an explicit description in the HEIF file. If the sub-image version is an image item referenced by an "altr" entity group, the list of all versions of the sub-image is the list of image items contained in this "altr" entity group. If the sub-image version is an image item with an associated progressive layer item property, the list of all versions of the sub-image is the list of coding layers indicated in this progressive layer item property. If the sub-image version is an image item referenced by an "altr" entity group and with an associated progressive layer item property, the list of all versions of the sub-image is a combination of the lists for both cases.

[0281] In some cases, the list of subimage versions is inferred from other structure from the "meta" box of the HEIF file, for example if the subimage version is an image item with an associated thumbnail, then this thumbnail is a version of the subimage.

[0282] Once this list of versions for each subimage is computed, the versions are ordered in progressive refinement order, from lowest to highest quality, and associated with a progressive step. For each subimage, its versions are associated with the progressive step that enumerates this subimage.

[0283] In this variant, step 640 checks whether all sub-image versions involved in the refinement of the progressive step have already been loaded. If the sub-image version corresponds to a coding layer of a multi-layered image item, it is checked whether the data of this coding layer and all previous coding layers have been loaded. Otherwise, it is checked whether the data of the image item has been loaded.

[0284] [Item Reference Variation] This third variant of the third embodiment, hereafter referred to as the item reference variant, is similar to the second variant: instead of storing a list of sub-images whose versions are included in the refinement of the progressive steps in the item properties, this list is stored in an entry of the item reference box.

[0285] An item reference box entry may have a "prli" 4cc. The from_item_ID field of this entry is an identifier of the main image, and the sub-image is specified using the to_item_ID field of this entry. This reference type indicates to the main image the list of image items included in the refinement of the progressive step. There is an item reference box entry of type "prli" for each progressive step, and these entries are ordered according to the progressive step order.

[0286] [Item count variation] In this fourth variant of the third embodiment (hereinafter referred to as the item count variant), for each progressive step, the number of sub-image versions included in the refinement of this progressive step is specified in the progressive information stored in the HEIF file.

[0287] The number of sub-image versions included in a progressive step refinement for an image item in a HEIF file may be indicated in an item property associated with this image item. This progressive item count item property may be described by the following structure: aligned(8) class ProgressiveItemCountProperty extends ItemFullProperty('pcnt', version = 0, flags = 0) { unsigned int(16) step_count; for (i=0; i < step_count; i++) { unsigned int(16) item_count; } }

[0288] In this structure, step_count indicates the number of progressive steps described in the progressive item count item property.

[0289] For each progressive step, the item_count value indicates how many subimages whose versions are included in the refinement of this progressive step.

[0290] In this embodiment, in step 510 of Fig. 5, the progressive specification includes some information to define the progressive steps, which may include, for example, the number of sub-image versions included in each progressive step refinement.

[0291] In step 520 of Figure 5, a progressive item count item property is generated based on the progressive step definition. This item property is written as part of the "meta" box of the HEIF file in step 540 and associated with the image item corresponding to the main image in the HEIF file. For example, this item property structure is written as part of the item property container box "ipco" and associated with the image item corresponding to the main image via the item property association box "ipma".

[0292] Preferably, for each sub-image that makes up the main image, a list of its versions is explicitly included as part of the "meta" box in the HEIF file. If the versions are based on the coding layers of a multi-layered sub-image, the progressive layer item property may be used to list the versions of this multi-layered sub-image. Otherwise, an "altr" entity group may be used to list all image items that correspond to versions of this sub-image. The highest quality version of a sub-image should be listed first in this "altr" entity group.

[0293] The number of subimages for which new versions have been loaded since the previous progressive step is counted in step 640. It is then checked whether this number is greater than or equal to the number associated with the current progressive step.

[0294] As a variant, in step 620, a list of versions of each sub-image may be calculated. Preferably, this list of versions is extracted from an explicit description in the HEIF file. If the sub-image version is an image item referenced by an "altr" entity group, the list of all versions of the sub-image is the list of image items contained in this "altr" entity group. If the sub-image version is an image item with an associated progressive layer item property, the list of all versions of the sub-image is the list of coding layers indicated in this progressive layer item property. If the sub-image version is an image item referenced by an "altr" entity group and with an associated progressive layer item property, the list of all versions of the sub-image is a combination of the lists for both cases.

[0295] In some cases, the list of subimage versions is inferred from other structure in the "meta" box of the HEIF file - for example, if a subimage version is an image item associated with a thumbnail, then this thumbnail is a version of the subimage.

[0296] Once this list of versions for each subimage is computed, the versions are ordered in progressive refinement order, from lowest to highest quality, and associated with a progressive step. For each subimage, its versions are associated with the progressive step that enumerates this subimage.

[0297] In this variant, step 640 checks whether all sub-image versions involved in the refinement of the progressive step have already been loaded. If the sub-image version corresponds to a coding layer of a multi-layered image item, it is checked whether the data of this coding layer and all previous coding layers have been loaded. Otherwise, it is checked whether the data of the image item has been loaded.

[0298] Instead of storing the number of sub-image versions contained in the refinement of a progressive step, in a variant the number of sub-image versions contained in the content of the progressive step is stored. In this variant, in step 640 the total number of loaded sub-image versions is counted and compared with the number associated with the progressive step. In a variant a single item_count value is used for all progressive steps.

[0299] In this variation, this progressive item count item property may be described by the following structure: aligned(8) class ProgressiveItemCountProperty extends ItemFullProperty('pcnt', version = 0, flags = 0) { unsigned int(16) item_count; }

[0300] In another variant, the number of subimages whose versions are included in the refinement of the progressive steps is specified as a percentage of the total number of subimage versions, possibly not taking into account lower quality subimage versions.

[0301] [Item Step Implementation] In a fourth embodiment of the invention (hereinafter referred to as the item step embodiment), progressive steps are indicated using representations of the sub-image versions they contain in their refinement, each sub-image version being indicated as a progressive step of a sub-image.

[0302] Advantageously, this fourth embodiment uses the same mechanism to describe the progressive steps of the derived image and the different versions of the sub-images.

[0303] In a first variant of this embodiment, for each progressive step, the list of sub-image versions included in the refinement of this progressive step is specified in the progressive information stored in the HEIF file.

[0304] The sub-image versions involved in the refinement of a progressive step for an image item in a HEIF file may be indicated in an item property associated with this image item. This progressive derived information item property may be described by the following structure: aligned(8) class ProgressiveDerivedInformationProperty extends ItemFullProperty('prdi', version = 0, flags = 0) { unsigned int(8) step_count; for (i=0; i < step_count; i++) { unsigned int(8) item_count; for (j=0; j < item_count; j++) { unsigned int(16) input_item_index; unsigned int(1) has_steps; unsigned int(7) reserved; if (has_steps) { unsigned int(8) step_index; } } } }

[0305] In this structure, step_count indicates the number of progressive steps described in the progressive derived information item property of the associated derived image item.

[0306] For each progressive step, the item_count value indicates the number of sub-image versions included in the refinement of this progressive step.

[0307] For each sub-image version, the input_item_index value indicates the identifier of the sub-image item that corresponds to this sub-image version, as an index of the sub-image in the list of input images used to construct the derived image item. The list of input images is provided by the "dimg" item reference from the derived item.

[0308] For each sub-image version, the has_steps flag indicates whether the input image item indicated by the input_item_index has one or more versions that are described as progressive steps. If the input image item has an associated progressive step, the has_steps flag has the value 1, otherwise it has the value 0. In some cases, if the input image item has only a single progressive step, i.e. the input image itself, the has_steps flag may have the value 0. Preferably, the index is 0-based, with the value 0 corresponding to the first progressive step of the input image.

[0309] If the has_steps flag has the value 1, the step_index value is the 0-based index of the progressive step in the input image item that corresponds to the version of the subimage used to construct the progressive step of the derived item.

[0310] A progressive step for an input image item may be signaled using an entity group of type "altr". An entity group of type "altr" may be used to group alternative versions of an input image item. In this case, the index associated with a progressive step of an input image item is its position within the entity group of type "altr".

[0311] Preferably, the progressive steps for an input image item are signaled using a more specific entity group type dedicated to this task, for example an entity group of type "prgr" (for progressive entity group), which will be used below in a non-limiting way to describe any entity group used to signal progressive steps of an item as a list of items.

[0312] Within a "prgr" entity group, the input image item versions are preferably listed in order of decreasing quality. Preferably, the ordering of the data corresponding to these input image item versions matches the ordering within the entity group. Any image item included in a "prgr" entity group may in particular be used as a substitute for another image item included in the same "prgr" entity group to achieve progressive rendering of one of the image items included in the entity group. Preferably, during progressive rendering of an image item included in a "prgr" entity group, image items occurring within the entity group are rendered as temporary substitutes. Thus, different image items included in a "prgr" entity group preferably correspond to similar images, but have different quality levels and / or resolutions.

[0313] The input image item for a derived item may be the last item in an entity group of type "prgr" associated with this input image item. This input image item does not have to be the last item in this entity group. For example, the entity group may contain a thumbnail of the input image item, the input image item itself as a low dynamic range image (LDR), and a high dynamic range version (HDR) of the input image. This entity group also signals that the thumbnail and the LDR image item may be used for progressive rendering of the HDR input image. Preferably, if the input image item is not the last item in an entity group of type "prgr", later occurring items in the entity group of type "prgr" are not considered versions of this input image item.

[0314] In this embodiment, in step 510 of Fig. 5, the progressive specification includes some information for defining the progressive steps, which may include, for example, a list of the sub-image versions included in each progressive step refinement.

[0315] In step 520 of Figure 5, progressive derived information item properties are generated based on the progressive step definition. The item properties are written as part of the "meta" box of the HEIF file in step 540 and associated with the image item corresponding to the derived image in the HEIF file. For example, the item property structure is written as part of the item property container box "ipco" and associated with the image item corresponding to the derived image via the item property association box "ipma".

[0316] Furthermore, for each input image item associated with a derived image item that has several versions based on the progressive step definition, an entity group of type "prgr" is generated to group these image item versions in order of high quality.

[0317] In this embodiment, in step 610, the structure extracted from the "meta" box also includes entity groups.

[0318] In step 620, progressive information is extracted from the derived information item properties and from the "prgr" entity group.

[0319] In step 640 it is checked whether all data for the image versions specified in the list of sub-image versions included in the refinement of the progressive step has already been loaded.

[0320] In some cases, the derived image item itself is included in an entity group of type "prgr" to indicate progressive steps that are based on other image items in addition to those that are based on different input image versions. For example, a grid item may be included in an entity group of type "prgr" with a thumbnail and a preview, and an item property of type "prdi" may be associated with the grid item. In this case, the progressive display of the grid starts with the thumbnail, followed by the preview, and then all the progressive steps described in the "prdi" item property.

[0321] In some cases, not all of these progressive steps are used during rendering of the derived item. In some cases, one or more of the initial progressive steps described in the "prdi" item property may not include versions of all sub-image items used by the derived item. In this case, rendering of the derived item may generate a partial result of the derived item. This rendering may replace the sub-image items with blank images or with transparent images without any versions. For example, the first progressive step described in the "prdi" item property may include only the sub-image items corresponding to the top row of the grid item. This partial result may then be the top row of the grid item. In some cases, when an entity group of type "prgr" includes the derived item, the renderer may combine another image item with the partial result to provide a complete rendering of the derived item even with variable quality. In the previous example, the renderer may combine the top row of the grid item with a thumbnail of the grid item to generate a complete image where the top row has good quality and the remaining rows have low quality.

[0322] In some cases, the has_steps flag may not be present. In this case, the value of the step_index field indicates the index of the progressive step plus one. A value of 0 is used to indicate that there is no progressive step associated with the input image. In some cases, a value of 0 is used to indicate that there is no progressive step associated with the input image and that the input image itself should be used as the input image version, or that there are progressive steps associated with the input image and the first progressive step should be used as the input image version.

[0323] In a second variation of this embodiment, the progressive derived item information item property may indicate a layer for the input image item having the following structure: aligned(8) class ProgressiveDerivedInformationProperty extends ItemFullProperty('prdi', version = 0, flags = 0) { unsigned int(8) step_count; for (i=0; i < step_count; i++) { unsigned int(8) item_count; for (j=0; j < item_count; j++) { unsigned int(16) input_item_index; unsigned int(1) has_steps; unsigned int(1) has_layers; unsigned int(6) reserved; if (has_steps) { unsigned int(8) step_index; } if (has_layers) { unsigned int(8) layer_index; } } } }

[0324] In this structure, the has_layers flag indicates whether the input image item indicated by the input_item_index has one or more versions described as layers. If the input image item has one or more versions described as layers, the has_layers flag has a value of 1, otherwise it has a value of 0. In some cases, if the input image item has only one layer corresponding to the entire input image, the has_layers flag may have a value of 0. Preferably, the index is 0-based, with the value 0 corresponding to the first layer of the input image item.

[0325] If the has_layers flag has the value 1, the layer_index value is the 0-based index of the layer or set of layers of the input image item to use to construct the progressive steps of the derived item.

[0326] The layers of an input image item may be described using a "play" progressive layer item property, or multiple "lsel" layer selection item properties, or any similar property that describes the layers that make up the input image item. The layers of an input image item may be described using a "liip" item property, which has the following structure: class LayeredImageIndexingProperty extends ItemFullProperty('liip', version = 0, flags) { field_length = ((flags & 1) + 1) * 16; unsigned int(field_length)[4] layer_size; }

[0327] The "liip" 4cc used here is an example 4cc for hierarchical image indexing properties; other 4ccs that do not conflict with existing 4ccs may be used.

[0328] The "liip" item property describes the layers of the associated image item by giving their size within the image item data.

[0329] The layer_size field indicates the number of bytes corresponding to each layer in the item payload. In some cases, this field may be replaced by an offset in bytes, either in the item payload, or in the box containing the item payload, or in the HEIF file itself.

[0330] In some cases, the encoded size of layer_size may be a constant, e.g. 32 bits, in which case the field_length value is not calculated.

[0331] In some cases, an item property for indicating layers for progressive rendering may have a different name, e.g., a ProgressiveLayerInformationProperty with an associated "plai" 4cc. In some cases, a "plai" progressive layer information item property may indicate only some of the layers of the image item with which it is associated, which are used for progressive rendering of the image item. As a variant, the "plai" item property may use the "prli" 4cc. The "plai" item property will be used in a non-limiting manner below to describe any item property associated with an image item to signal the layers used as progressive steps for this image item. In addition, except where noted, the "plai" item property may be replaced by a more general item property associated with an image item to signal the layers used in the encoding of this image item, such as the "liip" item property.

[0332] The progressive layer information item property may have the following structure: class ProgressiveLayerInformationProperty extends ItemFullProperty('plai', version = 0, flags) { field_length = ((flags & 1) + 1) * 16; unsigned int(8) step_count; for (i=0; i < step_count; i++) { unsigned int(field_length) step_size; } }

[0333] The step_count field indicates the number of progressive steps described by the progressive layer information property.

[0334] The step_size field indicates the number of bytes that correspond to each progressive step in the item payload.

[0335] In some cases, the size of the last progressive step may be omitted or set to a zero value to reduce the size of the progressive layer information item property.

[0336] In some cases, image items that have an associated "plai" item property may be included in the "prgr" entity group.

[0337] Preferably, an image item does not have both an associated 'prdi' item property and an associated 'plai' item property, where these item properties correspond to different types of image items.

[0338] The entity group of type "prgr" will be used below in a non-limiting manner to describe any entity group used to signal progressive steps of items as a list of items.

[0339] In this variant, in step 520 of FIG. 5, a progressive layer information item property "plai" is generated for each input image item for which several layers are used as progressive steps.

[0340] In this variant, in step 640 of Fig. 6, for each image item specified in the list of sub-image versions included in the refinement of the progressive step, it is checked whether the corresponding sub-image version has been loaded. If this image item does not have a layer identifier specified, e.g. via the "plai" item property, and all data for this image item has been loaded, then the corresponding sub-image version has been loaded. If this image item has a layer identifier specified and data for this layer and all previous layers have been loaded, then the corresponding sub-image version has been loaded.

[0341] In some cases, the has_layers flag may not be present. In this case, the value of the layer_index field indicates the layer's index plus 1. A value of 0 is used to indicate that there are no layers associated with the input image item. In some cases, a value of 0 is used to indicate either that there are no layers associated with the input image item and that the entire input image item should be used as the input image item version, or that there are layers associated with the input image item and that the first layer should be used as the input image item version.

[0342] Optionally, the layer_index field may be replaced by an offset value in bytes corresponding to the location of the end of the data corresponding to the indicated layer.

[0343] In some cases, the layer_index field may be replaced by the sizes of all layers used to generate the indicated version of the input item. In these two latter variations, when the has_layers flag is not present, a value of 0 may be used to indicate that the input image has no layers and that the input image is the entire input image.

[0344] In this second variation of this embodiment, the progressive derived item information item property may be simplified by removing the has_steps and has_layers flags, resulting in the following structure: aligned(8) class ProgressiveDerivedInformationProperty extends ItemFullProperty('prdi', version = 0, flags = 0) { unsigned int(8) step_count; for (i=0; i < step_count; i++) { unsigned int(8) item_count; for (j=0; j < item_count; j++) { unsigned int(16) input_item_index; unsigned int(8) step_index; unsigned int(8) layer_index; } } }

[0345] This structure describes the progressive rendering steps associated with a derived image item. Each progressive step may be seen as specifying an alternative image to use in place of the input image used by the derived image item. As seen before, the alternative image is a version of the input image and may be a blank image, a lower quality version of the input image, or the input image itself.

[0346] In this structure, each progressive rendering step is described as a difference from the previous step. This description lists the alternative images, or versions of the input image, that are used to reconstruct the derived image item. First, before the first progressive rendering step, all alternative images of the input image correspond to an empty image. Each progressive rendering step adds new images as alternative images and / or updates existing alternative images with other alternative images.

[0347] This structure may describe two different cases of alternative images, or versions of an input image. First, an input image may be replaced by a lower quality image associated with it by being included in the same "prgr" entity group. Second, an input image may be replaced by a lower quality version that corresponds to some layers that make up this input image and is described through associated "plai" item properties. In some cases, this input image may be included in a "prgr" entity group and have associated "plai" item properties.

[0348] Using this structure, if the input image is not contained in the "prgr" entity group, the step_index field is set to the value 0. If the input image is contained in the "prgr" entity group, the step_index value is the 0-based index within the "prgr" entity group of the image item to use as a replacement for this input image.

[0349] Using this structure, if an input image does not have an associated "plai" item property, the layer_index field is set to the value 0. If the input image has an associated "plai" item property, the layer_index value is a 0-based index within the associated "plai" item property of the set of layers to use as a substitute for this input image.

[0350] Using this structure, if an input image is neither contained in the "prgr" entity group nor associated with the "plai" item property, then both the step_index and layer_index fields will have a value of 0.

[0351] Using this structure, if an input image is contained in a "prgr" entity group and associated with a "plai" item property, the information in the "plai" item property indicated by the layer_index field is considered only if the step_index corresponds to the position of the input image in the "prgr" entity group, otherwise layer_index is set to 0 and the information in the "plai" item property is ignored.

[0352] Optionally, in this structure, the input_item_index may have a different name, for example item_index.

[0353] In some cases, in either the first or second variant of this embodiment, or elsewhere where applicable, the meaning of the step_index field may be extended to support input images that are themselves derived image items.

[0354] In this example variant extension, an input image may be replaced by its own progressive reconstruction step as described in the "prdi" item property associated with this input image. In this example variant extension, if the input image is contained in a "prgr" entity group, the step_index value is a zero-based index in the "prgr" entity group of the image item to use as a replacement for this input image. If the input image has a "prdi" item property associated with it, the step_index value is a zero-based index in the "prdi" item property of the progressive step to use to construct the replacement for this input image. If the input image is not contained in a "prgr" entity group and has no "prdi" item property associated with it, the step_index value is 0.

[0355] If the input image is contained in a "prgr" entity group and has a "prdi" item property associated with it, the step_index value is used to index either the "prgr" entity group or the "prdi" item property. Values ​​from 0 to the position of the input image minus one in the "prgr" entity group are used to index items in the "prgr" entity group. Values ​​starting with the input image's position in the "prgi" entity group are used to index progressive rendering steps in the "prdi" item property.

[0356] In some cases, a new field derived_index may be introduced in the definition of the progressive derived item information item information item property to indicate a progressive rendering step and may be used as a substitute for an input image that is a derived image item. Using this new field, if there is no "prdi" item property associated with the input image, the derived_index field is set to the value 0. If the input image has an associated "prdi" item property, the derived_index value is a 0-based index in the associated "prdi" item property of the progressive rendering step and is used as a substitute for this input image. This combination of use of the derived_index field and the step_index field may be similar to the combination of use of the layer_index field and the step_index field. Other possible variants described for the layer_index field may also be applied to the derived_index field. Preferably, the derived_index field and the layer_index field are not used with non-zero values, since they correspond to different types of input items.

[0357] In some cases, as described in the third variant, the step_index field may combine the use of these three different fields: step_index, derived_index, and layer_index.

[0358] In some cases, in either the first or second variant of this embodiment, or elsewhere where applicable, the meaning of the layer_index field may be extended to support the case of an input image that is a derived image having a single input image with several layers.

[0359] In this variant extension example, if the input image is a derived image item and has a single input image with an associated "plai" item property, then layer_index is the 0-based index within the associated "plai" item property of the set of layers to use to reconstruct an alternative to the input image.

[0360] In some cases, this use of the layer_index field may be extended to reference from the input image to the image item using a chain of references through the derived items. For example, the layer_index field may be used as a 0-based index within the "plai" item property associated with an image item that is the single input image of a derived image item, which is itself the single input image of the derived image item that is the input image indicated in the progressive derived item information item property by the input_item_index field.

[0361] In some cases, the meaning of the step_index or derived_index field may be extended to a reference from an input image to an image item, either using a single derived item or a series of references through derived items. For example, this derived_index field may be used as a 0-based index in a "prdi" item property associated with a grid item that is a single input image of a derived image item that is an input image indicated in the progressive derived item information item property by the input_item_index field.

[0362] In some cases, the meaning of the layer_index field may be extended to support the case of an input image that is a derived image having several input images with one or more layers.

[0363] In this variant extension example, if the input image is a derived image item and has several input images, at least one of these input images has an associated "plai" item property, then layer_index is used as a zero-based index inside each associated "plai" item property of each set of layers to use to reconstruct the respective alternative of the input image for all input images that have an associated "plai" item property. If layer_index is greater than or equal to the number of layer sets described in the associated "plai" item property of the input image, then the last layer set described by the associated "plai" item property is used. If the input image does not have an associated "plai" item property, then the input image itself is used when reconstructing the derived image item.

[0364] In some cases, in this variant extension example, the layer_index value may be used differently to select the set of layers to use for the alternative reconstruction of each input image. For example, if some input images have two sets of layers and other input images have three sets of layers, a layer_index value of 0 may be used to indicate the first set of layers for all input images. A value of 1 may be used to indicate the first set of layers for input images with two sets of layers and the second set of layers for input images with three sets of layers. A value of 2 may be used to indicate the last set of layers for all input images.

[0365] In some cases, in this variant extension example, the meaning of the layer_index field may be extended to a reference from the input image to several image items using a series of references through derived items. In some cases, the number of derived items linking the input image to different image items may vary depending on the image item.

[0366] In some cases, the meaning of the step_index or derived_index field may be extended to support the case of an input image that is a derived image with several input images as well. In some cases, the meaning of the step_index or derived_index field may be extended to a reference from the input image to several image items using a series of references through the derived items. In some cases, the number of derived items linking an input image to different image items may vary depending on the image item.

[0367] In some cases, two or more of the layer_index, step_index and derived_index fields may be used together to support the case of an input image that is a derived image, having several input images with several layers included in the "prgr" entity group and / or is a derived image item with an associated "prdi" item property. Depending on the progressive rendering information available for each of several input images, the corresponding field is used to determine the progressive step reconfiguration of the input image. For example, the input image may be an overlay with two input images. The first input image of the overlay may be an image item with two sets of layers indicated by an associated "plai" item property. The rendering step to use for this first input image is based on the value of the layer_index field. The second input image of the overlay may be an image item included in the thumbnail in the "prgr" entity group. The rendering step to use for this second input image is based on the value of the step_index field.

[0368] In a third variation of this embodiment, an indication of the version to use for an input item is indicated using a single field having the following structure: aligned(8) class ProgressiveDerivedInformationProperty extends ItemFullProperty('prdi', version = 0, flags = 0) { unsigned int(8) step_count; for (i=0; i < step_count; i++) { unsigned int(8) item_count; for (j=0; j < item_count; j++) { unsigned int(16) item_index; unsigned int(8) step_index; } } }

[0369] In this variant, the step_index value is the index of the progressive step corresponding to the version of the input image item used to build the progressive step of the derived item. This progressive step may be signaled by an entity group of type "prgr", or by the "plai" item property, or both. The item_index is similar to the input_item_index described in the previous variant.

[0370] The complete list of progressive steps associated with a target item to be displayed may be composed of several stages. First, if the target item is included in an entity group of type "prgr", then each item included in this entity group is added to the list as a progressive step. Preferably, items listed after the target item in an entity group are not included in this list. If the target item is not included in such an entity group, then the target item is added to the list.

[0371] Second, if the target item has an associated "plai" item property, then this target item is replaced in the list by the list of progressive steps that correspond to the layer indicated in the item property.

[0372] For example, for an image item, there is an entity group of type "prgr" that contains a thumbnail and an image item, and there is a "plai" item property associated with the image item that signals two layers. Then the progressive steps for this image item are the thumbnail, the first layer of the image item, and the second layer of the image item.

[0373] Third, if the target item is a derived item and has an associated "prdi" item property, then this target item is replaced in the list by the list of progressive steps that correspond to the progressive steps indicated in the "prdi" item property.

[0374] For example, for a grid item, there is an entity group of type "prgr" that contains the thumbnail and the grid item, and there is a "prdi" item property associated with the two steps of grid item signaling. Then the progressive steps for this grid item are the thumbnail, the first step of the grid item, and the second step of the grid item.

[0375] In some cases, if an item added to the list that is different from the target item has an associated "plai" item property, this item may be replaced in the list by a list of progressive steps that correspond to the layer indicated in the item property.

[0376] For example, for an image item, there is an entity group of type "prgr" that contains the preview and the image item, and there is a "plai" item property associated with the preview that signals two layers. Then the progressive steps for this image item are the first layer of the preview, the second layer of the preview, and the image item.

[0377] In some cases, if an item added to the list that differs from the target item is a derived item and has an associated item property of type "prdi", this item may be replaced in the list by a list of progressive steps that correspond to the progressive steps indicated in the "prdi" item property.

[0378] In some cases, if the target item is a derived image item that does not have an associated "prdi" item property and at least one of its input image items has an associated "prdi" item property, has an associated "pri" item property, and / or is included in an entity group of type "prgr", then the target item is replaced in the list by a list of progressive steps corresponding to the generation of the derived image using the progressive steps for the input image item indicated in the "prdi" item property, or corresponding to the layers indicated in the "plai" item property and / or signaled by the entity group of type "prgr". The list progressive steps of the input image items may be constructed as described herein.

[0379] In some cases, this mechanism for obtaining progressive steps for a derived image item is only used when the derived image item has a single input image item.

[0380] Preferably, derived image items that have several input image items have an associated 'prdi' item property to make them independent of this mechanism.

[0381] For example, for a derived item that is a rotation of a graphical item having two layers, the progressive steps are a rotation of the first layer of the graphical item and a rotation of the second layer of the graphical item.

[0382] As another example, for a derived item that is a rotation of an image item having two layers, the image item is included in an entity group of type "prgr" that has a thumbnail, and the progressive steps for the derived item are a rotation of the thumbnail, a rotation of the first layer of the image item, and a rotation of the second layer of the image item.

[0383] In some cases, there may be several successive references from a derived item to an input image item.

[0384] For example, for a derived item that is a crop of another derived item that is itself a rotation of an image item having two layers, the progressive steps are a crop of the rotation of the first layer of the image item, and a crop of the rotation of the second layer of the image item.

[0385] In some cases, if the item added to the list that differs from the target item is a derived item without an associated "prdi" item property, the same process may be used to replace the item by the list of progressive steps.

[0386] In a fourth variant, a single item property combines the information contained in the "prgr" entity group, the "prdi" item property and the "plai" item property. This progressive information item property may be described by the following structure: aligned(8) class ProgressiveInformationProperty extends ItemFullProperty('prif', version = 0, flags = 0) { / / Progressive step count unsigned int(8) step_count; for (i=0; i < step_count; i++) { unsigned int(32) step_item_id; unsigned int(8) layer_index; unsigned int(8) item_count; for (j=0; j < item_count; j++) { unsigned int(16) item_index; unsigned int(8) step_index; } } }

[0387] In this structure, step_count indicates the number of progressive steps described in the progressive information item property, which corresponds to the number of reconstruction steps of the image item associated with this property.

[0388] For each progressive step, step_item_id indicates the identifier of the graphical item corresponding to the step, i.e. item_ID.

[0389] For each progressive step, layer_index indicates the index of the layer that corresponds to the progressive step of the visual item identified by the step_item_id field. If the visual item has no layers, the value of layer_index is 0.

[0390] For each progressive step, if the identified image item is a derived item, the item_count value indicates the number of sub-image versions included in the refinement of this derived item's progressive step. If the identified image item is not a derived item, the value of item_count is 0.

[0391] The item_index and step_index fields are similar to those described in the "prdi" item property of the previous variant. However, in this variant, the complete list of progressive steps may be reconstructed directly from the "prif" item property associated with the image item identified by the item_index field. Thus, the step_index value may be an index into the list of progressive steps defined by the "prif" item property associated with the identified image item.

[0392] In a fifth variant, this fourth embodiment is combined with the second embodiment, and for each refinement step the position of the last byte corresponding to this refinement step is identified.

[0393] These positions may be indicated for each progressive step using an item property associated with the image item. This progressive step end property may be described by the following structure: aligned(8) class ProgressiveInformationProperty extends ItemFullProperty('prif', version = 0, flags) { unsigned int(1) layer_based; unsigned int(7) reserved; unsigned int(8) num_steps; for (i=0; i < num_steps; i++) { unsigned int(32) last_byte_position; } }

[0394] In this structure, the layer_based value is a flag that indicates whether the progressive reconstruction step of the associated item is based on the layer or version of the input image item of the derived image item. If the value of the layer_based field is 1, the progressive reconstruction step of the associated item is based on the coding layer of this image item. Otherwise, the associated item is a derived image item and its progressive reconstruction step is based on using different versions of its input image items.

[0395] The num_steps value indicates the number of reconstruction steps for the associated image item. The last_byte_position indicates the last byte in the HEIF file for the reconstruction step.

[0396] When the layer_based field is 1, this last_byte_position indicates the last byte of the layer of the associated pictorial item. When the layer_based field is 0, this last_byte_position indicates the last byte of the version of the input image for the associated pictorial item that is a derived pictorial item.

[0397] 9 shows an example using a "prgr" entity group 900 to illustrate progressive rendering of an image item 930. The entity group includes an image item, a thumbnail 910, and a preview 920. The thumbnail and / or the preview may be used for progressive rendering of the image item.

[0398] Figure 10 shows an example describing the progressive rendering of an image item 1020 that has been encoded with several layers 1030, 1040, and 1050. The image item 1020 is included in a "prgr" entity group 1000 along with a thumbnail 1010 to indicate that the thumbnail is an alternative version of the image item that may be used for progressive rendering of the image item.

[0399] In addition, a "plai" item property 1060 is associated with the image item 1020, indicating that two progressive steps for the image item may be used for progressive rendering. The first progressive step uses layer 0, 1030, and layer 1, 1040, of the image item. The second progressive step additionally uses layer 2, 1050, of the image item.

[0400] In this example, the main image corresponds to a reconstructed image from layer 2 (1050) of image item 1020 (possibly depending on a previous layer if there are some code dependencies between layers), and layers 1030, 1040, and 1050 correspond to sub-images of image item 1020. In this example, each layer or sub-image version is a sub-image itself.

[0401] 11 shows an example illustrating progressive rendering of a derived image item 1120 that is a grid. The derived image item is included in a "prgr" entity group 1100 along with a thumbnail 1110 to indicate that the thumbnail is an alternative version of the image item 1120 that may be used in the progressive rendering of the image item 1120.

[0402] In this example, the main image corresponds to the derived image item 1120 .

[0403] The grid image item 1120 is composed of four input image items arranged in a 2x2 grid. The input image items are C0 1132, C1 1142, C2 1152, and C3 1162. Each input image item also has lower quality versions, L0-L3, which are associated with the respective input image items by being included in the "prgr" entity group. Thus, input image item C0 1132 and its lower quality version L0 1131 are included in the "prgr" entity group 1130. Similarly, input image item C3 1162 and its lower quality version L3 1161 are included in the "prgr" entity group 1160.

[0404] In addition, a "prdi" item property 1125 is associated with the image item 1120, indicating that three progressive steps for the image item may be used for progressive rendering. The first progressive step uses lower quality versions L0, L1, L2, and L3 in place of the input image items C0, C1, C2, and C3. The second progressive step replaces L0 and L2 with the corresponding image items C0 and C2, resulting in full quality for the left column of the grid and lower quality for the right column. The third progressive step replaces L1 and L3 with the corresponding image items C1 and C3, resulting in full quality for the entire grid.

[0405] Figure 12 expands on the example shown by Figure 10 by adding a derived image item 1270, using the layered image 1020 as its input image item. In this example, the main image corresponds to the derived image item 1270. This derived image item 1270 applies a rotation to its input image item, as indicated by an associated "irot" item property 1280.

[0406] In this example, progressive rendering of the derived image item 1270 may be achieved by first rendering a thumbnail that is in the same "prgr" entity group as the input image item and applying to it the rotation indicated by the "irot" item property. A second rendering step may then be achieved by rendering layers 0 and 1 of the input image item, as indicated by the "plai" item property 1060, and applying the rotation indicated by the "irot" item property to these layers. The final rendering step may be a rendering of the complete input image item with the rotation applied.

[0407] A derived image item 1270 may be included alone in the "prgr" entity group to indicate that it supports progressive rendering, and this indication may also be achieved by certain item properties associated with it, such as the "prog" item property described below.

[0408] In some cases, in a variation of this example, the thumbnail 1010 may not be associated with the layered image item 1020 through the "prgr" entity group 1000, but may be directly associated with the derived image item 1270 through the "prgr" entity group. In this variation, the thumbnail 1010 would also have an "irot" property associated with it or would already contain a rotated image. In this variation, progressive rendering of the derived image item 1270 may be achieved by first rendering the thumbnail without applying the rotation indicated by the "irot" item property 1280 associated with the derived image item. The second and third rendering steps may then be achieved by using the layers of the input image items as described above.

[0409] In some cases, in another variation of this example, both the layered image item 1020 and the derived item 1270 are included in a "prgr" entity group that associates each of them with a different thumbnail. Preferably, in this variation, for progressive rendering of the derived item 1270, the thumbnail associated with it through the "prgr" entity group is used, and the thumbnail 1010 associated with the layered image item 1020 through the "prgr" entity group 1000 is ignored. In some cases, the thumbnail 1010 may be used instead of the thumbnail associated with the derived item 1270, for example, if this thumbnail 1010 is available before the thumbnail associated with the derived item 1270. In some cases, both thumbnails may be used for different progressive rendering steps of the derived item 1270, for example, if these thumbnails have different quality levels.

[0410] All variants described in the previous variant can also be used in this variant, where applicable.

[0411] [Combination of embodiments] In some cases, two or more embodiments of the present invention can be combined. In the combination of embodiments, structures describing progressive information related to different embodiments can be combined. These structures may be kept separate.

[0412] In particular, it may be advantageous to combine an embodiment of the refinement pattern with one of the other embodiments, in which the progressive refinement pattern indicates the progressive refinement method used in the HEIF file, while the offset or item instructions specify the refinement of each progressive step.

[0413] For example, an item property for specifying both a progressive refinement pattern and a location for each progressive step may have the following structure: aligned(8) class ProgressivePatternLocationProperty extends ItemFullProperty('pplo', version = 0, flags = 0) { unsigned int(8) pattern_type; unsigned int(1) reverse_flags; unsigned int(1) bidirectional_flag; unsigned int(1) parameter_flags; unsigned int(5) reserved; unsigned int(8) scale; if (parameter_flag == 1) { unsigned int (16) parameter; } unsigned int(16) step_count; for (i=0; i < step_count; i++) { if (flags & 1) { unsigned int(32) item_index; } else { unsigned int(16) item_index; } unsigned int(16) extent_index; } }

[0414] In some cases, some of the embodiments of the present invention can be combined with other embodiments of the present invention.

[0415] [Implementation Variations] In an alternative embodiment where the sub-images are indicated via their respective identifiers, instead of specifying an identifier for each sub-image, an index of the sub-image within the item reference entry that links the main image to the sub-image may be used.

[0416] In some cases, instead of specifying several progressive steps in a single item property, each progressive step may be specified in its own item property, and these item properties may be ordered according to the progressive step order.

[0417] In some cases, different item properties for describing the progressive information are used for each type of main image: for example, there may be a specific item property for describing progressive information linked to grid items, and a different item property for describing progressive information linked to overlay items, each with its own list of pattern_type values.

[0418] In some cases, the progressive information may be included in a new item, for example a "prog" type progressive item. This progressive item may have the same structure as one of the item properties described in this invention. This progressive item may be associated with an item reference, for example the main image item that applies to the "prog" 4cc. This item reference may be from the progressive item to the image item or from the image item to the progressive item.

[0419] In some cases, a progressive item may specify a single progressive step. In this case, the item reference may link a main image item to a progressive item that describes the progressive step for this main image. The progressive items may be ordered within the item reference entry according to the order of their respective progressive steps.

[0420] In some cases, the progressive information may be included in a new box, for example a "prog" 4cc progressive box. This new box may be placed in the "meta" box of the HEIF file. This progressive box may have the same structure as one of the item properties described in this invention. In addition, this progressive box may include an indication of the main image to which it applies, for example by specifying an identifier of the main image item. There may be as many instances of the progressive box as there are image items in the HEIF file that support progressive rendering.

[0421] In some cases, a progressive box may specify a single progressive step, in which case the progressive box may contain the progressive step number.

[0422] In some cases, the progressive step description may include a progressive step number.

[0423] In some cases, different sizes may be used for some of the various fields described herein.

[0424] In some cases, variable sizes may be used for some of the various fields described herein. In some cases, the size of a field may be based on the "flags" parameter of the box that contains the field, the "version" of this box, or a flags field that is included in this box. In some cases, the sizes of several fields may depend on the same representation.

[0425] In some cases, the progressive information may indicate the content of a progressive step instead of a refinement of the progressive step.

[0426] In some cases, the progressive steps of the main image are organized as a directed acyclic graph, rather than as a list. This means that a progressive step may have two or more subsequent progressive steps, and a progressive step may have two or more preceding progressive steps. This also means that a progressive step should not follow or precede itself, directly or indirectly. Such an organization is advantageous when HEIF files are not intended to be loaded sequentially, or when the main image content spans several files that may be loaded in an undetermined order.

[0427] [Progressive Step Variation] Preferably, the order of use of the progressive steps corresponds to the order of these progressive steps.

[0428] In some cases, some progressive steps may be empty or missing.

[0429] In some cases, the progressive information may define progressive steps for the progressive loading of lower quality versions of the sub-images. In the example of Fig. 4f, using a center-to-border progressive refinement pattern, a first progressive step may include only the lower quality version of the center sub-image "l5", then a second progressive step refinement may include lower quality versions of the four sub-images "l2", "l4", "l6" and "l8".

[0430] In some cases, after the last progressive step, some high quality versions of some sub-images may be missing.

[0431] In some cases, one or more versions of the subimage may be missing.

[0432] [File with multiple contents] In some cases, a HEIF file may contain several image items with associated progressive information. The data for these image items may be interleaved or sequentially ordered.

[0433] For example, a HEIF file may include a first image item with a progressive refinement pattern as described by FIG. 4f and a second image item with a progressive refinement pattern as described by FIG. 4g. The different sub-image versions for these image items may be organized as follows: First, lower quality versions of all of the sub-images of the first image item are stored, followed by lower quality versions of the sub-images of the second image item. Second, high quality versions of the first row of the first item are stored, followed by high quality versions of the first and second columns of the second item, etc.

[0434] For the same example, different sub-image versions of two image items may be organized differently, with all the sub-image versions of the first image item being stored, followed by all the sub-image versions of the second image item.

[0435] In some cases, the interleaved or sequential organization of a HEIF file for image items that support progressive rendering may be indicated. This indication may be achieved by a brand within the HEIF file. A brand may be present to indicate a sequential organization, e.g., "spro". A brand may be present to indicate an interleaved organization, e.g., "ipro".

[0436] In some cases, the item properties may indicate, for each image item that supports progressive rendering, whether it is stored sequentially or in an interleaved manner.

[0437] In some cases, other image items or other non-image items that do not support progressive rendering may be included in a HEIF file that contains one or more image items that do support progressive rendering, and in some cases, these other image items or non-image items are stored after all progressive image items.

[0438] In some cases, the progressive information associated with a visual item indicates where metadata associated with this item is stored relative to the progressive content of this visual item. For example, the progressive information may indicate that the metadata is stored after all the content of the visual item. As another example, the progressive information may indicate that the metadata is stored before a high level version of a sub-image of the visual item.

[0439] In some cases, a component of an image item may itself be composed of several sub-images: for example, an overlay item may use a grid item as one of its components. In such cases, the progressive information associated with the composited sub-image defines different versions of this sub-image.

[0440] [Entity Group Properties] In some cases, an item property describing a progressive step may be associated with an entity group (e.g., a "prgr" or "altr" entity group) that indicates that entities (e.g., items) declared within this group may be used for progressive rendering. For example, in Figure 10, the "plai" item property 1060 may be associated with the "prgr" entity group 1000 (instead of image 102, as shown). As another example, in Figure 11, the "prdi" item property 1125 may be associated with the "prgr" entity group 1100 (instead of grid image 1120, as shown).

[0441] In some cases, the item property describing the progressive step associated with an entity group signals the progressive step of the highest quality item contained in the entity group. In the case of the "prgr" entity group, this is the last item in this entity group. For example, in FIG. 11, if the "prdi" item property 1125 is associated with the "prgr" entity group 1100, it may describe the progressive step for the last item contained in this "prgr" entity group, i.e., the grid image 1120.

[0442] In some cases, an item property describing a progressive step associated with an entity group signals the progressive step of any items contained in the entity group to which it is applicable. For example, as described in the first embodiment, if the item property is a "ppat" item property, it may apply to any derived items contained in the associated entity group.

[0443] In some cases, an item property describing a progressive step associated with an entity group may indicate which item of the entity group it applies to. This indication may be the item_ID of the item it applies to, or the index of the item in the list of entities referenced by the entity group (i.e. the index of the entity_id that is equal to the item_ID of the item in the list of entity_ids in the entity group), or any other indication.

[0444] In some cases, an item property describing a progressive step associated with an entity group may describe the progressive steps of several items contained in this entity group. This property may have the following structure, similar to the "prif" item property of the fourth variant of the fourth embodiment: aligned(8) class ProgressiveGroupInformationProperty extends ItemFullProperty('prgi', version = 0, flags = 0) { / / Progressive step count unsigned int(8) step_count; for (i=0; i < step_count; i++) { unsigned int(8) step_item_index; unsigned int(8) layer_index; unsigned int(8) item_count; for (j=0; j < item_count; j++) { unsigned int(16) item_index; unsigned int(8) step_index; } } }

[0445] In this structure, step_count indicates the number of progressive steps described in the progressive information item property. This corresponds to the number of reconstruction steps of the entity group associated with this property (e.g., a "prgr", "altr" entity group, or other type of entity group). For each progressive step, step_item_index indicates the index of the image item in the entity group that corresponds to that step. This index may be 0-based.

[0446] For each progressive step, layer_index indicates the index of the layer that corresponds to the progressive step of the visual item identified by the step_item_index field. If the visual item has no layers, the value of layer_index is 0.

[0447] For each progressive step, if the identified image item is a derived item, the item_count value indicates the number of sub-image versions included in the refinement of this derived item's progressive step. If the identified image item is not a derived item, the value of item_count is 0. If the identified image item is a derived item, it has an associated "prif" item property.

[0448] The item_index and step_index fields are similar to those described for the "prdi" item property in the previous variant. The complete list of progressive steps may be reconstructed directly from the "prif" item property associated with the input image item identified by the item_index field. Thus, the step_index value may be an index into the list of progressive steps defined by the "prif" item property associated with the identified image item.

[0449] In this structure, the complete list of progressive steps for an input image used by a derived item contained in the associated 'prgr' entity group and identified by the item_index field may be reconstructed directly from the 'prgi' item property associated with the 'prgr' entity group that contains the input image.

[0450] In this structure, some items contained in an entity group may not be described if they correspond to a single progressive step, for example a thumbnail may be used during progressive rendering but may not have an associated progressive step described in this item property.

[0451] In some cases, this item property may list the progressive steps for each item contained in the entity group, in the following structure: aligned(8) class ProgressiveGroupItemInformationProperty extends ItemFullProperty('pgii', version = 0, flags = 0) { / / Loop over the entities in the group for (i=0; i < entity_count; i++) { / / Progressive step count unsigned int(8) step_count; for (j=0; j < step_count; j++) { unsigned int(8) layer_index; unsigned int(16) item_count; for (k=0; k < item_count; k++) { unsigned int(16) item_index; unsigned int(8) step_index; } } } }

[0452] In this structure, the layer_index, item_count, item_index, and step_index fields are similar to the "prgi" property. The entity_count value corresponds to the number of entities contained in the associated "prgr" entity group. In some cases, this value may be explicitly specified in a field of the "pgii" item property.

[0453] In this structure, a progressive step is described in order for each item contained in the associated "prgr" entity group.

[0454] In this structure, the step_count field indicates the number of progressive steps described in the progressive group item information property of the item contained in the associated "prgr" entity group being described (i.e., the item corresponding to the i-th entity in the entity group).

[0455] In this structure, the complete list of progressive steps for an input image used by the derived item contained in the associated 'prgr' entity group and identified by the item_index field may be reconstructed directly from the 'pgii' item property associated with the 'prgr' entity group that contains the input image.

[0456] If the item being described has a single progressive step that corresponds to the rendering of the item itself, the step_count value may be set to 0.

[0457] In some cases, item properties associated with entity groups and item properties associated with image items may be used together in the same HEIF file. For example, in FIG. 11, a "prdi" item property 1125 may be associated with the "prgr" entity group 1100, and another "prdi" item property (not shown) may be associated with the C0 image item 1132. In some cases, the item property used to describe the progressive steps associated with the entity group may be identical to the item property used to describe the progressive steps associated with the image item. For example, they may both be "prdi" item properties. In some cases, these item properties may be of different types and have different content. For example, the item property used to describe the progressive steps associated with the entity group may be a "prgi" item property, while the item property used to describe the progressive steps associated with the image item may be a "prdi" item property.

[0458] All these possibilities for associating item properties describing progressive steps with entity groups can also be applied to entity groups other than the "prgr" entity group, for example the "altr" entity group. Other item properties describing progressive steps may also be used.

[0459] [Multi-Progressive Rendering] In some cases, several progressive renderings may be described for the same image item. For example, in FIG. 9, the second "prgr" entity group may include only thumbnail 910 and image 930. Preferably, the ordering of image data in the HEIF file allows for all the described progressive renderings. For example, in FIG. 9, if the second "prgr" entity group includes thumbnail 910, second preview, and image 930, both the data corresponding to preview 920 and the data corresponding to second preview should be stored after the data corresponding to thumbnail 910 and before the data corresponding to image 930.

[0460] An image item that is intended to be rendered may have several progressive rendering descriptions described for it. For example, an image item may be included in several "prgr" entity groups. As another example, a derived image item may be associated with several "prdi" item properties. In such a case, the renderer may select one of these progressive rendering descriptions (e.g., a "prgr" group, a "prgr" group with a progressive step property, an item with "prdi") to realize the progressive rendering of the image item.

[0461] In some cases, the selection of which progressive rendering description to use may be based on the characteristics of the renderer: for example, a renderer configured to conserve CPU resources may select the progressive rendering description with the fewest rendering steps.

[0462] In some cases, the selection of which progressive rendering description to use may be based on the location of this description in the HEIF file. For example, if an image item is included in several "prgr" entity groups, the renderer may select the first "prgr" entity group. The renderer may select the last "prgr" entity group. As another example, the renderer may select the "prgr" entity group with the lowest group_id. The renderer may also select the "prgr" entity group with the highest group_id.

[0463] In some cases, the different progressive rendering descriptions may contain information to assist the renderer in selecting one of them. For example, a progressive rendering description may contain information about the intended download speed when using this progressive rendering description. This information may be stored in an item property for a progressive rendering description based on an item property, such as the "prdi" item property or other item properties mentioned above. This information may be stored in an item property associated with an entity group, for example, for a progressive rendering description based on an entity group, such as the "prgr" entity group.

[0464] In some cases, different progressive rendering descriptions may include preference related information. For example, a progressive rendering description may include a flag indicating that it is a preferred progressive rendering description. This information may be stored, for example, inside the item properties for progressive rendering descriptions based on item properties such as the "prdi" item property or any other item property mentioned above, or in a progressive download information box at the top level of the file when the primary item has a progressive rendering description, possibly with the initial_delay parameter set to zero so that the renderer starts rendering the selected version of the image without delay. This information may be stored, for example, in item properties associated with the entity group for progressive rendering descriptions based on entity groups such as the "prgr" entity group. This information may also be stored directly in the entity group for progressive rendering descriptions, such as, for example, using different grouping types or using different version or flag values. For example, the "prgr" entity group may be used to indicate a preferred progressive rendering, and the "prg2" entity group may be used to indicate an alternative progressive rendering. As another example, several alternative 'prgr' entity groups may be declared within an 'altr' entity group.

[0465] In some cases, an image item may correspond to a main image of a progressive rendering description and a rendering step of another progressive rendering description. In this case, to progressively render an image item, a renderer may select a progressive rendering description in which the image item is the main image of the description. For example, if a first "prgr" entity group contains a thumbnail and an LDR (low dynamic range, as opposed to high dynamic range) image and a second "prgr" entity group contains another thumbnail, an LDR image, and an HDR image, then when progressively rendering the LDR image, the renderer may select the first "prgr" entity group.

[0466] In some cases, a renderer may combine progressive rendering steps corresponding to different progressive rendering descriptions. For example, if a first "prgr" entity group includes a first thumbnail, a first preview and an image item, and a second "prgr" entity group includes a second thumbnail, a second preview and an image item, the renderer may progressively render the image item by first displaying the first thumbnail, then the second preview, and finally the image item.

[0467] In some cases, before selecting progressive rendering descriptions, the renderer may check whether these descriptions are compatible with the ordering of image data in the HEIF file. The renderer may select the progressive rendering description only if the progressive rendering description is compatible with the ordering of image data in the HEIF file.

[0468] In some cases, two or more progressive rendering descriptions corresponding to different image items may contain a common initial step: for example, a HEIF file containing a pair of stereo image items may contain each image item in its own "prgr" entity group and use the same thumbnail for the first rendering step for both image items, since these image items have only minor differences that are not visible on the thumbnails.

[0469] In some cases, not all initial steps are common to all different image items: for example, the thumbnails corresponding to the first rendering step of two stereo image items may be different, while the previews corresponding to the second rendering step may be the same.

[0470] In some cases, the progressive rendering of these image items may utilize a common initial step.

[0471] In some cases, when rendering an image item that exists as an intermediate step in several progressive rendering descriptions, the renderer may select one of these rendering descriptions, as described above.

[0472] When using different progressive rendering descriptions, steps 510 and 520 of FIG. 5 are modified to take multiple progressive specifications and generate multiple progressive information.

[0473] If different progressive rendering descriptions are used, step 620 in FIG. 6 is modified to extract multiple pieces of progressive information and select one of them.

[0474] It should be noted that all these possibilities described with reference to the variants of the fourth embodiment may also be applied to other variants or embodiments where applicable.

[0475] [Progressive rendering of entity groups] In some cases, progressive rendering information may be associated with an entity group. For example, a thumbnail may be used as a temporary replacement for a slideshow, a collection of image items, or an album. This thumbnail may be different from the thumbnail of the first image in the slideshow, collection, or album.

[0476] In some cases, the "prgr" entity group may be used to indicate that other items or entity groups may be used as temporary substitutes for rendering the entity group. For example, the "prgr" entity group may include a thumbnail and a slideshow entity group to indicate that thumbnails may be used in progressive rendering of the slideshow entity group. As another example, the "prgr" entity group may include a thumbnail, a first slideshow entity group, and a second slideshow entity group. The first slideshow entity group may include previews of image items included in the second slideshow entity group. Progressive rendering of the second slideshow entity group may be achieved by rendering the first thumbnail, then the first slideshow entity group, and finally the second slideshow entity group. As a third example, the "prgr" entity group may include a thumbnail and a slideshow entity group. Additionally, each image item included in the slideshow entity group is also included in the "prgr" entity group along with its thumbnail. Progressive rendering of the slideshow entity group may be achieved by first rendering its thumbnail and then the slideshow itself. As part of rendering a slide show, each image item of the slide show may be rendered progressively, for example by first displaying a thumbnail of the image item before displaying the image item itself when the next image of the slide show has not yet been received by the player. Instead of pausing, the user can watch the next image appear progressively by applying a reconstruction step to this next image.

[0477] In some cases, specific item properties may be used to describe the progressive rendering steps associated with an entity group.

[0478] It should be noted that all these possibilities described with reference to the variants of the fourth embodiment may also be applied to other variants or embodiments where applicable.

[0479] In some cases, progressive rendering of an image item may be used even when the entire data for the image item is already available to allow for faster rendering of the image item. For example, to quickly review all images in a slide show, an initial progressive rendering step of the image may be rendered instead of the image itself. This can be used to allow a user to quickly check a slide show or to achieve some fast forwarding of the slide show.

[0480] Figure 7 shows the main steps for progressive rendering of a HEIF file generated according to an embodiment of the invention. This is a more general alternative to the main steps shown in Figure 6. Note that although these steps are described with reference to image items, they may also be used for progressive rendering of tracks, especially tracks where time information is not necessarily meaningful, such as "pict" tracks or sequences of independent still images encapsulated within tracks.

[0481] In a first step 700, a graphical item is selected for rendering.

[0482] Potential entities for progressive refinement are then determined in steps 710 to 735. Firstly, the image item itself is considered as a potential entity for progressive refinement.

[0483] In step 710 it is checked whether the selected image item is included in an entity group of type "altr". If so, in step 715 the image items included in that entity group are considered as potential entities for progressive refinement. The next step is step 730.

[0484] If not, then in step 720 it is checked whether the selected image item has an associated thumbnail image item. If so, then in step 725 the associated thumbnail is considered as a potential entity for progressive refinement. The next step is step 730.

[0485] In step 730 it is checked whether the selected image item is coded as a multi-layer image having several layers. If so, the different coded layers are considered as potential entities for progressive refinement in step 735. If a progressive layer item property is associated with the selected image item, only the layers indicated in this item property are considered as potential entities for progressive refinement.

[0486] In step 740, the next potential entity is acquired. The loading of data is monitored, for example to check when data corresponding to a new item or a new layer has been loaded. If the loaded data makes it possible to acquire a new image item that is considered a potential entity, this image item is the next potential entity. If the loaded data makes it possible to acquire a new layer that is considered a potential entity, this layer is the next potential entity. If the loaded data makes it possible to acquire a new image item that is a sub-image of a potential entity, this potential entity is the next potential entity. If not, the monitoring continues.

[0487] Next, it is checked whether the entity has been rendered in step 750. If the entity has not been rendered, the next potential entity is rendered in step 755. The next step is step 790.

[0488] If not, then in step 760 it is checked whether an "altr" group was found in step 710, and if true it is checked whether the next potential entity is earlier in this "altr" group than the currently rendered entity. If true, then in step 765 the next potential entity is rendered. The next step is step 790.

[0489] If not, then in step 770 it is checked whether the currently rendered entity is a thumbnail and the next potential entity is its master image. If true, then in step 775 the next potential entity is rendered. The next step is step 790.

[0490] If not, then in step 780 it is checked whether the next potential entity is the same as the currently rendered entity and has a progressive step associated with it. If this is true, then it is further checked whether a new progressive step can be rendered. If this is true, then in step 785 this new progressive step is displayed. The next step is step 790.

[0491] In step 790 it is checked whether the end of the HEIF file has been reached, in which case no further steps are performed, otherwise the next step is step 740.

[0492] The "altr" grouping type may be used to associate an image item with other image items that may be used in progressive rendering of this image item. Some variations of embodiments of the invention rely on the "altr" entity group to indicate all versions of a sub-image. Other grouping types may be used to indicate which image items may be used in progressive rendering of a given image item. A specific grouping type, for example "prog", may be used to associate image items with progressive rendering in mind.

[0493] In some cases, a progressive refinement pattern item property may be associated with any image item to indicate the progressive refinement method selected for this image item.

[0494] In some cases, a progressive refinement item property may be associated with an image item to indicate that this image item supports progressive rendering. The structure of this progressive refinement item property may be as follows: aligned(8) class ProgressiveProperty extends ItemFullProperty('prog', version = 0, flags = 0) { }

[0495] In some cases, some progressive information may be associated with an entity group to indicate that the entire group supports progressive rendering. For example, a left-right progressive rendering pattern may be associated with a panorama group to indicate that the panorama itself may be rendered in a progressive manner.

[0496] 8 is a schematic block diagram of a computing device 800 for implementing one or more embodiments of the present invention. The computing device 800 may be a device such as a microcomputer, a workstation, or a light portable device. The computing device 800 comprises a communication bus connected to the following: A central processing unit 801, such as a microprocessor, referred to as a CPU. a random access memory 802, referred to as RAM, where the memory capacity can be expanded, for example by an optional RAM connected with an expansion port, for storing the executable code of the methods of the embodiments of the present invention, as well as registers adapted for recording variables or parameters necessary for implementing the methods according to the embodiments of the present invention. A read-only memory 803, called ROM, for storing a computer program for implementing an embodiment of the invention. - The network interface 804 is typically connected to a communications network over which the digital data to be processed is transmitted or received. The network interface 804 may be a single network interface or may be composed of a set of different network interfaces (e.g., wired and wireless interfaces, or different types of wired or wireless interfaces). Data packets are written to the network interface for transmission or read from the network interface for reception under the control of a software application running in the CPU 801. A graphical user interface 805 may be used to receive input from a user or to display information to a user. A hard disk 806, designated -HD, may be provided as a mass storage device. - I / O module 807 may be used for receiving / transmitting data from / to external devices such as video sources or displays.

[0497] The executable code may be stored either in the read-only memory 803, in the hard disk 806 or on a removable digital medium such as a disk. According to a variant, the executable code of the program may be received by the communications network via the network interface 804 to be stored in one of the storage means of the communications device 800, such as the hard disk 806, before being executed.

[0498] The central processing unit 801 is adapted to control and direct the execution of instructions or parts of software code of one or more programs according to embodiments of the present invention, the instructions of which are stored in one of the aforementioned storage means. After power-on, the CPU 801 is capable of executing instructions from the main RAM memory 802 relating to software applications after they have been read from the program ROM 803 or the hard disk 806. Such software applications, when executed by the CPU 801, cause the steps of the flowcharts of the present invention to be performed.

[0499] Any step of an algorithm of the present invention may be implemented in software by execution of a set of instructions or programs by a programmable computing machine such as a PC ("Personal Computer"), a DSP ("Digital Signal Processor") or a microcontroller, or else in hardware by mechanical or dedicated components such as an FPGA ("Field-Programmable Gate Array") or an ASIC ("Application-Specific Integrated Circuit").

[0500] While the present invention has been described above based on specific embodiments, the present invention is not limited to the above embodiments, and various modifications are possible without departing from the gist of the present invention.

[0501] Furthermore, many further modifications and changes will be suggested to those skilled in the art with reference to the exemplary embodiments described above, which are given by way of example only and are not intended to limit the scope of the invention, which is determined solely by the appended claims. In particular, different features from the various embodiments may be interchanged where appropriate.

[0502] Each of the above-described embodiments of the present invention may be implemented alone or in combination in several embodiments, and features from the various embodiments may be combined where necessary or where a combination of elements or features from the individual embodiments in a single embodiment is beneficial.

[0503] In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. The mere fact that different features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be used to advantage.

Claims

1. A method for performing progressive rendering on a plurality of sub-images by varying a step to be processed according to an order of rendering steps corresponding to the plurality of sub-images, comprising: obtaining progressive information indicating a rendering step corresponding to a sub-image of interest from a file including the sub-images; a determination step of determining whether the target sub-image is associated with the step to be processed based on the progressive information acquired in the acquisition step; a rendering step of rendering the sub-image of interest determined to be associated in the determining step; having The rendering method further comprises, in the determination process, when it is determined that the target sub-image is not associated with the step to be processed, making a determination on the other target sub-images based on progressive information of the other target sub-images contained in the multiple sub-images.

2. The plurality of sub-images includes at least two sub-images of different quality or resolution; 2. The method of claim 1, wherein the rendering step performs progressive rendering based on the at least two sub-images so that the quality or resolution increases.

3. The plurality of sub-images correspond to a plurality of regions of the reconstructed image, 2. The method of claim 1, wherein the rendering step generates the reconstructed image by progressively rendering the plurality of sub-images.

4. The rendering method of claim 1, wherein the multiple sub-images represent different subsets of layers of the reconstructed image.

5. The plurality of sub-images include a plurality of first sub-images respectively corresponding to a plurality of regions of the reconstructed image, and a plurality of second sub-images respectively corresponding to the plurality of regions and having a different quality layer or resolution layer from the plurality of first sub-images, 2. The method of claim 1, wherein the progressive information indicates a progressive step for each of the first plurality of sub-images and each of the second plurality of sub-images.

6. The rendering method described in Claim 5, characterized in that in the rendering step, after rendering a sub-image of a portion of the plurality of first sub-images, rendering a sub-image of a portion of the plurality of second sub-images is performed.

7. The rendering method of claim 5, wherein in the rendering step, the rendering of the plurality of first sub-images is followed by the rendering of the plurality of second sub-images.

8. A method of progressively rendering a plurality of sub-images, comprising: obtaining progressive information indicating a rendering step corresponding to each of the plurality of sub-images from a file containing the plurality of sub-images; a selection step of selecting, from among the plurality of sub-images, a sub-image corresponding to the progressive information indicating a rendering step to be processed; a rendering step of rendering the sub-image selected in the selection step; 13. A method of rendering comprising:

9. The rendering method according to claim 1, wherein the file is a file conforming to the ISO Base Media File Format (ISOBMFF).

10. The rendering method of claim 9, wherein the file is a High Efficiency Image File Format (HEIF) file.

11. The progressive information is included in a "meta" box of a HEIF file, 11. The rendering method according to claim 10, wherein in the obtaining step, the progressive information is obtained from the "meta" box.

12. The rendering method of claim 11, wherein the progressive information is described as a "prdi" item property in the "meta" box.

13. A method for performing progressive rendering on a plurality of sub-images by varying a step to be processed according to an order of rendering steps corresponding to the plurality of sub-images, comprising: an acquisition means for acquiring progressive information indicating a rendering step corresponding to a target sub-image among the plurality of sub-images from a file including the plurality of sub-images; a determination means for determining whether the target sub-image is associated with the step to be processed based on the progressive information acquired by the acquisition means; a rendering means for rendering the target sub-image determined to be associated by the determining means; having The device is further characterized in that, when it is determined that the target sub-image is not associated with the step to be processed, the determination means makes a determination regarding the other target sub-image based on progressive information of the other target sub-image contained in the plurality of sub-images.

14. The plurality of sub-images includes at least two sub-images of different quality or resolution; 14. The device of claim 13, wherein the rendering means performs progressive rendering of increasing quality or resolution based on the at least two sub-images.

15. The method of claim 1, wherein the plurality of sub-images correspond to a plurality of regions of the reconstructed image, 14. The device of claim 13, wherein the rendering means generates the reconstructed image by progressively rendering the plurality of sub-images.

16. The device of claim 13, wherein the multiple sub-images represent different subsets of layers of a reconstructed image.

17. The plurality of sub-images include a plurality of first sub-images respectively corresponding to a plurality of regions of the reconstructed image, and a plurality of second sub-images respectively corresponding to the plurality of regions and differing in quality layer or resolution layer from the plurality of first sub-images, 14. The device of claim 13, wherein the progressive information indicates a progressive step for each of the first plurality of sub-images and each of the second plurality of sub-images.

18. The device described in Claim 17, characterized in that the rendering means renders a sub-image of a portion of the plurality of second sub-images after rendering a sub-image of a portion of the plurality of first sub-images.

19. The device of claim 17, wherein the rendering means renders the plurality of second sub-images after rendering the plurality of first sub-images.

20. A device for progressively rendering a plurality of sub-images, comprising: an acquisition means for acquiring progressive information indicating a rendering step corresponding to each of the plurality of sub-images from a file including the plurality of sub-images; a selection means for selecting a sub-image corresponding to the progressive information indicating a rendering step to be processed from among the plurality of sub-images; rendering means for rendering the sub-image selected by said selection means; A device comprising:

21. A device described in any one of claims 13 to 20, characterized in that the file is a file that complies with the ISO Base Media File Format (ISOBMFF).

22. The device of claim 21, wherein the file is a High Efficiency Image File Format (HEIF) file.

23. The progressive information is included in a "meta" box of a HEIF file, 23. The device of claim 22, wherein the obtaining means obtains the progressive information from the "meta" box.

24. The device of claim 23, wherein the progressive information is described as a "prdi" item property in the "meta" box.

25. A program that, when executed by a computer, causes the computer to perform a rendering method for performing progressive rendering on a plurality of sub-images by changing a step to be processed according to an order of the rendering steps corresponding to the plurality of sub-images, the rendering method comprising: obtaining progressive information indicating a rendering step corresponding to a sub-image of interest from a file including the sub-images; a determination step of determining whether the target sub-image is associated with the step to be processed based on the progressive information acquired in the acquisition step; a rendering step of rendering the sub-image of interest determined to be associated in the determining step; having The program further comprises, in the judgment process, when it is judged that the target sub-image is not associated with the step to be processed, making a judgment on the other target sub-images based on progressive information of the other target sub-images contained in the multiple sub-images.

26. A program which, when executed by a computer, causes the computer to execute a rendering method for performing progressive rendering on a plurality of sub-images, the rendering method comprising: obtaining progressive information indicating a rendering step corresponding to each of the plurality of sub-images from a file containing the plurality of sub-images; a selection step of selecting, from among the plurality of sub-images, a sub-image corresponding to the progressive information indicating a rendering step to be processed; a rendering step of rendering the sub-image selected in the selection step; A program comprising:

Images