Representation of small images in a condensed, highly efficient image file format encapsulation format

Abstract

Embodiments of this disclosure provide HEIF-based encapsulation techniques optimized for the distribution of small images over communication networks such as the Internet. In such applications, the encoded image content may have a small content payload, the spatial area of ​​the image may be smaller compared to the range of image sizes otherwise supported by HEIF, and the image may contain only a limited number of representations. According to these embodiments, the overhead signaling of these images can be condensed into a separate box that is distinguishable from other types of HEIF boxes by a character code. This box may be a self-contained unit containing both the overhead signaling and the encoded content of the image itself.

Description

[Technical Field]

[0001] Priority Claim The present application claims the benefit of priority from U.S. Patent Application No. 63 / 487,280, entitled "Representation of Small Images in Condensed High Efficiency Image File-Style Encapsulation Formats", filed on February 28, 2023, the disclosure of which is incorporated herein by reference in its entirety. [Background Art]

[0002] The present disclosure relates to the distribution of image content via computer networks, and in particular to the distribution of images using container files such as those using the HEIF format.

[0003] Distribution of image and video content is ubiquitous in modern networking applications. Image content is typically encoded according to a compression algorithm and made available by a source device over a computer network. The image content is retrieved by a sink device, which decodes the encoded image content to obtain a restored version of the image content, and consumes the decoded image content, typically by display.

[0004] There is a vast variety of image content available for use. Image content can be represented as a single still image, a short image sequence, or a full video. Image content can be represented in relatively small image sizes (e.g., thumbnails) or as frames of high-definition 1080p, 4K, or 8K content. Image content can be represented using various bit depths and color spaces. Image content can be compressed using various compression algorithms. To ensure that sink devices can properly interpret image content provided by source devices, interface standards have been developed that define how source devices represent such information and how sink devices interpret it. The High Efficiency Image File (HEIF) format (ISO / IEC 23008-12) is one example of a container format that defines the syntax for representing such information. HEIF works in conjunction with a variety of compression algorithms, including HEVC and AV1, for example.

[0005] Currently, the HEIF image format is optimized for large and complex files, which typically consist of many tiles or sub-images sharing attributes. Many of the attributes and boxes are optimized for flexibility and extensibility. All of these optimizations come at the cost of header overhead. For example, the minimum size of HEIF boxes required for an AVIF-branded file (e.g., AV1 encoded image information encapsulated within an HEIF container file) is approximately 300 bytes. This overhead "cost" is usually not a problem for large images, but it becomes a significant cost when dealing with very small files. For an 80x60 pixel AVIF file, the overhead can amount to approximately 30% of the compressed size. [Brief explanation of the drawing]

[0006] [Figure 1]This is a block diagram of a communication system suitable for use in the embodiment of this proposal.

[0007] [Figure 2] This figure shows an encapsulated data structure image according to one embodiment of the present disclosure. [Modes for carrying out the invention]

[0008] Embodiments of this disclosure provide HEIF-based encapsulation techniques optimized for the distribution of small images over communication networks such as the Internet. In such applications, the encoded image content may have a small content payload, such as less than 64kB. Typically, in such applications, the spatial area of ​​the image is small compared to the range of image sizes otherwise supported by HEIF (e.g., less than 640x480 pixels). The image may contain only a limited number of representations. For example, it may consist of a main image and optionally an alpha plane. It may also contain an ICC profile and / or CICP color information. It may also contain Exif and XMP metadata.

[0009] According to these embodiments, the overhead signaling and payload content of an image may be condensed into a single, independent box. In HEIF, a box is identifiable by a four-character code (4CC) indicating the type of box. Therefore, the boxes proposed in this disclosure may be distinguishable from other types of HEIF boxes by their unique character code and box type. For the purposes of this specification, we assume that a box can be identified by the code "coni" indicating that it is a Condensed Image box. Alternative character codes may also be used, as long as they are distinguishable from other box identifiers as defined in the control standards.

[0010] The condensed image box proposed in this document can be used in place of other boxes currently used in HEIF (ISO / IEC 23008-12), such as the "meta" box. In HEIF, using the "meta" box requires the use of subboxes such as "hdlr," "dinf," "pitm," "iinf," "iprp" (which itself may contain subboxes), "iloc," "iref," and "mdat" (which contains the image payload). The condensed image box proposed in this document can provide the informational content of these other HEIF boxes for intended image use cases, but in a highly condensed format, which leads to greater efficiency in representation.

[0011] Figure 1 is a block diagram of a communication system 100 suitable for use in an embodiment of the present invention. Figure 1 shows a source terminal 110 and a sink terminal 120 provided in mutual communication over a network 130. The source terminal 110 may make image content available for download and consumption by the sink terminal 120. For this purpose, the source terminal 110 may generate image content on the fly, but more often it stores image information in a local storage device 140. For example, Figure 1 shows an image 150 stored by the source terminal 110 and made available for use by the sink terminal 120 over the network 130. The image 150 may be encapsulated according to the HEIF format representation technique, which includes signaling of image overhead information in a common box and the image content.

[0012] The source terminal 110 and the sink terminal 120 may operate according to interface specifications that define how image information is represented. In relation to this paper, such information can be classified as “payload” information, which includes the image content of the image information, and “overhead” information, which represents the characteristics of the payload information (such as the height and width of the payload image, the encoding protocol applied to the payload image, and the color space used to represent the payload image). The image information is usually encoded by a coder 112 that applies a selected compression algorithm to the image content before it is made available to the source terminal 110. The sink terminal 120 usually has one or more decoders 122 available to decode the image. When applied, a controller 124 in the sink terminal 120 examines the overhead information provided with the image 150 and causes the appropriate decoder 122 of the sink device to act in a way that reverses the encoding process applied by the encoder 112.

[0013] In Figure 1, the source terminal 110 is shown as a server and the sink terminal 120 is shown as a tablet computer, but the principles of this disclosure are not limited thereto. Embodiments of the present invention can be applied to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 130 represents any number of networks that transmit information between terminals 110 and 120, including, for example, wired communication networks and / or wireless communication networks. Communication network 130 can exchange data on circuit-switched and / or packet-switched channels. Typical networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of network 130 are not important to the operation of the present invention unless specifically described below in this specification.

[0014] Figure 2 shows a data structure 200 for an image 150 (Figure 1) according to one embodiment of the present disclosure. As shown, the image 150 may be represented by a pair of boxes 210 and 220. The first box 210 may be the "ftyp" (file type) box of the image 150. The data contained in the "ftyp" box 210 may indicate that the image consists of a "coni" box 220 and may specify the location of the "coni" box 220. The "coni" box 220 may contain all the overhead 222 necessary to interpret the encoded payload 224 of the image, which is itself contained within the "coni" box 220.

[0015] During operation, the sink terminal 120 (Figure 1) may access the "ftyp" box 210 and the "coni" box 220 of the image. Examining the "ftyp" box 210 sets the context for the sink terminal 120 to interpret the "coni" box 220. The controller 124 of the sink terminal can examine the overhead signaling 222 of the "coni" box 220 to determine how the encoded payload 224 is represented. The controller 124 then activates the appropriate decoder 122 of the sink terminal 120 to reconstruct the decoded image from the encoded payload 224.

[0016] As described above, embodiments of this disclosure provide overhead information for image 150 in a highly condensed form. In the first embodiment, the overhead element 222 of the "coni" box 220 is entirely self-contained and intended to provide all the information necessary for the decoder 122 to reconstruct the image information from the encoded payload 224. In one example, the overhead portion 222 of the "coni" box may be defined as follows: [Table 1] [Table 2] [Table 3] Here, these syntax elements convey the following information. [Table 4-1] [Table 4-2]

[0017] The overhead portion 222 that provides the aforementioned information is expected to provide sufficient information to the sink terminal 120 (FIG. 1) for the above-described use cases of encoding a large number of small images. Subsequent to this overhead portion 222, the encoded payload 224 can be expressed as follows. [Table 5] When a certain payload does not exist (e.g., an alpha image, Exif metadata, or XMP metadata), the length of the payload elements listed above can be zero.

[0018] As shown in FIG. 2, the overhead 222 and the payload 224 may be included in a common box.

[0019] As described, in practice, the source terminal 110 and the sink terminal 120 (FIG. 1) exchange the image 150 in accordance with a certain protocol, which not only defines the syntax for representing the overhead information 222, but also defines rules for interpreting these syntax elements. In one embodiment, the extended attribute data may store a set of item attributes applied to an item. In such embodiments, the interface protocol specifies the following. 1. Extended attributes shall not include "ispe", "pixi", "imir", "irot", or a codec settings box. 2. Extended attributes shall not include the "colr" box if hasICC or hasNCLX is true. 3. The order of the attributes is as follows: a) "ispe", codec settings, "pixi", "colr-ICC", and "colr-NCLX" synthesized from the "coni" box. b) Attributes extended in a given order c) "irot" and "imir" synthesized from the "coni" box.

[0020] As described, the embodiments of this proposal may yield a condensed representation of HEIF overhead information, which may result in higher efficiency than other HEIF-based encapsulation techniques. Table 1 compares the representations that may be obtained for an exemplary 80x60 pixel image encoded in an AVIF-branded representation, such as an AV1 encoded image contained within a HEIF-encapsulated image. For the purposes of this discussion, we assume that the image payload is encoded as 1,083 bytes, has 400 bytes of Exif metadata, has an NCLX color representation, and includes both rotation and mirroring.

[0021] Prior to this proposal, HEIF required 375 bytes of overhead information to transmit images, as shown below, but the aforementioned "coni" block requires only 44 bytes. [Table 6] As this example shows, overhead signaling in the HEIF representation is approximately 34% (375 bytes / 1083 bytes) of the size of the encoded image. By comparison, the "coni" representation is far more efficient. The overhead signaling is approximately 4% (44 bytes / 1083 bytes) of the size of the encoded image. Also, since the "coni" box contains the payload content 224 of the image 150, the 8-bit overhead of the "mdat" box can also be avoided.

[0022] As with typical HEIF cases, the principles of this disclosure apply to both codec-independent and codec-specific brands. When the "coni" box 220 is used in a codec-independent application, the explicitCodecTypes flag within box 220 is set to true, and the infeType and codecConfigType fields are specified. This information defines the type of decoder 122 that should be used when decoding the payload 224 for the sink terminal 120.

[0023] When the "coni" box 220 is used in a codec-specific application, the file's ftyp box 210 may contain a brand identifier that identifies both the file containing the coni box 220 and the coni box being associated with a particular codec type. For example, the brand "cna1" may identify that the coni box 220 is intended to be used with the AV1 coder. By using such a brand, the ftyp box 210 may define the default codec information for the "coni" box 220 as follows: infeType:av01 codecConfigType:av1C In this application example, these default values ​​may apply if the explicitCodecTypes flag is not set within the overhead 222 of the "coni" box. These defaults can be overridden by setting the explicitCodecTypes flag and identifying the codec parameters using additional signaling. In this embodiment, efficiency improvements can be achieved by omitting the overhead bytes consumed to transmit the codec information (8 bytes), and by omitting the need for the ftyp box 210 to have a separate brand to identify the codec.

[0024] As described above, in the primary use case, the "coni" box may be used as a self-contained box containing all the overhead 222 required for decoding and consuming the image information within the image payload 224 content. In this case, other boxes defined in HEIF, such as the "meta," "moov," and "moof" boxes, are not used, thus avoiding the signaling overhead they consume. Thus, the "coni" box provides a highly efficient representation of the overhead required to decode the encoded payload 224.

[0025] In the primary use case, the “coni” box is expected to eliminate the need for other boxes; however, the principles of this disclosure allow the “coni” box to be used in conjunction with such other boxes if desired by the content publisher. In one use case, it may be desirable to make a “coni” encoded image (e.g., 150 in Figure 1) available both as standalone image content and as part of a video sequence. In such applications, overhead corresponding to other parts of the video (e.g., frames at other temporal locations in the video sequence) may be represented using “moov” or “moof” boxes as needed. The payload content of these other parts of the video may be represented within an “mdat” box. The file’s ftyp box 210 is expected to indicate that the “coni” box 220 is a second box contained within the file and to identify the type of other boxes (not shown in Figure 2) that sequentially follow the “coni” box.

[0026] In such applications, the sink terminal 120 may operate according to predetermined rules to integrate the image content obtained from the decoding of the encoded payload 224 with the content obtained from the decoding of other parts of the video. First, the sink terminal 120 may operate according to interpretation rules that specify that each image contained within the "coni" box 220 will be assigned an identifier. For example, the main image decoded from payload 224 may be assigned item ID 1, the alpha image decoded from payload 224 may be assigned item ID 2, the Exif metadata item decoded from payload 224 may be assigned item ID 3, and the XMP metadata item decoded from payload 224 may be assigned item ID 4. According to the terminal's interpretation rules, the "meta" box is not permitted to define items having such identifiers (e.g., not to list them in the "infe" box), it is not permitted to add item attributes to the items provided by the "coni" box 220 (e.g., not to list them in the "ipma" box), and it is not permitted to describe the location of the items provided by the "coni" box 220.

[0027] However, the sink terminal 120 may generate a virtual "meta" box that represents the items in the "coni" box according to a predetermined protocol. Furthermore, the "meta" box may add item references from the items in the "meta" box to the items in the "coni" box. The "meta" box may, for example, add additional metadata to the items from the "coni" box, add additional thumbnails to the items from the "coni" box, use the items from the "coni" box as input to derived items in the "meta" box, and / or add grouping between the "coni" items and the "meta" items. Thus, the items recovered from the payload 224 portion and the items from these other boxes may be integrated into a final representation of the content according to a process defined in either the management interface specification or these other boxes. In this case, the "coni" box 220 remains a self-contained box containing all the data necessary to decrypt the payload 224 within box 220.

[0028] The above discussion describes the operation of embodiments of the present disclosure in the context of encoding and decoding systems provided within source and sink terminals. Generally, these components are provided as electronic devices. They can be embodied in integrated circuits such as application-specific integrated circuits, field-programmable gate arrays and / or digital signal processors. Alternatively, they can be embodied in computer programs stored in memory and executed on processing devices of mobile computing platforms such as personal computers, notebook computers, computer servers, or smartphones and tablet computers. Similarly, coders and decoders can be embodied in integrated circuits such as application-specific integrated circuits, field-programmable gate arrays and / or digital signal processors, or in computer programs stored in memory and executed on processing devices of mobile computing platforms such as personal computers, notebook computers, computer servers, or smartphones and tablet computers. Decoders are generally packaged within consumer electronic devices such as gaming systems, DVD players, portable media players, tablet computers, smartphones, smartwatches, virtual reality goggles, augmented reality goggles, automotive media systems, and aircraft media systems. Furthermore, these can be packaged within consumer software applications such as video games and browser-based media players. Naturally, these components may also be provided as a hybrid system, distributing functionality between dedicated hardware components and programmed general-purpose processors, as needed.

[0029] Some embodiments of this disclosure are specifically illustrated and / or described herein. However, modifications and variations of this disclosure will be understood to be covered by the above teachings and included within the scope of the appended claims without departing from the spirit and intended scope of this disclosure.

Claims

1. A method of representing image data, Encoding an image according to an image compression algorithm, A method comprising storing the encoded image on a network in an encapsulation representation including a pair of boxes, The first box of the pair of boxes identifies the second box of the pair of boxes by type, A method wherein the second box is a self-contained unit comprising an overhead portion and a payload portion, wherein the overhead portion provides sufficient metadata to identify the type of compression algorithm applied to the image, the size of the image, the applicable color space to the image, and the orientation of the image, and the payload portion comprises the encoded content of the image.

2. The method according to claim 1, wherein the overhead portion includes identification information of the bit depth of the image content represented by the payload portion.

3. The method according to claim 1, wherein the overhead portion includes an indication of whether the image content represented by the payload portion is monochrome.

4. The method according to claim 1, wherein the overhead portion includes indication of an ICC profile applicable to the image content represented by the payload portion.

5. The method according to claim 1, wherein the overhead portion includes indication of NCLX color information applicable to the image content represented by the payload portion.

6. The method according to claim 1, wherein the overhead portion includes a flag that identifies height and width size information related to the image content represented by the payload portion.

7. The method according to claim 1, wherein, in the case of image content including an alpha channel, the overhead portion includes an indication of whether the image content represented by the payload portion includes an alpha channel.

8. The method according to claim 1, wherein, in the case of image content containing Exif metadata, the overhead portion includes an indication of whether the image content represented by the payload portion contains Exif metadata.

9. The method according to claim 1, wherein, in the case of image content containing XMP metadata, the overhead portion includes an indication of whether the image content represented by the payload portion contains XMP metadata.

10. The method according to claim 1, wherein the second box further includes a box sequence of item attributes containing information that was not included in the overhead portion.

11. The method according to claim 1, wherein the first box includes a brand identifier that identifies a default codec type applicable to the payload portion of the second box.

12. A non-temporary computer-readable medium for storing encoded image data expressed according to a certain syntax, wherein the syntax is Encoding an image according to an image compression algorithm, A medium comprising storing the encoded image on a network in an encapsulated representation including a pair of boxes, The first box of the pair of boxes identifies the second box of the pair of boxes by type, A medium in which the second box is a self-contained unit comprising an overhead portion and a payload portion, wherein the overhead portion provides sufficient metadata to identify the type of compression algorithm applied to the image, the size of the image, the applicable color space to the image, and the orientation of the image, and the payload portion contains the encoded content of the image.

13. The medium according to claim 12, wherein the overhead portion includes identification information of the bit depth of the image content represented by the payload portion.

14. The medium according to claim 12, wherein the overhead portion includes an indication of whether the image content represented by the payload portion is monochrome.

15. The medium according to claim 12, wherein the overhead portion includes indication of an ICC profile applicable to the image content represented by the payload portion.

16. The medium according to claim 12, wherein the overhead portion includes indication of NCLX color information applicable to the image content represented by the payload portion.

17. The medium according to claim 12, wherein the overhead portion includes a flag that identifies height and width size information related to the image content represented by the payload portion.

18. The medium according to claim 12, wherein, in the case of image content including an alpha channel, the overhead portion includes an indication of whether the image content represented by the payload portion includes an alpha channel.

19. The medium according to claim 12, wherein, in the case of image content containing Exif metadata, the overhead portion includes an indication of whether the image content represented by the payload portion contains Exif metadata.

20. The medium according to claim 12, wherein, in the case of image content containing XMP metadata, the overhead portion includes an indication of whether the image content represented by the payload portion contains XMP metadata.

21. The medium according to claim 12, wherein the second box further comprises a box sequence of item attributes containing information that was not included in the overhead portion.

22. The medium according to claim 12, wherein the first box includes a brand identifier that identifies a default codec type applicable to the payload portion of the second box.

23. A method for decoding image data, The encoded image is taken from the network, a pair of boxes, and the pair of boxes is The first box of the pair of boxes, which identifies the second box of the pair of boxes by type, Extracting in an encapsulated representation comprising a pair of boxes, the second box being a self-contained unit comprising an overhead portion and a payload portion, wherein the overhead portion provides sufficient metadata to identify the type of compression algorithm applied to the image, the size of the image, the applicable color space to the image, and the orientation of the image, and the payload portion comprises the encoded content of the image, Interpreting the contents of the second box according to the type identified in the first box, Setting the image decoder according to the metadata provided to the overhead portion, A method comprising decoding the payload portion according to the configured image decoder.

24. The method according to claim 23, wherein the overhead portion includes identification information of the bit depth of the image content represented by the payload portion.

25. The method according to claim 23, wherein the overhead portion includes an indication of whether the image content represented by the payload portion is monochrome.

26. The method according to claim 23, wherein the overhead portion includes indication of an ICC profile applicable to the image content represented by the payload portion.

27. The method according to claim 23, wherein the overhead portion includes indication of NCLX color information applicable to the image content represented by the payload portion.

28. The method according to claim 23, wherein the overhead portion includes a flag that identifies height and width size information related to the image content represented by the payload portion.

29. The method according to claim 23, wherein, in the case of image content including an alpha channel, the overhead portion includes an indication of whether the image content represented by the payload portion includes an alpha channel.

30. The method according to claim 23, wherein, in the case of image content containing Exif metadata, the overhead portion includes an indication of whether the image content represented by the payload portion contains Exif metadata.

31. The method according to claim 23, wherein, in the case of image content containing XMP metadata, the overhead portion includes an indication of whether the image content represented by the payload portion contains XMP metadata.

32. The method according to claim 23, wherein the second box further includes a box sequence of item attributes containing information that was not included in the overhead portion.

33. The method according to claim 23, wherein the first box includes a brand identifier that identifies a default codec type applicable to the payload portion of the second box.

34. An encoding system, Processor and The method comprises a memory that stores instructions for causing the processor to execute an encoding method, when executed by the processor, Encoding an image according to an image compression algorithm, An encoding system comprising storing the encoded image in an encapsulation representation including a pair of boxes, The first box of the pair of boxes identifies the second box of the pair of boxes by type, An encoding system in which the second box is a self-contained unit comprising an overhead portion and a payload portion, wherein the overhead portion provides sufficient metadata to identify the type of compression algorithm applied to the image, the size of the image, the applicable color space to the image, and the orientation of the image, and the payload portion comprises the encoded content of the image.

35. A decoding system, Processor and The method, in response to receiving an encapsulated representation comprising a pair of boxes, the pair of boxes comprising a pair of boxes, the pair of boxes comprising a first box which identifies the second box of the pair of boxes by type, and the second box which is a self-contained unit comprising an overhead portion and a payload portion, the overhead portion providing sufficient metadata to identify the type of compression algorithm applied to the image, the size of the image, the color space applicable to the image, and the orientation of the image, and the payload portion comprising the second box which comprises the encoded content of the image, Interpreting the contents of the second box according to the type identified in the first box, Setting the image decoder according to the metadata provided to the overhead portion, A decoding system comprising decoding the payload portion according to the configured image decoder.

Images