File processing apparatus, file processing method, and program

Abstract

The present technology relates to a file processing device, a file processing method, and a program which enable a High Efficiency Image File Format (HEIF) file to be efficiently played back. There is generated an HEIF file in which a related-image related to an image in the HEIF file is arranged in a leading portion of an mdat box. In addition, the leading portion of the HEIF file is read. The present technology can be applied to the generation and playback of the HEIF file.

Description

Technical Field

[0001] The present technology relates to a file processing apparatus, a file processing method, and a program, and particularly to a file processing apparatus, a file processing method, and a program that enable efficient playback of, for example, HEIF files.

Background Art

[0002] There is HEIF (High Efficiency Image File Format) as a file format for efficiently storing images (see Non-Patent Document 1).

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the future, for HEIF files, for example, it is required to efficiently perform playback of HEIF files such as displaying thumbnail images at high speed.

[0005] The present technology has been made in view of such a situation, and enables efficient playback of HEIF files.

Means for Solving the Problems

[0006] The first file processing apparatus or program of the present technology is a file processing apparatus including a file control unit that generates the HEIF (High Efficiency Image File Format) file in which related images related to the images in the HEIF file are arranged at the head portion of the mdat box, or a program for causing a computer to function as such a file processing apparatus.

[0007] The first file processing method of the present technology is a file processing method including generating the HEIF (High Efficiency Image File Format) file in which related images related to the images in the HEIF file are arranged at the head portion of the mdat box.

[0008] In the first file processing apparatus, file processing method, and program of the present technology, the HEIF (High Efficiency Image File Format) file in which related images related to the images in the HEIF file are arranged at the head portion of the mdat box is generated.

[0009] The second file processing apparatus or program of the present technology is a file processing apparatus including a file control unit that reads the head portion of the HEIF (High Efficiency Image File Format) file in which related images related to the images in the HEIF file are arranged at the head portion of the mdat box, or a program for causing a computer to function as such a file processing apparatus.

[0010] The second file processing method of the present technology is a file processing method including reading the head portion of the HEIF (High Efficiency Image File Format) file in which related images related to the images in the HEIF file are arranged at the head portion of the mdat box.

[0011] In the second file processing apparatus, file processing method, and program of the present technology, the head portion of the HEIF (High Efficiency Image File Format) file in which related images related to the images in the HEIF file are arranged at the head portion of the mdat box is read.

[0012] The file processing apparatus may be an independent apparatus or an internal block constituting one apparatus.

[0013] The program can be provided by being recorded on a recording medium or transmitted via a transmission medium.

Brief Description of Drawings

[0014]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Embodiments for Carrying Out the Invention

[0015] <An Embodiment of a Digital Camera to which the Present Technology is Applied>

[0016] FIG. 1 is a block diagram showing a configuration example of an embodiment of a digital camera to which the present technology is applied.

[0017] The digital camera 10 includes an optical system 11, an image sensor 12, a signal processing unit 13, a medium 14, I / Fs (Interfaces) 15 and 16, buttons / keys 17, a touch panel 18, a liquid crystal panel 19, a viewfinder 20, and an I / F 21 and the like.

[0018] The optical system 11 condenses light from a subject onto the image sensor 12.

[0019] The image sensor 12 receives light from the optical system 11, performs imaging by photoelectric conversion, generates image data as an electrical signal, and supplies it to the signal processing unit 13.

[0020] The signal processing unit 13 includes an optical system / image sensor control unit 41, an encoding control unit 42, a file control unit 43, a medium control unit 44, an operation control unit 45, a display control unit 46, and a UI control unit 47.

[0021] The optical system / image sensor control unit 41 controls the optical system 11 and the image sensor 12, and supplies an image (data) obtained by imaging performed according to the control to the encoding control unit 42.

[0022] The encoding control unit 42 supplies the image from the optical system / image sensor control unit 41 to the display control unit 46, encodes it as necessary, and supplies it to the file control unit 43. Further, the encoding control unit 42 decodes the image supplied from the file control unit 43 as necessary and supplies it to the display control unit 46.

[0023] The file control unit 43 generates a file storing the image supplied from the encoding control unit 42 and supplies it to the media control unit 44. Also, the file control unit 43 performs reproduction of the file supplied from the media control unit 44, that is, reading out of data such as the image stored in the file. For example, the image read out from the file is supplied from the file control unit 43 to the encoding control unit 42.

[0024] The media control unit 44 controls the exchange of files between the media 14 and the I / Fs 15 and 16. For example, the media control unit 44 causes the file from the file control unit 43 to be recorded on the media 14 or transmitted from the I / Fs 15 and 16. Also, the media control unit 44 reads out a file from the media 14 or receives a file from the I / Fs 15 and 16 and supplies it to the file control unit 43.

[0025] The operation control unit 45 supplies an operation signal corresponding to the operation to the necessary blocks according to the operations of the buttons / keys 17 and the touch panel 18 by the user.

[0026] The display control unit 46 performs display control such as supplying the image etc. supplied from the encoding control unit 42 to the liquid crystal panel 19, the viewfinder 20, and the I / F 21 for display.

[0027] The UI control unit 47 is in charge of UI (User Interface) control.

[0028] The medium 14 is a storage medium such as an SD card or the like. The I / F 15 is an I / F of a LAN (Local Area Network) such as WiFi (registered trademark) or Ethernet (registered trademark). The I / F 16 is an I / F of USB (Universal Serial Bus) for example. The buttons / keys 17 and the touch panel 18 are operated by the user when inputting commands and other information into the digital camera 10. The touch panel 18 can be configured integrally with the liquid crystal panel 19. The liquid crystal panel 19 and the viewfinder 20 display images and the like supplied from the display control unit 46. The I / F 21 is an I / F for transmitting at least images such as HDMI (High-Definition Multimedia Interface) (registered trademark) or DP (Display Port).

[0029] In the digital camera 10 configured as described above, the optical system / image sensor control unit 41 generates a YUV image having the same resolution (number of pixels) (size) as that of the RAW image, for example, from the RAW data image (hereinafter also referred to as the RAW image) obtained by imaging with the image sensor 12, and supplies it to the encoding control unit 42 together with the RAW image. The encoding control unit 42 generates a master image of the HEIF file and the like from the YUV image from the optical system / image sensor control unit 41. For example, the YUV image from the optical system / image sensor control unit 41 can be used as the master image of the HEIF file as it is.

[0030] The symbol control unit 42 generates, from the main YUV image, for use in displaying on the liquid crystal panel 19 or an external display, a first other image based on the main image, for example, a YUV image with a lower resolution than the main image (hereinafter also referred to as a screen thumbnail image), and for use in index display (list display), a second other image based on the main image, for example, a YUV image with a lower resolution than the screen thumbnail image (hereinafter also referred to as a thumbnail image). The symbol control unit 42 supplies, for example, the screen thumbnail image to the liquid crystal panel 19 via the display control unit 46 and causes it to be displayed as a so-called through image. As the thumbnail image, for example, an image with a long side size of 320 pixels or less can be adopted. The ratio of the size (number of pixels) between the main image and the screen thumbnail image as the first other image based on the main image, or the thumbnail image as the second other image based on the main image can be, for example, 200 times or less. The ratio of the sizes between the screen thumbnail image as the first other image based on the main image and the thumbnail image as the second other image based on the main image can also be, similarly, 200 times or less. As the screen thumbnail image, for example, an image with a resolution of 4K or higher can be adopted. Also, as the screen thumbnail image, for example, a 4K (QFHD) or FHD image can be adopted according to the user's selection. Furthermore, as the main image and the screen thumbnail image, images with the same resolution can be adopted. When adopting images with the same resolution as the main image and the screen thumbnail image, both the main image and the screen thumbnail image can be stored in the HEIF file, or the main image can be stored without storing the screen thumbnail image. When storing the main image without storing the screen thumbnail image in the HEIF file, the main image can be resized and used as the screen thumbnail image.

[0031] Also, the symbol control unit 42 encodes the main image, the screen thumbnail image, and the thumbnail image (the main image, the screen thumbnail image, and the thumbnail image generated from the same RAW image) corresponding to the RAW image as necessary and supplies them to the file control unit 43 together with the RAW image.

[0032] The file control unit 43 generates, as necessary, a RAW file in which a RAW image is stored, a corresponding main image, a screen thumbnail image, and a HEIF file in which a thumbnail image (the main image, screen thumbnail image, and thumbnail image generated from the same RAW image) is stored, and / or a JPEG file, etc., and supplies them to the media control unit 44. The HEIF file is a file compliant with HEIF (High Efficiency Image File Format), and the JPEG file is a file compliant with JPEG (Joint Photographic Experts Group).

[0033] The media control unit 44 records (writes) the RAW file, HEIF file, and JPEG file from the file control unit 43 to the media 14 in accordance with the control from the file control unit 43, or causes them to be transmitted from the I / Fs 15 or 16.

[0034] The type of file (e.g., RAW file, HEIF file, JPEG file, etc.) generated by the file control unit 43 can be selected, for example, according to the operation (specification) of the user. Also, as will be described later, the HEIF file has an image item format and an image sequence format, and which of the image item format and the image sequence format to adopt can be selected, for example, according to the operation of the user. Furthermore, the file control unit 43 can perform mutual conversion between the HEIF file and the JPEG file according to the operation of the user.

[0035] Also, the file control unit 43 can generate a plurality of files with the same image content but different codecs, image sizes (resolutions), color formats, and bit depths.

[0036] When the file control unit 43 generates a plurality of files with the same image content, the encoding control unit 42 generates image streams (file streams) to be stored in each of the plurality of files from the YUV image from the optical system / image sensor control unit 41.

[0037] The encoding control unit 42 can generate image streams with different codecs, image sizes (resolutions), color formats, and bit depths.

[0038] For example, the encoding control unit 42 generates an image with a predetermined size, a predetermined color format, and a predetermined bit depth from the YUV image supplied from the optical system / image sensor control unit 41, and can generate a first stream obtained by encoding the image with a predetermined codec (encoding method). Further, the encoding control unit 42 generates an image with another size, another color format, and another bit depth from the same YUV image supplied from the optical system / image sensor control unit 41, and can generate a second stream obtained by encoding the image with another codec.

[0039] Then, the file control unit 43 can generate a file storing the first stream and a file storing the second stream.

[0040] <JPEG file>

[0041] FIG. 2 is a diagram showing an example of the format of a JPEG file compliant with JPEG (Joint Photographic Experts Group).

[0042] A JPEG file is composed of, for example, Exif (EXIF) (data), a thumbnail image, XMP (Extensible Metadata Platform) (registered trademark) (data), an MPF representing the storage location (position) of the main image and the simple display image, the main image, and the simple display image. As the simple display image, for example, a screen thumbnail image can be adopted.

[0043] <ISO Base Media File Format>

[0044] Figure 3 is a diagram showing an example of the ISO Base Media File Format.

[0045] HEIF (ISO / IEC 23008-12) is a file format compliant with the ISO Base Media File Format (ISO / IEC 14496-12). Therefore, a HEIF file complies with the ISO Base Media File Format.

[0046] The ISO Base Media File Format is composed of units called boxes, which are containers for storing data, and has a structure called a box structure.

[0047] A box has a type, actual data, etc. The type represents the type of actual data in the box. As the actual data, reproducible media data such as images (still images, videos), audio, and subtitles (subtitles), attribute names (field names) and attribute values (field values) of the variables represented by the attribute names, and various other data can be adopted.

[0048] Furthermore, as the actual data, a box can be adopted. That is, a box can have a box as the actual data, thereby enabling a hierarchical structure.

[0049] A base media file compliant with the ISO Base Media File Format can have an ftyp box, a moov box (MovieBox), a meta box (MetaBox), an mdat box (MediaDataBox), etc. The ftyp box stores identification information for identifying the file format. The moov box can store trak boxes, etc. The meta box can store an iinf box, an iprp box, an iref box, an iloc box, etc. The mdat box can store media data (AV data) and any other data.

[0050] HEIF complies with the ISO Base Media File Format as described above.

[0051] <HEIF file>

[0052] FIG. 4 is a diagram showing an example of the format of a HEIF file compliant with HEIF.

[0053] HEIF files are roughly divided into an image item format and an image sequence format. Furthermore, the image item format includes a single image format having only one item to be described later and an image collection format having a plurality of items.

[0054] A HEIF file in the image item format has an ftyp box, a meta box, and an mdat box.

[0055] A HEIF file in the image sequence format has an ftyp box, a moov box, and an mdat box.

[0056] Note that a HEIF file can have not only one of the meta box and the moov box but also both of them.

[0057] The ftyp box stores identification information for identifying the file format, such as that the file is a HEIF file in the image item format or the image sequence format, etc.

[0058] The meta box and the moov box store metadata necessary for playing and managing the media data stored in the mdat box, such as the storage location of the media data, etc.

[0059] The mdat box stores media data (AV data), etc.

[0060] In the digital camera 10, which HEIF file to generate among the HEIF files in the image item format and the image sequence format can be selected according to, for example, the user's operation. Also, when storing an encoded image in the mdat box of the HEIF file, for the image item format, only intra coding is allowed, and for the image sequence format, intra coding and inter coding are allowed. Therefore, for example, when prioritizing fast access to the data stored in the HEIF file, the generation of the HEIF file in the image item format can be selected, and when prioritizing reducing the size (data amount) of the HEIF file, the generation of the HEIF file in the image sequence format can be selected.

[0061] FIG. 5 is a diagram showing an example of the format of a HEIF file in the image item format.

[0062] In a HEIF file in the image item format, information indicating that it is a HEIF file in the image item format, such as mif1, etc., is stored (as an attribute value) in the ftyp box.

[0063] The meta box stores the iinf box, the iref box, the iprp box, and the iloc box.

[0064] The iinf box stores the number (the attribute name and attribute value representing) of items such as media data (AV data) stored in the mdat box. An item is one piece of data stored in the mdat box of a HEIF file in the image item format. For example, one (screen) image is an item. In this specification, regardless of whether it is a still image or a moving image, one image is also referred to as a frame. One frame is one item.

[0065] The iref box stores information representing the relationship between items. For example, in the mdat box, the corresponding main image, screen thumbnail image, and thumbnail image can each be stored as an item. When item I1 as the main image, item I2 as the screen thumbnail image, and item I3 as the thumbnail image are stored in the mdat box, the iref box stores information indicating that item I2 is the screen thumbnail image of the main image as item I1, and information indicating that item I3 is the thumbnail image of the main image as item I1.

[0066] The iprp box stores information regarding the properties of items.

[0067] The iloc box stores information regarding the storage location of items stored in the mdat box.

[0068] The mdat box of the image item format (of the HEIF file) stores, as an item, for example, a frame of an image. The mdat box can store one or more items. Also, the mdat box can store the frame as an item after encoding. However, the encoding of the frame as an item stored in the mdat box of the image item format is limited to intra encoding. As an encoding method (codec) for encoding the frame as an item, for example, HEVC or the like can be adopted.

[0069] FIG. 6 is a diagram showing an example of the iprp box in FIG. 5.

[0070] The iprp box stores an ipco box and an ipma box regarding the properties of items. The ipco box stores the properties of the items stored in the mdat box, for example, codec information regarding the codec of the image as an item and picture size information regarding the size. The ipma box stores an index (pointer) to the properties stored in the ipco box of the items stored in the mdat box.

[0071] FIG. 7 is a diagram showing an example of the format of a HEIF file in image sequence format.

[0072] In a HEIF file in image sequence format, the ftyp box stores information indicating that it is a HEIF file in image sequence format, for example, msf1, etc.

[0073] The moov box stores trak boxes. The trak box stores information regarding the tracks stored in the mdat box.

[0074] A track is composed of one independent piece of media data, such as an image or audio, that is played according to a timeline. For example, a track is composed of one or more frames of images that become elementary streams. Regarding the tracks stored in the mdat box, a plurality of tracks, for example, tracks of images and audio recorded simultaneously, can be played simultaneously.

[0075] The media data of a track is composed of units called samples. A sample is the smallest unit (access unit) when accessing the media data in the HEIF file. Therefore, it is not possible to access the media data in the HEIF file in units finer than a sample.

[0076] Regarding the media data of an image, for example, one frame or the like is one sample. Regarding the media data of audio, for example, one audio frame or the like defined by the standard of the media data of the audio is one sample.

[0077] In the mdat box of the image sequence format (HEIF file), the media data of a track is arranged in units called chunks. A chunk is a set of one or more samples arranged at logically consecutive addresses.

[0078] When a plurality of tracks as media data are stored in the mdat box, the plurality of tracks are interleaved and arranged in units of chunks.

[0079] As described above, the mdat box in the image sequence format stores one or more tracks composed of media data such as images and audio.

[0080] The mdat box can store the encoded frames of the image constituting the track. For the encoding of the frames constituting the track stored in the mdat box in the image sequence format, long GOP is adopted as GOP (Group of Picture), and both intra encoding and inter encoding can be adopted. As the codec for encoding the frames constituting the track, for example, HEVC or the like can be adopted.

[0081] FIG. 8 is a diagram showing an example of a trak box.

[0082] The trak box can store the tkhd box and the mdia box. The tkhd box stores the track header information such as the creation date and time of the track managed by the trak box. The mdia box stores the minf box and the like. The minf box stores the stbl box. The stbl box stores the stsd box, the stsc box, the stsz box, and the stco box, which are information for accessing the samples of the track and thus the chunks. The stsd box stores codec information regarding the codec of the track. The stsc box stores the chunk size (the number of samples in one chunk). The stsz box stores the sample size. The stco box stores the chunk offset, that is, the offset of the placement position of each chunk of the track stored in the mdat box.

[0083] Here, the HEIF file in the image item format is also called a collection file, and the HEIF file in the image sequence format is also called a sequence file.

[0084] The digital camera 10 can generate a HEIF file in which the main image, and further one or both of the necessary screen thumbnail images and the thumbnail images are stored.

[0085] <Collection file>

[0086] FIG. 9 is a diagram showing an example of a collection file in which the main image and the thumbnail image are stored.

[0087] Now, assume that the frames (items) in the mdat box of the collection file are encoded with HEVC and stored.

[0088] The ftyp box stores heic, which indicates that the file format is the image item format and the codec is HEVC, as identification information for identifying the file format.

[0089] The iinf box stores the number of items (number of items) stored in the mdat box. In FIG. 9, a total of 4 items (frames), namely, the main image identified by item ID #1 (hereinafter also referred to as main image Item #1), main image Item #2, the thumbnail image identified by item ID #101 (hereinafter also referred to as thumbnail image Item #101), and thumbnail image Item #102, are stored in the mdat box. Therefore, the number of items is 4. Note that thumbnail image Item #101 is the thumbnail image of main image Item #1, and thumbnail image Item #102 is the thumbnail image of main image Item #2.

[0090] The iinf box further stores, for example, an infe box for each item stored in the mdat box. The infe box registers an item ID for identifying an item and an item type. In FIG. 9, there are infe boxes for main images Item #1 and Item #2, and thumbnail images Item #101 and Item #102, respectively.

[0091] The iref box stores, for example, a thmb box as information for associating the items stored in the mdat box. The thmb box stores the source and destination associated as information for associating a main image and its thumbnail image. In the thmb box, the source represents the item ID of the main image, and the destination represents the item ID of the thumbnail image of the main image specified by the item ID of the source. Therefore, according to the destination associated with the source, the item ID of the thumbnail image of the main image specified by the item ID represented by the source can be recognized. Also, according to the source associated with the destination, the item ID of the main image of the thumbnail image specified by the item ID represented by the destination can be recognized.

[0092] As described with reference to FIG. 6, the iprp box stores the ipco box and the ipma box. As described with reference to FIG. 6, the ipco box stores properties of frames as items stored in the mdat box, such as codec information regarding a codec and picture size information regarding a size. As described with reference to FIG. 6, the ipma box stores indexes to the properties stored in the ipco box of the items stored in the mdat box.

[0093] As described with reference to FIG. 6, the iloc box stores information regarding storage locations of items in the mdat box. In FIG. 9, it is stored in the iloc box that the number of items is 4. Further, in the iloc box, offsets and sizes to the storage locations of the main images Item#1 and Item#2 and the thumbnail images Item#101 and Item#102 stored in the mdat box are stored in association with item IDs.

[0094] <Sequence File>

[0095] FIG. 10 is a diagram showing an example of a sequence file in which a track of a main image and a track of a thumbnail image of the main image are stored.

[0096] Now, assume that frames are encoded with HEVC and stored in the mdat box of the sequence file.

[0097] The ftyp box stores hevc indicating that the file format is an image sequence format and that the codec is HEVC as identification information for identifying the file format.

[0098] The moov box stores trak boxes that manage each of the tracks stored in the mdat box, as described with reference to FIG. 7. In FIG. 10, the track of the main image identified by track ID #1 (hereinafter also referred to as track #1), and track #2 of the thumbnail image of the main image of track #1 are stored in the mdat box. Therefore, the moov box stores a trak box that manages track #1 and a trak box that manages track #2. The n-th thumbnail image (frame) of track #2 (from the beginning) is the thumbnail image of the n-th main image of track #1.

[0099] The sequence file is useful, for example, when continuous shooting is performed with the digital camera 10, and the main images and thumbnail images of a plurality of frames obtained by the continuous shooting are recorded as one track each.

[0100] The tkhd box of the trak box that manages track #1 of the main image stores the track ID #1 that identifies track #1, the frame size of the main image that constitutes track #1, rotation information representing the orientation of the digital camera 10 when the main image was captured, and the creation date and time of track #1. The tkhd box of the trak box that manages track #2 of the thumbnail image stores the track ID #2 that identifies track #2 and the creation date and time of track #2.

[0101] In addition to the tkhd box and mdia box described with reference to FIG. 7, the trak box can store a tref box. The tref box stores a track ID for identifying other tracks related to the track managed by the trak box in which the tref box is stored, and information representing the content of the track. In FIG. 10, a tref box is provided in the trak box that manages track #2. The tref box stores information indicating that another track related to track #2 is track #1 (track_ID = 1) and that the data constituting track #2 is a thumbnail image (i.e., track #2 is a thumbnail image track) (type = thmb).

[0102] In addition to the minf box described with reference to FIG. 8, the mdia box of the trak box can store an hdlr box. The hdlr box stores information representing the type of data that constitutes the track managed by the trak box in which the hdlr box is stored. The hdlr box stored in the mdia box (stored in the trak box) that manages the main image track #1 stores information (pict) indicating that the data constituting track #1 is a picture (frame), and the hdlr box stored in the trak box that manages the thumbnail image track #2 stores information indicating that the data constituting track #2 is a picture.

[0103] The minf box is as described with reference to FIG. 8.

[0104] <Generation and Playback of HEIF Files>

[0105] FIG. 11 is a diagram for explaining a first example of the generation and playback of a HEIF file by the file control unit 43.

[0106] In the following, as an example, the case where a HEIF file in the image item format is adopted as the HEIF file to be generated and played back will be described. However, this technology can also be applied to HEIF files in the image sequence format. When this technology is applied to a HEIF file in the image sequence format, the meta box in the following description is replaced with a moov box. In addition, this technology can be applied not only to HEIF files but also to any file in which related images such as thumbnail images are stored as related images related to the images stored in the file.

[0107] FIG. 11 shows a HEIF file in which image-related data related to the main image of one frame is stored.

[0108] The image-related data related to the main image is a set of data related to the main image that is stored in the mdat box together with the main image. For example, it is a set of the main image main, the screen thumbnail image screen, the thumbnail image thum, and XMP (extensible metadata platform) and EXIF (exchangeable image file format) as metadata. The components such as the main image main that make up the image-related data are also referred to as element data.

[0109] Among the element data that make up the image-related data, the screen thumbnail image screen and the thumbnail image thum are images with the same content as the main image main, although they have a smaller data volume, the number of pixels, and the resolution than the main image main. It can be said that they are related images related to the main image main.

[0110] The file control unit 43 generates a HEIF file in which the main image main, the screen thumbnail image screen, the thumbnail image thum, XMP, and EXIF are arranged (stored) in the order of the main image main, XMP, the screen thumbnail image screen, the thumbnail image thum, and EXIF from the head of the mdat box, as a HEIF file in which image-related data related to the main image of one frame is stored.

[0111] When the file control unit 43 plays back a HEIF file in which the image-related data is arranged as described above, for example, when playing back a thumbnail image, the file control unit 43 reads out and parses the meta box of the HEIF file to identify the positions of the thumbnail image thum, XMP, and EXIF stored in the mdat box. Then, the file control unit 43 seeks the mdat box of the HEIF file (performs operations such as setting a desired read / write position to a file pointer, which is a variable representing the read / write position of the file), and reads out the thumbnail image thum, XMP, and EXIF from the position after seeking.

[0112] In the display control unit 46, the thumbnail image thum read out from the HEIF file, as well as the necessary metadata (such as the imaging date and time) in XMP and EXIF, are displayed as described above.

[0113] In HEIF, the order in which the element data constituting the image-related data is stored in the mdat box is not particularly specified. Therefore, the element data constituting the image-related data can be arranged in the mdat box in any order.

[0114] In this technology, by specifying the arrangement of the element data, in use cases during playback, such as displaying a thumbnail image, displaying a screen thumbnail image, or performing partial transfer where only some types of element data of the image-related data are transferred, the playback of the HEIF file can be efficiently performed, the element data such as the thumbnail image can be quickly acquired, and the high-speed display of the thumbnail image can be achieved.

[0115] FIG. 12 is a diagram for explaining a second example of generation and reproduction of a HEIF file by the file control unit 43.

[0116] Similar to FIG. 11, FIG. 12 shows a HEIF file in which image-related data associated with the main image of one frame is stored.

[0117] In FIG. 12, as the image-related data associated with the main image of one frame, the arrangement order of the main image main, the screen thumbnail image screen, the thumbnail image thum, XMP, and EXIF is defined in the order of XMP, EXIF, thumbnail image thum, main image main, and screen thumbnail image screen.

[0118] Therefore, the file control unit 43 generates, as a HEIF file in which image-related data associated with the main image of one frame is stored, a HEIF file in which main image main, screen thumbnail image screen, thumbnail image thum, XMP, and EXIF are arranged in the order of XMP, EXIF, thumbnail image thum, main image main, and screen thumbnail image screen from the beginning of the mdat box.

[0119] As described above, in FIG. 12, a HEIF file is generated in which XMP and EXIF as metadata, and the thumbnail image thum are arranged in the head portion of the mdat box.

[0120] At the time of reproducing the HEIF file, as preprocessing, the file control unit 43 reads data of a predetermined data amount (for example, 128 KB, 256 KB, etc.) from the head portion of the HEIF file, for example, from the head of the HEIF file to a position exceeding at least the meta box, thereby reading the data of the meta box and the head portion of the mdat box following the meta box, and expanding (storing) it in a built-in memory (not shown) such as a built-in RAM.

[0121] When the image-related data are arranged in the order of XMP, EXIF, thumbnail image thum, main image main, and screen thumbnail image screen from the beginning of the mdat box, in the file control unit 43, the XMP, EXIF, and thumbnail image thum arranged at the beginning part of the mdat box are expanded together with the meta box into the built-in memory by preprocessing.

[0122] Therefore, in the file control unit 43, by parsing the meta box in on-memory and reading out the XMP, EXIF, and thumbnail image thum arranged at the beginning part of the mdat box on the built-in memory according to the result of the parsing, it can be quickly acquired. As a result, the display of the XMP, EXIF, and thumbnail image thum can be performed at high speed, and the reproduction of the HEIF file can be efficiently performed.

[0123] FIG. 13 is a diagram for explaining a third example of the generation and reproduction of a HEIF file by the file control unit 43.

[0124] FIG. 13 shows a HEIF file in which image-related data related to, for example, three frames of main images as a plurality of frames are stored.

[0125] In FIG. 13, similar to the case of FIG. 12, the arrangement order of the element data constituting the image-related data is defined in the order of XMP, EXIF, thumbnail image thum, main image main, and screen thumbnail image screen.

[0126] Therefore, in FIG. 13, the file control unit 43 generates a HEIF file in which a set of the main image main(1), the screen thumbnail image screen(1), the thumbnail image thum(1), XMP(1), and EXIF(1), which are the image-related data of the main image main(1); a set of the main image main(2), the screen thumbnail image screen(2), the thumbnail image thum(2), XMP(2), and EXIF(2), which are the image-related data of the main image main(2); and a set of the main image main(3), the screen thumbnail image screen(3), the thumbnail image thum(3), XMP(3), and EXIF(3), which are the image-related data of the main image main(3) are all arranged in the mdat box in the order of XMP, EXIF, the thumbnail image thum, the main image main, and the screen thumbnail image screen.

[0127] However, in FIG. 13, the image-related data of the main image main(1), the image-related data of the main image main(2), and the image-related data of the main image main(3) are stored together for each piece of image-related data in the order of the image-related data of the main images main(1), main(2), and main(3) from the head of the mdat box.

[0128] Now, assume that the file control unit 43 plays the HEIF file in FIG. 13 without performing the preprocessing described in FIG. 12, for example, by playing a thumbnail image.

[0129] In this case, the file control unit 43 sequentially identifies the positions of the thumbnail images thum(1), thum(2), and thum(3) stored in the mdat box by reading and parsing the meta box of the HEIF file.

[0130] As the parsing of the meta box progresses, when the position of the thumbnail image thum(m) (where m = 1, 2, 3) is identified, the file control unit 43 seeks the mdat box of the HEIF file and reads the thumbnail image thum(m) from the position after seeking.

[0131] In FIG. 13, since the thumbnail image thum(m) is arranged together with other element data of the image-related data of the main image main(m), the thumbnail image thum(m) of a certain main image main(m) and the thumbnail image thum(m+1) of the next main image main(m+1) are arranged at positions far apart. Therefore, when reading the thumbnail image, a seek occurs every time a thumbnail is read, and it takes time to read all the thumbnail images from the HEIF file.

[0132] Next, the file control unit 43 performs the preprocessing described in FIG. 12, and as the reproduction of the HEIF file in FIG. 13, for example, the thumbnail image is reproduced.

[0133] In this case, when reproducing the HEIF file, the file control unit 43 reads data of a predetermined data amount from the head of the HEIF file in the preprocessing, reads the data of the meta box and the head part of the mdat box following the meta box, and expands them in the internal memory.

[0134] In the preprocessing, the meta box and, for example, the XMP(1), EXIF(1), and thumbnail image thum(1) of the main image main(1) at the head part of the mdat box following the meta box are expanded in the internal memory. However, since the XMP(2), EXIF(2), thumbnail image thum(2) of the main image main(2), and the XMP(2), EXIF(3), thumbnail image thum(3) of the main image main(3) are arranged at positions far from the head part of the mdat box, they are not expanded in the internal memory by the preprocessing.

[0135] Therefore, for the thumbnail image thum(1) expanded in the internal memory, the file control unit 43 can read it from the internal memory and obtain it quickly in the same manner as in the case of FIG. 12.

[0136] On the other hand, for the thumbnail images thum(2) and thum(3) that are not expanded in the built-in memory, similar to the case where no preprocessing is performed, it is necessary to seek and read the mdat box of the HEIF file, which takes time.

[0137] FIG. 14 is a diagram for explaining a fourth example of generation and reproduction of a HEIF file by the file control unit 43.

[0138] Similar to FIG. 13, FIG. 14 shows a HEIF file in which image-related data related to three-frame main images as a plurality of frames is stored.

[0139] In FIG. 14, the arrangement order of the element data within the image-related data is not defined, and for all the image-related data stored in the HEIF file, XMP, EXIF, and the thumbnail images thum among the element data of the image-related data are arranged at the head portion of the mdat box.

[0140] Therefore, in FIG. 14, the file control unit 43 generates a HEIF file in which XMP(1), EXIF(1), thumbnail image thum(1), XMP(2), EXIF(2), thumbnail image thum(2), XMP(3), EXIF(3), and thumbnail image thum(3) among the set of the main image main(1), screen thumbnail image screen(1), thumbnail image thum(1), XMP(1), and EXIF(1) which are the image-related data of the main image main(1), the set of the main image main(2), screen thumbnail image screen(2), thumbnail image thum(2), XMP(2), and EXIF(2) which are the image-related data of the main image main(2), and the set of the main image main(3), screen thumbnail image screen(3), thumbnail image thum(3), XMP(3), and EXIF(3) which are the image-related data of the main image main(3) are arranged at the head portion of the mdat box.

[0141] Note that the main image main(1), screen thumbnail image screen(1), main image main(2), screen thumbnail image screen(2), main image main(3), and screen thumbnail image screen(3) are arranged later.

[0142] When playing the HEIF file as described above, the file control unit 43 performs the preprocessing described with reference to FIG. 12.

[0143] In the preprocessing, by reading a predetermined amount of data from the beginning of the HEIF file, the data of the meta box and the beginning part of the mdat box following the meta box are read and expanded into the built-in memory of the file control unit 43.

[0144] As a result, the meta box and, for example, XMP(1), EXIF(1), thumbnail image thum(1), XMP(2), EXIF(2), thumbnail image thum(2), XMP(3), EXIF(3), and thumbnail image thum(3) following the meta box are expanded into the built-in memory.

[0145] Therefore, the file control unit 43 parses the meta box in memory, and according to the result of the parsing, reads the thumbnail images thum(1), thum(2), thum(3), XMP(1), EXIF(1), XMP(2), EXIF(2), XMP(3), and EXIF(3) arranged at the beginning part of the mdat box on the built-in memory, so that the thumbnail images, XMP, and EXIF in the HEIF file can be quickly acquired without seeking the HEIF file. As a result, the display of XMP, EXIF, and the thumbnail image thum can be performed at high speed, and the playback of the HEIF file can be efficiently performed.

[0146] Here, for all the image-related data stored in the HEIF file as shown in FIG. 14, among the element data of the image-related data, at least an image with a smaller number of pixels (lower resolution) than the main image, such as the thumbnail image thum, is placed at the beginning of the mdat box in the HEIF file, and this HEIF file is referred to as a high-speed HEIF file. In a high-speed HEIF file, in addition to all the thumbnail images stored in the high-speed HEIF file, as shown in FIG. 14, all of the EXIF and XMP, which are the metadata of the image, can also be placed at the beginning of the mdat box.

[0147] The HEIF file in FIG. 12 stores image-related data related to the main image of one frame, and among the element data of the image-related data, XMP, EXIF, and the thumbnail image thum are placed at the beginning of the mdat box, so it is a high-speed HEIF file.

[0148] Therefore, whether the HEIF file stores image-related data related to the main image of one frame or image-related data related to the main images of multiple frames, a high-speed HEIF file can be applied. However, when image-related data related to the main images of multiple frames is stored in the HEIF file, seeking occurs frequently, so the effect of adopting a high-speed HEIF file is significant.

[0149] Note that in a high-speed HEIF file, the main image main is not placed at the beginning of the mdat box.

[0150] Also, in a high-speed HEIF file, as an image with fewer pixels than the main image main, together with the thumbnail image thum, or instead of the thumbnail image thum, the screen thumbnail image screen can be arranged at the head part of the mdat box. However, since the data amount of the screen thumbnail image screen is larger than that of the thumbnail image thum, when the screen thumbnail image screen is arranged at the head part of the mdat box, the data amount read in the preprocessing and the memory capacity required for the built-in memory of the file control unit 43 will increase significantly.

[0151] In the preprocessing, as the data amount of the data read from the head of the high-speed HEIF file, for example, a fixed data amount set in advance can be adopted.

[0152] Also, for example, an upper limit of the number of image-related data (the number of main images) stored in the high-speed HEIF file can be set, and the data amount corresponding to the upper limit number can be adopted as the data amount of the data read from the head of the high-speed HEIF file in the preprocessing. For example, for the image-related data of the upper limit number, when the XMP, EXIF, and thumbnail image thum are arranged at the head part of the mdat box, the data amount that can read all of the XMP, EXIF, and thumbnail image thum can be adopted as the data amount of the data read from the head of the high-speed HEIF file in the preprocessing.

[0153] Furthermore, the data amount corresponding to the capacity of the built-in memory of the file control unit 43, for example, the data amount that can be tolerated by the built-in memory, can be adopted as the data amount of the data read from the head of the high-speed HEIF file in the preprocessing.

[0154] Here, for high-speed HEIF files, in preprocessing, by reading and parsing the meta box, it is possible to identify the data volumes of all of XMP, EXIF, and the thumbnail image thum arranged at the beginning of the mdat box. Further, by reading data of the identified data volume from the beginning of the mdat box, all of XMP, EXIF, and the thumbnail image thum arranged at the beginning of the mdat box can be read out and expanded in the built-in memory of the file control unit 43.

[0155] However, in this case, the reading from the beginning of the high-speed HEIF file is performed in two steps: reading the meta box and reading the beginning part of the mdat box. Therefore, in this case, the number of times of reading data from the high-speed HEIF file increases compared to simply reading data of a predetermined data volume from the beginning of the high-speed HEIF file.

[0156] FIG. 15 is a flowchart for explaining an example of the process of generating the high-speed HEIF file of FIG. 14.

[0157] In step S111, the file control unit 43 generates an mdat box in which XMP, EXIF, and the thumbnail image of each main image stored in the HEIF file are arranged at the beginning, and then the main image and the screen thumbnail image are arranged, and the process proceeds to step S112.

[0158] In step S112, the file control unit 43 generates a meta box storing metadata regarding the mdat box (data) generated in step S111, and the process proceeds to step S112.

[0159] In step S113, the file control unit 43 generates a high-speed HEIF file in which the mdat box generated in step S111 is arranged after the meta box generated in step S112, and ends the process.

[0160] FIG. 16 is a flowchart for explaining an example of the process of playing back a high-speed HEIF file in FIG. 14.

[0161] In step S121, as preprocessing, the file control unit 43 reads data of a predetermined data amount (for example, 256 KB or the like) at the head portion of the high-speed HEIF file, thereby reading the meta box and the head portion of the mdat box following the meta box. Further, the file control unit 43 expands the meta box and the head portion of the mdat box read from the high-speed HEIF file in the built-in memory, and the process proceeds from step S121 to step S122.

[0162] In step S122, the file control unit 43 determines, for example, according to a user operation or the like, whether there is a playback request for element data, that is, XMP, EXIF, thumbnail image, main image, or screen thumbnail image.

[0163] In step S122, if it is determined that there is no playback request, the process proceeds to step S127.

[0164] Also, in step S122, if it is determined that there is a playback request, the process proceeds to step S123.

[0165] In step S123, the file control unit 43 parses the meta box expanded in the built-in memory to identify the position of the target element data (hereinafter also referred to as the playback request target) having a playback request in the mdat box, and the process proceeds to step S124.

[0166] In step S124, the file control unit 43 determines whether the playback request target exists (is stored) in the built-in memory according to the position of the playback request target identified by parsing the meta box.

[0167] In step S124, if it is determined that the playback request target exists in the built-in memory, the process proceeds to step S125. In step S125, the file control unit 43 reads the playback request target from the built-in memory, and the process proceeds to step S127.

[0168] Also, in step S124, if it is determined that the playback request target does not exist in the built-in memory, the process proceeds to step S126. In step S126, the file control unit 43 seeks the mdat box of the HEIF file according to the result of parsing, reads the playback request target, and the process proceeds to step S127.

[0169] In step S127, the file control unit 43 determines whether to end the playback of the high-speed HEIF file.

[0170] In step S127, if it is determined not to end the playback of the high-speed HEIF file, the process returns to step S122, and the following similar processes are repeated.

[0171] Also, in step S127, if it is determined to end the playback of the high-speed HEIF file, for example, when the digital camera 10 is operated so that the user ends the playback, the process ends.

[0172] FIG. 17 is a flowchart for explaining an example of a process of playing back all the thumbnail images stored in the high-speed HEIF file of FIG. 14.

[0173] In step S131, as a preprocess, the file control unit 43 reads a predetermined amount of data at the head part of the high-speed HEIF file to read the meta box and the head part of the mdat box following the meta box. Further, the file control unit 43 expands the meta box and the head part of the mdat box read from the high-speed HEIF file in the built-in memory, and the process proceeds from step S131 to step S132.

[0174] In step S132, the file control unit 43 parses the meta box expanded in the built-in memory to identify the position of the mdat box where the thumbnail image exists, and the process proceeds to step S133.

[0175] In step S133, the file control unit 43 determines whether all of the thumbnail images exist in the built-in memory according to the result of parsing the meta box.

[0176] In step S133, if it is determined that all of the thumbnail images exist in the built-in memory, the process proceeds to step S134, and then all of the thumbnail images stored in the built-in memory are reproduced.

[0177] That is, in step S134, the file control unit 43 determines whether there is a thumbnail image that has not been decoded yet in the built-in memory.

[0178] In step S134, if it is determined that there is a thumbnail image that has not been decoded yet in the built-in memory, the process proceeds to step S135.

[0179] In step S135, the file control unit 43 reads out one of the thumbnail images that have not been decoded from the built-in memory, and the process proceeds to step S136.

[0180] In step S136, the file control unit 43 causes the encoding control unit 42 to decode the thumbnail image read out in the previous step S135 and display it on the display control unit 46. Then, the process returns from step S136 to step S134, and hereinafter, the processes of steps S134 to S136 are repeated until it is determined in step S134 that there is no thumbnail image that has not been decoded yet in the built-in memory.

[0181] In step S134, if it is determined that there is no thumbnail image that has not been decoded yet in the built-in memory, the process ends.

[0182] On the other hand, in step S133, if it is determined that at least a part of the thumbnail image does not exist in the built-in memory, the process proceeds to step S137, and hereinafter, all of the thumbnail images are reproduced from the HEIF file.

[0183] That is, in step S137, the file control unit 43 determines whether there is a thumbnail image that has not yet been decoded in the HEIF file.

[0184] In step S137, if it is determined that there is a thumbnail image that has not yet been decoded in the built-in memory, the process proceeds to step S138.

[0185] In step S138, the file control unit 43 seeks from the current position in the HEIF file to the position where the next thumbnail image exists, and the process proceeds to step S139. Note that in the first step S138, the seek is performed from the head of the HEIF file or the head of the mdat box, etc., to the position where the first thumbnail image exists.

[0186] In step S139, the file control unit 43 reads out one thumbnail image from the position after the seek in the HEIF file, and the process proceeds to step S140.

[0187] In step S140, the file control unit 43 causes the encoding control unit 42 to decode the thumbnail image read out in the immediately preceding step S139 and display it on the display control unit 46. Then, the process returns from step S140 to step S137, and hereinafter, the processing of steps S137 to S140 is repeated until it is determined in step S137 that there is no thumbnail image that has not been decoded in the HEIF file.

[0188] In step S140, if it is determined that there is no thumbnail image that has not been decoded in the HEIF file, the process ends.

[0189] FIG. 18 is a diagram for further explaining a fourth example of generation and reproduction of a HEIF file by the file control unit 43.

[0190] Similar to FIG. 14, FIG. 18 shows a HEIF file in which image-related data is stored. However, in FIG. 14, image-related data related to the main images of 3 frames is stored in the HEIF file, while in FIG. 18, image-related data related to the main images of M frames, which is a number of frames exceeding 3 frames, is stored in the HEIF file.

[0191] The HEIF file of FIG. 18 is generated in the same manner as that of FIG. 14.

[0192] When reproducing the HEIF file as described above, the file control unit 43 performs preprocessing in the same manner as when reproducing the HEIF file of FIG. 14.

[0193] In the preprocessing, by reading data of a predetermined data amount from the beginning of the HEIF file, the meta box and the data of the head portion of the mdat box following the meta box are read and expanded in the built-in memory of the file control unit 43.

[0194] In FIG. 18, since image-related data related to the main images of M frames, which is a large number of frames, is stored in the HEIF file, in the preprocessing, it may not be possible to read all of the XMP, EXIF, and thumbnail image thum of the main image of the M frames at the head portion of the mdat box.

[0195] Now, assume that a request is made from the user or the like to reproduce the N-th main image main(N) (or screen thumbnail image screen(N)) among the main images of M frames together with its XMP(N) and EXIF(N).

[0196] In this case, the file control unit 43 parses the meta box expanded in the built-in memory during preprocessing to identify the positions of the main image main(N) (or screen thumbnail image screen(N)), XMP(N), and EXIF(N) in the HEIF file. As a result, it can be recognized that the main image main(N) (or screen thumbnail image screen(N)) and XMP(N) and EXIF(N) are arranged at distant positions in the HEIF file, and that the XMP(N) and EXIF(N) of the main image main(N) are not stored (not read out) in the built-in memory.

[0197] The file control unit 43 seeks from the position where the reading of the HEIF file ended during preprocessing to the position (starting position) where the XMP(N) and EXIF(N) of the main image main(N) are arranged, and reads out the XMP(N) and EXIF(N). Further, the file control unit 43 seeks from the position where the reading of the XMP(N) and EXIF(N) of the HEIF file ended to the position where the main image main(N) is arranged, and reads out the main image main(N).

[0198] Thereafter, the main image main(N) is decoded and displayed together with the XMP(N) and EXIF(N) of the main image main(N) read out earlier.

[0199] As described above, when the set of XMP, EXIF, and thumbnail image thum is arranged at the head part of the mdat box for all of the image-related data of the main image main stored in the HEIF file, when the number of main images main stored in the HEIF file is large, the number of sets of XMP, EXIF, and thumbnail image thum arranged at the head part of the mdat box increases, and in preprocessing, it may happen that some of the sets of XMP, EXIF, and thumbnail image thum arranged at the head part of the mdat box are not read out.

[0200] When it is required to reproduce the main image main(N) corresponding to the XMP(N) and EXIF(N), together with the XMP(N) and EXIF(N) that were not read out during preprocessing, two seeks occur to read the XMP(N) and EXIF(N) from the HEIF file and to read the main image main(N). As a result, it takes time to display the main image main(N) together with the XMP(N) and EXIF(N).

[0201] FIG. 19 is a diagram for explaining a fifth example of generation and reproduction of a HEIF file by the file control unit 43.

[0202] Similar to FIG. 18, FIG. 19 shows a HEIF file in which image-related data related to the main images of M frames, which are a large number of frames, is stored.

[0203] In FIG. 19, similar to FIGS. 14 and 18, the arrangement order of the element data within the image-related data is not defined, and for all the image-related data stored in the HEIF file, XMP(m), EXIF(m), and thumbnail image thum(m) among the element data of the image-related data are arranged at the head portion of the mdat box (m = 1, 2, ···, N, ···, M).

[0204] Furthermore, in FIG. 19, similar to FIGS. 14 and 18, for all the image-related data stored in the HEIF file, the main image main(m) and the screen thumbnail image screen(m) among the element data of the image-related data are arranged after the head portion of the mdat box. The head portion of the mdat box in the case of "after the head portion of the mdat box" means the head portion of the mdat box where XMP, EXIF, and the thumbnail image thum for all the image-related data stored in the HEIF file are arranged.

[0205] However, in FIG. 19, the main image main(m) and the screen thumbnail image screen(m) arranged after the head portion of the mdat box are arranged together with XMP(m) and EXIF(m).

[0206] Therefore, in FIG. 19, XMP(m) and EXIF(m) are arranged overlappingly at the head portion of the mdat box and after the head portion thereof. Note that the thumbnail image thum(m) can also be arranged overlappingly with XMP(m) and EXIF(m).

[0207] As described above, in FIG. 19, the file control unit 43 generates a HEIF file in which XMP(m), EXIF(m), and the thumbnail image thum(m) are arranged at the head portion of the mdat box for all the image-related data related to the main image of the M frame, and XMP(m), EXIF(m), the main image main(m), and the screen thumbnail image screen(m) are arranged together in a combined form after the head portion of the mdat box.

[0208] Note that in FIG. 19, XMP(m), EXIF(m), the main image main(m), and the screen thumbnail image screen(m), which are the element data of the image-related data of the main image main(m) arranged together after the head portion of the mdat box, are arranged in that order, but the arrangement order of the element data is not limited to this.

[0209] Regarding the HEIF file of FIG. 19, when XMP(N), EXIF(N), and the thumbnail image thum(N) are played back, for example, in preprocessing, when XMP(N), EXIF(N), and the thumbnail image thum(N) arranged at the head portion of the mdat box are read out and expanded in the built-in memory, XMP(N), EXIF(N), and the thumbnail image thum(N) are read out from the built-in memory. In this case, the display of XMP(N), EXIF(N), and the thumbnail image thum(N) can be performed at high speed, and the playback of the HEIF file can be performed efficiently.

[0210] Also, in the preprocessing, when XMP(N), EXIF(N), and thumbnail image thum(N) arranged at the head part of the mdat box are not read, XMP(N), EXIF(N), and thumbnail image thum(N) arranged at the head part are read from the HEIF file.

[0211] Here, when a set of XMP(m), EXIF(m), and thumbnail image thum(m) is arranged at the head part of the mdat box for all of the image-related data of the main image main(m) stored in the HEIF file, when the number of main images main(m) stored in the HEIF file is large, the number of sets of XMP(m), EXIF(m), and thumbnail image thum(m) arranged at the head part of the mdat box increases, and in the preprocessing, it may happen that some sets of the sets of XMP(m), EXIF(m), and thumbnail image thum(m) arranged at the head part of the mdat box are not read.

[0212] In the HEIF file of FIG. 18, when XMP(N), EXIF(N), and thumbnail image thum(N) constituting the image-related data of a certain main image main(N) are not read in the preprocessing, as described with reference to FIG. 18, when playing the main image main(N) (or the screen thumbnail image screen(N)) together with XMP(N) and EXIF(N) of the main image main(N), two seeks occur to read XMP(N) and EXIF(N) from the HEIF file and to read the main image main(N).

[0213] On the other hand, in the HEIF file of FIG. 19, XMP(N) and EXIF(N) are arranged not only at the head part of the mdat box but also after the head part of the mdat box in a form combined with the main image main(N) and the like.

[0214] Therefore, for the HEIF file of FIG. 19, in the reading of XMP(N), EXIF(N), and the main image (N), by, for example, reading all of XMP(N), EXIF(N), the main image main(N), and the screen thumbnail image screen(N) that are collectively arranged after the start portion of the mdat box at once, XMP(N), EXIF(N), and the main image (N) can be quickly read out in one seek.

[0215] FIG. 20 is a flowchart for explaining an example of a process of reproducing the main image (or screen thumbnail image), XMP, and EXIF stored in the HEIF file of FIG. 18 or FIG. 19.

[0216] In step S151, as preprocessing, the file control unit 43 reads a predetermined amount of data from the start portion of the high-speed HEIF file to read the meta box and the start portion of the mdat box following the meta box. Further, the file control unit 43 expands the meta box and the start portion of the mdat box read from the high-speed HEIF file in the built-in memory, and the process proceeds from step S151 to step S152.

[0217] In step S152, the file control unit 43 parses the meta box expanded in the built-in memory to identify the position in the mdat box where the desired main image (or screen thumbnail image), for example, the main image requested for reproduction by the user, and the XMP and EXIF of the main image exist, and the process proceeds to step S153.

[0218] In step S153, the file control unit 43 determines whether the desired main image (and screen thumbnail image), XMP, and EXIF are arranged continuously (collectively) in the HEIF file according to the result of parsing the meta box.

[0219] In step S153, if it is determined that the desired main image, XMP, and EXIF are arranged continuously in the HEIF file, that is, the HEIF file is a HEIF file as shown in FIG. 19, and XMP and EXIF are arranged overlappingly at the beginning part of the mdat box of the HEIF file and after that beginning part, the process proceeds to step S154.

[0220] In step S154, the file control unit 43 seeks to the position where the desired main image, XMP, and EXIF are arranged continuously in the HEIF file, that is, the position (the beginning) of the desired main image, XMP, and EXIF arranged after the beginning part of the mdat box, and the process proceeds to step S155.

[0221] In step S155, the file control unit 43 reads out the desired main image (and the screen thumbnail image), XMP, and EXIF arranged together at that position from the position after seeking in the HEIF file, and the process proceeds to step S156.

[0222] In step S156, the file control unit 43 causes the encoding control unit 42 to decode the main image read out in the immediately preceding step S155. Further, the file control unit 43 causes the display control unit 46 to display the decoded main image together with the XMP and EXIF read out in the immediately preceding step S155, and ends the process.

[0223] On the other hand, in step S153, if it is determined that the desired main image, XMP, and EXIF are not arranged continuously in the HEIF file, that is, the HEIF file is a HEIF file as shown in FIG. 18, and XMP and EXIF are arranged only at the beginning part of the mdat box of the HEIF file, the process proceeds to step S157.

[0224] In step S157, the file control unit 43 seeks to the position (the start) of the XMP of the desired main image, which is arranged at the start portion of the mdat box of the HEIF file, and the process proceeds to step S158.

[0225] In step S158, the file control unit 43 reads out the XMP of the desired main image arranged at that position from the position after seeking within the HEIF file, and the process proceeds to step S159.

[0226] In step S159, the file control unit 43 seeks to the position (the start) of the EXIF of the desired main image, which is arranged at the start portion of the mdat box of the HEIF file, and the process proceeds to step S160.

[0227] In step S160, the file control unit 43 reads out the EXIF of the desired main image arranged at that position from the position after seeking within the HEIF file, and the process proceeds to step S161.

[0228] In step S161, the file control unit 43 seeks to the position (the start) of the desired main image (or screen thumbnail image), which is arranged after the start portion of the mdat box of the HEIF file, and the process proceeds to step S162.

[0229] In step S162, the file control unit 43 reads out the desired main image arranged at that position from the position after seeking within the HEIF file, and the process proceeds to step S156.

[0230] In this case, in step S156, the file control unit 43 causes the encoding control unit 42 to decode the main image read out in the immediately preceding step S162. Further, the file control unit 43 causes the display control unit 46 to display the decoded main image together with the XMP and EXIF read out in the immediately preceding steps S158 and S160 respectively, and ends the process.

[0231] <Description of a computer to which the present technology is applied>

[0232] Next, the above-described series of processes can be performed by hardware or by software. When the series of processes is performed by software, the program constituting the software is installed in a general-purpose computer or the like.

[0233] FIG. 21 is a block diagram showing a configuration example of an embodiment of a computer in which a program for executing the above-described series of processes is installed.

[0234] The program can be pre-recorded in a hard disk 905 or a ROM 903 as a recording medium built in the computer.

[0235] Alternatively, the program can be stored (recorded) in a removable recording medium 911 driven by a drive 909. Such a removable recording medium 911 can be provided as so-called package software. Here, examples of the removable recording medium 911 include a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, a semiconductor memory, and the like.

[0236] In addition to installing the program from the removable recording medium 911 as described above into the computer, the program can be downloaded to the computer via a communication network or a broadcast network and installed in the built-in hard disk 905. That is, the program can be wirelessly transferred to the computer from, for example, a download site via an artificial satellite for digital satellite broadcasting, or can be wiredly transferred to the computer via a network such as a LAN (Local Area Network) or the Internet.

[0237] The computer incorporates a CPU (Central Processing Unit) 902, and an input / output interface 910 is connected to the CPU 902 via a bus 901.

[0238] When a command is input by the user operating the input unit 907 etc. via the input / output interface 910, the CPU 902 executes the program stored in the ROM (Read Only Memory) 903 accordingly. Alternatively, the CPU 902 loads and executes the program stored in the hard disk 905 into the RAM (Random Access Memory) 904.

[0239] Thereby, the CPU 902 performs the processing according to the above-described flowchart or the processing performed according to the configuration of the above-described block diagram. And the CPU 902 outputs the processing result from the output unit 906 via, for example, the input / output interface 910 as necessary, or transmits it from the communication unit 908, or further records it on the hard disk 905.

[0240] Note that the input unit 907 is composed of a keyboard, a mouse, a microphone, etc. Also, the output unit 906 is composed of an LCD (Liquid Crystal Display), a speaker, etc.

[0241] Here, in this specification, the processing performed by the computer according to the program does not necessarily have to be performed in time series in the order described as a flowchart. That is, the processing performed by the computer according to the program also includes processing that is executed in parallel or individually (for example, parallel processing or processing by an object).

[0242] Also, the program may be processed by one computer (processor), or may be processed in a distributed manner by a plurality of computers. Furthermore, the program may be transferred to a remote computer and executed.

[0243] Furthermore, in this specification, a system means a collection of a plurality of components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and one device in which a plurality of modules are housed in one housing are both systems.

[0244] Note that the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0245] For example, the present technology can take a configuration of cloud computing in which one function is shared and jointly processed by a plurality of devices via a network.

[0246] In addition, each step described in the above flowchart can be executed by one device or can be shared and executed by a plurality of devices.

[0247] Furthermore, when a plurality of processes are included in one step, the plurality of processes included in that one step can be executed by one device or can be shared and executed by a plurality of devices.

[0248] Also, the effects described in this specification are merely examples and are not limiting, and there may be other effects.

[0249] Note that the present technology can take the following configuration.

[0250] <1> A file processing apparatus comprising a file control unit that generates the HEIF file in which related images related to the images in the HEIF (High Efficiency Image File Format) file are arranged at the head portion of the mdat box. File processing apparatus. <2> The file control unit generates the HEIF file in which a plurality of the related images related to the plurality of the images are arranged at the head portion of the mdat box. The file processing apparatus according to <1>. <3> The file control unit generates the HEIF file in which the metadata of the image is further arranged at the head portion of the mdat box. The file processing apparatus according to <1> or <2>. <4> The file control unit generates the HEIF file in which the metadata is arranged at the head portion of the mdat box and after the head portion of the mdat box. The file processing apparatus according to <3>. <5> The related image is an image having a smaller data amount than the image. The file processing apparatus according to any one of <1> to <4>. <6> The related image is a thumbnail image. The file processing apparatus according to <5>. <7> Generating the HEIF file in which the related image related to the image in the HEIF (High Efficiency Image File Format) file is arranged at the head portion of the mdat box. File processing method. <8> A file control unit that generates the HEIF file in which the related image related to the image in the HEIF (High Efficiency Image File Format) file is arranged at the head portion of the mdat box. A program for causing a computer to function as the file control unit. <9> A file processing apparatus including a file control unit that reads the head portion of the HEIF file in which the related image related to the image in the HEIF (High Efficiency Image File Format) file is arranged at the head portion of the mdat box. File processing apparatus. <10> The file control unit reads the head portion of the HEIF file in which a plurality of the associated images related to the plurality of the images are arranged at the head portion of the mdat box. The file processing apparatus according to <9>. <11> The file control unit reads the head portion of the HEIF file in which the metadata of the image is further arranged at the head portion of the mdat box. The file processing apparatus according to <9> or <10>. <12> The file control unit reads the head portion of the HEIF file in which the metadata is arranged at the head portion of the mdat box and after the head portion of the mdat box. The file processing apparatus according to <11>. <13> The associated image is an image having a smaller data amount than the image. The file processing apparatus according to any one of <9> to <12>. <14> The associated image is a thumbnail image. The file processing apparatus according to <13>. <15> The file control unit reads the box in which the metadata of the data stored in the mdat box is stored in the HEIF file and the head portion of the mdat box following the box. The file processing apparatus according to any one of <9> to <14>. <16> The file control unit reads data of a predetermined data amount from the head portion of the HEIF file. The file processing apparatus according to any one of <9> to <15>. <17> The file control unit reads data of a data amount corresponding to the number of the images stored in the HEIF file from the head of the HEIF file. The file processing apparatus according to <16>. <18> The file control unit reads data in an amount corresponding to the capacity of the memory in which data read from the beginning of the HEIF file is stored, from the beginning of the HEIF file. The file processing apparatus according to <16>. <19> Including reading the head portion of the HEIF file in which related images related to the images in the HEIF (High Efficiency Image File Format) file are arranged in the head portion of the mdat box File processing method. <20> A file control unit that reads the head portion of the HEIF file in which related images related to the images in the HEIF (High Efficiency Image File Format) file are arranged in the head portion of the mdat box As a program for causing a computer to function.

Explanation of Signs

[0251] 10 Digital camera, 11 Optical system, 13 Signal processing unit, 14 Medium, 15, 16 I / F, 17 Button / key, 18 Touch panel, 19 Liquid crystal panel, 20 Viewfinder, 21 I / F, 41 Optical system / image sensor control unit, 42 Encoding control unit, 43 File control unit, 44 Medium control unit, 45 Operation control unit, 46 Display control unit, 47 UI control unit, 901 Bus, 902 CPU, 903 ROM, 904 RAM, 905 Hard disk, 906 Output unit, 907 Input unit, 908 Communication unit, 909 Drive, 910 Input / output interface, 911 Removable recording medium

Claims

1. A file control unit that generates a HEIF (High Efficiency Image File Format) file in which related images related to an image in the HEIF file and metadata of the image are arranged at the head portion of the mdat box. A file processing apparatus.

2. The file control unit generates the HEIF file in which a plurality of the related images related to a plurality of the images are arranged at the head portion of the mdat box. The file processing apparatus according to Claim 1.

3. The file control unit generates the HEIF file in which the metadata is arranged at the head portion of the mdat box and after the head portion of the mdat box. The file processing apparatus according to Claim 1.

4. The related image is an image having a smaller data amount than the image. The file processing apparatus according to Claim 1.

5. The related image is a thumbnail image. The file processing apparatus according to Claim 4.

6. A file processing method including generating a HEIF (High Efficiency Image File Format) file in which related images related to an image in the HEIF file and metadata of the image are arranged at the head portion of the mdat box. A file processing method.

7. A program for causing a computer to function as a file control unit that generates a HEIF (High Efficiency Image File Format) file in which related images related to an image in the HEIF file and metadata of the image are arranged at the head portion of the mdat box.

8. A file control unit that reads the beginning part of the HEIF file in which related images related to the image in the HEIF (High Efficiency Image File Format) file and the metadata of the image are arranged at the beginning part of the mdat box. A file processing apparatus.

9. The file control unit reads the beginning part of the HEIF file in which a plurality of the related images related to the plurality of the images are arranged at the beginning part of the mdat box. The file processing apparatus according to claim 8.

10. The file control unit reads the beginning part of the HEIF file in which the metadata is arranged at the beginning part of the mdat box and after the beginning part of the mdat box. The file processing apparatus according to claim 8.

11. The related image is an image having a smaller data amount than the image. The file processing apparatus according to claim 8.

12. The related image is a thumbnail image. The file processing apparatus according to claim 11.

13. The file control unit reads the box in which the metadata of the data stored in the mdat box of the HEIF file is stored and the beginning part of the mdat box following that box. The file processing apparatus according to claim 8.

14. The file control unit reads data of a predetermined data amount from the beginning part of the HEIF file. The file processing apparatus according to claim 8.

15. The file control unit reads data of a data amount corresponding to the number of the images stored in the HEIF file from the beginning of the HEIF file. The file processing apparatus according to claim 14.

16. The file control unit reads data in an amount corresponding to the capacity of the memory in which data read from the head of the HEIF file is stored, from the head of the HEIF file The file processing apparatus according to claim 14.

17. Reading the head portion of the HEIF file in which related images related to the image in the HEIF (High Efficiency Image File Format) file and the metadata of the image are arranged at the head portion of the mdat box including File processing method.

18. A file control unit that reads the head portion of the HEIF file in which related images related to the image in the HEIF (High Efficiency Image File Format) file and the metadata of the image are arranged at the head portion of the mdat box A program for causing a computer to function as such.

Images