File processing device, file processing method, and program product

Abstract

The present technology relates to a file processing device, a file processing method, and a program that enable efficient playback of HEIF files. A HEIF file is generated in which related images related to the images in the HEIF file are arranged at the beginning of the mdat box. Furthermore, the beginning of the HEIF file is read. The present technology can be applied to the generation and playback of HEIF files.

Description

Technical Field

[0001] The present technology relates to a file processing device, a file processing method, and a program, and particularly to a file processing device, a file processing method, and a program that enable efficient reproduction of HEIF files, for example. Background Art

[0002] As a file format for efficiently storing images, there is High Efficiency Image File Format (HEIF) (see Non-Patent Literature 1).

[0003] Reference List

[0004] Non-patent literature

[0005] Non-patent document 1: ISO / IEC 23008-12:2017, Information technology -- Efficient coding and media delivery in heterogeneous environments -- Part 12: Image file formats Summary of the Invention

[0006] Problems to be solved by the present invention

[0007] In the future, for HEIF files, there will be a need to efficiently reproduce HEIF files, such as displaying thumbnail images at high speed.

[0008] The present technology has been made in view of such circumstances and enables efficient reproduction of HEIF files.

[0009] Solution to the problem

[0010] The first file processing device or first program of the present technology is a file processing device including: a file control unit that generates a HEIF file in which related images associated with images in a high-efficiency image file format HEIF file are arranged at the beginning of an mdat box; or a program for enabling a computer to be used as such a file processing device.

[0011] A first file processing method of the present technology is a file processing method including generating a High Efficiency Image File Format (HEIF) file in which related images associated with images in the HEIF file are arranged at a head portion of an mdat box.

[0012] In the first file processing device, the first file processing method, and the first program of the present technology, a High Efficiency Image File Format HEIF file is generated in which related images associated with images in the HEIF file are arranged at the beginning of an mdat box.

[0013] The second file processing device or second program of the present technology is a file processing device including: a file control unit that reads the beginning part of a HEIF file in which related images associated with images in a high-efficiency image file format HEIF file are arranged at the beginning part of an mdat box; or a program for causing a computer to be used as such a file processing device.

[0014] A second file processing method of the present technology is a file processing method including: reading a leading portion of a High Efficiency Image File Format HEIF file in which related images associated with images in the HEIF file are arranged at a leading portion of an mdat box.

[0015] In the second file processing device, the second file processing method and the second program of the present technology, the beginning part of the HEIF file in which related images associated with images in the HEIF file are arranged at the beginning part of the mdat box is read.

[0016] Note that the file processing device may be an independent device, or may be an internal block included in one device.

[0017] The program can be provided by being recorded on a recording medium or transmitted via a transmission medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a block diagram showing a configuration example of one embodiment of a digital camera to which the present technology is applied.

[0019] Figure 2 is a diagram showing an example of the format of a JPEG file conforming to the Joint Photographic Experts Group (JPEG).

[0020] Figure 3 is a diagram showing an example of the ISO base media file format.

[0021] Figure 4 is a diagram illustrating an example of the format of a HEIF file conforming to HEIF.

[0022] Figure 5 is a diagram showing an example of the format of a HEIF file in the image item format.

[0023] Figure 6 is a diagram showing an example of an iprp box.

[0024] Figure 7 is a diagram showing an example of the format of a HEIF file in the image sequence format.

[0025] Figure 8 is a diagram illustrating an example of a trak box.

[0026] Figure 9 is a diagram illustrating an example of a collection file storing a main image and thumbnail images.

[0027] Figure 10 : is a diagram showing an example of a sequence file storing a track of a main image and a track of thumbnail images of the main image.

[0028] Figure 11 is a diagram showing a first example of generating and reproducing a HEIF file by the file control unit 43 .

[0029] Figure 12 is a diagram showing a second example of generating and reproducing a HEIF file by the file control unit 43 .

[0030] Figure 13 is a diagram showing a third example of generating and reproducing a HEIF file by the file control unit 43 .

[0031] Figure 14 is a diagram illustrating a fourth example of generating and reproducing a HEIF file by the file control unit 43 .

[0032] Figure 15 is a flowchart illustrating an example of a process for generating a high-speed HEIF file.

[0033] Figure 16 is a flowchart illustrating an example of a process for reproducing a high-speed HEIF file.

[0034] Figure 17 is a flowchart illustrating an example of a process of reproducing all thumbnail images stored in a high-speed HEIF file.

[0035] Figure 18 4 is a diagram further illustrating a fourth example of generating and reproducing a HEIF file by the file control unit 43 .

[0036] Figure 19 is a diagram illustrating a fifth example of generating and reproducing a HEIF file by the file control unit 43 .

[0037] Figure 20 : is a flowchart illustrating an example of a process of reproducing a main image, XMP, and EXIF ​​stored in a HEIF file.

[0038] Figure 21 is a block diagram showing a configuration example of one embodiment of a computer to which the present technology is applied. DETAILED DESCRIPTION

[0039] <One embodiment of a digital camera to which the present technology is applied>

[0040] Figure 1 is a block diagram showing a configuration example of one embodiment of a digital camera to which the present technology is applied.

[0041] The digital camera 10 includes an optical system 11 , an image sensor 12 , a signal processing unit 13 , a medium 14 , interfaces (I / F) 15 and 16 , buttons / keys 17 , a touch panel 18 , a liquid crystal panel 19 , a viewfinder 20 , an I / F 21 , and the like.

[0042] The optical system 11 condenses light from a subject onto the image sensor 12 .

[0043] The image sensor 12 receives light from the optical system 11 , performs imaging through photoelectric conversion, generates image data as an electric signal, and supplies the image data to the signal processing unit 13 .

[0044] The signal processing unit 13 includes an optical system / image sensor control unit 41 , an encoding control unit 42 , a file control unit 43 , a medium control unit 44 , an operation control unit 45 , a display control unit 46 , and a UI control unit 47 .

[0045] The optical system / image sensor control unit 41 controls the optical system 11 and the image sensor 12 , and supplies (data of) an image obtained by imaging performed according to the control to the encoding control unit 42 .

[0046] The encoding control unit 42 supplies the image from the optical system / image sensor control unit 41 to the display control unit 46, encodes the image as needed, and supplies the image to the file control unit 43. Furthermore, the encoding control unit 42 decodes the image supplied from the file control unit 43 as needed, and supplies the image to the display control unit 46.

[0047] The file control unit 43 generates a file storing the image supplied from the encoding control unit 42 and supplies the file to the media control unit 44. Furthermore, the file control unit 43 reproduces the file supplied from the media control unit 44, that is, reads data such as the image stored in the file. For example, the file control unit 43 supplies the encoding control unit 42 with the image read from the file.

[0048] The medium control unit 44 controls file exchange with the medium 14 and the I / Fs 15 and 16. For example, the medium control unit 44 causes a file from the file control unit 43 to be recorded on the medium 14 or to be sent from the I / Fs 15 and 16. In addition, the medium control unit 44 reads a file from the medium 14 or causes the I / Fs 15 and 16 to receive a file and provide the file to the file control unit 43.

[0049] The operation control unit 45 supplies an operation signal corresponding to the operation to a necessary block according to the operation of the button / key 17 or the touch panel 18 by the user.

[0050] The display control unit 46 performs display control and the like to supply the image and the like supplied from the encoding control unit 42 to the liquid crystal panel 19 , the viewfinder 20 , and the I / F 21 for display.

[0051] The UI control unit 47 manages user interface (UI) control.

[0052] For example, the medium 14 is a storage medium such as an SD card. For example, the I / F 15 is an I / F of a local area network (LAN) such as Wi-Fi (registered trademark) or Ethernet (registered trademark). For example, the I / F 16 is a universal serial bus (USB) I / F. When a command or other information is input to the digital camera 10, the button / key 17 and the touch panel 18 are operated by the user. The touch panel 18 can be formed integrally with the liquid crystal panel 19. The liquid crystal panel 19 and the viewfinder 20 display an image provided from the display control unit 46, etc. The I / F 21 is an I / F that transmits at least an image, such as a High Definition Multimedia Interface (HDMI) (registered trademark) or a display port (DP).

[0053] In the digital camera 10 configured as described above, the optical system / image sensor control unit 41 generates a YUV image having, for example, the same resolution (number of pixels) (size) as that of the original image from an image of raw data obtained by imaging by the image sensor 12 (hereinafter also referred to as a raw image), and supplies the image together with the raw image to the encoding control unit 42. The encoding control unit 42 generates a main image of a HEIF file or the like from the YUV image from the optical system / image sensor control unit 41. For example, the YUV image from the optical system / image sensor control unit 41 can be directly used as the main image of the HEIF file.

[0054] The encoding control unit 42 generates, from the YUV main image, a YUV image (hereinafter also referred to as a screenshot image), for example, having a lower resolution than the main image, as a first other image based on the main image, for display on the liquid crystal panel 19 or an external display. It also generates, for example, a YUV image (hereinafter also referred to as a thumbnail image), for example, having a lower resolution than the screenshot image, as a second other image based on the main image, for index display (list display). For example, the encoding control unit 42 provides the screenshot image to the liquid crystal panel 19 via the display control unit 46 to display the screenshot image as a so-called through-the-lens image. For example, an image with a size of 320 pixels or less on its long side can be used as the thumbnail image. The size (pixel count) ratio between the main image and the screenshot image (first other image based on the main image) or the thumbnail image (second other image based on the main image) can be, for example, 200 times or less. The ratio of the size between the screenshot image as the first other image based on the main image and the thumbnail image as the second other image based on the main image may also be 200 times or less. For example, as the screenshot image, an image with a resolution of 4K or greater may be adopted. In addition, for example, as the screenshot image, a 4K (QFHD) or FHD image may be adopted according to the user's selection. In addition, images with the same resolution may be adopted as the main image and the screenshot image. In the case of adopting images with the same resolution as the main image and the screenshot image, both the main image and the screenshot image may be stored in the HEIF file, or the main image may be stored without storing the screenshot image. In the case of storing the main image in the HEIF file without storing the screenshot image, the main image may be resized and used as the screenshot image.

[0055] Furthermore, the encoding control unit 42 encodes a main image, screenshot image, and thumbnail image corresponding to the original image (main image, screenshot image, and thumbnail image generated from the same original image) as needed, and supplies them to the file control unit 43 together with the original image.

[0056] For example, the file control unit 43 generates an original file storing an original image, a HEIF file storing corresponding main images, screenshot images, and thumbnail images (main images, screenshot images, and thumbnail images generated from the same original image), and / or a JPEG file as needed, and supplies the generated files to the medium control unit 44. The HEIF file is a file compliant with the High Efficiency Image File Format (HEIF), and the JPEG file is a file compliant with the Joint Photographic Experts Group (JPEG).

[0057] Under the control of the file control unit 43, the medium control unit 44 records (writes) the original file, HEIF file, or JPEG file from the file control unit 43 on the medium 14, or transmits the original file, HEIF file, or JPEG file from the I / F 15 or 16.

[0058] For example, the type of file to be generated by the file control unit 43 (e.g., original file, HEIF file, JPEG file, etc.) can be selected according to the user's operation (specification). In addition, as described later, the HEIF file includes an image item format and an image sequence format. For example, which of the image item format and the image sequence format to adopt can be selected according to the user's operation. In addition, the file control unit 43 can perform mutual conversion between the HEIF file and the JPEG file according to the user's operation.

[0059] In addition, the file control unit 43 can generate multiple files having the same image content and different codecs, image sizes (resolutions), color formats, and bit depths.

[0060] When the file control unit 43 generates multiple files having the same image content, the encoding control unit 42 generates an image stream (file stream) to be stored in each of the multiple files based on the YUV image from the optical system / image sensor control unit 41.

[0061] The encoding control unit 42 can generate image streams having different codecs, image sizes (resolutions), color formats, and bit depths.

[0062] For example, the encoding control unit 42 can generate an image having a predetermined size, a predetermined color format, and a predetermined bit depth based on the YUV image provided from the optical system / image sensor control unit 41, and generate a first stream obtained by encoding the image using a predetermined codec (encoding method). In addition, the encoding control unit 42 can generate an image having another size, another color format, and another bit depth based on the same YUV image provided from the optical system / image sensor control unit 41, and generate a second stream obtained by encoding the image using another codec.

[0063] Then, the file control unit 43 can generate a file storing the first stream and a file storing the second stream.

[0064] <JPEG file>

[0065] Figure 2 is a diagram showing an example of the format of a JPEG file conforming to the Joint Photographic Experts Group (JPEG).

[0066] The JPEG file is configured to store data such as Exif (EXIF), a thumbnail image, data of the Extensible Metadata Platform (XMP) (registered trademark), an MPF indicating the storage locations (positions) of the main image and the simplified display image, and the main image and the simplified display image. For example, as the simplified display image, a screenshot image can be adopted.

[0067] <ISO Base Media File Format>

[0068] Figure 3 is a diagram showing an example of the ISO Base Media File Format.

[0069] HEIF (ISO / IEC 23008-12) is a file format compliant with the ISO Base Media File Format (ISO / IEC 14496-12), so HEIF files comply with the ISO Base Media File Format.

[0070] The ISO Base Media File Format includes units called boxes as containers for storing data and has a structure called a box structure.

[0071] A box includes a type (box type), actual data (data), etc. The type indicates the type of the actual data in the box. As the actual data, reproducible media data (such as images (still images, moving images), audio, and subtitles), attribute names (field names), and attribute values (field values) of attribute names (variables represented by the attribute names), and various other data can be adopted.

[0072] In addition, as the actual data, a box can be adopted. That is, a box can have a box as the actual data, so a hierarchical structure can be formed.

[0073] A base media file compliant with the ISO Base Media File Format can include a ftyp box, a moov box (MovieBox), a meta box (MetaBox), a mdat box (MediaDataBox), etc. The ftyp box stores identification information for identifying the file format. The Moov box can store trak boxes, etc. The meta box can store iinf boxes, iprp boxes, iref boxes, iloc boxes, etc. The mdat box can store media data (AV data) and any other data.

[0074] HEIF complies with the above ISO Base Media File Format.

[0075] <HEIF File>

[0076] Figure 4 [[ID=3

[0077] HEIF files are roughly divided into image item format and image sequence format. In addition, the image item format includes a single image format with only one item and an image collection format with multiple items, which will be described later.

[0078] The HEIF file in the image item format includes an ftyp box, a meta box, and an mdat box.

[0079] The HEIF file in the image sequence format includes an ftyp box, a moov box, and an mdat box.

[0080] Note that a HEIF file may include not only one of the meta box and the moov box, but both of them.

[0081] The ftyp box stores identification information for identifying the file format, such as whether the file is a HEIF file in an image item format or a HEIF file in an image sequence format.

[0082] The meta box and the moov box store metadata required for reproduction, management, etc. of the media data stored in the mdat box, such as metadata of a storage location of the media data.

[0083] The mdat box stores media data (AV data) and the like.

[0084] In the digital camera 10, for example, it is possible to select which of a HEIF file in an image item format and a HEIF file in an image sequence format to generate according to a user's operation. In addition, in the case of encoding an image and storing it in the mdat box of a HEIF file, only intra-frame encoding is allowed for the image item format, and intra-frame encoding and inter-frame encoding are allowed for the image sequence format. Therefore, for example, in the case where high-speed access to data stored in a HEIF file is prioritized, it is possible to select the generation of a HEIF file in an image item format, and in the case where reducing the size (data amount) of the HEIF file is prioritized, it is possible to select the generation of a HEIF file in an image sequence format.

[0085] Figure 5 is a diagram showing an example of the format of a HEIF file in the image item format.

[0086] In a HEIF file in the image item format, for example, the ftyp box stores information indicating that the HEIF file is in the image item format (such as mif1) (as an attribute value).

[0087] The meta box stores the iinf box, the iref box, the iprp box, and the iloc box.

[0088] The iinf box stores (indicates) the number of items (attribute names and attribute values) such as media data (AV data) stored in the mdat box. An item is a piece of data stored in the mdat box of a HEIF file in the image item format, and for example, one image (picture) is an item. In this specification, an image is also referred to as a frame, regardless of whether it is a still image or a moving image. One frame is one item.

[0089] The iref box stores information indicating the association between items. For example, the mdat box may store each of the corresponding main image, screenshot image, and thumbnail image as an item. If the mdat box stores item I1 as the main image, item I2 as the screenshot image, and item I3 as the thumbnail image, the iref box stores information indicating that item I2 is the screenshot image of the main image of item I1, and information indicating that item I3 is the thumbnail image of the main image of item I1.

[0090] The iprp box stores information about the properties of an item.

[0091] The iloc box stores information about the storage location of the items stored in the mdat box.

[0092] For example, the mdat box (of a HEIF file) in the image item format stores frames of an image as items. The mdat box can store one or more items. In addition, the mdat box can encode frames and store them as items. However, it is noted that encoding frames as items stored in the mdat box in the image item format is limited to intra-frame encoding. For example, as an encoding method (codec) for encoding frames as items, HEVC or the like can be used.

[0093] Figure 6 It shows Figure 5 Figure 1 shows an example of an iprp box in .

[0094] The iprp box stores the ipco box and ipma box related to the attributes of the item. The ipco box stores the attributes of the item stored in the mdat box, such as codec information about the codec of the image as the item and image size information about the size of the image as the item. The ipma box stores an index (pointer) from the item stored in the mdat box to the attributes stored in the ipco box.

[0095] Figure 7 is a diagram showing an example of the format of a HEIF file in the image sequence format.

[0096] In a HEIF file in an image sequence format, for example, the ftyp box stores information indicating that the HEIF file is in an image sequence format, such as msf1.

[0097] The moov box stores the trak box, which stores information about the track stored in the mdat box.

[0098] A track is a piece of independent media data, such as an image or sound, that is reproduced according to a timeline. For example, a track includes one or more image frames that form an elementary stream. For tracks stored in an mdat box, multiple tracks can be reproduced simultaneously, such as tracks for simultaneously recorded images and sound.

[0099] The media data of a track is composed of units called samples. A sample is the smallest unit (access unit) when accessing media data in a HEIF file. Therefore, media data in a HEIF file cannot be accessed in units smaller than a sample.

[0100] For example, for image media data, one frame or the like is one sample. Also, for example, for audio media data, one audio frame or the like defined in the audio media data standard is one sample.

[0101] In the mdat box of the image sequence format (HEIF file), the media data of a track is arranged in units called chunks. A chunk is a collection of one or more samples arranged at logically consecutive addresses.

[0102] In the case where the mdat box stores a plurality of tracks as media data, the plurality of tracks are interleaved and arranged in units of blocks.

[0103] As described above, the mdat box of the image sequence format stores one or more tracks of media data including images, sounds, and the like.

[0104] The mdat box can encode and store image frames included in the track. When encoding the frames included in the track stored in the mdat box in the image sequence format, a long group of pictures (GOP) can be used as the GOP, and both intra-frame coding and inter-frame coding can be used. As a codec for encoding the frames included in the track, for example, HEVC can be used.

[0105] Figure 8 is a diagram illustrating an example of a trak box.

[0106] The trak box can store a tkhd box and an mdia box. The tkhd box stores track header information, such as the creation date and time of the track managed by the trak box. The mdia box stores the minf box, etc. The minf box stores the stbl box. The stbl box stores the stsd box, stsc box, stsz box, and stco box, which store samples of the track and therefore store information for accessing blocks. The stsd box stores codec information about the codec of the track. The stsc box stores the block size (the number of samples per block). The stsz box stores the sample size. The stco box stores the block offset, that is, the offset of the layout position of each block of the track stored in the mdat box.

[0107] Here, the HEIF file in the image item format is also referred to as a collection file, and the HEIF file in the image sequence format is also referred to as a sequence file.

[0108] In the digital camera 10 , a HEIF file may be generated, which stores a main image and one or both of a required screenshot image and a thumbnail image.

[0109] <Collection File>

[0110] Figure 9 is a diagram showing an example of a set file storing a main image and thumbnail images.

[0111] Here, it is assumed that the frame (item) is encoded by HEVC and stored in the mdat box of the collection file.

[0112] The ftyp box stores heic as identification information for identifying a file format, the heic indicating that the format is an image item format and the codec is HEVC.

[0113] The iinf box stores the number of items stored in the mdat box. Figure 9 In the mdat box, a total of four items (frames) are stored, including a main image identified by item ID #1 (hereinafter also described as main image item #1), main image item #2, a thumbnail image identified by item ID #101 (hereinafter also described as thumbnail image item #101), and thumbnail image item #102. Therefore, the number of items is four. Note that thumbnail image item #101 is a thumbnail image of main image item #1, and thumbnail image item #102 is a thumbnail image of main image item #2.

[0114] For example, the iinf box also stores an infe box for each item stored in the mdat box. In the infe box, the item ID and item type for identifying the item are registered. Figure 9In the image, there are information boxes of the main image item #1 and the main image item #2 and the thumbnail image item #101 and the thumbnail image item #102.

[0115] For example, the iref box stores the thmb box as information for associating the items stored in the mdat box. The thmb box stores a reference source (referrer) and a reference destination (reference) in association with each other as information for associating a main image with a thumbnail image of the main image. In the thmb box, the reference source represents the item ID of the main image, and the reference destination represents the item ID of the thumbnail image of the main image identified by the item ID of the reference source. Therefore, based on the reference destination associated with the reference source, the item ID of the thumbnail image of the main image identified by the item ID represented by the reference source can be identified. In addition, based on the reference source associated with the reference destination, the item ID of the main image of the thumbnail image specified by the item ID represented by the reference destination can be identified.

[0116] like Figure 6 As described in , the iprp box stores the ipco box and the ipma box. Figure 6 As described in , the ipco box stores attributes of the frame as items stored in the mdat box, such as codec information about the codec and image size information about the size. Figure 6 As described in , the ipma box stores indexes from items stored in the mdat box to attributes stored in the ipco box.

[0117] like Figure 6 As described in , the iloc box stores information about the storage location of the items in the mdat box. Figure 9 , the number of storage items of the iloc box is 4. Furthermore, the iloc box stores the offset and size of the storage location of each of the main image item #1 and main image item #2 and the thumbnail image item #101 and thumbnail image item #102 stored in the mdat box in association with the item ID.

[0118] <sequence file>

[0119] Figure 10 : is a diagram showing an example of a sequence file storing a track of a main image and a track of thumbnail images of the main image.

[0120] Here, it is assumed that the frame is encoded by HEVC and stored in the mdat box of the sequence file.

[0121] The ftyp box stores hevc as identification information for identifying a file format, the hevc indicating that the format is an image sequence format and the codec is HEVC.

[0122] like Figure 7As described in , the moov box stores the trak box, which manages each track stored in the mdat box. Figure 10 In the [ ], the mdat box stores the track of the main image identified by track ID #1 (hereinafter also referred to as track #1) and track #2 of the thumbnail image of the main image of track #1. Therefore, the moov box stores the trak box that manages track #1 and the trak box that manages track #2. The (frame) of the nth thumbnail image (from the beginning) of track #2 is the thumbnail image of the nth main image of track #1.

[0123] For example, when continuous shooting is performed by the digital camera 10 , a sequence file is useful in the case of recording main images and thumbnail images of a plurality of frames obtained by the continuous shooting as one track.

[0124] The tkhd box of the trak box for track #1 managing main images stores track ID #1 identifying track #1, the image size of the main image included in track #1, rotation information indicating the orientation of the digital camera 10 when the main image was captured, and the creation date and time of track #1. The tkhd box of the trak box for track #2 managing thumbnail images stores track ID #2 identifying track #2, and the creation date and time of track #2.

[0125] Apart from Figure 7 In addition to the tkhd box and mdia box described in

[15] , the trak box can also store a tref box. The tref box stores a track ID for identifying another track associated with the track managed by the trak box storing the tref box, information indicating the content of the track, and the like. Figure 10 , the tref box is set in the trak box that manages track #2. Thus, the tref box stores information indicating that another track related to track #2 is track #1 (track #ID=1) and that the data included in track #2 is a thumbnail image (track #2 is a track of thumbnail images) (type=thmb).

[0126] Apart from Figure 8 In addition to the minf box described in [ 15 ], the mdia box of the trak box can also store an hdlr box. The hdlr box stores information indicating the type of data included in the track managed by the trak box storing the hdlr box. The hdlr box stored in the trak box of track 1 that manages the main image (stored in the mdia box) stores information (pict) indicating that the data included in track 1 is a picture (frame), and the hdlr box stored in the trak box of track 2 that manages the thumbnail image stores information indicating that the data included in track 2 is a picture.

[0127] minf box such as Figure 8 As described in.

[0128] <Generation and reproduction of HEIF file>

[0129] Figure 11 It is a diagram showing a first example of generating and reproducing a HEIF file by a file control unit 43.

[0130] Note that although the case where a HEIF file in an image item format is used as the HEIF file to be generated and reproduced will be described as an example below, this technology can also be applied to a HEIF file in an image sequence format. Note that in the case where this technology is applied to a HEIF file in an image sequence format, the meta box in the following description is replaced with a moov box. In addition, in addition to the HEIF file, this technology can also be applied to any file that stores, for example, a thumbnail image or the like as a related image associated with the image stored in the file.

[0131] Figure 11 A HEIF file storing image-related data associated with the main image of one frame is shown.

[0132] The image-related data associated with the main image is a set of data associated with the main image stored in the mdat box, and is, for example, a set of the main image main (primary), screenshot image screen (screen), thumbnail image thum, and extensible metadata platform (XMP) and exchangeable image file format (EXIF) as metadata. Components such as the main image main (primary) included in the image-related data are also called element data.

[0133] Among the element data included in the image-related data, the screenshot image screen (screen) and the thumbnail image thum have a smaller data volume, number of pixels, and resolution than the main image main (primary), but are images having the same content as the main image main (primary), and can be regarded as related images associated with the main image main (primary).

[0134] The file control unit 43 generates a HEIF file in which the main image main (primary), screenshot image screen (screen), thumbnail image thum, XMP, and EXIF are arranged (arranged and stored) in this order from the beginning part of the mdat box as a HEIF file storing image-related data associated with the main image of one frame.

[0135] As a reproduction of the HEIF file in which the image-related data is arranged as described above, for example, in the case of reproducing a thumbnail image, the file control unit 43 reads and parses (analyzes) the meta box of the HEIF file to identify the location of the thumbnail image thum, XMP, and EXIF ​​stored in the mdat box. Then, the file control unit 43 searches the mdat box of the HEIF file (for example, sets the desired read / write location as a file pointer that is a variable indicating the read / write location of the file) and reads the thumbnail image thum, XMP, and EXIF ​​from the searched location.

[0136] As described above, the display control unit 46 displays the thumbnail image thum read from the HEIF file and necessary metadata (such as imaging date and time) in XMP and EXIF.

[0137] HEIF does not specifically specify the order in which the element data included in the image-related data is stored in the mdat box. Therefore, the element data included in the image-related data can be arranged in the mdat box in any order.

[0138] In the present technology, by specifying the arrangement of element data, in usage scenarios during reproduction, such as displaying thumbnail images, displaying screenshot images, performing partial transmission of only certain types of element data in image-related data, etc., HEIF files can be efficiently reproduced, and element data such as thumbnail images can be quickly acquired to perform high-speed display of thumbnail images.

[0139] Figure 12 is a diagram showing a second example of generating and reproducing a HEIF file by the file control unit 43 .

[0140] Similar to Figure 11 , Figure 12 A HEIF file storing image-related data associated with a main image of one frame is shown.

[0141] exist Figure 12 , the arrangement order of the main image main, thumbnail image thum, XMP and EXIF, which are image-related data associated with the main image of one frame, is specified in the order of XMP, EXIF, thumbnail image thum, main image main and thumbnail image screen.

[0142] Therefore, the file control unit 43 generates a HEIF file in which the main image main, the screenshot image screen, the thumbnail image thum, XMP and EXIF ​​are arranged in the order of XMP, EXIF, thumbnail image thum, the main image main and the screenshot image screen from the beginning of the mdat box, as a HEIF file storing image-related data associated with the main image of one frame.

[0143] As mentioned above, in Figure 12 In the above, a HEIF file is generated in which XMP and EXIF ​​as metadata and a thumbnail image thum are arranged at the beginning of the mdat box.

[0144] When reproducing a HEIF file, as preprocessing, the file control unit 43 reads a predetermined amount of data (for example, 128K bytes, 256K bytes, etc.) from the beginning of the HEIF file (such as from the beginning of the HEIF file to at least beyond the meta box) to read the data of the beginning of the meta box and the mdat box after the meta box, and expands (stores) the read data in a built-in memory (not shown) (such as built-in RAM).

[0145] In the case where image-related data is arranged in the order of XMP, EXIF, thumbnail image thum, main image main, and screenshot image screen from the beginning of the mdat box, the file control unit 43 expands the XMP, EXIF, and thumbnail image thum arranged at the beginning of the mdat box together with the meta box in the built-in memory through preprocessing.

[0146] Therefore, the file control unit 43 can parse the metadata box on the storage and read the XMP, EXIF, and thumbnail image Thumb located at the beginning of the mdat box on the built-in storage based on the parsed result, thereby quickly acquiring the XMP, EXIF, and thumbnail image Thumb. As a result, the XMP, EXIF, and thumbnail image Thumb can be displayed at high speed, and HEIF files can be efficiently reproduced.

[0147] Figure 13 is a diagram showing a third example of generating and reproducing a HEIF file by the file control unit 43 .

[0148] Figure 13 An HEIF file storing image-related data associated with a main image that is a plurality of frames, for example, three frames, is shown.

[0149] exist Figure 13 In, similar to Figure 12 In the case of , the arrangement order of element data included in the image-related data is specified in the order of XMP, EXIF, thumbnail image thum, main image main, and screenshot image screen.

[0150] Therefore, in Figure 13 In the embodiment, the file control unit 43 generates the following HEIF file, in which a set of the main image main(main)(1), the screenshot image screen(1), the thumbnail image thum(1), XMP(1) and EXIF(1) as image-related data of the main image main(main)(1), a set of the main image main(main)(2), the screenshot image screen(2), the thumbnail image thum(2), XMP(2) and EXIF(2) as image-related data of the main image main(main)(2), and a set of the main image main(main)(3), the screenshot image screen(3), the thumbnail image thum(3), XMP(3) and EXIF(3) as image-related data of the main image main(main)(3) are each arranged in the order of XMP, EXIF, thumbnail image thum, main image main(main) and screenshot image screen(screen) in the mdat box.

[0151] However, note that in Figure 13 In the mdat box, the image-related data of the main image main(1), the image-related data of the main image main(2) and the image-related data of the main image main(3) are stored collectively for each image-related data in the order of the image-related data of the main image main(1), the main image main(2) and the main image main(3), starting from the beginning of the mdat box.

[0152] Here, it is assumed that the file control unit 43 does not execute Figure 12 The preprocessing described in the example reproduces the thumbnail image, as Figure 13 Reproduction of HEIF files.

[0153] In this case, the file control unit 43 reads and parses a meta box of the HEIF file to sequentially identify the positions of the thumbnail images thum(1), thum(2), and thum(3) stored in the mdat box.

[0154] When the position of the thumbnail image thum(m) (where m=1, 2, 3) is identified through the process of parsing the meta box, the file control unit 43 searches the mdat box of the HEIF file and reads the thumbnail image thum(m) from the searched position.

[0155] exist Figure 13 In the HEIF file, since the thumbnail image thum(m) is arranged together with other element data of the image-related data of the main image main(m), the thumbnail image thum(m) of a certain main image main(m) and the thumbnail image thum(m+1) of the next main image main(m+1) are arranged at positions far apart from each other. Therefore, when reading thumbnail images, a search is performed each time a thumbnail image is read, and it takes time to read all thumbnail images from the HEIF file.

[0156] Next, it is assumed that the file control unit 43 performs a reference Figure 12 The preprocessing described, to reproduce for example thumbnails, as Figure 13 Reproduction of HEIF files.

[0157] In this case, when reproducing the HEIF file, the file control unit 43 reads a predetermined amount of data from the beginning of the HEIF file in preprocessing, and thereby reads the data of the meta box and the beginning part of the mdat box after the meta box, and expands the read data in the built-in memory.

[0158] In pre-processing, the meta box and, for example, the XMP(1), EXIF(1), and thumbnail image thum(1) of the main image main(1) at the beginning of the mdat box following the meta box are expanded in the built-in memory. However, note that since the XMP(2), EXIF(2), and thumbnail image thum(2) of the main image main(2) and the XMP(2), EXIF(3), and thumbnail image thum(3) of the main image main(3) are arranged at a position away from the beginning of the mdat box, they are not expanded in the built-in memory according to the pre-processing.

[0159] Therefore, with Figure 12 Similar to the case of , the file control unit 43 can read the thumbnail image thum(1) expanded in the built-in memory from the built-in memory, and can quickly acquire the thumbnail image thum(1).

[0160] On the other hand, similar to the case where preprocessing is not performed, the file control unit 43 needs to search and read the mdat box of the HEIF file for the thumbnail images thum(2) and thum(3) that are not expanded in the built-in memory, and reading the thumbnail images thum(2) and thum(3) requires time.

[0161] Figure 14 is a diagram illustrating a fourth example of generating and reproducing a HEIF file by the file control unit 43 .

[0162] Similar to Figure 13 , Figure 14 A HEIF file storing image-related data associated with a main image of three frames as a plurality of frames is shown.

[0163] exist Figure 14 In the mdat file, the arrangement order of element data in the image-related data is not specified, and for all image-related data stored in the HEIF file, the XMP, EXIF ​​and thumbnail image thum of the element data of the image-related data are arranged at the beginning of the mdat box.

[0164] Therefore, in Figure 14 In the embodiment, the file control unit 43 generates the following HEIF file, wherein, among the set of the main image main(1), the screenshot image screen(1), the thumbnail image thum(1), XMP(1) and EXIF(1) as the image related data of the main image main(1), the set of the main image main(2), the screenshot image screen(2), the thumbnail image thum(2), XMP(2) and EXIF(2) as the image related data of the main image main(2), and the set of the main image main(3), the screenshot image screen(3), the thumbnail image thum(3), XMP(3) and EXIF(3) as the main image main(3), XMP(1), EXIF(1), the thumbnail image thum(1), XMP(2), EXIF(2), the thumbnail image thum(2), XMP(3), EXIF(3) and the thumbnail image thum(3) are arranged at the beginning of the mdat box.

[0165] Note that the main image main (1), the screenshot image screen (1), the main image main (2), the screenshot image screen (2), the main image main (3), and the screenshot image screen (3) are arranged subsequently.

[0166] When reproducing the HEIF file as described above, the file control unit 43 executes Figure 12 The preprocessing described in .

[0167] In preprocessing, a predetermined amount of data is read from the beginning of the HEIF file so that the data of the beginning portion of the meta box and the mdat box following the meta box are read and expanded in the built-in memory of the file control unit 43 .

[0168] As a result, the meta box and, for example, XMP(1), EXIF(1), thumbnail image thum(1), XMP(2), EXIF(2), thumbnail image thum(2), XMP(3), EXIF(3), and thumbnail image thum(3) at the beginning of the mdat box following the meta box are expanded in the built-in memory.

[0169] Therefore, the file control unit 43 can parse the meta box on the memory and read the thumbnail image thum(1), thumbnail image thum(2), thumbnail image thum(3), XMP(1), EXIF(1), XMP(2), EXIF(2), XMP(3), and EXIF(3) at the beginning of the mdat box arranged on the built-in memory based on the parsed result, thereby quickly acquiring the thumbnail image, XMP, and EXIF ​​in the HEIF file without searching the HEIF file. As a result, XMP, EXIF, and thumbnail image thum can be displayed at high speed, and the HEIF file can be efficiently reproduced.

[0170] Here, about Figure 14 A HEIF file that stores all image-related data as shown in , wherein at least an image (such as a thumbnail image thum) having a smaller number of pixels (lower resolution) than a main image main among the element data of the image-related data is arranged at the beginning of the mdat box is called a high-speed HEIF file. In a high-speed HEIF file, not only all thumbnail images stored in the high-speed HEIF file, but also all EXIF ​​and XMP as image metadata can be arranged at the beginning of the mdat box, as shown in Figure 14 As shown in .

[0171] Figure 12 The HEIF file is a high-speed HEIF file because image-related data related to the main image of one frame is stored, and XMP, EXIF, and thumbnail image thumb in the element data of the image-related data are arranged at the beginning of the mdat box.

[0172] Therefore, regardless of whether the HEIF file stores image-related data associated with a main image of one frame or image-related data associated with main images of multiple frames, high-speed HEIF files can be applied. However, it should be noted that in the case of a HEIF file storing image-related data associated with main images of multiple frames, searches may occur frequently, and therefore, the effect of using high-speed HEIF files is large.

[0173] Note that in a high-speed HEIF file, the main image main is not placed at the beginning of the mdat box.

[0174] In addition, in a high-speed HEIF file, as an image with a smaller number of pixels than the main image main, a screenshot image screen can be placed at the beginning of the mdat box together with or instead of the thumbnail image thum. However, it is noted that since the data amount of the screenshot image screen is larger than the data amount of the thumbnail image thum, when the screenshot image screen is placed at the beginning of the mdat box, the amount of data read in preprocessing and the storage capacity required for the built-in memory of the file control unit 43 become larger.

[0175] For example, in preprocessing, a preset fixed data amount may be adopted as the data amount of data read from the beginning of a high-speed HEIF file.

[0176] In addition, for example, an upper limit may be set for the amount of image-related data (the number of main images) stored in a high-speed HEIF file, and the amount of data corresponding to the upper limit may be adopted as the amount of data to be read from the beginning of the high-speed HEIF file in preprocessing. For example, with respect to the upper limit amount of image-related data, in the case where XMP, EXIF, and thumbnail image thum are arranged at the beginning of the mdat box, the amount of data that can read all of XMP, EXIF, and thumbnail image thum may be adopted as the amount of data to be read from the beginning of the high-speed HEIF file in preprocessing.

[0177] In addition, the amount of data corresponding to the capacity of the built-in memory of the file control unit 43, for example, the amount of data that the built-in memory can allow, can be used as the amount of data read from the beginning of the high-speed HEIF file in preprocessing.

[0178] Here, for high-speed HEIF files, by reading and parsing the meta box in preprocessing, the data amount of all XMP, EXIF, and thumbnail image thum arranged at the beginning of the mdat box can be identified. In addition, by reading the data of the identified data amount from the beginning of the mdat box, all XMP, EXIF, and thumbnail image thum arranged at the beginning of the mdat box can be read and expanded in the built-in memory of the file control unit 43.

[0179] However, note that in this case, reading from the beginning of the high-speed HEIF file is performed twice, including reading the meta box and reading the beginning of the mdat box. Therefore, in this case, the number of times data is read from the high-speed HEIF file increases compared to simply reading a predetermined amount of data from the beginning of the high-speed HEIF file.

[0180] Figure 15 It shows that the generated Figure 14 Flowchart of an example of high-speed HEIF file processing in .

[0181] In step S111, the file control unit 43 generates an mdat box in which the XMP, EXIF, and thumbnail image of each main image stored in the HEIF file are arranged at the beginning, and the main image and screenshot image are arranged thereafter, and the process proceeds to step S112.

[0182] In step S112 , the file control unit 43 generates a meta box storing metadata (of data) about the mdat box generated in step S111 , and the process proceeds to step S112 .

[0183] In step S113 , the file control unit 43 generates a high-speed HEIF file in which the mdat box generated in step S111 is arranged after the meta box generated in step S112 , and ends the processing.

[0184] Figure 16 Is to show the reproduction Figure 14 Flowchart of an example of high-speed HEIF file processing.

[0185] In step S121, as pre-processing, the file control unit 43 reads a predetermined amount of data (e.g., 256 KB, etc.) of data at the beginning of the high-speed HEIF file, and thereby reads the meta box and the beginning of the mdat box following the meta box. Furthermore, the file control unit 43 expands the meta box and the beginning of the mdat box read from the high-speed HEIF file in the built-in memory, and the process proceeds from step S121 to step S122.

[0186] In step S122 , the file control unit 43 determines whether there is a reproduction request for element data (ie, XMP, EXIF, thumbnail image, main image, or screenshot image) according to, for example, a user's operation or the like.

[0187] If it is determined in step S122 that there is no reproduction request, the process proceeds to step S127.

[0188] Furthermore, if it is determined in step S122 that there is a reproduction request, the process proceeds to step S123.

[0189] In step S123, the file control unit 43 parses the meta box expanded in the built-in memory to identify the position of the target element data (hereinafter also referred to as the reproduction request target) for which the reproduction request has been made in the mdat box, and the processing proceeds to step S124.

[0190] In step S124, the file control unit 43 determines whether the reproduction request target exists (is stored) in the built-in memory based on the position of the reproduction request target identified by parsing the meta box.

[0191] If it is determined in step S124 that the reproduction request target exists in the built-in memory, the process proceeds to step S 125. In step S125, the file control unit 43 reads the reproduction request target from the built-in memory, and the process proceeds to step S127.

[0192] If the target of the reproduction request is not present in the built-in memory in step S124, the process proceeds to step S126. In step S126, the file control unit 43 searches the mdat box of the HEIF file according to the parsing result and reads the target of the reproduction request, and the process proceeds to step S127.

[0193] In step S127 , the file control unit 43 determines whether to end the reproduction of the high-speed HEIF file.

[0194] If it is determined in step S127 that the reproduction of the high-speed HEIF file is not to be ended, the process returns to step S122, and similar processing is repeated thereafter.

[0195] Furthermore, if it is determined in step S127 that the reproduction of the high-speed HEIF file is to be ended, such as the case where the user operates the digital camera 10 in order to end the reproduction, the process ends.

[0196] Figure 17 Is to show the reproduction Figure 14 Flowchart of an example of processing all thumbnail images stored in a high-speed HEIF file.

[0197] In step S131, as pre-processing, the file control unit 43 reads a predetermined amount of data from the beginning of the high-speed HEIF file, and thereby reads the meta box and the beginning of the mdat box following the meta box. Furthermore, the file control unit 43 expands the meta box and the beginning of the mdat box read from the high-speed HEIF file in the built-in memory, and the process proceeds from step S131 to step S132.

[0198] In step S132 , the file control unit 43 parses the meta box expanded in the built-in memory to recognize the position in the mdat box where the thumbnail image exists, and the process proceeds to step S133 .

[0199] In step S133 , the file control unit 43 determines whether all thumbnail images exist in the built-in memory based on the parsing result of the meta box.

[0200] If it is determined in step S133 that all thumbnail images exist in the built-in memory, the process proceeds to step S134, and thereafter, all thumbnail images stored in the built-in memory are reproduced.

[0201] That is, in step S134 , the file control unit 43 determines whether an undecoded thumbnail image still exists in the built-in memory.

[0202] If it is determined in step S134 that an undecoded thumbnail image still exists in the built-in memory, the process proceeds to step S135.

[0203] In step S135 , the file control unit 43 reads one of the undecoded thumbnail images from the built-in memory, and the process proceeds to step S136 .

[0204] In step S136, the file control unit 43 causes the encoding control unit 42 to decode the thumbnail image read in the immediately preceding step S135, and causes the display control unit 46 to display the thumbnail image. Then, the process returns from step S136 to step S134, and thereafter, the processes of steps S134 to S136 are repeated until it is determined in step S134 that there is no undecoded thumbnail image in the built-in memory.

[0205] If it is determined in step S134 that there is no undecoded thumbnail image in the built-in memory, the process ends.

[0206] On the other hand, if it is determined in step S133 that at least a part of the thumbnail images does not exist in the built-in memory, the process proceeds to step S137, and all thumbnail images are reproduced from the HEIF file.

[0207] That is, in step S137 , the file control unit 43 determines whether there is an undecoded thumbnail image in the HEIF file.

[0208] If it is determined in step S137 that an undecoded thumbnail image still exists in the built-in memory, the process proceeds to step S138.

[0209] In step S138, the file control unit 43 searches from the current position in the HEIF file to the position where the next thumbnail image exists, and the process proceeds to step S139. Note that in step S138, which is initially performed, the search is performed from the beginning of the HEIF file, the beginning of the mdat box, etc. to the position where the first thumbnail image exists.

[0210] In step S139 , the file control unit 43 reads one of the thumbnail images from the searched position in the HEIF file, and the process proceeds to step S140 .

[0211] In step S140, the file control unit 43 causes the encoding control unit 42 to decode the thumbnail image read in the immediately preceding step S139, and causes the display control unit 46 to display the thumbnail image. The process then returns from step S140 to step S137, and thereafter, the processes of steps S137 to S140 are repeated until it is determined in step S137 that there is no undecoded thumbnail image in the HEIF file.

[0212] If it is determined in step S140 that there is no undecoded thumbnail image in the HEIF file, the process ends.

[0213] Figure 18 4 is a diagram for further describing a fourth example of generating and reproducing a HEIF file by the file control unit 43 .

[0214] Similar to Figure 14 , Figure 18 shows a HEIF file storing image related data. However, note that Figure 14 In HEIF files, the image-related data associated with the main image of three frames is stored, while in Figure 18 In the HEIF file, the image-related data is stored in association with a main image of M frames, which is a large number of frames exceeding three frames.

[0215] Similar to Figure 14 generate Figure 18 HEIF files.

[0216] When reproducing the HEIF file as described above, the file control unit 43 performs the same Figure 14Pre - processing similar to the reproduction of the HEIF file in

[0217] In the pre - processing, a predetermined amount of data is read from the beginning of the HEIF file so that the data of the meta - box and the beginning part of the mdat - box after the meta - box are read and expanded in the built - in memory of the file control unit 43.

[0218] In Figure 18 , since the HEIF file stores image - related data associated with the main images of M frames (which are a large number of frames), it may be impossible to read all of the XMP, EXIF, and thumbnail image thum of the main images of the M frames at the beginning part of the mdat - box in the pre - processing.

[0219] Here, it is assumed that a user or the like requests to reproduce the N - th main image main(primary)(N) (or screenshot image screen(screen)(N)) among the main images of M frames together with XMP(N) and EXIF(N) of the main image main(primary)(N).

[0220] In this case, the file control unit 43 analyzes the meta - box expanded in the built - in memory in the pre - processing to identify the positions of the main image main(primary)(N) (or screenshot image screen(screen)(N)), XMP(N), and EXIF(N) in the HEIF file. As a result, it can be identified that the main image main(primary)(N) (or screenshot image screen(screen)(N)) is arranged far from XMP(N) and EXIF(N) in the HEIF file, and XMP(N) and EXIF(N) of the main image main(primary)(N) are not stored in the built - in memory (not read in the pre - processing).

[0221] The file control unit 43 searches the HEIF file from the position where the reading was completed in the pre - processing to the position (starting position) where XMP(N) and EXIF(N) of the main image main(primary)(N) are arranged, and reads XMP(N) and EXIF(N). In addition, the file control unit 43 searches the HEIF file from the position where the reading of XMP(N) and EXIF(N) was completed to the position where the main image main(primary)(N) is arranged, and reads the main image main(primary)(N).

[0222] Thereafter, the main image main(primary)(N) is decoded and displayed together with XMP(N) and EXIF(N) of the previously read main image main(primary)(N).

[0223] As described above, for all image-related data of the main image stored in the HEIF file, in the case where a set of XMP, EXIF, and thumbnail image thum is arranged at the beginning of the mdat box, when the number of main images main stored in the HEIF file is large, the number of sets of XMP, EXIF, and thumbnail image thum arranged at the beginning of the mdat box increases, and there may be a case where a part of the set of XMP, EXIF, and thumbnail image thum arranged at the beginning of the mdat box is not read during preprocessing.

[0224] Then, when a request is made to reproduce XMP(N) and EXIF(N) that were not read in pre-processing together with the reproduction of the main image (main)(N) corresponding to XMP(N) and EXIF(N), two searches are performed to read XMP(N) and EXIF(N), and the main image (main)(N) is read from the HEIF file. Therefore, it takes time to display the main image (main)(N) together with XMP(N) and EXIF(N).

[0225] Figure 19 is a diagram illustrating a fifth example of generating and reproducing a HEIF file by the file control unit 43 .

[0226] Similar to Figure 18 , Figure 19 A HEIF file is shown storing image-related data associated with a main image of M frames, which is a large number of frames.

[0227] exist Figure 19 In, similar to Figure 14 and Figure 18 , the arrangement order of element data in image-related data is not specified, and for all image-related data stored in the HEIF file, XMP(m), EXIF(m) and thumbnail image thum(m) in the element data of the image-related data are arranged at the beginning of the mdat box (m=1, 2,…, N,…, M).

[0228] In addition, Figure 19 In, similar to Figure 14 and Figure 18 For all image-related data stored in the HEIF file, the main image (main) (m) and the screenshot image (screen) (m) in the element data of the image-related data are arranged after the beginning of the mdat box. The beginning of the mdat box in "after the beginning of the mdat box" refers to the beginning of the mdat box in which the XMP, EXIF, and thumbnail images of all image-related data stored in the HEIF file are arranged.

[0229] However, note that in Figure 19 In , the main image main(m) and the screenshot image screen(m) arranged after the leading portion of the mdat box are collectively arranged together with XMP(m) and EXIF(m).

[0230] Therefore, in Figure 19 In the .mdat file, XMP(m) and EXIF(m) are redundantly arranged at the beginning of the mdat box and after the beginning of the mdat box. Note that the thumbnail image thum(m) may also be redundantly arranged together with XMP(m) and EXIF(m).

[0231] As mentioned above, in Figure 19 , the file control unit 43 generates the following HEIF file, wherein, for all image-related data associated with the main image of M frames, XMP(m), EXIF(m) and the thumbnail image thum(m) are arranged at the beginning of the mdat box, and XMP(m), EXIF(m), the main image main(m) and the screenshot image screen(m) are arranged in a concentrated form after the beginning of the mdat box.

[0232] Note that in Figure 19 In the image, XMP(m), EXIF(m), main image main(m) and screenshot image screen(m) are arranged in sequence, which are element data of image-related data of the main image main(m) centrally arranged after the beginning of the mdat box, but the arrangement order of the element data is not limited to this.

[0233] exist Figure 19 In the case of reproducing XMP(N), EXIF(N), and thumbnail image thum(N) in a HEIF file, for example, XMP(N), EXIF(N), and thumbnail image thum(N) arranged at the beginning of the mdat box are read in preprocessing, and when the XMP(N), EXIF(N), and thumbnail image thum(N) are expanded in the built-in memory, the XMP(N), EXIF(N), and thumbnail image thum(N) are read from the built-in memory. In this case, XMP(N), EXIF(N), and thumbnail image thum(N) can be displayed at high speed, and the HEIF file can be reproduced efficiently.

[0234] In addition, when XMP(N), EXIF(N) and thumbnail image thum(N) arranged at the beginning of the mdat box are not read in preprocessing, XMP(N), EXIF(N) and thumbnail image thum(N) arranged at the beginning of the mdat box are read from the HEIF file.

[0235] Here, for all image-related data of the main image main(m) stored in the HEIF file, in a case where a set of XMP(m), EXIF(m) and thumbnail image thum(m) is arranged at the beginning of the mdat box, when the number of main images main(m) stored in the HEIF file is large, the number of sets of XMP(m), EXIF(m) and thumbnail image thum(m) arranged at the beginning of the mdat box increases, and there may be a case where a part of the set of XMP(m), EXIF(m) and thumbnail image thum(m) arranged at the beginning of the mdat box is not read in preprocessing.

[0236] exist Figure 18 In the case where the XMP(N), EXIF(N) and thumbnail image thum(N) included in the image-related data of a main image main(N) are not read in the preprocessing, as shown in the reference Figure 18 As described above, when the main image main(N) (or the screenshot image screen(N)) is reproduced together with the XMP(N) and EXIF(N) of the main image main(N), two searches are performed to read the XMP(N) and EXIF(N) from the HEIF file and read the main image main(N).

[0237] On the other hand, Figure 19 In the HEIF file, XMP(N) and EXIF(N) are not only arranged at the beginning of the mdat box, but also arranged in a concentrated form after the beginning of the mdat box together with the main image main(N) and the like.

[0238] Therefore, in Figure 19 In a HEIF file, when reading XMP(N), EXIF(N), and main image(N), for example, all XMP(N), EXIF(N), main image main(N), and screenshot image screen(N) arranged collectively after the beginning of the mdat box are read so that XMP(N), EXIF(N), and main image(N) can be quickly read in one search.

[0239] Figure 20 Is to show the reproduction Figure 18 or Figure 19 Flowchart of an example of processing of a main image (or screenshot image), XMP, and EXIF ​​stored in a HEIF file.

[0240] In step S151, as pre-processing, the file control unit 43 reads a predetermined amount of data from the beginning of the high-speed HEIF file, and thereby reads the meta box and the beginning of the mdat box following the meta box. Furthermore, the file control unit 43 expands the meta box and the beginning of the mdat box read from the high-speed HEIF file in the built-in memory, and the process proceeds from step S151 to step S152.

[0241] In step S152, the file control unit 43 parses the meta box expanded in the built-in memory to identify the presence of the desired main image (or screenshot image) (such as the main image requested to be reproduced by the user) and the location of the XMP and EXIF ​​of the main image in the mdat box, and the processing proceeds to step S153.

[0242] In step S153 , the file control unit 43 determines whether a desired main image (and screenshot image), XMP, and EXIF ​​are continuously (collectively) arranged in the HEIF file based on the parsing result of the meta box.

[0243] In step S153, if it is determined that the desired main image, XMP, and EXIF ​​are continuously arranged in the HEIF file, that is, if the HEIF file is as follows Figure 19 , and XMP and EXIF ​​are redundantly arranged at the beginning and after the beginning of the mdat box of the HEIF file, the process proceeds to step S154.

[0244] In step S154, the file control unit 43 searches for the position where the desired main image, XMP and EXIF ​​are continuously arranged in the HEIF file, that is, the position (beginning) of the desired main image, XMP and EXIF ​​arranged after the beginning of the mdat box, and the processing proceeds to step S155.

[0245] In step S155 , at the searched position in the HEIF file, the file control unit 43 reads the desired main image (and screenshot images), XMP, and EXIF ​​collectively arranged at the position, and the process proceeds to step S156 .

[0246] In step S156, the file control unit 43 causes the encoding control unit 42 to decode the main image read in the immediately preceding step S155. Furthermore, the file control unit 43 causes the display control unit 46 to display the decoded main image together with the XMP and EXIF ​​read in the immediately preceding step S155, and ends the process.

[0247] On the other hand, if it is determined in step S153 that the desired main image, XMP, and EXIF ​​are not continuously arranged in the HEIF file, that is, if the HEIF file is as follows Figure 18 , and XMP and EXIF ​​are arranged only in the beginning portion of the mdat box of the HEIF file, the process proceeds to step S157.

[0248] In step S157 , the file control unit 43 searches for the position (the beginning) of the XMP of the desired main image arranged in the beginning portion of the mdat box in the HEIF file, and the process proceeds to step S158 .

[0249] In step S158 , at the searched position in the HEIF file, the file control unit 43 reads the XMP of the desired main image arranged at the position, and the process proceeds to step S159 .

[0250] In step S159 , the file control unit 43 searches for the position (the beginning) of the EXIF ​​of the desired main image arranged at the beginning portion of the mdat box of the HEIF file, and the process proceeds to step S160 .

[0251] In step S160 , at the sought position in the HEIF file, the file control unit 43 reads the EXIF ​​of the desired main image arranged at the position, and the process proceeds to step S161 .

[0252] In step S161 , the file control unit 43 searches for the position (the beginning) of a desired main image (or screenshot image) arranged after the beginning portion of the mdat box in the HEIF file, and the process proceeds to step S162 .

[0253] In step S162 , at the sought-after position in the HEIF file, the file control unit 43 reads the desired main image arranged at the position, and the process proceeds to step S156 .

[0254] In this case, in step S156, the file control unit 43 causes the encoding control unit 42 to decode the main image read in the immediately preceding step S162. Furthermore, the file control unit 43 causes the display control unit 46 to display the decoded main image together with the XMP and EXIF ​​read in the immediately preceding steps S158 and S160, and ends the processing.

[0255] <Description of Computer Using Conventional Technology>

[0256] Next, the above-described series of processing can be executed by hardware or software. In the case of executing the series of processing by software, the program included in the software is installed on a general-purpose computer or the like.

[0257] Figure 21 : is a block diagram showing a configuration example of one embodiment of a computer on which a program for executing the above-described series of processes is installed.

[0258] The program can be recorded in advance in the hard disk 905 or the ROM 903 which is a recording medium built in the computer.

[0259] Alternatively, the program may be stored (recorded) in a removable recording medium 911 driven by the drive 909. Such a removable recording medium 911 may be provided as so-called package software. Examples of the removable recording medium 911 include a floppy disk, a compact disk read-only memory (CD-ROM), a magneto-optical disk (MO) disk, a digital versatile disk (DVD), a magnetic disk, a semiconductor memory, and the like.

[0260] Note that the program can be installed on the computer from the removable recording medium 911 as described above, or can be downloaded to the computer via a communication network or a broadcast network and installed in the built-in hard disk 905. That is, for example, the program can be wirelessly transmitted from a download site to the computer via an artificial satellite used for digital satellite broadcasting, or can be transmitted to the computer in a wired manner via a network such as a local area network (LAN) or the Internet.

[0261] The computer includes a central processing unit (CPU) 902 , and an input / output interface 910 is connected to the CPU 902 through a bus 901 .

[0262] When a command is input by the user by operating the input unit 907 or the like through the input / output interface 910, the CPU 902 executes a program stored in the read-only memory (ROM) 903 according to the command. Alternatively, the CPU 902 loads a program stored in the hard disk 905 into the random access memory (RAM) 904 and executes the program.

[0263] As a result, the CPU 902 executes the processing according to the above-described flowchart or the processing executed by the configuration of the above-described block diagram. Then, the CPU 902 performs control as needed to output the processing result from the output unit 906 through the input / output interface 910 or transmit the processing result from the communication unit 908, or to record the processing result in the hard disk 905, for example.

[0264] Note that the input unit 907 includes a keyboard, a mouse, a microphone, etc. Furthermore, the output unit 906 includes a liquid crystal display (LCD), a speaker, and the like.

[0265] Here, in this specification, the processing executed by the computer according to the program is not necessarily executed in time series in the order described in the flowchart. That is, the processing executed by the computer according to the program also includes processing executed in parallel or individually (for example, parallel processing or object processing).

[0266] In addition, the program may be processed by one computer (processor), or may be processed in a distributed manner by a plurality of computers. In addition, the program may be transferred to a remote computer for execution.

[0267] In addition, in this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), and it does not matter whether all components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, as well as a single device that houses multiple modules in a single housing, are both systems.

[0268] Note that embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present technology.

[0269] For example, the present technology may have a cloud computing configuration in which a function is shared and processed by multiple devices over a network.

[0270] Furthermore, each step described in the above flowchart may be performed by one device, or may be performed by a plurality of devices in a shared manner.

[0271] Furthermore, in the case where a plurality of processes are included in one step, the plurality of processes included in one step may be executed by one device, or may be executed by a plurality of devices in a shared manner.

[0272] Furthermore, the effects described in this specification are merely illustrative and not restrictive, and therefore, other effects may be obtained.

[0273] Note that the present technology can also be configured in the following manner.

[0274] <1>

[0275] A file processing device, comprising:

[0276] A file control unit that generates a High Efficiency Image File Format (HEIF) file in which related images associated with an image in the HEIF file are arranged at a beginning portion of an mdat box.

[0277] <2>

[0278] according to <1> The file processing device, wherein:

[0279] The file control unit generates a HEIF file in which a plurality of related images associated with a plurality of images are arranged at a head portion of an mdat box.

[0280] <3>

[0281] according to <1> or <2> The file processing device, wherein:

[0282] The file control unit generates a HEIF file in which metadata of the image is also arranged at the beginning of the mdat box.

[0283] <4>

[0284] according to <3> The file processing device, wherein:

[0285] The file control unit generates a HEIF file in which metadata is arranged at a leading portion of an mdat box and after the leading portion of the mdat box.

[0286] <5>

[0287] according to <1> to <4> The file processing device as described in any one of the preceding claims, wherein:

[0288] The related image includes an image having a smaller data amount than that of the image.

[0289] <6>

[0290] according to <5> The file processing device, wherein:

[0291] Related images include thumbnail images.

[0292] <7>

[0293] A file processing method, comprising:

[0294] A High Efficiency Image File Format (HEIF) file is generated in which related images associated with an image in the HEIF file are arranged at the beginning of an mdat box.

[0295] <8>

[0296] A program that causes a computer to:

[0297] A file control unit that generates a High Efficiency Image File Format (HEIF) file in which related images associated with an image in the HEIF file are arranged at a beginning portion of an mdat box.

[0298] <9>

[0299] A file processing device, comprising:

[0300] A file control unit that reads a head portion of a High Efficiency Image File Format HEIF file in which related images associated with images in the HEIF file are arranged at a head portion of an mdat box.

[0301] <10>

[0302] according to <9> The file processing device, wherein:

[0303] The file control unit reads a head portion of a HEIF file in which a plurality of related images associated with a plurality of images are arranged at a head portion of an mdat box.

[0304] <11>

[0305] according to <9> or <10> The file processing device, wherein:

[0306] The file control unit reads the beginning portion of the HEIF file in which metadata of the image is also arranged in the beginning portion of the mdat box.

[0307] <12>

[0308] according to <11> The file processing device, wherein:

[0309] The file control unit reads a leading portion of a HEIF file in which metadata is arranged at a leading portion of an mdat box and after the leading portion of the mdat box.

[0310] <13>

[0311] according to <9> to <12> The file processing device as described in any one of the preceding claims, wherein:

[0312] The related image includes an image having a smaller data amount than that of the image.

[0313] <14>

[0314] according to <13> The file processing device, wherein:

[0315] Related images include thumbnail images.

[0316] <15>

[0317] according to <9> to <14> The file processing device as described in any one of the preceding claims, wherein:

[0318] The file control unit reads a box storing metadata of data stored in an mdat box in the HEIF file and a beginning portion of an mdat box following the box.

[0319] <16>

[0320] according to <9> to <15> The file processing device as described in any one of the preceding claims, wherein:

[0321] The file control unit reads a predetermined amount of data from a leading portion of the HEIF file.

[0322] <17>

[0323] according to <16> The file processing device, wherein:

[0324] The file control unit reads data of an amount corresponding to the number of images stored in the HEIF file from the beginning of the HEIF file.

[0325] <18>

[0326] according to <16> The file processing device, wherein:

[0327] The file control unit reads data of an amount corresponding to a capacity of a memory storing the data read from the beginning of the HEIF file from the beginning of the HEIF file.

[0328] <19>

[0329] A file processing method, comprising:

[0330] The head portion of a High Efficiency Image File Format (HEIF) file in which related images associated with an image in the HEIF file are arranged at the head portion of an mdat box is read.

[0331] <20>

[0332] A program that causes a computer to:

[0333] A file control unit that reads a head portion of a High Efficiency Image File Format HEIF file in which related images associated with images in the HEIF file are arranged at a head portion of an mdat box.

[0334] Reference Mark List

[0335] 10. Digital Camera

[0336] 11 Optical System

[0337] 13 Signal Processing Unit

[0338] 14 Medium

[0339] 15, 16 I / F

[0340] 17 buttons / keys

[0341] 18 Touch Panel

[0342] 19 LCD panel

[0343] 20 Viewfinder

[0344] 21 I / F

[0345] 41 Optical system / image sensor control unit

[0346] 42 Encoding Control Unit

[0347] 43 File Control Unit

[0348] 44 Media Control Unit

[0349] 45 Operation control unit

[0350] 46 Display Control Unit

[0351] 47 UI control unit

[0352] 901 Bus

[0353] 902 CPU

[0354] 903 ROM

[0355] 904 RAM

[0356] 905 Hard Drive

[0357] 906 Output Unit

[0358] 907 Input Unit

[0359] 908 Communication Unit

[0360] 909 Driver

[0361] 910 Input / Output Interface

[0362] 911 Removable Recording Media

Claims

1. A file processing device comprising: A file control unit that generates a High Efficiency Image File Format (HEIF) file in which related images associated with images in the HEIF file are arranged at a beginning portion of an mdat box. The file control unit generates a HEIF file in which metadata of an image is arranged at a leading portion of an mdat box and redundantly arranged after the leading portion of the mdat box.

2. The file processing device according to claim 1, wherein The file control unit generates a HEIF file in which a plurality of related images associated with a plurality of images are arranged at a head portion of an mdat box.

3. The file processing device according to claim 1, wherein The related image includes an image having a smaller data amount than that of the image. The file processing device according to claim 3 , wherein: Related images include thumbnail images.

5. A file processing method, comprising: Generate a HEIF file in which related images associated with an image in a High Efficiency Image File Format HEIF file are arranged at the beginning of an mdat box, The file processing method includes generating a HEIF file in which metadata of an image is arranged at the beginning of an mdat box and is redundantly arranged after the beginning of the mdat box.

6. A program product causing a computer to: A file control unit that generates a High Efficiency Image File Format (HEIF) file in which related images associated with images in the HEIF file are arranged at a beginning portion of an mdat box. in, The file control unit generates a HEIF file in which metadata of an image is arranged at a leading portion of an mdat box and is redundantly arranged after the leading portion of the mdat box.

7. A file processing device comprising: A file control unit that reads a head portion of a High Efficiency Image File Format (HEIF) file in which related images associated with images in the HEIF file are arranged in a head portion of an mdat box, The file control unit reads a leading portion of a HEIF file in which metadata of an image is arranged at a leading portion of an mdat box and redundantly arranged after the leading portion of the mdat box.

8. The file processing device according to claim 7, wherein: The file control unit reads a head portion of a HEIF file in which a plurality of related images associated with a plurality of images are arranged at a head portion of an mdat box.

9. The file processing device according to claim 7, wherein: The related image includes an image having a smaller data amount than that of the image.

10. The file processing device according to claim 9, wherein: Related images include thumbnail images.

11. The file processing device according to claim 7, wherein: The file control unit reads a box storing metadata of data stored in an mdat box in the HEIF file and a beginning portion of an mdat box following the box.

12. The file processing device according to claim 7, wherein: The file control unit reads a predetermined amount of data from a leading portion of the HEIF file.

13. The file processing device according to claim 12, wherein: The file control unit reads data of an amount corresponding to the number of images stored in the HEIF file from the beginning of the HEIF file.

14. The file processing device according to claim 12, wherein: The file control unit reads data of an amount corresponding to a capacity of a memory storing the data read from the beginning of the HEIF file from the beginning of the HEIF file.

15. A file processing method, comprising: Reading a head portion of a High Efficiency Image File Format (HEIF) file in which related images associated with an image in the HEIF file are arranged at a head portion of an mdat box, The file processing method includes reading a leading portion of a HEIF file in which metadata of an image is arranged at a leading portion of an mdat box and redundantly arranged after the leading portion of the mdat box.

16. A program product causing a computer to: A file control unit that reads a head portion of a High Efficiency Image File Format (HEIF) file in which related images associated with images in the HEIF file are arranged in a head portion of an mdat box, in, The file control unit reads a leading portion of a HEIF file in which metadata of an image is arranged at a leading portion of an mdat box and is redundantly arranged after the leading portion of the mdat box.

Images