File processing device, file processing method, and program

Abstract

The present technology relates to a file processing device, a file processing method, and a program which enable index display to be rapidly performed. A file control unit generates a proxy file that is another high efficiency image file format (HEIF) file in which a related image, which is related to an image in the HEIF file, is stored. In addition, a file control unit detects an image corresponding to a selection-related image selected from the related image stored in the proxy file. The present technology can be applied to, for example, such as a case where index display for a thumbnail image is performed.

Description

[Technical field]

[0001] The present technology relates to a file processing device, a file processing method, and a program, and in particular to a file processing device, a file processing method, and a program that enable, for example, index display to be performed quickly. [Background technology]

[0002] High Efficiency Image File Format (HEIF) is a file format for efficiently storing images (see Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] ISO / IEC 23008-12:2017, Information technology -- High efficiency coding and media delivery in heterogeneous environments -- Part 12: Image File Format Summary of the Invention [Problem to be solved by the invention]

[0004] It is expected that in the future, there will be a demand for faster indexing of HEIF files that comply with the High Efficiency Image File Format (HEIF).

[0005] The present technology has been made in consideration of such circumstances, and is intended to enable rapid index display. [Means for solving the problem]

[0006] The first file processing device or program of the present technology is A method for generating a HEIF file conforming to the High Efficiency Image File Format (HEIF) in which an image is stored, and A file control unit that generates a proxy file, which is another HEIF file that stores related images related to the image in the The file control unit generates, as the proxy file, a collection file that is a HEIF file in an image item format in which the related images are stored as items, and generates the proxy file in which association information that associates the related images with specific information that identifies the images corresponding to the related images is stored. A file processing device, or a program for causing a computer to function as such a file processing device.

[0007] A first file processing method of the present technology includes: A method for generating a HEIF file conforming to the High Efficiency Image File Format (HEIF) in which an image is stored, and a method for generating a plurality of the HEIF files Generate a proxy file, which is another HEIF file that contains related images related to the image in generating, as the proxy file, a collection file that is a HEIF file in an image item format in which the related images are stored as items; and generating the proxy file in which association information that associates the related images with specific information that identifies the images corresponding to the related images is stored. A file processing method comprising the steps of:

[0008] In the first file processing device, file processing method, and program of the present technology, A HEIF file conforming to the High Efficiency Image File Format (HEIF) is generated, and the HEIF file in which an image is stored is generated. A proxy file is generated, which is another HEIF file that contains related images related to the image in A collection file that is a HEIF file in an image item format in which the related images are stored as items is generated as the proxy file, and the proxy file is generated in which association information that associates the related images with specific information that identifies the images corresponding to the related images is stored. .

[0009] The second file processing device or program of the present technology supports HEIF (High Efficiency Image File Format) HEIF compliant Related images related to the images in the file association information that associates the related image with specific information that identifies the image corresponding to the related image; stored in a proxy file, which is another HEIF file that contains From among the plurality of related images The image corresponding to the selected related image is The image is identified based on the identification information associated with the selected related image, and the identified image is regarded as the image corresponding to the selected related image. Equipped with a file control unit that detects The proxy file is a HEIF file in image item format in which the related image is stored as an item. A file processing device, or a program for causing a computer to function as such a file processing device.

[0010] The second file processing method of this technology is HEIF (High Efficiency Image File Format) HEIF compliant Related images related to the images in the file association information that associates the related image with specific information that identifies the image corresponding to the related image; stored in a proxy file, which is another HEIF file that contains From among the plurality of related images The image corresponding to the selected related image is identifying the selected related image based on the identification information associated with the selected related image; and determining the identified image as the image corresponding to the selected related image. Detect and wherein the proxy file is a HEIF file in an image item format in which the related image is stored as an item.A file processing method.

[0011] In the second file processing device, file processing method, and program of the present technology, HEIF (High Efficiency Image File Format) HEIF compliant Related images related to the images in the file association information that associates the related image with specific information that identifies the image corresponding to the related image; stored in a proxy file, which is another HEIF file that contains From among the plurality of related images The image corresponding to the selected related image is The image is identified by the identification information associated with the selected related image, and the identified image is the image corresponding to the selected related image. It is detected. In addition, the proxy file is a HEIF file in image item format in which the related images are stored as items.

[0012] The first and second file processing devices may be independent devices, or may be internal blocks constituting a single device.

[0013] In addition, the first and second programs can be provided by being recorded on a recording medium or transmitted via a transmission medium. [Brief description of the drawings]

[0014] [Figure 1] 1 is a block diagram showing an example of the configuration of an embodiment of a digital camera to which the present technology is applied. [Diagram 2] FIG. 1 is a diagram showing an example of a JPEG file format that complies with JPEG (Joint Photographic Experts Group). [Diagram 3] FIG. 1 is a diagram illustrating an example of an ISO base media file format. [Figure 4] FIG. 2 is a diagram showing an example of a format of a HEIF file that complies with HEIF. [Diagram 5] FIG. 1 is a diagram showing an example of the format of a HEIF file in image item format. [Figure 6] FIG. 13 is a diagram showing an example of an iprp box. [Figure 7] FIG. 1 is a diagram showing an example of the format of a HEIF file in image sequence format. [Figure 8]FIG. 1 is a diagram showing an example of a trak box. [Figure 9] FIG. 13 is a diagram showing an example of a normal collection file in which main images and thumbnail images are stored. [Figure 10] FIG. 13 is a diagram showing an example of a first association-type collection file. [Figure 11] FIG. 13 is a diagram showing an example of a second association-type collection file. [Figure 12] FIG. 13 is a diagram showing an example of a third association-type collection file. [Figure 13] FIG. 13 is a diagram showing an example of a normal sequence file in which a track of main images and a track of thumbnail images of the main images are stored. [Figure 14] FIG. 13 is a diagram showing an example of an associated sequence file. [Figure 15] 11 is a flowchart outlining an example of a generation process for generating an associated HEIF file. [Figure 16] 11 is a flowchart outlining an example of a playback process for playing an associated HEIF file. [Figure 17] 11 is a flowchart illustrating an example of a reproduction process for reproducing a collection file. [Figure 18] 13 is a flowchart illustrating an example of a process for reading an image to be reproduced in step S32. [Figure 19] 13 is a flowchart illustrating a first example of a process for obtaining a playback target item ID in step S31. [Figure 20] 13 is a flowchart illustrating a second example of the process of acquiring a playback target item ID in step S31. [Figure 21] 13 is a flowchart illustrating an example of a process for acquiring a uuid as identification information of a RAW file of a predetermined main image from a first associative collection file. [Figure 22] 13 is a flowchart illustrating an example of a process for acquiring a uuid as identification information of a RAW file of a predetermined main image from a second associative collection file. [Figure 23] 13 is a flowchart illustrating an example of a process for acquiring a uuid as identification information of a RAW file of a predetermined main image from a third associative collection file. [Figure 24] 13 is a flowchart illustrating an example of a process for acquiring a list of item IDs of main images from a collection file. [Diagram 25] 11 is a flowchart illustrating an example of a process for reproducing a thumbnail image of a main image (a frame) corresponding to predetermined time information from a sequence file. [Figure 26] 13 is a flowchart illustrating an example of a process for acquiring a uuid as identification information of a RAW file of a predetermined main image (frame) from an associative sequence file. [Figure 27] FIG. 13 is a diagram showing an example of storing a uuid in a RAW file when a RAW file of a main image is used as external data and an associated collection file is generated. [Figure 28] FIG. 13 is a diagram showing an example of storing a uuid in a RAW file when a RAW file of a main image is used as external data and an associated sequence file is generated. [Figure 29] FIG. 13 is a diagram showing an example of storing a uuid in a WAV file when a WAV file of a main image is used as external data and an associated collection file is generated. [Diagram 30] 13 is a diagram showing an example of storing a uuid in a WAV file when a WAV file of a main image is used as external data and an associated sequence file is generated. FIG. [Diagram 31] 1A to 1C are diagrams illustrating an example of index display on the digital camera 10. [Diagram 32] 13 is a diagram illustrating an example of index display in an external device other than the digital camera 10. FIG. [Diagram 33] FIG. 13 is a diagram showing an example of an index display for a first associative collection file. [Diagram 34]FIG. 13 is a diagram showing another example of an index display targeting the first associative collection file. [Diagram 35] FIG. 13 is a diagram showing an example of an index display for an associated sequence file. [Diagram 36] FIG. 11 is a diagram illustrating a first example of a proxy file. [Figure 37] FIG. 11 is a diagram illustrating a second example of a proxy file. [Figure 38] FIG. 13 is a diagram illustrating a third example of a proxy file. [Figure 39] FIG. 13 is a diagram illustrating a fourth example of a proxy file. [Diagram 40] 13 is a diagram illustrating an example of writing uuid as specific information of a main image as external data to sequence file #1 as a proxy target file. FIG. [Diagram 41] 13 is a diagram illustrating an example of generation of a proxy file in which a first association-type collection file is a proxy target file. FIG. [Diagram 42] 13 is a diagram illustrating another example of generation of a proxy file in which a first association-type collection file is a proxy target file. FIG. [Diagram 43] 11A and 11B are diagrams illustrating an example of generation of a proxy file in which an associated sequence file is a proxy target file. [Diagram 44] 11 is a diagram for explaining the form taken by a pair of a proxy target file and a proxy file generated by a proxy file generation process using the proxy target file; FIG. [Diagram 45] FIG. 13 is a diagram showing a detailed example of a second form taken by a pair of a proxy target file and a proxy file. [Diagram 46] FIG. 13 is a diagram showing a detailed example of a third form taken by a pair of a proxy target file and a proxy file. [Figure 47] 1A and 1B are diagrams illustrating an example of index display using a proxy file in the digital camera 10. [Figure 48]13 is a diagram illustrating an example of index display using a proxy file in an external device other than the digital camera 10. FIG. [Figure 49] 13A to 13C are diagrams illustrating an example of selection of an HEIF file to be a proxy target file. [Figure 50] 13A to 13C are diagrams illustrating another example of selection of an HEIF file to be a proxy target file. [Figure 51] 11A and 11B are diagrams illustrating a proxy file generation process for generating a proxy file of a first type, and an addition process for adding a thumbnail image to the proxy file. [Figure 52] 11 is a flowchart illustrating an example of a proxy file generation process for generating a proxy file in a first embodiment. [Diagram 53] 13 is a flowchart illustrating an example of an adding process for adding a thumbnail image to a proxy file in the first form. [Figure 54] 13A and 13B are diagrams illustrating a proxy file generation process for generating a proxy file of the second or third type, and an addition process for adding a thumbnail image to the proxy file. [Figure 55] 13 is a flowchart illustrating an example of a proxy file generation process for generating a proxy file of the second or third type. [Figure 56] 13 is a flowchart illustrating an example of an adding process for adding a thumbnail image to a proxy file in the second or third form. [Figure 57] 13A and 13B are diagrams illustrating an example of a detection process for detecting a main image corresponding to a thumbnail image stored in a proxy file of a second form. [Figure 58] 13A and 13B are diagrams further illustrating an example of a detection process for detecting a main image corresponding to a thumbnail image stored in a proxy file of the second form. [Figure 59] 13 is a flowchart illustrating an example of a detection process for detecting a main image corresponding to a thumbnail image stored in a proxy file in a second form. [Figure 60]13A and 13B are diagrams illustrating an example of a detection process for detecting a main image corresponding to a thumbnail image stored in a proxy file of a third form. [Figure 61] FIG. 13 is a diagram further illustrating an example of a detection process for detecting a main image corresponding to a thumbnail image stored in a proxy file of the third form. [Figure 62] FIG. 13 is a diagram illustrating a first example of a detection process for detecting a main image corresponding to a thumbnail image stored in a proxy file of a third form. [Figure 63] 13 is a flowchart illustrating a first example of a detection process for detecting a main image corresponding to a thumbnail image stored in a proxy file in a third form. [Figure 64] FIG. 13 is a diagram illustrating a second example of the detection process for detecting a main image corresponding to a thumbnail image stored in a proxy file of the third form. [Figure 65] FIG. 13 is a diagram illustrating a third example of a detection process for detecting a main image corresponding to a thumbnail image stored in a proxy file of a third form. [Figure 66] FIG. 13 is a diagram illustrating a fourth example of the detection process for detecting a main image corresponding to a thumbnail image stored in a proxy file of the third form. [Figure 67] 13A and 13B are diagrams for explaining a proxy file generation process and an addition process in the case where a thumbnail image to be stored in a proxy file of the first type is generated when a proxy target file is generated. [Figure 68] 13 is a flowchart illustrating an example of a proxy file generation process in which a thumbnail image to be stored in a proxy file of the first type is generated when a proxy target file is generated. [Figure 69] 13 is a flowchart illustrating an example of additional processing in the case where a thumbnail image to be stored in a proxy file of the first type is generated when a proxy target file is generated. [Figure 70] 13A and 13B are diagrams illustrating a proxy file generation process and an addition process in the case where a thumbnail image to be stored in a proxy file of a third type is generated when a proxy target file is generated. [Figure 71] 13 is a flowchart illustrating an example of a proxy file generation process in which a thumbnail image to be stored in a proxy file of a third type is generated when a proxy target file is generated. [Figure 72] 13 is a flowchart illustrating an example of additional processing in the case where a thumbnail image to be stored in a proxy file of a third type is generated when a proxy target file is generated. [Figure 73] 13 is a flowchart illustrating an example of a proxy file generation process in which an already-existing HEIF file is a proxy target file. [Figure 74] 13 is a flowchart illustrating an example of adding a pre-existing HEIF file as a proxy target file. [Figure 75] 1 is a block diagram showing an example of the configuration of an embodiment of a computer to which the present technology is applied. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] <One embodiment of a digital camera to which the present technology is applied>

[0016] FIG. 1 is a block diagram showing an example of the configuration of an embodiment of a digital camera to which the present technology is applied.

[0017] The digital camera 10 has an optical system 11, an image sensor 12, a signal processing unit 13, a medium 14, interfaces 15 and 16, buttons / keys 17, a touch panel 18, a liquid crystal panel 19, a viewfinder 20, and an interface 21, etc.

[0018] The optical system 11 collects light from a subject onto the image sensor 12 .

[0019] The image sensor 12 receives light from the optical system 11 and performs photoelectric conversion to capture an image, thereby generating image data as an electrical signal, and supplies the data to a signal processing unit 13 .

[0020] The signal processing unit 13 has an optical system / image sensor control unit 41, an encoding control unit 42, a file control unit 43, a media control unit 44, an operation control unit 45, a display control unit 46, and a UI control unit 47.

[0021] The optical system / image sensor control unit 41 controls the optical system 11 and the image sensor 12, and supplies the image (data) obtained by imaging performed under that control to the encoding control unit .

[0022] The encoding control unit 42 supplies the image from the optical system / image sensor control unit 41 to the display control unit 46, and also encodes the image as necessary and supplies it to the file control unit 43. The encoding control unit 42 also decodes the image supplied from the file control unit 43 as necessary and supplies it to the display control unit 46.

[0023] The file control unit 43 generates a file storing the image supplied from the encoding control unit 42, and supplies the file to the media control unit 44. The file control unit 43 also plays the file supplied from the media control unit 44, i.e., reads data such as images stored in the file. For example, an image read from a file is supplied from the file control unit 43 to the encoding control unit 42.

[0024] The media control unit 44 controls the exchange of files between the medium 14 and the interfaces 15 and 16. For example, the media control unit 44 causes a file from the file control unit 43 to be recorded on the medium 14 or transmitted from the interfaces 15 and 16. The media control unit 44 also reads a file from the medium 14 or causes the interfaces 15 and 16 to receive a file and supply it to the file control unit 43.

[0025] In response to a user's operation of the buttons / keys 17 or the touch panel 18, the operation control unit 45 supplies an operation signal corresponding to the operation to a necessary block.

[0026] The display control unit 46 performs display control and the like to supply the images and the like supplied from the encoding control unit 42 to the liquid crystal panel 19, the viewfinder 20, and the interface 21 for display.

[0027] The UI control unit 47 is in charge of UI (User Interface) control.

[0028] The medium 14 is, for example, a storage medium such as an SD card. The interface 15 is, for example, a LAN (Local Area Network) interface such as WiFi (registered trademark) or Ethernet (registered trademark). The interface 16 is, for example, a USB (Universal Serial Bus) interface. The buttons / keys 17 and the touch panel 18 are operated by a user when inputting commands or other information to the digital camera 10. The touch panel 18 can be configured integrally with a liquid crystal panel 19. The liquid crystal panel 19 and the viewfinder 20 display images and the like supplied from the display control unit 46. The interface 21 is an interface that transmits at least images, such as HDMI (High-Definition Multimedia Interface) (registered trademark) or DP (Display Port).

[0029] In the digital camera 10 configured as described above, the optical system / image sensor control unit 41 generates, from the RAW data image (hereinafter also referred to as the RAW image) obtained by imaging with the image sensor 12, for example, a YUV image having the same resolution (number of pixels) as the RAW image, as the master image of the HEIF file, and supplies this to the encoding control unit 42.

[0030] The encoding control unit 42 generates, from the YUV main image, a YUV image (hereinafter also referred to as a screen nail image) having a lower resolution than the main image as a first other image based on the main image for display on the liquid crystal panel 19 or an external display, and generates, for example, a YUV image (hereinafter also referred to as a thumbnail image) having a lower resolution than the screen nail image as a second other image based on the main image for index display (list display). The encoding control unit 42 supplies, for example, the screen nail image to the liquid crystal panel 19 via the display control unit 46, and displays it as a so-called through image. As the thumbnail image, for example, an image with a long side of 320 pixels or less can be used. The ratio of the size (number of pixels) between the main image and the screen nail image as the first other image based on the main image, or the thumbnail image as the second other image based on the main image can be, for example, 200 times or less. Similarly, the ratio of the size between the screen nail image as the first other image based on the main image and the thumbnail image as the second other image based on the main image can be 200 times or less. As the screen nail image, for example, an image with a resolution of 4K or more can be adopted. Furthermore, as the screen nail image, for example, an image with a resolution of 4K (QFHD) or FHD can be adopted depending on the user's selection. Furthermore, images with the same resolution can be adopted as the main image and the screen nail image. When images with the same resolution are adopted as the main image and the screen nail image, the HEIF file can store both the main image and the screen nail image, or the main image can be stored without storing the screen nail image. When the main image is stored without storing the screen nail image in the HEIF file, the main image can be resized and used as the screen nail image.

[0031] In addition, the encoding control unit 42 encodes the main image, screennail image, and thumbnail image corresponding to the RAW image (the main image, screennail image, and thumbnail image generated from the same RAW image) as necessary, and supplies them to the file control unit 43 together with the RAW image.

[0032] The file control unit 43 generates a RAW file in which a RAW image is stored, and also generates a HEIF file or a JPEG file in which a corresponding main image, screennail image, and thumbnail image (a main image, screennail image, and thumbnail image generated from the same RAW image) are stored, and supplies the files to the media control unit 44. A HEIF file is a file that conforms to HEIF (High Efficiency Image File Format), and a JPEG file is a file that conforms to JPEG (Joint Photographic Experts Group).

[0033] The media control unit 44 records the RAW file and the HEIF file or JPEG file from the file control unit 43 on the media 14 or transmits them from the interface 15 or 16.

[0034] In the file control unit 43, whether to generate a HEIF file or a JPEG file can be selected according to, for example, a user operation. Also, as described later, HEIF files come in an image item format and an image sequence format, and whether to adopt the image item format or the image sequence format can be selected according to, for example, a user operation. Furthermore, the file control unit 43 can perform mutual conversion between a HEIF file and a JPEG file according to a user operation.

[0035] Furthermore, in generating a HEIF file, the file control unit 43 can associate internal data (data stored in the HEIF file) in the HEIF file that is associated with external data outside the HEIF file (data not stored in the HEIF file) with specific information that identifies the external data and store it in the HEIF file. A HEIF file in which internal data and specific information of external data associated with the internal data are stored in association with each other is also called an associated HEIF file. In an associated HEIF file, for example, the internal data and specific information can be stored in association with each other by storing association information that associates the internal data and the specific information.

[0036] <JPEGファイル>

[0037] FIG. 2 is a diagram showing an example of a JPEG file format that complies with JPEG (Joint Photographic Experts Group).

[0038] A JPEG file is configured by storing, for example, Exif metadata, a thumbnail image, XMP (Extensible Metadata Platform) (registered trademark) metadata, an MPF ​​indicating the storage location (position) of the main image and the simplified display image, etc., a main image, and an image for simplified display. For example, a screen nail image can be used as the image for simplified display.

[0039] <ISOベースメディアファイルフォーマット>

[0040] FIG. 3 is a diagram showing an example of the ISO base media file format.

[0041] HEIF (ISO / IEC 23008-12) is a file format that conforms to the ISO Base Media File Format (ISO / IEC 14496-12), and therefore HEIF files conform to the ISO Base Media File Format.

[0042] The ISO base media file format is composed of units called boxes, which serve as containers for storing data, and has a structure called a box structure.

[0043] A box has a type (box type) and actual data (data). The type indicates the type of actual data in the box. Actual data can be playable media data such as images (still images, videos), audio, subtitles, etc., attribute names (field names) and attribute values ​​(field values) of the attribute names (variables represented by the attribute names), and various other data.

[0044] Furthermore, a box can be used as actual data, that is, a box can have another box as actual data, which allows a hierarchical structure to be created.

[0045] A base media file that complies with the ISO base media file format can have an ftyp box, a moov box (MovieBox), a meta box (MetaBox), an mdat box (MediaDataBox), and the like. The ftyp box stores identification information that identifies the file format. The moov box can store a trak box, and the like. The meta box can store an iinf box, an iprp box, an iref box, an iloc box, and the like. The mdat box can store media data (AV data) and any other data.

[0046] HEIF complies with the above ISO Base Media File Formats.

[0047] <HEIFファイル>

[0048] FIG. 4 is a diagram showing an example of a format of a HEIF file that complies with HEIF.

[0049] HEIF files are broadly divided into an image item format and an image sequence format, and the image item format is further divided into a single image format that has only one item, which will be described later, and an image collection format that has multiple items.

[0050] A HEIF file in the image item format has an ftyp box, a meta box, and an mdat box.

[0051] A HEIF file in image sequence format has an ftyp box, a moov box, and an mdat box.

[0052] Note that a HEIF file can have both a meta box and a moov box, not just one of them.

[0053] The ftyp box stores identification information for identifying the file format, for example, that the file is a HEIF file in image item format or image sequence format.

[0054] The meta box and moov box store metadata necessary for playback and management of the media data stored in the mdat box, such as the storage location of the media data.

[0055] The mdat box stores media data (AV data) and the like.

[0056] In the digital camera 10, which HEIF file to generate, the image item format or the image sequence format, can be selected according to, for example, a user's operation. Also, when encoding and storing an image in the mdat box of the HEIF file, only intra-encoding is permitted for the image item format, and intra-encoding and inter-encoding are permitted for the image sequence format. Therefore, for example, when high-speed access to data stored in the HEIF file is prioritized, generation of a HEIF file in the image item format can be selected, and when reducing the size (amount of data) of the HEIF file is prioritized, generation of a HEIF file in the image sequence format can be selected.

[0057] FIG. 5 is a diagram showing an example of the format of a HEIF file in image item format.

[0058] In a HEIF file in the image item format, information indicating that it is a HEIF file in the image item format, such as mif1, is stored (as an attribute value) in the ftyp box.

[0059] The meta box stores an iinf box, an iref box, an iprp box, and an iloc box.

[0060] The iinf box stores the number of items (attribute names and attribute values ​​representing) that are media data (AV data) stored in the mdat box. An item is a piece of data stored in the mdat box of a HEIF file in image item format; for example, one image (screen) is an item. In this specification, one image, whether a still image or a video, is also called a frame. One frame is one item.

[0061] The iref box stores information indicating the relationship between items. For example, the mdat box can store the corresponding main image, screennail image, and thumbnail image as items. When the mdat box stores item I1 as the main image, item I2 as the screennail image, and item I3 as the thumbnail image, the iref box stores information indicating that item I2 is the screennail image of the main image as item I1, and information indicating that item I3 is the thumbnail image of the main image as item I1.

[0062] The iprp box stores information about the properties of an item.

[0063] The iloc box stores information about the storage location of the items stored in the mdat box.

[0064] In the mdat box of the image item format (HEIF file), for example, a frame of an image is stored as an item. One or more items can be stored in the mdat box. Also, a frame as an item can be encoded and stored in the mdat box. However, encoding of a frame as an item stored in the mdat box of the image item format is limited to intra encoding. As an encoding method (codec) for encoding a frame as an item, for example, HEVC or the like can be adopted.

[0065] FIG. 6 is a diagram showing an example of the iprp box of FIG.

[0066] The iprp box stores an ipco box and an ipma box related to item properties. The ipco box stores the properties of the item stored in the mdat box, such as codec information related to the codec of the image as the item and image size information related to the size. The ipma box stores an index (pointer) to the properties stored in the ipco box of the item stored in the mdat box.

[0067] FIG. 7 is a diagram showing an example of the format of a HEIF file in image sequence format.

[0068] In a HEIF file in the image sequence format, information indicating that it is a HEIF file in the image sequence format, such as msf1, is stored in the ftyp box.

[0069] The moov box stores a trak box, which stores information about the track stored in the mdat box.

[0070] A track consists of one independent piece of media data, such as images and audio, that is played according to a timeline. For example, a track consists of one or more frames of images that form an elementary stream. For tracks stored in an mdat box, multiple tracks, for example, image and audio tracks recorded at the same time, can be played simultaneously.

[0071] The media data of a track is composed of units called samples. A sample is the smallest unit (access unit) for accessing media data in a HEIF file. Therefore, it is not possible to access media data in a HEIF file in units smaller than a sample.

[0072] For image media data, one sample is, for example, one frame, etc. For audio media data, one sample is, for example, one audio frame or the like defined by the standard of the audio media data.

[0073] In the mdat box of the image sequence format (HEIF file), the media data of a track is arranged in units called chunks. A chunk is a set of one or more samples arranged at logically consecutive addresses.

[0074] When multiple tracks of media data are stored in the mdat box, the multiple tracks are arranged in an interleaved manner in chunk units.

[0075] As described above, an mdat box in the image sequence format stores one or more tracks made up of media data such as images and audio.

[0076] In the mdat box, the frames of the images that make up the track can be encoded and stored. To encode the frames that make up the track stored in the mdat box in the image sequence format, long GOP can be used as the GOP (Group of Picture), and both intra-encoding and inter-encoding can be used. As a codec for encoding the frames that make up the track, for example, HEVC can be used.

[0077] FIG. 8 is a diagram showing an example of a trak box.

[0078] The trak box can store a tkhd box and an mdia box. The tkhd box stores header information of the track managed by the trak box, such as the creation date and time of the track. The mdia box stores the minf box and the like. The minf box stores the stbl box. The stbl box stores the stsd box, stsc box, stsz box, and stco box, which store the track samples and, in turn, information for accessing the chunks. The stsd box stores codec information related to the codec of the track. The stsc box stores the chunk size (the number of samples in one chunk). The stsz box stores the sample size. The stco box stores the chunk offset, i.e., the offset of the placement position of each chunk of the track stored in the mdat box.

[0079] Here, a HEIF file in the image item format is also called a collection file, and a HEIF file in the image sequence format is also called a sequence file. Furthermore, an associated HEIF file in the image item format is also called an associated collection file, and an associated HEIF file in the image sequence format is also called an associated sequence file.

[0080] The digital camera 10 can generate a HEIF file (including an associated HEIF file) that stores the main image and also any required screennail and / or thumbnail images.

[0081] <Collection file>

[0082] FIG. 9 is a diagram showing an example of a normal collection file in which main images and thumbnail images are stored.

[0083] Here, a normal collection file refers to a collection file in which internal data within the collection file is not associated with specific information of external data.

[0084] Now, it is assumed that frames (items) are encoded in HEVC and stored in the mdat box of the collection file.

[0085] In the ftyp box, heic, which indicates that the file format is an image item format and that the codec is HEVC, is stored as identification information for identifying the file format.

[0086] The number of items (item count) stored in the mdat box is stored in the iinf box. In Fig. 9, a total of four items (frames) are stored in the mdat box: the main image (hereinafter also referred to as main image Item#1) identified by item ID#1, the main image Item#2, the thumbnail image (hereinafter also referred to as thumbnail image Item#101) identified by item ID#101, and the thumbnail image Item#102. Therefore, the number of items is four. Note that the thumbnail image Item#101 is the thumbnail image of the main image Item#1, and the thumbnail image Item#102 is the thumbnail image of the main image Item#2.

[0087] The iinf box further stores an infe box for each item stored in the mdat box. The infe box stores an item ID that identifies the item and an item type. In FIG. 9, there are infe boxes for the main images Item#1 and Item#2, and the thumbnail images Item#101 and Item#102.

[0088] In the iref box, for example, a thmb box is stored as information relating items stored in the mdat box. In the thmb box, a reference source and a reference destination are stored in association with each other as information relating a main image and a thumbnail image of the main image. In the thmb box, the reference source represents the item ID of the main image, and the reference destination represents the item ID of the thumbnail image of the main image specified by the item ID of the reference source. Therefore, according to the reference destination associated with the reference source, it is possible to recognize the item ID of the thumbnail image of the main image specified by the item ID represented by the reference source. Also, according to the reference source associated with the reference destination, it is possible to recognize the item ID of the main image of the thumbnail image specified by the item ID represented by the reference destination.

[0089] The iprp box stores an ipco box and an ipma box, as described in Fig. 6. The ipco box stores properties of a frame as an item stored in the mdat box, such as codec information related to the codec and image size information related to the size, as described in Fig. 6. The ipma box stores an index to the properties stored in the ipco box of the item stored in the mdat box, as described in Fig. 6.

[0090] The iloc box stores information about the storage location of an item in the mdat box, as described in Fig. 6. In Fig. 9, the iloc box stores that the number of items is 4. Furthermore, the iloc box stores the offsets to the storage locations and sizes of the main images Item#1 and Item#2 and the thumbnail images Item#101 and Item#102 stored in the mdat box, in association with the item IDs.

[0091] Hereinafter, an associated collection file in which internal data and specific information of external data are stored in association with each other in the normal collection file of FIG. 9 will be described.

[0092] FIG. 10 is a diagram showing an example of the first association-type collection file.

[0093] Hereinafter, the external data to be associated with a main image as internal data within a HEIF file will be, for example, a RAW image of the main image (or a RAW file in which the RAW image is stored).

[0094] In the first associative collection file, by storing association information that associates the main image as internal data with specific information of the RAW file in which the RAW image is stored (the RAW image stored in the RAW file), the main image and specific information of the RAW file in which the RAW image is stored are stored in association with each other. Furthermore, in the first associative collection file, the association information is stored in a meta box.

[0095] The identifying information for a RAW file in which a RAW image as external data is stored can include the file name of the RAW file, a universally unique identifier (uuid) issued to the RAW file, a uniform resource locator (URL), or any other information that can identify the RAW file (the RAW image stored in it).

[0096] For the first association type collection file, an association information storage box in which association information is stored is defined as a new box stored in the meta box, and is stored in the meta box. The association information storage box of the first association type collection file stores association information in which, for example, an item ID that identifies a main image and a uuid as identification information that identifies a RAW file (in which a RAW image is stored) associated with the main image are associated. Furthermore, the association information storage box stores the number of main images (the number of main images) associated with the RAW file (the RAW image stored in the RAW file). The number of main images stored in the association information storage box is the number of main images associated with the RAW file, and is therefore equal to or less than the number of main images stored in the mdat box.

[0097] In Fig. 10, the uuid of the RAW file of the main image Item#1 (the uuid of the RAW image associated with the main image Item#1) is UUID#1, and the uuid of the RAW file of the main image Item#2 is UUID#2. If a RAW file with a uuid of UUID#i is described as a RAW file UUID#i, then in Fig. 10, the item ID#1 of the main image Item#1 is associated with the uuid of the RAW file UUID#1, and the item ID#2 of the main image Item#2 is associated with the uuid of the RAW file UUID#2, and association information is stored in the association information storage box.

[0098] FIG. 11 is a diagram showing an example of the second association-type collection file.

[0099] In the second associative collection file, similarly to the first associative collection file, the main image as internal data is stored in association information that associates the main image with the specific information of the RAW file as external data, and the main image and the specific information of the RAW file are stored in association with each other. However, in the second associative collection file, the association information is stored in the mdat box.

[0100] For the second association collection file, for example, the same association information as that of the first association collection file is stored as an item in the mdat box. In Fig. 11, the association information is stored in the mdat box as an item of item ID #201.

[0101] As described above, in the second associative collection file, the association information for the item Item#201 is stored in the mdat box, and accordingly the information stored in the meta box differs from that in the normal collection file of Fig. 9. In the second associative collection file, metadata of the association information for the item Item#201 is stored in the meta box.

[0102] Specifically, in the second associative collection file, the number of items stored in the iinf box and the iloc box changes from 4 in the case of FIG. 9 to 5, which is 4 plus 1 for item Item#201. Furthermore, an infe box for item Item#201 is added to the iinf box, and an offset to the storage location and size of item Item#201 are added to the iloc box. The infe box for item Item#201 stores item ID#201 of item Item#201 and an item type IDIF (identifying data info) indicating that item Item#201 is associative information. IDIF is a newly defined attribute value (field value) indicating that the item is associative information.

[0103] FIG. 12 is a diagram showing an example of the third association-type collection file.

[0104] In the third associative collection file, the specific information of the RAW file as external data is stored as an item for each specific information in the mdat box, and the association information that associates the main image as internal data with the specific information of the RAW file as external data is stored in the meta box, so that the main image and the specific information of the RAW file are stored in association with each other. However, in the third associative collection file, the association information is information that associates the item ID of the main image as an item with the item ID of the specific information (of the RAW file) as an item, and is stored in the cdsc box stored in the iref box in the meta box.

[0105] In the cdsc box, a reference source and a reference destination can be stored in association with each other as information relating items as a main image and specific information of a RAW file of the main image. In the cdsc box, the reference source indicates the item ID of the main image, and the reference destination indicates the item ID of specific information as an item of the RAW file of the main image specified by the item ID of the reference source.

[0106] 12, UUID#1, which is the uuid as the identifying information of the RAW file of the main image Item#1, is stored in the mdat box as item Item#201, and UUID#2, which is the uuid as the identifying information of the RAW file of the main image Item#2, is stored in the mdat box as item Item#202. Furthermore, a cdsc box storing association information that associates the item ID#1 of the main image Item#1 with the item ID#201 of the identifying information UUID#1 as the reference source and the reference destination, respectively, is stored in the iref box, and a cdsc box storing association information that associates the item ID#2 of the main image Item#2 with the item ID#202 of the identifying information UUID#2 as the reference source and the reference destination, respectively, is stored in the iref box.

[0107] <Sequence file>

[0108] FIG. 13 is a diagram showing an example of a normal sequence file in which a track of a main image and a track of thumbnail images of the main image are stored.

[0109] Here, a normal sequence refers to a sequence file in which internal data within the sequence file is not associated with specific information of external data.

[0110] Now, let us assume that frames are encoded in HEVC and stored in the mdat box of the sequence file.

[0111] In the ftyp box, hevc, which indicates that the file format is an image sequence format and that the codec is HEVC, is stored as identification information for identifying the file format.

[0112] As described in Fig. 7, the moov box stores a trak box that manages each track stored in the mdat box. In Fig. 13, the track of the main image identified by track ID #1 (hereinafter also referred to as track #1) and track #2 of the thumbnail image of the main image in track #1 are stored in the mdat box. Therefore, the moov box stores a trak box that manages track #1 and a trak box that manages track #2. The (frame of) n-th thumbnail image (from the beginning) of track #2 is the thumbnail image of the n-th main image in track #1.

[0113] A sequence file is useful, for example, when continuous shooting is performed with the digital camera 10, in which case the main images and thumbnail images of a plurality of frames obtained by the continuous shooting are each recorded as one track.

[0114] The tkhd box of the trak box that manages track #1 of the main images stores track ID #1 that identifies track #1, the image size of the main images that make up track #1, rotation information that indicates the orientation of the digital camera 10 when the main images were captured, and the creation date and time of track #1. The tkhd box of the trak box that manages track #2 of the thumbnail images stores track ID #2 that identifies track #2 and the creation date and time of track #2.

[0115] In addition to the tkhd box and mdia box described in Fig. 7, the trak box can store a tref box. The tref box stores a track ID that identifies other tracks related to the track managed by the trak box in which the tref box is stored, and information that indicates the contents of the track. In Fig. 13, a tref box is provided in a trak box that manages track #2. The tref box stores information that indicates that the other track related to track #2 is track #1 (track_ID=1) and that the data that constitutes track #2 is a thumbnail image (track #2 is a thumbnail image track) (type=thmb).

[0116] In addition to the MINF box described in Fig. 8, the mdia box of the trak box can store an hdlr box. The hdlr box stores information indicating the type of data that constitutes the track managed by the trak box in which the hdlr box is stored. The hdlr box stored in the trak box that manages track #1 of the main image (in the mdia box in which it is stored) stores information (pict) indicating that the data that constitutes track #1 is a picture (frame), and the hdlr box stored in the trak box that manages track #2 of the thumbnail images stores information indicating that the data that constitutes track #2 is a picture.

[0117] The minf box is as explained in FIG.

[0118] Hereinafter, an associated sequence file in which internal data and specific information of external data are stored in association with each other in the normal sequence file of FIG. 13 will be described.

[0119] FIG. 14 is a diagram showing an example of an associated sequence file.

[0120] In an associated sequence file, track #3 of the (elementary) stream (Meta ES) of the uuid as specific information of the RAW file as external data is added to the mdat box, and the trak box that manages track #3 is added to the moov box.

[0121] Here, track #1 is a time series of one or more frames of the main image lined up on the timeline, and track #3 is a time series of the uuids of the RAW files of each frame of the main image lined up on the timeline.

[0122] The nth uuid (from the beginning) of track #3 is specific information of the RAW file of the nth frame of the main image of track #1. In addition, (data of) a plurality of tracks stored in the mdat box can be played back in synchronization according to time information on one timeline. Therefore, by storing track #1 of the main image and track #3 of the uuid (stream) of the RAW file of each frame of the main image constituting track #1 in the mdat box, the nth frame of the main image of track #1 and the uuid of the RAW file of the main image (frame) are stored in association with each other. In this case, it can be said that the frame of the main image of track #1 and the uuid of the RAW file of the main image (frame) are associated with each other by time information on the timeline.

[0123] In addition, the nth uuid (from the beginning) of track #3 is specific information for the RAW file of the nth frame of track #1, and the main images (frames) that make up track #1 and the uuids that make up track #3 can also be considered to be related by the order in which they are arranged on the tracks.

[0124] In an associative sequence file, when track #3 of the uuid of the RAW file is added to the mdat box, the trak box that manages that track #3 is added to the moov box.

[0125] The trak box that manages track #3 of the uuid of the RAW file stores a tkhd box, a tref box, an mdia box, and the like.

[0126] The tkhd box of the trak box that manages track #3 stores track ID #3 that identifies track #3 and the creation date and time of track #3.

[0127] The tref box of the trak box managing track #3 stores a track ID that identifies other tracks related to track #3 managed by the trak box in which the tref box is stored, and information that indicates the contents of track #3, etc. The uuid that constitutes track #3 is specific information of the RAW file of the main image that constitutes track #1, and since track #3 is related to track #1, the tref box of the trak box managing track #3 in Fig. 14 stores information that indicates that the other track related to track #3 is track #1 (track_ID=1) and that track #3 is a track of metadata (specific information here) (type=cdsc).

[0128] In the mdia box of the trak box that manages Track #3, an hdlr box and a minf box are stored. In the trak box that manages Track #3, the hdlr box stores information indicating that the data constituting Track #3 is metadata (of the main image), and the minf box stores an stsc box, an stsc box, an stsz box, and an stco box for Track #3.

[0129] <Generation and Playback of HEIF Files>

[0130] Figure 15 is a flowchart for explaining an outline of an example of a generation process for generating an associated type HEIF file.

[0131] In the generation process, in step S11, the file control unit 43 generates a uuid as specific information of the RAW file (RAW image) of the frame of the main image, and the process proceeds to step S12.

[0132] In step S12, the file control unit 43 assigns the uuid generated in step S11 to the RAW file (RAW image) of the frame of the main image, and the process proceeds to step S13.

[0133] In step S13, the file control unit 43 generates an associated type HEIF file in which the frame of the main image and the uuid of the RAW file of the frame are associated and stored in the HEIF file, and ends the generation process.

[0134] Figure 16 is a flowchart for explaining an outline of an example of a playback process for playing back an associated type HEIF file.

[0135] In the playback process, in step S21, the file control unit 43 generates, for example, a handle list of handles for identifying individual frames of the main image stored in the HEIF file stored in the medium 14, and the process proceeds to step S22.

[0136] Here, the handle of the main image frame includes the file name of the HEIF file in which the frame is stored. The handle of the main image frame (item) stored in the collection file further includes the item ID of the frame. The handle of the main image frame stored in the sequence file further includes the time information of the frame. The handle of the main image frame allows the frame for that handle to be uniquely identified (specified).

[0137] In addition, the handle of a main image frame stored in a sequence file can include the track ID of the track that contains the frame and the order of the frame in that track (the number of the frame) instead of the frame's time information.

[0138] Whether the sequence file contains one or multiple tracks made up of frames of the main image, the time information of each frame is unique. Therefore, even if multiple tracks are stored in the sequence file, the frame of the time information included in the handle can be uniquely identified from the frames that make up each of the multiple tracks, according to the frame time information. Therefore, if the handle of the main image frame contains the time information of that frame, the frame for the time information can be uniquely identified even if there is no track ID of the track in which the frame exists.

[0139] The handle list can be generated for all frames of the main image stored in the HEIF file stored on media 14, or it can be generated for only frames narrowed down by specific criteria, such as frames with a specific creation date and time.

[0140] After the file control unit 43 generates the handling list, the HEIF file is accessed by referring to the handling list as necessary.

[0141] In step S22, for example, upon waiting for the user to operate the digital camera 10 to display thumbnail images, the UI control unit 47 requests the file control unit 43 to display thumbnail images. In response to the request to display thumbnail images from the UI control unit 47, the file control unit 43 reads out from the HEIF file (a frame of) a thumbnail image of the main image frame identified by the handle in the handle list. The file control unit 43 then causes a list of thumbnail images read out from the HEIF file to be displayed, for example, on the liquid crystal panel 19 (FIG. 1), and the process proceeds from step S22 to step S23.

[0142] In step S23, for example, after the user selects a desired thumbnail (frame) from a list of thumbnail images, the UI control unit 47 requests the file control unit 43 for a main image corresponding to the thumbnail image selected by the user. In response to the request for the main image from the UI control unit 47, the file control unit 43 reads the main image from the HEIF file. The file control unit 43 can display the main image read from the HEIF file on the liquid crystal panel 19 as necessary.

[0143] Alternatively, the UI control unit 47 requests the uuid of the RAW file of the main image corresponding to the thumbnail image selected by the user from the file control unit 43. In response to the request for the uuid from the UI control unit 47, the file control unit 43 reads the uuid from the associated HEIF file. The file control unit 43 can access the RAW file identified by the uuid read from the associated HEIF file as necessary.

[0144] FIG. 17 is a flowchart illustrating an example of a playback process for playing back a collection file.

[0145] In step S31, the file control unit 43 acquires the item ID of the image to be reproduced (hereinafter, also referred to as the reproduction target item ID) which is the image (item) to be reproduced, and the process proceeds to step S32.

[0146] When obtaining the playback target item ID, for example, a main image identified by an arbitrary handle in the handle list, a thumbnail image of that main image, a thumbnail image selected by the user from a list of thumbnail images (hereinafter also referred to as a selected thumbnail image), or the main image of the selected thumbnail image is taken as the playback target image, and the item ID (playback target item ID) of that playback target image is obtained.

[0147] In step S32, the file control unit 43 reads out the image to be played back in accordance with the playback target item ID acquired in step S31.

[0148] In reading the image to be played back, the image to be played back specified by the playback target item ID is read out from the collection file.

[0149] FIG. 18 is a flowchart illustrating an example of the process of reading the image to be reproduced in step S32 of FIG.

[0150] In step S41, the file control unit 43 searches the iloc box of the collection file (FIGS. 9 to 12) for the playback target item ID, and the process proceeds to step S42.

[0151] In step S42, the file control unit 43 reads out the offset and size associated with the playback target item ID found in the iloc box in step S41, and the process proceeds to step S43.

[0152] In step S43, the file control unit 43 reads out the image to be played back stored in the mdat box of the collection file in accordance with the offset and size associated with the playback target item ID, and the process ends.

[0153] FIG. 19 is a flowchart illustrating a first example of the process of acquiring the playback target item ID in step S31 of FIG.

[0154] That is, FIG. 19 shows an example of acquiring the item ID of a thumbnail image which is an image to be played back.

[0155] In FIG. 19, it is assumed that the file control unit 43 recognizes the item ID of the main image of the thumbnail image as the image to be played back from the handle, for example.

[0156] In step S51, the file control unit 43 searches for a thmb box whose reference source matches the item ID of the main image from among the thmb boxes in the iref box of the collection file (FIGS. 9 to 12), and the process proceeds to step S52.

[0157] In step S52, file control unit 43 reads out the reference in the thmb box whose reference source matches the item ID of the main image found in step S51 as the item ID of the thumbnail image to be played back, and then the process ends.

[0158] FIG. 20 is a flowchart illustrating a second example of the process of acquiring the playback target item ID in step S31 of FIG.

[0159] That is, FIG. 20 shows an example of acquiring the item ID of the main image, which is the image to be reproduced, with the main image being the image to be reproduced.

[0160] In FIG. 20, for example, it is assumed that the user selects a thumbnail image (selected thumbnail image) from a list of thumbnail images, and file control unit 43 recognizes the item ID of the selected thumbnail image.

[0161] In step S61, the file control unit 43 searches the thmb boxes in the iref box of the collection file (FIGS. 9 to 12) for a thmb box whose reference matches the item ID of the selected thumbnail image, and the process proceeds to step S62.

[0162] In step S62, the file control unit 43 reads out the reference source in the thmb box whose reference destination matches the item ID of the selected thumbnail image found in step S61 as the item ID of the main image to be played back, and the process ends.

[0163] FIG. 21 is a flowchart illustrating an example of a process for acquiring the uuid as identifying information of a RAW file of a predetermined main image from the first associative collection file of FIG.

[0164] In FIG. 21, for example, it is assumed that the file control unit 43 recognizes the item ID of a predetermined main image by a handle list or the like.

[0165] In step S71, the file control unit 43 searches for the item ID of a predetermined main image from the association information in the association information storage box of the first association-type collection file (FIG. 10), and the process proceeds to step S72.

[0166] In step S72, the file control unit 43 reads out the uuid associated with the item ID of the predetermined main image searched for in step S71 from the association information, and the process ends.

[0167] The file control unit 43 can access the RAW file of a predetermined main image using the uuid read out as described above.

[0168] FIG. 22 is a flowchart illustrating an example of a process for acquiring the uuid as identifying information of a RAW file of a predetermined main image from the second associative collection file of FIG.

[0169] In FIG. 22, for example, it is assumed that the file control unit 43 recognizes the item ID of a predetermined main image by a handle list or the like.

[0170] In step S81, the file control unit 43 searches for an infe box of item type IDIF indicating that the item is association information from the infe box in the iinf box of the second associative collection file (FIG. 11), and the process proceeds to step S82.

[0171] In step S82, the file control unit 43 reads out the item ID of the association information as an item from the infe box of the item type IDIF searched for in step S81, and the process proceeds to step S83.

[0172] In step S83, the file control unit 43 searches the iloc box of the second associative collection file for the item ID of the association information read out in step S82, and the process proceeds to step S84.

[0173] In step S84, the file control unit 43 reads out the offset and size associated with the item ID of the association information searched for in step S83 in the iloc box, and the process proceeds to step S85.

[0174] In step S85, the file control unit 43 reads out the association information as an item stored in the mdat box of the second association type collection file according to the offset and size associated with the item ID of the association information read in step S84, and the process proceeds to step S86.

[0175] In step S86, the file control unit 43 searches for the item ID of a predetermined main image from the association information read out in step S85, and the process proceeds to step S87.

[0176] In step S87, the file control unit 43 reads out the uuid associated with the item ID of the predetermined main image searched for in step S86 from the association information, and the process ends.

[0177] The file control unit 43 can access the RAW file of a predetermined main image using the uuid read out as described above.

[0178] FIG. 23 is a flowchart illustrating an example of a process for acquiring the uuid as identifying information of a RAW file of a predetermined main image from the third associative collection file of FIG.

[0179] In FIG. 23, for example, it is assumed that the file control unit 43 recognizes the item ID of a predetermined main image by a handle list or the like.

[0180] In step S91, the file control unit 43 searches for a cdsc box whose reference source matches the item ID of a predetermined main image from the cdsc boxes in the iref box of the third associative collection file (FIG. 12), and the process proceeds to step S92.

[0181] In step S92, the file control unit 43 reads out the reference in the cdsc box searched for in step S91, whose reference source matches the item ID of the specified main image, as the item ID of the specific information of the RAW file of the specified main image as an item, and the processing proceeds to step S93.

[0182] In step S93, the file control unit 43 searches the iloc box of the third associative collection file for the item ID of the specific information as the item read out in step S92, and the process proceeds to step S94.

[0183] In step S94, the file control unit 43 reads out the offset and size associated with the item ID of the specific information searched for in step S93 in the iloc box, and the process proceeds to step S95.

[0184] In step S95, the file control unit 43 reads out the uuid as specific information of the RAW file of a specified main image stored in the mdat box of the third associative collection file, based on the offset and size associated with the item ID of the specific information read in step S94, and the processing ends.

[0185] The file control unit 43 can access the RAW file of a predetermined main image using the uuid read out as described above.

[0186] FIG. 24 is a flowchart illustrating an example of a process for acquiring a list of item IDs of main images from a collection file.

[0187] The process of acquiring the list of item IDs of main images from the collection file is performed, for example, when generating a handle list.

[0188] In step S101, the file control unit 43 reads out item IDs from all infe boxes in the iinf box of the collection file (Figures 9 to 12) and registers them in a list of item IDs of main images (hereinafter also referred to as the main image list), and the process proceeds to step S102.

[0189] In step S102, the file control unit 43 reads out the item IDs that are referenced from all boxes in the iref box of the collection file, and removes them from the main image list, and the process ends.

[0190] After the above processing, the item ID registered in the main image list becomes the item ID of the main image.

[0191] FIG. 25 is a flowchart illustrating an example of a process for reproducing a thumbnail image of a main image (a frame thereof) corresponding to predetermined time information from a sequence file.

[0192] In FIG. 25, for example, it is assumed that the file control unit 43 recognizes the time information (or order) of a predetermined main image by a handle list or the like.

[0193] In step S111, the file control unit 43 searches the trak boxes in the moov box of the sequence file (Figures 13 and 14) for a trak box in which information indicating that the data constituting the track is a thumbnail image is stored in the tref box, that is, a trak box in which the type in the tref box is thmb, as the trak box that manages the track of the thumbnail image of the main image for the specified time information, and the processing proceeds to step S112.

[0194] In step S112, the file control unit 43 reads out the track ID in the tkhd box in the trak box searched for in step S111 as the track ID of the track of the thumbnail image of the main image for the specified time information, and the process proceeds to step S113.

[0195] In step S113, the file control unit 43 plays the track with the track ID read out in step S112, and obtains from that track a thumbnail image (frame) for the specified time information (or sequence) as a thumbnail image of the main image for the specified time information, and then the processing ends.

[0196] The process of playing back image tracks stored in a sequence file is similar to the process of playing back video from an MP4 file.

[0197] FIG. 26 is a flowchart illustrating an example of a process for acquiring the uuid as identification information of the RAW file of a predetermined main image (frame) from an associative sequence file.

[0198] In FIG. 26, for example, it is assumed that the file control unit 43 recognizes the time information (or order) of a predetermined main image by a handle list or the like.

[0199] In step S121, the file control unit 43 searches the trak boxes in the moov box of the associative sequence file (Figure 14) for a trak box in which information indicating that the data constituting a track is specific information is stored in the tref box, that is, a trak box in which the type in the tref box is cdsc, as the trak box that manages the track of specific information, and the processing proceeds to step S122.

[0200] In step S122, the file control unit 43 reads out the track ID in the tkhd box in the trak box searched for in step S121 as the track ID of the specific information track, and the process proceeds to step S123.

[0201] In step S123, the file control unit 43 obtains the uuid as specific information for the time information (or order) of the specified main image from the track of the track ID read out in step S122, as the uuid of the RAW file of the specified main image, and the processing ends.

[0202] The file control unit 43 can access the RAW file of a predetermined main image using the uuid acquired as described above.

[0203] As described above, the file control unit 43 generates and plays an associated HEIF file in which a HEIF file conforming to HEIF is stored in association with a main image within the HEIF file and specific information identifying external data outside the HEIF file that is associated with the main image, so that the main image stored in the HEIF file can be associated with external data outside the HEIF file.

[0204] Furthermore, when a uuid is used as the identification information, even if the file name of the external data is changed, the uuid can maintain the association between the main image in the HEIF file and the external data after the file name is changed.

[0205] <Storage of specific information assigned to external data>

[0206] FIG. 27 is a diagram showing an example of storing uuid in a RAW file when a RAW file of a main image is used as external data (a file in which the external data is stored) and an associative collection file is generated.

[0207] In FIG. 27, the first associative collection file is used as the associative collection file.

[0208] A RAW file has an area called a MakerNote as part of an area for storing Exif attribute information as metadata.

[0209] The file control unit 43 can store the uuid assigned to a RAW file (RAW image) in, for example, a marker note of the RAW file.

[0210] 27, main images Item#1, Item#2, Item#3, and Item#4 are stored as four items in the association collection file, and RAW files #1, #2, #3, and #4 are generated in which RAW images of the main images Item#1, Item#2, Item#3, and Item#4 are stored. A UUID#i is assigned to the RAW file #i (RAW image), and as association information associating the main image Item#i with the UUID#i of the RAW file #i of the main image Item#i, association information in which an item ID#i identifying the main image Item#i is associated with the UUID#i of the RAW file #i associated with the main image Item#i is stored in an association information storage box.

[0211] FIG. 28 is a diagram showing an example of storing uuid in a RAW file when a RAW file of a main image is used as external data and an associated sequence file is generated.

[0212] When generating an associated sequence file, the file control unit 43 can store the uuid assigned to the RAW file in the marker note of the RAW file, just as when generating the associated collection file described in Figure 27.

[0213] 28, track #1 consisting of four frames of main images #1, #2, #3, #4 is stored in the associative sequence file, and RAW files #1, #2, #3, #4 are generated in which RAW images of main images #1, #2, #3, #4 are stored. UUID #i is assigned to RAW file #i, and track #3 is stored in the associative sequence file, in which UUID #i of RAW file #i is arranged to have the same time information as main image #i corresponding to RAW file #i (RAW image).

[0214] As described above, track #3 is constructed by arranging the UUID #i of RAW file #i so that it has the same time information as the main image #i corresponding to RAW file #i, and the i-th main image #i of track #1 and the i-th UUID #i of track #3, i.e., the UUID #i of the RAW file #i of main image #i, are associated and stored in an associated sequence file.

[0215] In the above, a RAW file (RAW image) of the main image is used as the external data, but other data can be used as the external data. For example, audio (sound) recorded together with capturing the main image can be used as the external data. For example, a WAV file in WAV format or an MP4 file in MP4 format can be used as the file storing the audio. In the following, for example, a WAV file will be used as the file storing the audio.

[0216] FIG. 29 is a diagram showing an example of storing uuid in a WAV file when a WAV file of a main image is adopted as external data (a file in which the external data is stored) and an associative collection file is generated.

[0217] In FIG. 29, the first associative collection file is used as the associative collection file.

[0218] A WAV file has an area called a List chunk as a part of the area in which metadata is written.

[0219] The file control unit 43 can store the uuid assigned to a WAV file (audio) in, for example, a List chunk of the WAV file.

[0220] 29, main images Item#1, Item#2, Item#3, and Item#4 are stored as four items in the associated collection file, and WAV files #1, #2, #3, and #4 for the main images Item#1, Item#2, Item#3, and Item#4 are generated. A UUID#i is assigned to the WAV file #i (audio), and as association information associating the main image Item#i with the UUID#i of the WAV file #i of the main image Item#i, association information in which the item ID#i identifying the main image Item#i corresponds to the UUID#i of the WAV file #i associated with the main image Item#i is stored in the association information storage box.

[0221] FIG. 30 is a diagram showing an example of storing uuid in a WAV file when a WAV file of a main image is used as external data and an associated sequence file is generated.

[0222] When generating an associative sequence file, the file control unit 43 can store the uuid assigned to a WAV file in the List chunk of the WAV file, just as when generating an associative collection file as described with reference to FIG. 29.

[0223] In Fig. 30, track #1 consisting of four frames of main images #1, #2, #3, #4 is stored in the associative sequence file, and WAV files #1, #2, #3, #4 of main images #1, #2, #3, #4 are generated. UUID #i is assigned to WAV file #i, and track #3 is stored in the associative sequence file, which is configured by arranging the UUID #i of WAV file #i so as to have the same time information as the main image #i corresponding to WAV file #i.

[0224] As described above, track #3 is constructed by arranging the UUID #i of WAV file #i so that it has the same time information as the main image #i corresponding to WAV file #i, and the i-th main image #i of track #1 and the i-th UUID #i of track #3, i.e., the UUID #i of WAV file #i of main image #i, are associated and stored in the associated sequence file.

[0225] In addition to HEIF files, this technology can also be applied to files other than HEIF files that have a box structure, such as ISO-based media files, MP4 files, and Miaf files.

[0226] Furthermore, the present technology can also be applied to, for example, a file that does not have a box structure and stores an image (main image) and another image that is the main image with a lowered resolution.

[0227] Furthermore, this technology can be applied to associating external data with the main image in a HEIF file, as well as with screennail and thumbnail images in a HEIF file.

[0228] The technology can also be applied in other applications, for example, when associating external data with internal data other than images, such as the main image in a HEIF file.

[0229] <Index display> FIG. 31 is a diagram for explaining an example of index display on the digital camera 10. In FIG.

[0230] The digital camera 10 can display a list of images stored in various files such as HEIF files (including associated HEIF files) as an index display. For ease of explanation, it is assumed that the files to be displayed as indexes are HEIF files.

[0231] In index display, the digital camera 10 reads out the thumbnail images Thm#i stored in the HEIF file, and displays on the liquid crystal panel 19 a list screen in which the thumbnail images Thm#i are arranged in a matrix.

[0232] In FIG. 31, a list screen is generated in which thumbnail images Thm#i are arranged in a 3×3 matrix.

[0233] If the total number of thumbnail images Thm#i stored in the HEIF file to be indexed exceeds the number of thumbnail images that can be arranged on the list screen (here, 9), the required number of tabs are provided at the top of the list screen. In Fig. 31, the black tab is the selected tab.

[0234] By default, the first tab (from the left) is selected, and a list screen is displayed showing thumbnail images Thm#1 to Thm#9 stored in the first nine HEIF files to be indexed.

[0235] When the user selects the second tab, a list screen is displayed on which thumbnail images Thm#10 to Thm#18 stored in the next nine HEIF files to be indexed are arranged.

[0236] Similarly, a list screen is displayed on which thumbnail images stored in 9 or less HEIF files are arranged according to the tab selected by the user. Here, the arrangement of thumbnail images on the list screen is not limited to 3x3.

[0237] In FIG. 31, the digital camera 10 is provided with a share button and a RAW edit button.

[0238] When a share button is operated while a thumbnail image Thm#i is selected on the list screen, the main image corresponding to the thumbnail image Thm#i selected on the list screen is transmitted to an external device other than the digital camera 10 and shared with the external device.

[0239] When the RAW edit button is operated while thumbnail image Thm#i is selected on the list screen, for example, in digital camera 10, a RAW image corresponding to thumbnail image Thm#i selected on the list screen (associated with the main image) is read from the RAW file and displayed on liquid crystal panel 19 in place of the list screen. This puts digital camera 10 into a state in which the RAW image can be edited.

[0240] With the index display described above, if the number of HEIF files to be indexed increases, it may be necessary to open and close many of these HEIF files, which may take some time before the list screen is displayed.

[0241] Furthermore, if the HEIF file to be displayed in the index contains a sequence file, the processing performed for index display may become redundant if the thumbnail image Thm#i stored in that sequence file is encoded in long GOP.

[0242] For example, if thumbnail images Thm#1 to Thm#9 for the 1st to 9th frames of thumbnail images Thm#i of 15 frames that make up a GOP are arranged on the first list screen, and thumbnail images Thm#10 to Thm#15 for the 10th to 15th frames are arranged on the second list screen, when the list screen display is switched from a list screen other than the second to the second list screen, in order to decode thumbnail images Thm#10 to Thm#15 for the 10th to 15th frames that are arranged on the second list screen, thumbnail images Thm#i required for decoding thumbnail images Thm#10 to Thm#15 for the 10th to 15th frames of thumbnail images Thm#1 to Thm#9 for the 1st to 9th frames that are not arranged on the second list screen must be decoded.

[0243] FIG. 32 is a diagram for explaining an example of index display on an external device other than the digital camera 10. In FIG.

[0244] Images obtained by digital camera 10 and stored in a HEIF file can be indexed and displayed on an external device other than digital camera 10.

[0245] When index display is performed on an external device, for example, the HEIF file to be the subject of index display is transmitted to the external device via a network, and the index display is performed on the external device in the same manner as the index display on the digital camera 10.

[0246] The HEIF file can be transmitted from the digital camera 10 to an external device via the Internet, a LAN, a USB, or any other wired or wireless network. The HEIF file can also be transmitted from the digital camera 10 to an external device via a cloud computer.

[0247] When an external device displays an index of images stored in a HEIF file obtained by digital camera 10, it may take some time to display the list screen, and the processing performed to display the index may be redundant, just as it does when the index is displayed on digital camera 10.

[0248] Furthermore, when displaying an index on an external device, it takes time for the digital camera 10 to transmit the HEIF files required to generate the list screen to the external device, which can result in poor responsiveness between the time a user requests index display on the external device and the time the list screen is actually displayed.

[0249] FIG. 33 is a diagram showing an example of an index display for the first association-type collection file.

[0250] Here, overlapping item IDs can be used between a certain HEIF file A and another HEIF file B different from the HEIF file A. For example, the item ID of one item in the HEIF file A can be 1, and the item ID of one item in the HEIF file B can be 1. However, in this embodiment, for ease of explanation, it is assumed that non-overlapping item IDs are used in the HEIF file A and the HEIF file B as appropriate.

[0251] In FIG. 33, four first associative collection files (HEIF files) #1 to #4 are the targets of index display.

[0252] In the first associated collection file #i in Fig. 33, one frame of a main image Item#i and a thumbnail image Item#(1000+i) of the main image Item#i are stored in the mdat box. Furthermore, the number of main images is 1, and an associated information storage box having associated information in which the item ID#i of the main image Item#i is associated with the UUID#i as identifying information of external data associated with the main image Item#i is stored in the meta box.

[0253] When index display is performed for the four first associative collection files #1 to #4, for example, thumbnail images Item#1001 to Item#1004 are read out from the four first associative collection files #1 to #4, respectively. Then, a list screen is generated on which the thumbnail images Item#1001 to Item#1004 are arranged.

[0254] Therefore, when index display is performed in the digital camera 10 and the external device, the responsiveness of the index display may be poor because each of the four first associative collection files #1 to #4 must be opened / closed in order to read out the thumbnail images Item#1001 to Item#1004 from the first associative collection files #1 to #4, respectively.

[0255] Furthermore, when index display is performed on an external device, multiple first association-type collection files #1 to #4 must be transmitted from the digital camera 10 to the external device, which takes time and may result in poor responsiveness of the index display.

[0256] FIG. 34 is a diagram showing another example of the index display for the first association-type collection file.

[0257] In FIG. 34, one first associative collection file (HEIF file) #1 is the subject of index display.

[0258] In the first associative collection file #1 in Figure 34, multiple main images of four frames, Item#1 to Item#4, and thumbnail images, Item#1001 to Item#1004, of each of the main images, Item#1 to Item#4, are stored in the mdat box.

[0259] Furthermore, the number of main images is four, and for each of the four frames of main images Item#1 to Item#4, an association information storage box is stored in the meta box, which contains association information that corresponds the item ID#i of main image Item#i to the UUID#i as specific information of external data associated with the main image.

[0260] When index display is performed for the first associative collection file #1 as described above, for example, thumbnail images Item#1001 to Item#1004 are read from the first associative collection file #1. Then, a list screen on which the thumbnail images Item#1001 to Item#1004 are arranged is generated.

[0261] Therefore, when index display is performed in the digital camera 10 and the external device, a seek is performed in the first associative collection file #1 to read only the thumbnail images Item#1001 to Item#1004 from the first associative collection file #1, resulting in poor responsiveness of the index display.

[0262] Furthermore, the first associative collection file in FIG. 34 has a large file size because it stores a plurality of main images, Item#1 to Item#4, each consisting of four frames.

[0263] Therefore, when index display is performed on an external device, the first association-type collection file #1, which has a large file size, must be transmitted from the digital camera 10 to the external device, which may take time and result in poor responsiveness of the index display.

[0264] Incidentally, an index display for a collection file other than an associative collection file can be performed in the same manner as an index display for an associative collection file.

[0265] FIG. 35 is a diagram showing an example of an index display for an associated sequence file.

[0266] In FIG. 35, one associated sequence file (HEIF file) #1 is the subject of index display.

[0267] In the associated sequence file #1 of Figure 35, track #1 consisting of multiple main images of four frames, track #2 consisting of thumbnail images of four frames of each of the main images, and track #3 consisting of UUID #1 to UUID #4 as specific information of external data associated with each of the four frame main images are stored in the mdat box.

[0268] Furthermore, the trak boxes that manage tracks #1 to #3 are stored in the moov box.

[0269] Each of the trak boxes that manage tracks #1 to #3 has a tkhd box that stores the track ID of the track that the trak box manages. The trak box that manages track #2 further has a tref box that stores (information indicating) that another track related to track #2 is track #1 (track_ID=1) and that track #2 is a thumbnail image track (type=thmb). The trak box that manages track #3 further has a tref box that stores (information indicating) that another track related to track #3 is track #1 (track_ID=1) and that track #3 is a metadata (specific information) track (type=cdsc).

[0270] When index display is performed for the above-described associated sequence file #1, for example, four frames of thumbnail images (encoded data) are read from the associated sequence file #1 and decoded. Then, a list screen on which the four frames of thumbnail images are arranged is generated.

[0271] In an associative sequence file, thumbnail images are stored in a coded state in, for example, a long GOP.

[0272] Therefore, to decode (play back) an inter-coded thumbnail image, it is necessary to decode other thumbnail images that the thumbnail image references, and if the other thumbnail images refer to yet other thumbnail images, it is necessary to decode those thumbnail images.

[0273] Associative sequence files can store inter-coded images, which allows them to be smaller in file size than collection files that do not allow inter-coded images. However, due to inter-coding, decoding of a thumbnail image may require decoding of one or more other thumbnail images.

[0274] Therefore, in the digital camera 10 and the external device, for example, when scrolling a list screen or switching tabs, it may take time to decode the thumbnail images to be displayed after scrolling or switching tabs, which may result in poor responsiveness of the index display.

[0275] Incidentally, index display for sequence files other than associative sequence files can be performed in the same manner as index display for associative sequence files.

[0276] Therefore, the digital camera 10 can generate another HEIF file that stores a related image related to the main image, such as a thumbnail image of the main image stored in one or more HEIF files, as a proxy file that represents the one or more HEIF files.The digital camera 10 or an external device can then use the proxy file to display an index.

[0277] By using the proxy file to perform the index display, it is possible to reduce the time required for the digital camera 10 and the external device to display the list screen as the index display, and to quickly display the index.

[0278] Moreover, the process performed for index display can be prevented from becoming redundant.

[0279] Furthermore, by sending a proxy file from the digital camera 10 to an external device, the time required for transmission can be shortened compared to sending a HEIF file required to generate a list screen, and deterioration of the responsiveness of the external device can be suppressed.

[0280] In addition, a proxy file contains thumbnail images of the main images stored in one or more HEIF files, making it easier to manage the thumbnail images.

[0281] <proxy file>

[0282] FIG. 36 is a diagram illustrating a first example of a proxy file.

[0283] The file control unit 43 can generate a proxy file in which an associated image related to a main image stored in a HEIF file (including an associated HEIF file) is stored as the main image of the HEIF file.

[0284] The proxy file may be, for example, a HEIF file, particularly a collection file, or may be a sequence file or a file other than a HEIF file.

[0285] In addition to storing related images related to the main image stored in the HEIF file, the proxy file can also store related images related to images stored in other than the HEIF file, such as JPEG files or RAW files.

[0286] Associated images related to a main image stored in a HEIF file can be images with the same content as the main image but a different number of pixels (size), i.e., for example, thumbnail images with a reduced number of pixels (size) as the main image, images with the same content as the main image but with a different YUV signal format, images with the same content as the main image but with a different bit allocation (gradation) per pixel than the main image, or other images related to the main image.

[0287] For example, if a HEIF file stores a main image and a thumbnail image, the thumbnail image as an associated image related to the main image may be generated by thinning out pixels from the main image, or the thumbnail image stored in the HEIF file may be copied and used.

[0288] That is, when a thumbnail image of the main image is used as an image related to the main image, if the main image and thumbnail image are stored in the HEIF file, a new thumbnail image can be generated from the main image stored in the HEIF file and stored in the proxy file as a related image, or a (copy of) the thumbnail image stored in the HEIF file can be stored in the proxy file as a related image. In the following, for example, a thumbnail image of the main image is used as the related image of the main image.

[0289] Here, HEIF file items that serve as proxy files are also referred to as proxy items to distinguish them from HEIF file items that are not proxy files.

[0290] In Figure 36, there are four HEIF files, #1 to #4, which are collection files, and the mdat box of each HEIF file #i stores one main image Item#i and a thumbnail image Item#(1000+i) of that main image Item#i.

[0291] In Figure 36, a proxy file has been generated in which thumbnail images Item#1001 to Item#1004 (copies) stored in HEIF files #1 to #4 are stored as proxy items Item#1 to Item#4.

[0292] As an image, proxy item Item#i is a thumbnail image of the main image Item#i of HEIF file#i, but is the main image of the proxy file (as the HEIF file).

[0293] For the first example of a proxy file, for example, by matching the thumbnail image as proxy item #i in the proxy file with the main images Item#1 to Item#4 in HEIF files #1 to #4, or the thumbnail images Item#1001 to Item#1004, respectively, it is possible to detect the main image Item#i' that best matches the thumbnail image as proxy item #i, or the main image Item#i' corresponding to the thumbnail image Item#(1000+i') that best matches the thumbnail image as proxy item #i, as the main image corresponding to the thumbnail image as proxy item #i.

[0294] FIG. 37 is a diagram illustrating a second example of the proxy file.

[0295] In Figure 37, as in Figure 36, a proxy file has been generated in which thumbnail images Item#1001 to Item#1004 stored in HEIF files #1 to #4 are stored as proxy items Item#1 to Item#4.

[0296] Furthermore, in Figure 37, a thumbnail image (main image of the proxy file) as proxy item Item#i is treated as internal data within the proxy file, and the main image Item#i corresponding to that thumbnail image is treated as external data outside the proxy file, and an associated HEIF file is generated as the proxy file in which the internal data (thumbnail image as proxy item Item#i) and specific information of the external data (main image Item#i) are stored in association with each other.

[0297] That is, in Fig. 37, an association information storage box that stores association information that associates the item ID#i of a thumbnail image (main image of the proxy file) as a proxy item Item#i with the hash value #i as specific information of the main image Item#i corresponding to that thumbnail image is stored in the meta box of the HEIF file as a proxy file. Therefore, the proxy file in Fig. 37 is the first associative collection file (Fig. 10).

[0298] As the hash value #i as the specific information of the main image Item#i, for example, a hash value in which the main image Item#i is input can be used.

[0299] Also, in Figure 37, in the proxy file association information storage box, the number of main images is 4, which is the number of thumbnail images of main images Item#1 to Item#4 as proxy items Item#1 to Item#4 stored in the proxy file, i.e., the number of sets of associated item IDs and hash values ​​as specific information.

[0300] In addition, as the proxy file (file format), an associative HEIF file other than the first associative collection file can be used, i.e., the second associative collection file (Figure 11), the third associative collection file (Figure 12), or the associative sequence file (Figure 14).

[0301] Regarding the second example of a proxy file, for example, the main image identified by the hash value (the hash value associated with the item ID of the thumbnail image) as specific information associated with the thumbnail image as proxy item #i, that is, the main image Item#i' that has a hash value matching hash value #i among the hash values ​​generated using each of the main images Item#1 to Item#4 in HEIF files #1 to #4 as input, can be detected as the main image corresponding to the thumbnail image as proxy item #i.

[0302] FIG. 38 is a diagram illustrating a third example of a proxy file.

[0303] In Figure 38, as in Figure 37, a proxy file has been generated in which thumbnail images Item#1001 to Item#1004 stored in HEIF files #1 to #4, which are collection files, are stored as proxy items Item#1 to Item#4.

[0304] Furthermore, in Figure 38, similar to Figure 37, the thumbnail image (main image of the proxy file) as proxy item Item#i is treated as internal data within the proxy file, and the main image Item#i corresponding to that thumbnail image is treated as external data outside the proxy file, and an associated HEIF file is generated as the proxy file in which the internal data (thumbnail image as proxy item Item#i) and specific information of the external data (main image Item#i) are stored in association with each other.

[0305] That is, in Fig. 38, similar to Fig. 37, an association information storage box that stores association information that associates the item ID#i of a thumbnail image as a proxy item Item#i with specific information of the main image Item#i corresponding to that thumbnail image is stored in the meta box of a HEIF file as a proxy file. Therefore, the proxy file in Fig. 38 is a first associative collection file.

[0306] However, in FIG. 38, uuid is used as the identification information for the main image Item#i instead of the hash value.

[0307] When uuid is used as identification information for the main image Item#i as external data, as explained in Figures 27 to 30, the UUID#i as identification information for the main image Item#i must be written (stored) in the HEIF file #i in which the main image Item#i as external data is stored.

[0308] Therefore, the file control unit 43 writes the UUID#i of the main image Item#i as external data into the HEIF file #i in which the main image Item#i is stored.

[0309] The UUID#i can be written to the HEIF file #i in which the main image Item#i is stored as external data in the meta box or mdat box of the HEIF file #i. The UUID#i can be written to the HEIF file #i in which the main image Item#i is stored as external data in accordance with the format of an associated HEIF file, i.e., so that the format after writing the UUID#i is the format of an associated HEIF file.

[0310] In FIG. 38, the UUID#i is written to the HEIF file #i in which the main image Item#i is stored as external data, in accordance with the file format of the first associative collection file.

[0311] That is, in Figure 38, an association information storage box in which the UUID#i of the main image Item#i and the item ID#i of that main image Item#i are stored in correspondence with each other is stored in the meta box of the HEIF file #i, and the HEIF file #i becomes the first associative collection file (conforms to the format of the first associative collection file).

[0312] Also, in Figure 38, in the association information storage box of HEIF file #i (which has become the first associated collection file), the number of main images is 1, which is the number of main images Item #i stored in HEIF file #i, i.e., the number of sets of associated item IDs and uuids as identifying information.

[0313] Here, the HEIF file in which the main image for which a thumbnail image is stored as an associated image in the proxy file is stored is the HEIF file that is the target for generating the proxy file, and hereinafter is also referred to as the proxy target file.

[0314] Writing the UUID #i to the HEIF file #i as the proxy target file can be performed according to any one of the first to third associative collection file formats if the HEIF file #i is a collection file, and can be performed according to the associative sequence file format if the HEIF file #i is a sequence file.

[0315] Note that for HEIF file #i after UUID#i has been written to the HEIF file #i as the proxy target file, the main image Item#i in that HEIF file #i is internal data of the HEIF file #i, and the thumbnail image as the proxy item Item#i of the proxy file is external data of the HEIF file #i.

[0316] The UUID#i written to the HEIF file #i as the proxy target file is specific information for the main image #Item#i, which is external data for the proxy file, but can also be considered as specific information that identifies the proxy item #i.

[0317] In this case, the HEIF file #i as the proxy target file after UUID#i is written can be considered to be a first associated HEIF file in which the main image Item#i as internal data and the proxy item Item#i as external data, i.e., the specific information UUID#i of the thumbnail image (of the main image Item#i) in the proxy file, are stored in association with each other.

[0318] Regarding the third example of a proxy file, for example, the main image identified by a uuid (a uuid associated with the item ID of the thumbnail image) as specific information associated with the thumbnail image as proxy item #i, i.e., the main image Item#i of item ID#i associated with UUID#i among the main images Item#1 to Item#4 in HEIF files #1 to #4, can be detected as the main image corresponding to the thumbnail image as proxy item #i.

[0319] FIG. 39 is a diagram illustrating a fourth example of a proxy file.

[0320] In Figure 38, the HEIF files serving as proxy target files are four collection files, but in Figure 39, the HEIF file serving as the proxy target file is one sequence file #1, which is why the fourth example of a proxy file in Figure 39 differs from the third example of a proxy file in Figure 38.

[0321] Here, in FIG. 39, collection file #1 as a proxy target file is not an associated sequence file but a normal sequence file.

[0322] In sequence file #1 of Figure 39, track #1 consisting of multiple main images of four frames, and track #2 consisting of four thumbnail images of each of the four main images are stored in the mdat box.

[0323] Furthermore, the trak boxes that manage tracks #1 and #2 are stored in the moov box.

[0324] Each of the trak boxes that manage tracks #1 to #2 has a tkhd box in which the track ID of the track that the trak box manages is stored.

[0325] The trak box that manages track #2 further has a tref box that stores the information that the other track associated with track #2 is track #1 (track_ID=1) and that track #2 is a thumbnail image track (type=thmb).

[0326] In FIG. 39, for example, four frames of thumbnail images stored in sequence file #1 are decoded and intra-encoded.

[0327] Then, a proxy file is generated in which thumbnail images of four frames in an intra-encoded state are stored as proxy items Item#1 to Item#4. This proxy file further stores a thumbnail image (main image of the proxy file) as proxy item Item#i as internal data within the proxy file, and a main image of the i-th frame (i-th frame) of track #1 of sequence file #1 corresponding to the thumbnail image as external data outside the proxy file, with the internal data (thumbnail image as proxy item Item#i) and specific information of the external data (main image of the i-th frame) being stored in association with each other.

[0328] That is, in Fig. 39, similar to Fig. 38, an association information storage box that stores association information that associates the item ID#i of a thumbnail image as a proxy item Item#i with specific information of the main image of the i-th frame corresponding to that thumbnail image is stored in the meta box of a HEIF file as a proxy file. Therefore, the proxy file in Fig. 39 is a first associative collection file.

[0329] In Fig. 39, uuid is used as identification information for the main image as external data, similarly to Fig. 38. The uuid for the main image of the i-th frame as external data is UUID#i.

[0330] As described above, in Figure 39, the uuid is used as identifying information for the main image as external data, so similarly to Figure 38, the uuid is written (stored) as identifying information for the main image as external data in sequence file #1, which is a proxy target file in which the main image as external data is stored.

[0331] FIG. 40 is a diagram illustrating an example of writing uuid as specific information of the main image as external data to sequence file #1 as a proxy target file.

[0332] The uuid of the main image as external data can be written to the sequence file #i in which the main image is stored in the moov box or mdat box of the sequence file #i. The uuid of the main image can be written to the sequence file #i in which the main image is stored in accordance with the format of an associated HEIF file, i.e., the format after writing the uuid becomes the format of an associated HEIF file.

[0333] In FIG. 40, the uuid is written to sequence file #1, which stores the main image as external data, in accordance with the file format of the associative sequence file.

[0334] That is, in Figure 40, track #3 of UUID #1 to UUID #4 of the main images of the first to fourth frames that make up track #1, and a trak box that manages track #3 are generated, and track #3 is stored in the mdat box and the trak box that manages track #3 is stored in the moov box, making sequence file #1 an associative sequence file (following the format of an associative sequence file).

[0335] Track #3 is configured by arranging UUID#1 to UUID#4 for the main images of the first to fourth frames, respectively, so that the main image of the i-th frame of track #1 is associated with the UUID#i of that main image, based on the time information on the timeline or the order in which they are arranged on track #3.

[0336] The trak box that manages track #3 has a tkhd box in which the track ID of track #3 that is managed by the trak box is stored.

[0337] The trak box that manages track #3 further has a tref box that stores the information that the other track associated with track #3 is track #1 (track_ID=1) and that track #3 is a metadata (specific information) track (type=cdsc).

[0338] Regarding the fourth example of a proxy file, for example, of the four main images that make up track #1 of (associative) sequence file #1, the main image of the i-th frame of track #1 that is associated with UUID #i can be detected as the main image that corresponds to the thumbnail image as proxy item #i.

[0339] After UUID#i is written to sequence file #1 as the proxy target file, the main image in sequence file #1 is internal data of sequence file #1, and the thumbnail image as proxy item Item#i of the proxy file is external data of sequence file #1.

[0340] The UUID #i written to sequence file #1 as the proxy target file is specific information for the main image of the i-th frame, which is external data for the proxy file, but can also be considered as specific information that identifies proxy item #i.

[0341] In this case, sequence file #1 as the proxy target file after UUID#i is written can be regarded as an associated sequence file in which the main image of the i-th frame as internal data and the proxy item Item#i as external data, i.e., the specific information UUID#i of the thumbnail image of the main image of the i-th frame, are stored in association with each other.

[0342] The proxy file described above can quickly display the index of thumbnail images, for example, as related images of the main image stored in the HEIF file. Furthermore, the main image corresponding to the thumbnail image as the related image in the index display can be easily detected.

[0343] FIG. 41 is a diagram for explaining an example of generating a proxy file in which the first associated collection file is the proxy target file.

[0344] Figures 36 to 40 have described proxy files generated using regular HEIF files (HEIF files other than associated HEIF files) as proxy target files, but proxy files can also be generated using associated HEIF files as proxy target files in addition to regular HEIF files.

[0345] In FIG. 41, the proxy target files are the four first associative collection files #1 to #4.

[0346] In the first associated collection file #i in Fig. 41, one frame of a main image Item#i and a thumbnail image Item#(1000+i) of the main image Item#i are stored in the mdat box. Furthermore, the number of main images is 1, and an associated information storage box is stored in the meta box, which has associated information that associates the item ID#i of the main image Item#i with the UUID#i as identifying information of external data (not shown) associated with the main image Item#i.

[0347] As described above, in the first associative collection file #i, an associative information storage box having associative information in which the item ID#i of the main image Item#i corresponds to the UUID#i as identifying information of some external data associated with the main image Item#i is stored in the meta box, and the main image Item#i is already associated with some external data with the UUID#i as identifying information of that external data. In the first associative collection file #i, some external data already associated with the main image Item#i is also referred to as associated external data.

[0348] When performing a proxy file generation process to generate a proxy file using the above-described first associated collection file #i as the proxy target file, the file control unit 43 can generate the proxy file by using the UUID#i that is already used as identification information for the associated external data as well as identification information for the main image Item#i, which is external data for the proxy file.

[0349] That is, in the proxy file generation process, the file control unit 43 treats the thumbnail images Item#1001 to Item#1004 stored in the first associative collection files #1 to #4 as proxy items Item#1 to Item#4, treats the thumbnail image (main image of the proxy file) as proxy item Item#i as internal data within the proxy file, and treats the main image Item#i corresponding to that thumbnail image as external data outside the proxy file, and generates an associative HEIF file as a proxy file in which the internal data (thumbnail image as proxy item Item#i) and specific information of the external data (main image Item#i) are stored in association with each other.

[0350] For example, the file control unit 43 generates a first associated collection file as a proxy file in which an association information storage box is stored in a meta box, the association information storing box storing association information that associates the item ID#i of a thumbnail image as a proxy item Item#i with specific information of the main image Item#i corresponding to that thumbnail image.

[0351] In the proxy file generation process in which the first associated collection file #i as described above is the proxy target file, the UUID#i serving as identifying information for the main image Item#i, which is external data for the proxy file, can be used in the first associated collection file #i as identifying information for the associated external data associated with the main image Item#i.

[0352] In this case, for the UUID#i serving as specific information of the main image Item#i, which is external data for the proxy file, there is no need to write the UUID#i to the first associative collection file #i as described in Fig. 38. This is because the UUID#i has already been written to the first associative collection file #i.

[0353] As with the third example of the proxy file in Figure 38, for the proxy file in Figure 41, for example, the main image Item#i of item ID#i associated with UUID#i among the main images Item#1 to Item#4 in the first HEIF files #1 to #4 can be detected as the main image corresponding to the thumbnail image as proxy item #i.

[0354] FIG. 42 is a diagram for explaining another example of generation of a proxy file in which the first association-type collection file is the proxy target file.

[0355] In FIG. 42, the proxy target file is one first associative collection file #1.

[0356] In the first associated collection file #1 in Fig. 42, a plurality of main images Item#1 to Item#4 of four frames and thumbnail images Item#1001 to Item#1004 of the main images Item#1 to Item#4 are stored in the mdat box. Furthermore, the number of main images is four, and an associated information storage box is stored in the meta box, which has associated information in which the item ID#i of the main image Item#i is associated with the UUID#i as specific information of the associated external data associated with the main image for each of the four main images Item#1 to Item#4 of four frames.

[0357] When performing the proxy file generation process using the above-described first associated collection file #1 as the proxy target file, the file control unit 43 can generate the proxy file by using the UUID#i that is already used as identification information for the associated external data, as well as the identification information for the main image Item#i, which is external data for the proxy file, as in the case of FIG. 41.

[0358] That is, in the proxy file generation process, the file control unit 43 treats the thumbnail images Item#1001 to Item#1004 stored in the first associative collection file #1 as proxy items Item#1 to Item#4, treats the thumbnail image (main image of the proxy file) as proxy item Item#i as internal data within the proxy file, and treats the main image Item#i corresponding to that thumbnail image as external data outside the proxy file, and generates an associative HEIF file as a proxy file in which the internal data (thumbnail image as proxy item Item#i) and specific information of the external data (main image Item#i) are stored in association with each other.

[0359] For example, the file control unit 43 generates a first associated collection file as a proxy file in which an association information storage box is stored in a meta box, the association information storing box storing association information that associates the item ID#i of a thumbnail image as a proxy item Item#i with specific information of the main image Item#i corresponding to that thumbnail image.

[0360] In the proxy file generation process in which the first associated collection file #i as described above is the proxy target file, the UUID#i serving as identifying information for the main image Item#i, which is external data for the proxy file, can be used in the first associated collection file #i as identifying information for the associated external data associated with the main image Item#i.

[0361] In this case, for the UUID#i serving as specific information of the main image Item#i, which is external data for the proxy file, there is no need to write the UUID#i to the first associative collection file #i as described in Fig. 38. This is because the UUID#i has already been written to the first associative collection file #i.

[0362] As with the third example of the proxy file in Figure 38, for the proxy file in Figure 42, for example, the main image Item#i of item ID#i associated with UUID#i among the main images Item#1 to Item#4 in the first HEIF files #1 to #4 can be detected as the main image corresponding to the thumbnail image as proxy item #i.

[0363] FIG. 43 is a diagram for explaining an example of generating a proxy file in which an associated sequence file is a proxy target file.

[0364] In FIG. 43, the proxy target file is one associated sequence file #1.

[0365] In the associated sequence file #1 of Figure 43, track #1 consisting of multiple main images of four frames, track #2 consisting of thumbnail images of four frames of each of the main images, and track #3 consisting of UUID #1 to UUID #4 as specific information of associated external data associated with each of the four frame main images are stored in the mdat box.

[0366] Furthermore, in the associative sequence file #1 in FIG. 43, the trak boxes that manage tracks #1 to #3 are each stored in a moov box.

[0367] Each of the trak boxes that manage tracks #1 to #3 has a tkhd box in which the track ID of the track that the trak box manages is stored.

[0368] The trak box that manages track #2 further has a tref box that stores the information that the other track associated with track #2 is track #1 (track_ID=1) and that track #2 is a thumbnail image track (type=thmb).

[0369] The trak box that manages track #3 further has a tref box that stores the information that the other track associated with track #3 is track #1 (track_ID=1) and that track #3 is a metadata (specific information) track (type=cdsc).

[0370] When performing the proxy file generation process using the above-mentioned associated sequence file #1 as the proxy target file, the file control unit 43 can generate the proxy file by using the UUID#i that is already used as identification information for the associated external data, as well as identification information for the main image Item#i, which is external data for the proxy file, as in the case of Figure 41.

[0371] That is, in the proxy file generation process, the file control unit 43 causes the encoding control unit 42 to decode and further intra-encode the four-frame thumbnail images stored in the associative sequence file #1.

[0372] Then, the file control unit 43 generates, as a proxy file, a first associative collection file in which thumbnail images of four frames in an intra-encoded state are stored as proxy items Item#1 to Item#4.

[0373] This proxy file further stores a thumbnail image (main image of the proxy file) as proxy item Item#i as internal data within the proxy file, and a main image of the i-th frame of track #1 of sequence file #1 corresponding to that thumbnail image as external data outside the proxy file, with the internal data (thumbnail image as proxy item Item#i) and specific information of the external data (main image of the i-th frame) stored in association with each other.

[0374] In other words, an association information storage box that stores association information that associates the item ID#i of the thumbnail image as the proxy item Item#i with specific information of the main image of the i-th frame corresponding to that thumbnail image is stored in the meta box of the proxy file.

[0375] In the proxy file generation process in which the above-described associated sequence file #1 is the proxy target file, UUID #i can be used as identifying information for the main image in the i-th frame of track #1, which is external data for the proxy file, as identifying information for the associated external data associated with the main image in the i-th frame in the associated sequence file #1.

[0376] In this case, for UUID#i as specific information of the main image of the i-th frame, which is external data for the proxy file, there is no need to write UUID#i to the associative sequence file #1 as described in Figures 39 and 40. This is because UUID#i is already written in the associative sequence file #1.

[0377] As for the proxy file of Figure 43, as in the fourth example of the proxy file of Figures 39 and 40, for example, of the four main images that make up track #1 of associated sequence file #1, the main image of the i-th frame of track #1 that is associated with UUID #i can be detected as the main image corresponding to the thumbnail image as proxy item #i.

[0378] FIG. 44 is a diagram for explaining the form of a pair of a proxy target file and a proxy file generated by a proxy file generation process using the proxy target file.

[0379] FIG. 44A shows an example of a first form that a pair of a proxy target file and a proxy file takes.

[0380] In the first form, the proxy file does not have specific information of the main image in the proxy target file that corresponds to the thumbnail image as the proxy item in the proxy file. Similarly, the proxy target file does not have specific information of the main image in the proxy target file.

[0381] In the first form, for example, by matching a thumbnail image that is a proxy item in a proxy file with a main image in a proxy target file, the main image that best matches the thumbnail image as a proxy item can be detected as the main image that corresponds to the thumbnail image that is a proxy item.

[0382] FIG. 44B shows an example of a second form that a pair of a proxy target file and a proxy file takes.

[0383] In the second form, the proxy file has specific information of the main image in the proxied file that corresponds to the thumbnail image as a proxy item in the proxy file.

[0384] On the other hand, the proxy target file does not have identifying information of the main image in the proxy target file. In the second form, the identifying information of the main image may be, for example, information that can be generated using the main image, such as a hash value that uses the main image as an input.

[0385] In the second form, for example, among the hash values ​​generated by inputting the main image in the proxy target file, a main image that has a hash value that matches a hash value that is specific information associated with a thumbnail image as a proxy item in the proxy file can be detected as the main image corresponding to the thumbnail image as a proxy item.

[0386] FIG. 44C shows an example of a third form that a pair of a proxy target file and a proxy file takes.

[0387] In a third embodiment, the proxy file has specific information of the main image in the proxied file that corresponds to the thumbnail image as a proxy item in the proxy file.

[0388] Similarly, the proxy target file has identification information of the main image in the proxy target file. In the third embodiment, the identification information of the main image may be, for example, information capable of identifying the main image, such as a uuid.

[0389] In the third form, for example, a main image in a proxy target file that is associated with specific information in the proxy target file that is identical to the specific information associated with a thumbnail image as a proxy item in the proxy file can be detected as a main image corresponding to the thumbnail image as a proxy item.

[0390] FIG. 45 is a diagram showing a detailed example of the second form taken by a pair of a proxy target file and a proxy file.

[0391] In FIG. 45, a proxy file is generated using two HEIF files #1 and #2 as proxy target files.

[0392] 45, HEIF file #1 as a proxy target file stores two main images A and B, as well as thumbnail images A and B of the main images A and B. HEIF file #2 as a proxy target file stores two main images C and D, as well as thumbnail images C and D of the main images C and D.

[0393] The proxy file stores thumbnail images A to D as the main images of the proxy file, and further stores specific information of the main images A to D corresponding to the thumbnail images A to D. The specific information of the main images A to D employs hash values ​​A to D using the main images A to D as input, respectively.

[0394] In the second form, when a thumbnail image stored in a proxy file is selected, a main image that produces a hash value that matches the hash value serving as specific information associated with the thumbnail image selected in the proxy file among the hash values ​​generated using as input each of main images A to D in HEIF files #1 and #2 as proxy target files can be detected as the main image corresponding to the thumbnail image selected in the proxy file.

[0395] FIG. 46 is a diagram showing a detailed example of the third form taken by a pair of a proxy target file and a proxy file.

[0396] In FIG. 46, a proxy file is generated using two HEIF files #1 and #2 as proxy target files.

[0397] 46, HEIF file #1 as a proxy target file stores two main images A and B, as well as thumbnail images A and B of the main images A and B. HEIF file #2 as a proxy target file stores two main images C and D, as well as thumbnail images C and D of the main images C and D.

[0398] The proxy file stores thumbnail images A to D as the main images of the proxy file, and further stores specific information of the main images A to D corresponding to the thumbnail images A to D. The specific information of the main images A to D uses uuid (UUID#A, UUID#B, UUID#C, UUID#D).

[0399] UUID#A, UUID#B, UUID#C, and UUID#D as identifying information for main images A to D are stored (written) in the proxy file as well as in the proxy target file. That is, the UUID#A and UUID#B of main images A and B are stored in HEIF file #1 as the proxy target file storing main images A and B, and the UUID#C and UUID#D of main images C and D are stored in HEIF file #2 as the proxy target file storing main images C and D.

[0400] In a third form, when a thumbnail image stored in a proxy file is selected, a main image in the proxy target file that is associated with a UUID in the proxy target file that is identical to the UUID associated with the thumbnail image selected in the proxy file can be detected as the main image corresponding to the thumbnail image selected in the proxy file.

[0401] <Index display using proxy files>

[0402] FIG. 47 is a diagram for explaining an example of index display using a proxy file in the digital camera 10. In FIG.

[0403] In index display, the digital camera 10 reads out the thumbnail images Thm#i stored in the proxy file, and displays a list screen in which the thumbnail images Thm#i are arranged in a matrix on the liquid crystal panel 19. In Fig. 47, a list screen in which the thumbnail images Thm#i are arranged in a 3×3 matrix is ​​generated, similar to Fig. 31. If the total number of thumbnail images Thm#i stored in the proxy file to be the subject of index display exceeds the number of thumbnail images that can be arranged on the list screen (here, 9), a required number of tabs are provided at the top of the list screen.

[0404] By default, the first tab is selected, and a list screen is displayed on which thumbnail images Thm#1 through Thm#9 stored in the first nine proxy files to be indexed. When the user selects the second tab, a list screen is displayed on which thumbnail images Thm#10 through Thm#18 stored in the next nine proxy files to be indexed. Similarly, a list screen is displayed on which thumbnail images stored in nine (or fewer) proxy files according to the tab selected by the user are arranged.

[0405] Here, similarly to the case of Fig. 31, the arrangement of thumbnail images on the list screen is not limited to 3 x 3. Also, the share button and RAW edit button are as described in Fig. 31.

[0406] According to the index display using the proxy file as described above, the index display can be performed appropriately.

[0407] That is, in index display using a proxy file, it is sufficient to open / close the proxy file, and there is no need to open / close the HEIF file that stores the main image corresponding to the thumbnail image Thm#i in the proxy file. Therefore, compared to opening / closing a HEIF file that stores the main image, the time until the list screen is displayed is shortened, and the list screen can be displayed quickly.

[0408] Furthermore, since thumbnail image Thm#i is stored in the proxy file to be displayed as an index without being inter-encoded, that is, in this embodiment, it is stored as being intra-encoded, it is possible to prevent the process performed for displaying the index from becoming redundant. Specifically, when a thumbnail image Thm#i is displayed on a list screen, such as when displaying a thumbnail image Thm#i that is stored in a sequence file and inter-encoded in a long GOP as described in Fig. 31, there is no need to decode other thumbnail images Thm#i' in order to decode the thumbnail image Thm#i.

[0409] FIG. 48 is a diagram for explaining an example of index display using a proxy file in an external device other than the digital camera 10. In FIG.

[0410] When index display is performed on an external device other than the digital camera 10, for example, instead of the HEIF file to be the target of index display, a proxy file generated using the HEIF file as a proxy target file can be transmitted to the external device via a network.

[0411] The external device receives the proxy file from the digital camera 10. Then, in the external device, a list screen is generated and displayed using the thumbnail images Thm#i stored in the proxy file, in the same manner as when the digital camera 10 performs index display using the proxy file.

[0412] Therefore, in the external device, the index display can be properly performed in the same way as in the case where the index display is performed using the proxy file in the digital camera 10. In other words, the time required to display the list screen can be shortened, and the list screen can be displayed quickly.

[0413] Furthermore, the process performed for index display can be prevented from becoming redundant.

[0414] Furthermore, the amount of data in a proxy file in which thumbnail images are stored is smaller than the total amount of data in the HEIF files in which the main images corresponding to the thumbnail images stored in the proxy file are stored. Therefore, the proxy file can be transmitted from the digital camera 10 to an external device in a shorter time than the transmission of a HEIF file as described in FIG.

[0415] As a result, it is possible to improve the responsiveness from when a user requests index display on an external device until the list screen is actually displayed.

[0416] After the list screen is displayed on the external device, the user can select any thumbnail image Thm#i from the list screen. When the user selects a thumbnail image Thm#i, the external device can transmit to the digital camera 10 specification information of the main image Main#i corresponding to the thumbnail image Thm#i, which is associated with the thumbnail image Thm#i in the proxy file. The digital camera 10 can obtain the main image Main#i in the HEIF file specified by the specification information from the external device, or, as associated external data, for example, a RAW image in a RAW file corresponding to the main image Main#i, and transmit it to the external device for display.

[0417] FIG. 49 is a diagram illustrating an example of selection of an HEIF file to be a proxy target file.

[0418] In the proxy file generation process, for example, as shown in FIG. 41, a plurality of collection files in which one frame (or more) of main image and thumbnail images are stored can be selected as proxy target files to generate a proxy file.

[0419] Also, for example, as shown in FIG. 42, one (or more) collection files in which main images and thumbnail images of a plurality of frames are stored can be selected as proxy target files to generate a proxy file.

[0420] Furthermore, for example, as shown in FIG. 43, one (or more) sequence files can be selected as proxy target files to generate a proxy file.

[0421] Alternatively, a proxy file can be generated by selecting an HEIF file stored in a specific directory as the proxy target file, for example, as shown in FIG. 49.

[0422] In Figure 49, the root directory contains a DCIM directory, which in turn contains a 100MSDCF directory. The HEIF file in the 100MSDCF directory is selected as the proxy target file, and a proxy file is generated.

[0423] 49, the 100MSDCF directory stores two collection files (HEIF files) with file names "DSC00001.HEIF" and "DSC00002.HEIF", each containing one frame of a main image and a thumbnail image. Proxy files are generated using these two collection files as proxy target files.

[0424] FIG. 50 is a diagram illustrating another example of selection of an HEIF file to be a proxy target file.

[0425] A proxy file can be generated, for example, by selecting an HEIF file stored on a specific medium (for example, the medium 14 attached to the digital camera 10) as the proxy target file, as shown in FIG.

[0426] 50, the root directory of the media contains a DCIM directory, which contains a 100MSDCF directory, a 101MSDCF directory, ..., and a 999MSDCF directory. The HEIF files in the 100MSDCF directory to the 999MSDCF directory are selected as the proxy target files, and proxy files are generated.

[0427] In Figure 50, the 100MSDCF directory to the 999MSDCF directory store a total of N collection files (HEIF files) each containing one frame of a main image and a thumbnail image, and proxy files are generated using these N collection files as proxy target files.

[0428] <Proxy file generation process for generating a proxy file of the first type, and additional process for adding a thumbnail image to the proxy file>

[0429] FIG. 51 is a diagram for explaining a proxy file generation process for generating a proxy file of the first type, and an addition process for adding a thumbnail image to the proxy file.

[0430] FIG. 51 shows an example of the structure of a proxy file in the first mode.

[0431] A in Fig. 51 shows an example of the structure of a proxy file when the proxy file is generated. Note that, here, when the proxy file is generated, a thumbnail image of one main image stored in the proxy target file is stored in the proxy file.

[0432] In generating a proxy file, the file control unit 43 generates a normal collection file as the proxy file, for example, having a free box with an empty area (padding area) and a thumbnail image #1 of one main image stored in the proxy target file stored in an mdat box as a proxy item.

[0433] A free box is a box in which any data can be stored. The capacity of the free box (its empty area) is determined according to, for example, the number of thumbnail images to be stored (or planned to be stored) in the proxy file.

[0434] In Fig. 51, the free box is provided in the file hierarchy between the meta box and the mdat box. However, the free box can also be provided within the meta box.

[0435] FIG. 51B shows an example of the structure of a proxy file after thumbnail images of other main images have been added.

[0436] A thumbnail image can be added to a proxy file that has already been generated. The process of adding a thumbnail image to a proxy file that has already been generated is also called an adding process.

[0437] When storing thumbnail image #2 of another main image in an already generated proxy file such as A in Figure 51, the file control unit 43 stores thumbnail image #2 of the other main image as a proxy item by adding it after thumbnail image #1 already stored in the mdat box of the proxy file.

[0438] Furthermore, the file control unit 43 rewrites the meta box to a state in which the metadata of thumbnail image #2 as a proxy item is added, and reduces the capacity (size) of the free box by the increase in the capacity of the meta box caused by the rewriting. Addition of further thumbnail images of the main image to the proxy file is performed in a similar manner.

[0439] When adding a thumbnail image as a proxy item after generating a proxy file, the metadata of the thumbnail image to be added must be added to the meta box of the proxy file. In this case, if no measures are taken to add metadata, adding metadata to the proxy file may cause the offset of data already stored in the mdat box to shift.

[0440] If the offset of the data stored in the mdat box is shifted, it is necessary to calculate the amount of offset shift for each piece of data stored in the mdat box and rewrite the iloc box with an offset that reflects that amount of shift, which increases the load of the additional process of adding a thumbnail image to the proxy file.

[0441] Therefore, in this embodiment, a free box is provided in the proxy file when the proxy file is generated. When performing an additional process to add a thumbnail image to the proxy file, the free box is used to add metadata, thereby preventing offset deviation of data already stored in the mdat box.

[0442] In the addition process, thumbnail image #2 is stored (written) in a manner that adds it to the back of thumbnail image #1 already stored in the mdat box of the proxy file. Therefore, the addition of thumbnail image #2 to the mdat box does not consume the free box (its free space), and does not cause an offset shift in thumbnail image #1 already stored in the mdat box.

[0443] FIG. 52 is a flowchart illustrating an example of a proxy file generation process for generating a proxy file in the first mode.

[0444] In step S211, the file control unit 43 acquires a thumbnail image of the main image stored in the proxy target file as a thumbnail image to be stored as a proxy item in the proxy file, and the process proceeds to step S212.

[0445] For example, when the proxy target file is a collection file or a sequence file in which all thumbnail images are stored in an intra-encoded form, the file control unit 43 reads out the thumbnail images from the collection file or sequence file serving as the proxy target file, thereby obtaining the thumbnail images to be stored as proxy items in the proxy file.

[0446] Also, for example, if the proxy target file is a sequence file in which thumbnail images are stored inter-encoded in long GOP, the file control unit 43 obtains the thumbnail images by generating them from the sequence file as the proxy target file.

[0447] Specifically, the file control unit 43 obtains a thumbnail image to be stored as a proxy item in the proxy file by, for example, generating a YUV thumbnail image from a main image (or a RAW image corresponding to the main image) stored in a sequence file as a proxy target file and intra-encoding the thumbnail image. Alternatively, the file control unit 43 obtains a thumbnail image to be stored as a proxy item in the proxy file by decoding an inter-encoded YUV thumbnail image stored in a sequence file as a proxy target file and intra-encoding the thumbnail image.

[0448] In step S212, the file control unit 43 calculates the size of the free box to be provided in the proxy file, and the process proceeds to step S213.

[0449] For example, the file control unit 43 determines the maximum number of thumbnail images to be stored in the proxy file (hereinafter also referred to as the maximum number of thumbnails). Then, according to the maximum number of thumbnails, the file control unit 43 calculates the size of a free box that can store at least the metadata of the maximum number of thumbnail images so that the maximum number of thumbnail images can be stored in the proxy file without offsetting the data stored in the mdat box of the proxy file.

[0450] The maximum number of thumbnails can be determined according to, for example, the remaining capacity of the medium (such as the medium 14) on which the proxy target files and the proxy files are recorded.

[0451] In addition, in digital camera 10, if taking one photograph is defined as a single shot, as opposed to continuous shooting, which is a process of taking multiple photographs (images) in response to a single shutter operation, then the maximum number of thumbnails can be determined, for example, depending on whether the imaging mode of digital camera 10 is continuous shooting mode or single shooting mode, which can be set for each HEIF file as a proxy target file.

[0452] Furthermore, the maximum number of thumbnails can be determined, for example, by defining the maximum number of main images that can be stored in one HEIF file that is to be a proxy target file, and determining the maximum number according to that maximum number.

[0453] Furthermore, the maximum number of thumbnails can be determined, for example, in the file control unit 43, by evaluating the main images stored in the proxy target file and determining the number of main images with good ratings. In evaluating the main images, for example, information regarding the image quality of the main images, such as the S / N ratio and sharpness, can be obtained, and main images with S / N ratios, sharpness, etc. above a threshold can be determined to be main images with good ratings.

[0454] In addition, the maximum number of thumbnails can be determined according to a number of factors, such as the remaining capacity of the media mentioned above.

[0455] Furthermore, when a HEIF file stored in a specified directory or on a specified medium is used as a proxy target file, the maximum number of thumbnails can be determined according to a maximum number that defines the maximum number of main images that can be stored in one HEIF file that is used as a proxy target file stored in a specified directory or on a specified medium.

[0456] In step S213, the file control unit 43 generates a normal collection file as a proxy file, which has a free box of the size calculated in step S212 and in which a thumbnail image of the main image stored in the proxy target file is stored in an mdat box as a proxy item, and terminates the proxy file generation process.

[0457] FIG. 53 is a flowchart illustrating an example of the adding process for adding a thumbnail image to a proxy file in the first mode.

[0458] When a proxy file already exists and a thumbnail image is to be added to the proxy file, the adding process is performed with the proxy file as the target of the adding process.

[0459] In step S221, the file control unit 43 acquires a thumbnail image of the main image stored in the proxy target file as a thumbnail image to be added as a proxy item to the proxy file to be added (hereinafter also referred to as an added thumbnail image), and the process proceeds to step S222. The acquisition of the added thumbnail image in step S221 can be performed in the same manner as in step S211 in FIG.

[0460] In step S222, the file control unit 43 adds the additional thumbnail image as a proxy item to the mdat box of the proxy file to be added, and the process proceeds to step S223.

[0461] In step S223, the file control unit 43 reads and acquires the meta box from the proxy file to be added, and the process proceeds to step S224.

[0462] In step S224, the file control unit 43 acquires the remaining capacity in the free box of the proxy file to be added, and confirms that the remaining capacity is sufficient for adding the metadata of the added thumbnail image, and then the process proceeds to step S225.

[0463] In step S224, if the remaining capacity of the free box is insufficient to add the metadata of the added thumbnail image, for example, a message to that effect is displayed on the liquid crystal panel 19, and the adding process ends. Furthermore, in this case, a proxy file generation process can be performed to generate a new proxy file in which the added thumbnail image is stored.

[0464] In step S225, the file control unit 43 regenerates the meta box acquired in step S223 into a new meta box to which metadata of the added thumbnail image has been added in response to the addition of the added thumbnail image as a proxy item to the mdat box of the proxy file to be added. Furthermore, the file control unit 43 rewrites the meta box of the proxy file to be added with the new meta box, and the process proceeds from step S225 to step S226.

[0465] In step S226, the file control unit 43 generates a new free box whose size is reduced by the increase in the size (data amount) of the new meta box relative to the meta box before rewriting. Furthermore, the file control unit 43 rewrites the free box of the proxy file to be added with the new free box, and the adding process ends.

[0466] <Proxy file generation process for generating a proxy file of the second or third type, and additional process for adding a thumbnail image to a proxy file>

[0467] FIG. 54 is a diagram for explaining a proxy file generation process for generating a proxy file of the second or third type, and an addition process for adding a thumbnail image to the proxy file.

[0468] FIG. 54 shows an example of the structure of a proxy file in the second or third form.

[0469] A in Fig. 54 shows an example of the structure of a proxy file when the proxy file is generated. Note that, here, when the proxy file is generated, a thumbnail image of one main image stored in the proxy target file is stored in the proxy file.

[0470] In generating a proxy file, the file control unit 43 generates, for example, an associated collection file having a free box and in which thumbnail image #1 of one main image stored in the proxy target file is stored in an mdat box as a proxy item.

[0471] In the associative collection file serving as a proxy file, thumbnail image #1 is stored in a form associated with (specific information of) that thumbnail image #1. In other words, in the associative collection file serving as a proxy file, relationship information relating to the association between the thumbnail image in the proxy file and specific information of the main image corresponding to that thumbnail image as external data is stored.

[0472] The relationship information includes, for example, the association information stored in the association information storage box of the first association-type collection file in FIG.

[0473] Furthermore, the relationship information includes, for example, association information for item Item#201 stored in the mdat box of the second association type collection file in Figure 11, as well as an infe box for item Item#201 stored in the iinf box of the meta box, and an offset to the storage location of item Item#201 stored in the iloc box of the iprp box of the meta box.

[0474] Further, the relationship information includes, for example, specific information as an item stored in the mdat box in the third associated collection file of Figure 12, and association information that associates the item ID of the main image as an item stored in the cdsc box of the iref box of the meta box with the item ID of the specific information as an item.

[0475] Further, the related information includes, for example, track #3 of specific information stored in the mdat box of the associated sequence file in FIG. 14, and a trak box that manages track #3 of specific information stored in the moov box.

[0476] In FIG. 54, a third associative collection file is generated as a proxy file.

[0477] In A of FIG. 54, specific information #1 is specific information of the main image corresponding to thumbnail image #1, and is part of the related information.

[0478] In the proxy file, which is the third associative collection file, the specific information #1 is stored in the mdat box. Among the relational information related to the relation between the thumbnail image #1 and the specific information #1, information other than the specific information #1, for example, the relational information that associates the item ID of the thumbnail image #1 with the item ID of the specific information #1, which is stored in the cdsc box of the iref box of the meta box described in Fig. 12, and metadata such as the offset of the storage location of the specific information #1, are stored in the meta box of the proxy file, which is the third associative collection file.

[0479] FIG. 54B shows an example of the structure of a proxy file after thumbnail images of other main images have been added.

[0480] In the process of adding an additional thumbnail image #2 of another main image to an already generated proxy file such as A in Figure 54, the file control unit 43 stores the thumbnail image #2 of the other main image as a proxy item by adding it after the thumbnail image #1 and specific information #1 already stored in the mdat box of the proxy file.

[0481] Furthermore, the file control unit 43 rewrites the meta box to a state in which the metadata of the thumbnail image #2 as a proxy item has been added.

[0482] The storage of thumbnail image #2 of another main image is performed in a manner associated with specific information #2 of the main image corresponding to thumbnail image #2. That is, the file control unit 43 stores specific information #2 as a proxy item by adding it to the mdat box of the proxy file.

[0483] Furthermore, the file control unit 43 rewrites the meta box to include metadata for thumbnail image #2 as a proxy item, and information other than specific information #2 from among the relationship information relating to the association between thumbnail image #2 and specific information #2.

[0484] The file control unit 43 reduces the capacity of the free box by the amount of increase in the capacity of the meta box that has been increased by rewriting the meta box. Addition of further thumbnail images of the main image to the proxy file is performed in a similar manner.

[0485] In the addition process, thumbnail image #2 and specific information #2 are written in a manner that adds them to the back of thumbnail image #1 and specific information #1 already stored in the mdat box. Therefore, the addition of thumbnail image #2 and specific information #2 to the mdat box does not consume the free box, and does not cause a shift in the offset of thumbnail image #1 and specific information #1 already stored in the mdat box.

[0486] FIG. 55 is a flowchart illustrating an example of a proxy file generation process for generating a proxy file of the second or third type.

[0487] In step S241, similar to step S211 in FIG. 52, the file control unit 43 acquires a thumbnail image of the main image stored in the proxy target file as a thumbnail image to be stored as a proxy item in the proxy file, and the process proceeds to step S242.

[0488] In step S242, the file control unit 43 acquires specific information of the main image corresponding to the thumbnail image acquired in step S241, and the process proceeds to step S243.

[0489] When a HEIF file serving as a proxy target file in which a main image is stored is a collection file, examples of the identification information for the main image include a UUID, a hash value calculated using the main image as input, a set of a URL assigned to the proxy target file in which the main image is stored and an item ID of the main image, etc.

[0490] When the HEIF file serving as the proxy target file in which the main image is stored is a sequence file, examples of the identifying information for the main image include a uuid, a URL assigned to the proxy target file in which the main image is stored, a track ID of the track of the main image frame and a set of the order of the main image frame from the beginning of the track (frame offset), a set of a URL assigned to the proxy target file in which the main image is stored and time information on the timeline of the main image frame, etc.

[0491] In addition, if the specific information is information that can identify the data (main image) in the proxy target file by writing the specific information to the proxy target file, for example, if the specific information is a uuid, in step S242, after generating the specific information for the main image, the specific information is written to the proxy target file in which the main image to be identified by the specific information is stored.

[0492] Furthermore, if the proxy target file is an associated HEIF file, and the identifying information of the external data associated with the main image stored in the associated HEIF file is stored in the associated HEIF file as the proxy target file, in step S242, the identifying information of the external data can be acquired as the identifying information of the main image. In this case, since the identifying information of the main image has already been written in the associated HEIF file as the proxy target file, there is no need to write it again.

[0493] In step S243, the file control unit 43 calculates the size of the free box to be provided in the proxy file, similarly to step S212 in FIG. 52, and the process proceeds to step S244.

[0494] In addition, when additional processing is performed on a proxy file of the second or third form as the proxy file to be added, the meta box of the proxy file stores not only the metadata of the thumbnail image to be added, but also relationship information (which should be stored in the meta box) relating to the association between the thumbnail image to be added and the specific information of the main image corresponding to that thumbnail image.

[0495] Therefore, in step S243 of FIG. 55, even if the maximum number of thumbnails for thumbnail images to be stored in a proxy file of the second or third form is the same as the maximum number of thumbnails for thumbnail images to be stored in a proxy file of the first form, a size larger than that in step S212 of FIG. 52 is calculated as the size of the free box.

[0496] In step S244, the file control unit 43 generates an associated collection file as a proxy file, which has a free box of the size calculated in step S243 and in which a thumbnail image of the main image stored in the proxy target file is stored in an mdat box as a proxy item, and terminates the proxy file generation process.

[0497] In generating the proxy file in step S244, an associative collection file is generated as a proxy file in which thumbnail images as proxy items are stored in association with specific information of the main image corresponding to the thumbnail images, i.e., the specific information acquired in step S242. For example, a proxy file is generated in which, together with the thumbnail images as proxy items, relationship information relating to the association between the thumbnail images and the specific information of the main image corresponding to the thumbnail images is stored.

[0498] FIG. 56 is a flowchart illustrating an example of the adding process for adding a thumbnail image to a proxy file in the second or third format.

[0499] When a proxy file already exists and a thumbnail image is to be added to the proxy file, the adding process is performed on the proxy file. In this example, the proxy file to be added is the third associative collection file (FIG. 12), for example.

[0500] In step S251, the file control unit 43 acquires a thumbnail image of the main image stored in the proxy target file as an additional thumbnail image to be added as a proxy item to the proxy file to be added, and the process proceeds to step S252. The acquisition of the additional thumbnail image in step S251 can be performed in the same manner as in step S241 in FIG.

[0501] In step S252, the file control unit 43 performs the same processes as in step S242 in FIG. 55, such as obtaining specific information of the main image corresponding to the added thumbnail image obtained in step S241, and the process proceeds to step S253.

[0502] In step S253, the file control unit 43 adds the additional thumbnail image and specific information of the main image corresponding to the additional thumbnail image as proxy items to the mdat box of the proxy file to be added, and the process proceeds to step S254.

[0503] In step S254, the file control unit 43 obtains the meta box by reading it from the proxy file to be added, and the process proceeds to step S255.

[0504] In step S255, the file control unit 43 obtains the remaining capacity in the free box of the proxy file to be added, and confirms that the remaining capacity is sufficient for adding metadata for the added thumbnail image and for adding relationship information relating to the association between the added thumbnail image and the specific information of the main image corresponding to the added thumbnail image (however, in this case, excluding the specific information already added to the mdat box in step S253), and the process proceeds to step S256.

[0505] In step S255, if the remaining capacity of the free box is insufficient for addition, the same process as that described in step S224 of FIG. 53 can be performed.

[0506] In step S256, file control unit 43 regenerates the meta box acquired in step S254 into a new meta box.

[0507] A new meta box is generated by adding, to the meta box obtained in step S254, the metadata of the added thumbnail image as a proxy item added to the mdat box of the proxy file to be added, and relationship information relating to the association between the added thumbnail image and the specific information of the main image corresponding to the added thumbnail image (however, in this case, excluding the specific information already added to the mdat box in step S253).

[0508] The file control unit 43 rewrites the meta box of the proxy file to be added with the new meta box, and the process proceeds from step S256 to step S257.

[0509] In step S257, the file control unit 43 generates a new free box whose size is reduced by the increase in the size (data amount) of the new meta box relative to the meta box before rewriting. Furthermore, the file control unit 43 rewrites the free box of the proxy file to be added with the new free box, and the adding process ends.

[0510] <Detection process for detecting a main image corresponding to a thumbnail image stored in a proxy file of the second embodiment>

[0511] FIG. 57 is a diagram for explaining an example of a detection process for detecting a main image corresponding to a thumbnail image stored in a proxy file in the second format.

[0512] FIG. 57 shows an example of a pair of a proxy target file and a proxy file in the second mode.

[0513] 57, a proxy file is generated with collection files #1 and #2 as proxy target files. As the proxy file, for example, the third associative collection file is adopted.

[0514] In collection file #i, main image #i and thumbnail image #i of main image #i are stored in the mdat box.

[0515] In the proxy file, thumbnail images #1 and #2, and specific information #1 and #2 of the main images corresponding to thumbnail images #1 and #2 are stored in a manner that associates thumbnail image #i with specific information #i.

[0516] The collection file #i as the proxy target file in the second form does not have the specific information #i of the main image #i. Therefore, in the second form, as the specific information #i of the main image #i, information capable of identifying the main image #i, such as a hash value using the main image #i as an input, is adopted even if the specific information #i of the main image #i is not written in the collection file #i.

[0517] Assume now that, for example, a hash value in which the main image #i is input is adopted as the specific information #i of the main image #i.

[0518] In this case, among the hash values ​​generated using main images #1 and #2 in collection files #1 and #2 as proxy target files, respectively, the main image #i that produces a hash value that matches the hash value of specific information #i associated with thumbnail image #i as a proxy item in the proxy file is detected as the main image corresponding to thumbnail image #i in the proxy file.

[0519] FIG. 58 is a diagram further illustrating an example of a detection process for detecting a main image corresponding to a thumbnail image stored in a proxy file of the second type.

[0520] FIG. 58 shows an example of a pair of a proxy target file and a proxy file in the second mode.

[0521] In Fig. 58, a proxy file is generated with collection files #1 and #2 as proxy target files. For example, a first associated collection file is used as the proxy file. A normal collection file is used as collection files #1 and #2 as the proxy target files.

[0522] In collection file #1, main images A and B and thumbnail images A and B of main images A and B are stored in the mdat box. In collection file #2, main images C and D and thumbnail images C and D of main images C and D are stored in the mdat box.

[0523] In the proxy file, thumbnail images A to D and hash values ​​#A to #D as specific information for main images A to D are stored in a form that associates thumbnail images A to D with hash values ​​#A to #D as specific information, respectively. That is, in the proxy file, association information that associates item IDs of thumbnail images A to D with hash values ​​#A to #D as specific information, respectively, is stored in an association information storage box. The hash values ​​#A to #D as specific information are hash values ​​that have main images A to D as input, respectively.

[0524] In this case, among the hash values ​​generated using as input each of the main images A to D in collection files #1 and #2 as the proxy target files, the main image that has a hash value that matches the hash value serving as specific information associated with a thumbnail image X (X is any of A to D) as a proxy item in the proxy file is detected as the main image corresponding to thumbnail image X in the proxy file.

[0525] FIG. 59 is a flowchart illustrating an example of a detection process for detecting a main image corresponding to a thumbnail image stored in a proxy file in the second format.

[0526] For example, the UI control unit 47 controls the file control unit 43 and the like to generate a list screen using thumbnail images stored in the proxy file and display the list screen on the liquid crystal panel 19 as an index.

[0527] For example, the file control unit 43 generates a handle list of handles that identify each thumbnail image (main image for the proxy file) stored in the proxy file. The UI control unit 47 requests the file control unit 43 to display an index of the thumbnail images stored in the proxy file in response to, for example, a user operation.

[0528] In response to a request for index display from the UI control unit 47, the file control unit 43 reads out a thumbnail image (a frame of the thumbnail image) identified by a handle in the handle list from the proxy file and supplies it to the display control unit 46 via the encoding control unit 42.

[0529] The display control unit 46 generates a list screen of the thumbnail images from the file control unit 43 and displays it on the liquid crystal panel 19 .

[0530] Thereafter, for example, when the user selects a desired thumbnail image from the list screen, in step S271, the file control unit 43 obtains, by reading, from the proxy file, specific information associated with the thumbnail image selected from the list screen (hereinafter also referred to as the selected thumbnail image), and the process proceeds to step S272.

[0531] That is, for example, the file control unit 43 identifies the handle of the selected thumbnail image (selected related image) in the handle list, and obtains the specific information associated with the selected thumbnail image identified by the handle in the proxy file.

[0532] In step S272, the file control unit 43 selects the HEIF file to be searched for detecting the main image, and the process proceeds to step S273.

[0533] Here, in the proxy file generation process (and addition process), as described in Figures 49 and 50, etc., one or more collection files, one or more sequence files, HEIF files stored in a specific directory, or HEIF files in a specific media can be selected as proxy target files.

[0534] The file control unit 43 selects the HEIF file selected as the proxy target file for the proxy file as the search target. Information required to identify the proxy target file, for example, when a HEIF file stored in a specific directory or medium is selected as the proxy target file, information identifying the specific directory or medium can be stored in, for example, the meta box or mdat box of the proxy file.

[0535] In step S273, the file control unit 43 searches for a main image within the HEIF file to be searched, detects the main image identified by the identification information associated with the selected thumbnail image, and the detection process ends.

[0536] For example, if a hash value calculated using SHA-256 or the like is used as identification information for the main image, the file control unit 43 calculates a hash value for each HEIF file to be searched, using the main image (respectively) in the HEIF file as input.

[0537] That is, for example, if the HEIF files to be searched are collection files, the file control unit 43 obtains, for each of the collection files to be searched, a list of item IDs of main images in the collection files (main image list).

[0538] Furthermore, for each item ID in the main image list, the file control unit 43 reads (the data of) the main image as the item specified by the item ID from the mdat box according to the information in the iloc box of the collection file to be searched. The file control unit 43 calculates a hash value using the main image read from the mdat box of the collection file to be searched as an input.

[0539] Then, the file control unit 43 detects the main image #i for which a hash value matching the hash value serving as specific information associated with the selected thumbnail image is obtained, as the main image corresponding to the selected thumbnail image.

[0540] <Detection process for detecting a main image corresponding to a thumbnail image stored in a proxy file of the third embodiment>

[0541] FIG. 60 is a diagram for explaining an example of a detection process for detecting a main image corresponding to a thumbnail image stored in a proxy file of the third form.

[0542] FIG. 60 shows an example of a pair of a proxy target file and a proxy file in the third form.

[0543] In FIG. 60, a proxy file is generated with collection files #1 and #2 as proxy target files.

[0544] As the proxy file, for example, a third associative collection file is adopted. As collection files #1 and #2 as proxy target files, the third associative collection file is also adopted.

[0545] In collection file #i, as in the case of FIG. 57, main image #i and thumbnail image #i of main image #i are stored in the mdat box.

[0546] Furthermore, in the collection file #i, the main image #i and specific information #i of the main image #i are stored in such a manner that the main image #i and the specific information #i are associated with each other.

[0547] In the proxy file, as in the case of Figure 57, thumbnail images #1 and #2 and main image specific information #1 and #2 are stored in a manner that associates thumbnail image #i with specific information #i.

[0548] The collection file #i as the proxy target file in the third form has the specific information #i of the main image #i. Therefore, in the third form, if the collection file #i as the proxy target file is a file in which the specific information of the main image can be written, any information can be used as the specific information of the main image.

[0549] Assume now that, for example, uuid is used as the specific information #i of the main image #i.

[0550] In this case, among main images #1 and #2 in collection files #1 and #2, respectively, as proxy target files, the main image #i associated with a uuid that matches the uuid as specific information #i associated with thumbnail image #i as a proxy item in the proxy file is detected as the main image corresponding to thumbnail image #i in the proxy file.

[0551] FIG. 61 is a diagram further illustrating an example of a detection process for detecting a main image corresponding to a thumbnail image stored in a proxy file of the third form.

[0552] FIG. 61 shows an example of a pair of a proxy target file and a proxy file in the third form.

[0553] In FIG. 61, a proxy file is generated with collection files #1 and #2 as proxy target files.

[0554] As the proxy file, for example, a first associative collection file is adopted. As collection files #1 and #2 as proxy target files, the first associative collection file is also adopted.

[0555] In collection file #1, main images A and B and thumbnail images A and B of the main images A and B are stored in the mdat box.

[0556] In addition, in collection file #1, association information that associates the item IDs of main images A and B with UUIDs #A and #B as specific information for main images A and B, respectively, is stored in an association information storage box in the meta box.

[0557] In collection file #2, main images C and D and thumbnail images C and D of the main images C and D are stored in the mdat box.

[0558] In addition, in collection file #2, association information that associates the item IDs of main images C and D with UUIDs #C and #D as specific information for main images C and D, respectively, is stored in an association information storage box in the meta box.

[0559] In the proxy file, thumbnail images A to D and hash values ​​#A to #D as specific information for main images A to D are stored in such a way that thumbnail images A to D are associated with hash values ​​#A to #D as specific information, respectively.

[0560] That is, in the proxy file, association information that associates the item IDs of thumbnail images A through D with UUID#A through UUID#D as identification information is stored in an association information storage box.

[0561] In this case, among the main images A to D in collection files #1 and #2 as proxy target files, specific information associated with a thumbnail image X as a proxy item in the proxy file, i.e., the main image associated with a uuid that matches the uuid associated with the item ID of thumbnail image X in the association information storage box of the proxy file, is detected as the main image corresponding to thumbnail image X in the proxy file.

[0562] FIG. 62 is a diagram for explaining a first example of a detection process for detecting a main image corresponding to a thumbnail image stored in a proxy file of the third form.

[0563] FIG. 62 shows an example of a proxy target file when the first associative collection file (FIG. 10) is adopted as the proxy target file in a pair of a proxy target file and a proxy file in the third form.

[0564] In a first example of the detection process, it is assumed that a first association-type collection file is adopted as a proxy target file.

[0565] In the first associative collection file as a proxy target file, an association information storage box is stored in the meta box. In Fig. 62, the association information storage box stores the information that the number of main images is 1, and that the item ID of the main image is 1 and the UUID#1 as the specific information of the main image in association with each other.

[0566] When the first associated collection file is adopted as the proxy target file, in the detection process, the file control unit 43 detects a proxy target file whose uuid matches the uuid as specific information associated with the selected thumbnail image in the proxy file and is stored in the association information storage box.

[0567] Then, the file control unit 43 detects, in the proxy target file, the main image of the item ID associated with the uuid that matches the uuid as the specific information associated with the selected thumbnail image, as the main image corresponding to the selected thumbnail image.

[0568] FIG. 63 is a flowchart illustrating a first example of a detection process for detecting a main image corresponding to a thumbnail image stored in a proxy file in the third form.

[0569] For example, the UI control unit 47 generates a list screen using thumbnail images stored in the proxy file and causes the liquid crystal panel 19 to perform index display, as described with reference to FIG.

[0570] Thereafter, for example, when the user selects a desired thumbnail image from the list screen, in step S281, the file control unit 43 obtains, by reading out from the proxy file, specific information associated with the selected thumbnail image selected from the list screen, similar to step S271 in FIG. 59, and the process proceeds to step S282.

[0571] In step S282, similarly to step S272 in FIG. 59, the file control unit 43 selects the HEIF file to be searched for to detect the main image, and the process proceeds to step S283.

[0572] In step S283, the file control unit 43 detects, from the HEIF files to be searched, HEIF files in which specific information (hereinafter also referred to as matching specific information) that matches the specific information associated with the selected thumbnail image is stored in the association information storage box, as detection target files for detecting the main image corresponding to the selected thumbnail image.

[0573] Furthermore, the file control unit 43 obtains access information for accessing the main image (main image identified by the matching identification information) associated with the matching identification information in the file to be detected by reading it from the association information storage box of the file to be detected, and processing proceeds from step S283 to step S284.

[0574] In the first example of the detection process, the first association-type collection file is adopted as the proxy target file, as described in Fig. 62. Therefore, in the first example of the detection process, the detection target file is the first association-type collection file.

[0575] In the first associated collection file as the detection target file, as described in Figure 62, the item ID of the main image and specific information (uuid, etc.) of that main image are stored in association information storage box in correspondence with each other.

[0576] The file control unit 43 acquires, as access information, the item ID (of the main image) associated with the match specifying information in the association information storage box of the first association-type collection file as the detection target file.

[0577] In step S284, the file control unit 43 uses the access information to detect, from the detection target file, the main image identified by the identification information associated with the selected thumbnail image, and the detection process ends.

[0578] That is, the file control unit 43 detects the main image specified by the item ID as the access information from the detection target file by reading it as the main image specified by the identification information associated with the selected thumbnail image. Reading the main image specified by the item ID as the access information from the detection target file can be performed as described in FIG.

[0579] FIG. 64 is a diagram for explaining a second example of the detection process for detecting a main image corresponding to a thumbnail image stored in a proxy file of the third form.

[0580] FIG. 64 shows an example of a proxy target file when the second associative collection file (FIG. 11) is adopted as the proxy target file in a pair of a proxy target file and a proxy file in the third form.

[0581] In a second example of the detection process, it is assumed that a second association-type collection file is adopted as the proxy target file.

[0582] In the second associated collection file as a proxy target file, the number of main images stored in the association information storage box in the first associated collection file, and association information in which the item ID of the main image is associated with the uuid as specific information of that main image, are stored as items in the mdat box.

[0583] When the second associated collection file is adopted as the proxy target file, in the detection process, the file control unit 43 detects the proxy target file in which association information including a uuid that matches the uuid as specific information associated with the selected thumbnail image in the proxy file is stored in the mdat box.

[0584] Then, the file control unit 43 detects, in the proxy target file, the main image of the item ID associated with the uuid that matches the uuid as the specific information associated with the selected thumbnail image, as the main image corresponding to the selected thumbnail image.

[0585] In a second example of the detection process for detecting a main image corresponding to a thumbnail image stored in a proxy file of the third form, the same process as the first example of the detection process in FIG. 63 is performed.

[0586] However, in the second example of the detection process, in step S283 of FIG. 63, the file control unit 43 acquires the item ID in the Infe box, in which the item type in the infe box of the iinf box is IDIF, for the HEIF file to be searched for (the second associative collection file (FIG. 11)), as the item ID of the association information as an item.

[0587] Furthermore, the file control unit 43 searches the iloc box for the HEIF file to be searched for, and reads out the item ID of the association information as an item, and the offset and size associated with the item ID.

[0588] Then, the file control unit 43 reads out the association information as an item stored in the mdat box of the HEIF file to be searched, according to the offset and size associated with the item ID of the association information as an item.

[0589] As described above, the file control unit 43 detects association information having identifying information (matching identifying information) that matches identifying information associated with the selected thumbnail image from the association information read from the HEIF file to be searched.

[0590] Then, the file control unit 43 detects, as a detection target file, a search target HEIF file having association information with matching identification information as an item. Then, the file control unit 43 acquires an item ID (of the main image) associated with the matching identification information in the association information of the detection target file as access information for accessing the main image.

[0591] FIG. 65 is a diagram for explaining a third example of the detection process for detecting a main image corresponding to a thumbnail image stored in a proxy file of the third form.

[0592] FIG. 65 shows an example of a proxy target file when the third associative collection file (FIG. 12) is adopted as the proxy target file in a pair of a proxy target file and a proxy file in the third form.

[0593] In a third example of the detection process, it is assumed that a third association-type collection file is adopted as the proxy target file.

[0594] In the third associative collection file as the proxy target file, the uuid as specific information of the main image is stored as an item in the mdat box.

[0595] Furthermore, association information in which the item ID of the main image is associated with the item ID of the specific information of the main image is stored in the cdsc box of the iref box in the meta box (FIG. 12).

[0596] When the third associated collection file is adopted as the proxy target file, in the detection process, the file control unit 43 detects a proxy target file whose uuid stored in the mdat box matches the uuid as specific information associated with the selected thumbnail image in the proxy file.

[0597] Then, the file control unit 43 detects, in the proxy target file, the main image of the item ID associated with the uuid that matches the uuid as the specific information associated with the selected thumbnail image, as the main image corresponding to the selected thumbnail image.

[0598] In the third example of the detection process for detecting a main image corresponding to a thumbnail image stored in a proxy file of the third form, the same process as the first example of the detection process in FIG. 63 is performed.

[0599] However, in the third example of the detection process, in step S283 of FIG. 63, the file control unit 43 acquires the item ID in the Infe box, in which the item type in the infe box of the iinf box is IDIF, for the HEIF file to be searched for (the third associative collection file (FIG. 12)), as the item ID of the specific information as an item.

[0600] Furthermore, the file control unit 43 searches the iloc box for the HEIF file to be searched for, and reads out the item ID of the specific information as an item, and the offset and size associated with the item ID.

[0601] Then, the file control unit 43 reads out the specific information as an item stored in the mdat box of the HEIF file to be searched, according to the offset and size associated with the item ID of the specific information as an item.

[0602] In the manner described above, the file control unit 43 detects, from the identifying information read from the HEIF file to be searched, identifying information that matches the identifying information associated with the selected thumbnail image (matching identifying information).

[0603] Thereafter, the file control unit 43 detects the HEIF file to be searched that has the matching specifying information as an item as the detection target file.

[0604] Then, the file control unit 43 acquires the item ID (of the main image) associated with the item ID of the matching identification information as an item in the association information in the cdsc box (Figure 12) of the iref box of the meta box of the file to be detected, that is, the item ID associated with the item ID of the matching identification information as an item as a reference destination as the reference source, as access information for accessing the main image.

[0605] FIG. 66 is a diagram for explaining a fourth example of the detection process for detecting a main image corresponding to a thumbnail image stored in a proxy file of the third form.

[0606] FIG. 66 shows an example of a proxy target file when an associated sequence file (FIG. 14) is adopted as the proxy target file in a pair of a proxy target file and a proxy file in the third form.

[0607] In the fourth example of the detection process, it is assumed that an associated sequence file is adopted as the proxy target file.

[0608] In an associative sequence file as a proxy target file, a track of UUID as specific information of the main image is stored in the mdat box. In an associative sequence file as a proxy target file, the main image (frame) in the main image track (hereinafter also referred to as the main image track) and the UUID as specific information of the main image in the specific information track (hereinafter also referred to as the specific information track) are stored in association with each other by time information on the timeline (or the order of arrangement on the track).

[0609] When an associated sequence file is adopted as the proxy target file, in the detection process, the file control unit 43 detects a proxy target file whose UUID stored in the mdat box matches the UUID as specific information associated with the selected thumbnail image in the proxy file.

[0610] Then, the file control unit 43 detects, in the proxy target file, the main image specified by the uuid serving as the identification information associated with the selected thumbnail image, as the main image corresponding to the selected thumbnail image.

[0611] In the fourth example of the detection process for detecting a main image corresponding to a thumbnail image stored in a proxy file of the third form, the same process as the first example of the detection process in FIG. 63 is performed.

[0612] However, in the fourth example of the detection process, in step S283 of Figure 63, the file control unit 43 searches for the HEIF file (associative sequence file (Figure 14)) to be searched for, from the trak boxes in the moov box, a trak box in the tref box whose type is cdsc, as the trak box that manages the specific information track.

[0613] Furthermore, the file control unit 43 reads the track ID of the specific information track from the tkhd box of the trak box that manages the specific information track, and reads the track of that track ID, i.e., the specific information of the specific information track. In this way, the file control unit 43 detects specific information that matches the specific information associated with the selected thumbnail image (matching specific information) from the specific information read from the HEIF file to be searched.

[0614] Thereafter, the file control unit 43 detects, as a detection target file, a search target HEIF file having a specific information track including matching specific information.

[0615] Then, the file control unit 43 obtains the track ID (track ID of the main image track) in the tref box (Figure 14) of the trak box that manages the specific information tracks including the matching specific information of the file to be detected, and the time information on the timeline of the matching specific information (or the order in which it is arranged on the specific information track) as access information for accessing the main image.

[0616] <Proxy file generation process and additional process when generating thumbnail images to be stored in the proxy file when generating the proxy target file>

[0617] FIG. 67 is a diagram for explaining the proxy file generation process and the addition process in the case where a thumbnail image to be stored in the proxy file of the first type is generated when a proxy target file is generated.

[0618] FIG. 67 shows an example of the structure of a proxy file in the first mode.

[0619] FIG. 67A shows an example of the structure of a proxy file generated by the proxy file generation process.

[0620] In the proxy file generation process when generating a thumbnail image to be stored in a proxy file of the first type when the proxy target file is generated, the file control unit 43 generates a thumbnail image to be stored in the proxy file of the first type (hereinafter also referred to as a proxy thumbnail image) using an original image used to generate the thumbnail image to be stored in the proxy target file (hereinafter also referred to as a main thumbnail image) when the proxy target file is generated.

[0621] The proxy thumbnail image may be an image (copy) that is identical to the original thumbnail image, or an image with a different image quality or data amount from the original thumbnail image (an image with lower image quality or data amount).

[0622] For example, if the YUV signal format of the original thumbnail image is 4:2:2, the signal format of the proxy thumbnail image can be 4:2:0. Also, if the gradation of the original thumbnail image is 10 bits, the signal format of the proxy thumbnail image can be 8 bits. Furthermore, for example, the proxy thumbnail image can be an image with a smaller size (number of pixels) than the original thumbnail image.

[0623] Also, for example, the proxy thumbnail image may be encoded with a different compression rate than the main thumbnail image, for example, with a higher compression rate than the main thumbnail image.

[0624] Furthermore, for example, a different type of codec may be used for the proxy thumbnail image than for the main thumbnail image, for example HEVC may be used for the main thumbnail image and JPEG may be used for the proxy thumbnail image.

[0625] The file control unit 43 generates, for example, a normal collection file having a free box and in which a proxy thumbnail image as a thumbnail image of the main image stored in the proxy target file is stored in an mdat box as a proxy item, as a proxy file of the first type.

[0626] In A of FIG. 67, a proxy file in which two proxy thumbnail images A and B are stored (as main images) is generated by the proxy file generation process.

[0627] The free box is provided in the meta box. Metadata of the proxy thumbnail images A and B are stored in the meta box of the proxy file generated by the proxy file generation process.

[0628] FIG. 67B shows an example of the structure of a proxy file after proxy thumbnail images of other main images have been added by the addition process.

[0629] In the additional process of generating a proxy thumbnail image to be stored in a proxy file of the first type when a proxy target file is generated, for example, the HEIF file generated after the proxy file is generated is treated as a new proxy target file, and a proxy thumbnail image of the main image to be stored in the new proxy target file is added to the proxy file that has already been generated (hereinafter also referred to as the existing proxy file).

[0630] In other words, when a new proxy target file is generated, the file control unit 43 generates a proxy thumbnail image to be added to the proxy file of the first form, using the original image used to generate the original thumbnail image to be stored in the new proxy target file.

[0631] In B of Figure 67, the add process adds two proxy thumbnail images C and D (as main images) after the already stored proxy thumbnail images A and B in the mdat box of the existing proxy file in A of Figure 67.

[0632] When adding the proxy thumbnail images C and D, the file control unit 43 rewrites the meta box to a state in which the metadata of the proxy thumbnail images C and D have been added, and reduces the capacity of the free box by the increase in the capacity of the meta box caused by the rewriting. Addition of a proxy thumbnail image of the main image stored in a new proxy target file to an existing proxy file is performed in a similar manner.

[0633] FIG. 68 is a flowchart illustrating an example of a proxy file generation process in which a thumbnail image to be stored in a proxy file of the first type is generated when a proxy target file is generated.

[0634] When the file control unit 43 generates a HEIF file, if a proxy file generation process execution condition is satisfied for executing a proxy file generation process in which the HEIF file is a proxy target file, the file control unit 43 executes the following proxy file generation process.

[0635] The proxy file generation process execution condition may be, for example, that the proxy file does not exist in the directory or on the medium storing the proxy target file. Also, the proxy file generation process execution condition may be, for example, that when the proxy file exists in the directory or on the medium storing the proxy target file, the number of thumbnail images stored in the proxy file reaches an upper limit.

[0636] In step S311, the file control unit 43 generates a proxy thumbnail image to be stored in the proxy file of the first form, for example, using the original image used to generate the real thumbnail image stored in the proxy target file, and the process proceeds to step S312.

[0637] In step S311, instead of generating a proxy thumbnail image from the original image, the main thumbnail image generated from the original image can be directly acquired (copied) as the proxy thumbnail image.

[0638] In step S312, the file control unit 43 calculates the size of the free box to be provided in the proxy file, similarly to step S212 in FIG. 52, and the process proceeds to step S313.

[0639] In step S313, the file control unit 43 generates a proxy file as a normal collection file having a free box of the size calculated in step S312 and in which the proxy thumbnail image generated in step S311, i.e., for example, the thumbnail image of the main image stored in the proxy target file, is stored in the mdat box as a proxy item, and terminates the proxy file generation process.

[0640] FIG. 69 is a flowchart for explaining an example of additional processing in the case where a thumbnail image to be stored in a proxy file of the first type is generated when a proxy target file is generated.

[0641] When an HEIF file is generated, if an additional process execution condition for executing additional process of making the HEIF file a proxy target file is satisfied, the file control unit 43 executes the following additional process.

[0642] The condition for executing the additional process may be, for example, that a proxy file already exists in the directory or media that stores the proxy target file, and that the number of thumbnail images stored in the proxy file has not reached the upper limit.

[0643] The addition process is performed on an already existing proxy file (existing proxy file) as the target for addition process.

[0644] In step S321, the file control unit 43 generates a proxy thumbnail image as an additional thumbnail image, similarly to step S311 in FIG. 68, and the process proceeds to step S322.

[0645] In steps S322 through S326, the same processes as in steps S222 through S226 in FIG. 53 are carried out, respectively, and the addition process ends.

[0646] FIG. 70 is a diagram for explaining the proxy file generation process and the addition process in the case where a thumbnail image to be stored in a proxy file of the third type is generated when a proxy target file is generated.

[0647] Note that the proxy file generation process and the addition process for generating a thumbnail image to be stored in the proxy file of the second form when generating a proxy target file are similar to those for the proxy file of the third form, except that no specific information is written to the proxy target file, and that information that does not need to be written to the proxy target file is used as the specific information, and therefore will not be described here.

[0648] FIG. 70 shows an example of the structure of a proxy file in the third form.

[0649] FIG. 70A shows an example of the structure of a proxy file generated by the proxy file generation process.

[0650] In the proxy file generation process for generating a thumbnail image to be stored in a proxy file of the third type when the proxy target file is generated, the file control unit 43 generates a proxy thumbnail image to be stored in the proxy file of the third type when the proxy target file is generated, using the original image used to generate the main thumbnail image to be stored in the proxy target file, similar to the proxy file generation process for generating a proxy file of the first type (Figure 67).

[0651] The file control unit 43 generates, for example, an associated collection file as a third type of proxy file, in which a proxy thumbnail image as a thumbnail image of the main image stored in the proxy target file is stored in an mdat box as a proxy item, and the associated collection file has a free box.

[0652] In A of Fig. 70, a proxy file in which two proxy thumbnail images A and B are stored (as main images) is generated by the proxy file generation process. Also, a free box is provided in the meta box. Metadata of the proxy thumbnail images A and B are stored in the meta box of the proxy file generated by the proxy file generation process.

[0653] 70, a first association type collection file is adopted as the proxy file, and the proxy file has an association information storage box in the meta box. The association information storage box stores association information that associates item ID=1 of proxy thumbnail image A with UUID#A as identifying information of the main image corresponding to proxy thumbnail image A, and association information that associates item ID=2 of proxy thumbnail image B with UUID#B as identifying information of the main image corresponding to proxy thumbnail image B.

[0654] FIG. 70B shows an example of the structure of a proxy file after proxy thumbnail images of other main images have been added by the addition process.

[0655] In the additional processing when generating a proxy thumbnail image to be stored in a proxy file of the third form when a proxy target file is generated, for example, the HEIF file generated after the proxy file is generated is treated as a new proxy target file, and the proxy thumbnail image of the main image to be stored in the new proxy target file is added to the existing proxy file that has already been generated.

[0656] In other words, when a new proxy target file is generated, the file control unit 43 generates a proxy thumbnail image to be added to the proxy file of the third form, using the original image used to generate the real thumbnail image to be stored in the new proxy target file.

[0657] In B of Figure 70, the add process adds two proxy thumbnail images C and D (as main images) after the already stored proxy thumbnail images A and B in the mdat box of the existing proxy file in A of Figure 70.

[0658] When adding the proxy thumbnail images C and D, the file control unit 43 rewrites the meta box to a state in which the metadata of the proxy thumbnail images C and D has been added.

[0659] Furthermore, the file control unit 43 rewrites the meta box (the association information storage box) to a state in which association information that associates the item ID=3 of the proxy thumbnail image C with UUID#C as the specific information of the main image corresponding to the proxy thumbnail image C, and association information that associates the item ID=4 of the proxy thumbnail image D with UUID#D as the specific information of the main image corresponding to the proxy thumbnail image D have been added.

[0660] Then, the file control unit 43 reduces the capacity of the free box by the amount of increase in the capacity of the meta box that has been increased by rewriting the meta box. Addition of a proxy thumbnail image of the main image stored in a new proxy target file to an existing proxy file is performed in a similar manner.

[0661] FIG. 71 is a flowchart illustrating an example of a proxy file generation process in which a thumbnail image to be stored in a proxy file of the third type is generated when a proxy target file is generated.

[0662] When the file control unit 43 generates an HEIF file, it sets the HEIF file as a proxy target file, and when the conditions for executing the proxy file generation process described in Figure 68 are satisfied, it executes the following proxy file generation process.

[0663] In step S341, file control unit 43 generates a proxy thumbnail image in the same manner as in step S311 of FIG. 68, and the process proceeds to step S342.

[0664] In step S342, the file control unit 43 acquires, for example, a uuid by generating it, as identification information of the main image corresponding to the proxy thumbnail image generated in step S341, and the process proceeds to step S343.

[0665] In step S343, the file control unit 43 calculates the size of the free box to be provided in the proxy file, similarly to step S243 in FIG. 55, and the process proceeds to step S344.

[0666] In step S344, the file control unit 43 generates, as a proxy file, for example, a first associative collection file having a free box of the size calculated in step S343 and in which the proxy thumbnail image generated in step S341, i.e., the thumbnail image of the main image stored in the proxy target file, is stored in an mdat box as a proxy item, and terminates the proxy file generation process.

[0667] In generating the proxy file in step S344, a first associative collection file is generated in which the proxy thumbnail image is stored in association with specific information of the main image corresponding to the proxy thumbnail image, such as the uuid obtained in step S342.

[0668] In other words, a first association type collection file is generated as a proxy file, in which a proxy thumbnail image is stored in an mdat box and association information that associates the item ID of the proxy thumbnail image with the uuid of the main image corresponding to the proxy thumbnail image is stored in an association information storage box.

[0669] FIG. 72 is a flowchart for explaining an example of additional processing in the case of generating a thumbnail image to be stored in a proxy file of the third type when a proxy target file is generated.

[0670] When the file control unit 43 generates an HEIF file, the HEIF file is treated as a proxy target file, and when the additional process execution condition described with reference to FIG. 69 is satisfied, the file control unit 43 executes the following additional process.

[0671] The adding process is performed on an already existing proxy file (existing proxy file) as the target of the adding process. Here, the existing proxy file is assumed to be, for example, the first associative collection file, as in the cases of Figures 70 and 71.

[0672] In step S351, the file control unit 43 generates a proxy thumbnail image as an additional thumbnail image, similarly to step S321 in FIG. 69, and the process proceeds to step S352.

[0673] In step S352, the file control unit 43 acquires, for example, uuid as identification information of the main image corresponding to the added thumbnail image generated in step S351, and the process proceeds to step S353.

[0674] In step S353, the file control unit 43 adds the added thumbnail image as a proxy item to the mdat box of the proxy file to be added, and the process proceeds to step S354.

[0675] In step S354, the file control unit 43 obtains the meta box by reading it from the proxy file to be added, and the process proceeds to step S355.

[0676] In step S355, the file control unit 43 obtains the remaining capacity in the free box of the proxy file to be added, and confirms that the remaining capacity is sufficient for adding metadata for the added thumbnail image and for adding association information that associates the item ID of the added thumbnail image with the uuid as specific information of the main image corresponding to the added thumbnail image, and then the processing proceeds to step S356.

[0677] In step S355, if the remaining capacity of the free box is insufficient for addition, the same process as that described in step S224 of FIG. 53 can be performed.

[0678] In step S356, file control unit 43 regenerates the meta box acquired in step S354 into a new meta box.

[0679] A new meta box is generated by adding metadata of the added thumbnail image as a proxy item that was added to the mdat box of the proxy file to be added to the meta box obtained in step S354, and by adding association information that associates the item ID of the added thumbnail image with the uuid as specific information of the main image corresponding to the added thumbnail image to the association information storage box of the meta box.

[0680] The file control unit 43 rewrites the meta box of the proxy file to be added with the new meta box, and the process proceeds from step S356 to step S357.

[0681] In step S357, the file control unit 43 generates a new free box whose size is reduced by the amount of increase in the size of the new meta box relative to the meta box before rewriting. Furthermore, the file control unit 43 rewrites the free box of the proxy file to be added with the new free box, and the adding process ends.

[0682] <Proxy file generation and addition processing for an already existing HEIF file as the proxy target file>

[0683] FIG. 73 is a flowchart illustrating an example of a proxy file generation process in which an already-existing HEIF file is the proxy target file.

[0684] For example, when a request for generating a proxy file is received from a user or an application and the conditions for executing the proxy file generation process described in Fig. 68 are satisfied, the file control unit 43 executes the following proxy file generation process. Here, it is assumed that, for example, a proxy file of the second or third type of the first associative collection file is generated in the proxy file generation process.

[0685] In step S371, the file control unit 43 selects, for example, an HEIF file (all or part) that already exists in a predetermined directory or medium as a proxy target file. Furthermore, the file control unit 43 acquires a thumbnail image of the main image in the proxy target file as a proxy thumbnail image, and the process proceeds from step S371 to step S372.

[0686] The file control unit 43, for example, obtains a copy of the original thumbnail image of the main image stored in the proxy target file as a proxy thumbnail image. The file control unit 43 also generates a proxy thumbnail image by, for example, thinning out the main image or the original thumbnail image stored in the proxy target file, converting the signal format of the main image or the original thumbnail image, or re-encoding the image.

[0687] In step S372, the file control unit 43 acquires, for example, a uuid by generating it, as identification information of the main image corresponding to the proxy thumbnail image generated in step S371, and the process proceeds to step S373.

[0688] In step S373, the file control unit 43 calculates the size of the free box to be provided in the proxy file, similarly to step S243 in FIG. 55, and the process proceeds to step S374.

[0689] In step S374, the file control unit 43 generates, as a proxy file, for example, a first associative collection file having a free box of the size calculated in step S373 and in which the proxy thumbnail image generated in step S371 is stored in an mdat box as a proxy item, and terminates the proxy file generation process.

[0690] In generating the proxy file in step S374, a first associative collection file is generated in which the proxy thumbnail image is stored in association with the uuid obtained in step S372 as identification information of the main image corresponding to the proxy thumbnail image.

[0691] In other words, a first association type collection file is generated as a proxy file, in which a proxy thumbnail image is stored in an mdat box and association information that associates the item ID of the proxy thumbnail image with the uuid of the main image corresponding to the proxy thumbnail image is stored in an association information storage box.

[0692] FIG. 74 is a flowchart illustrating an example of adding a pre-existing HEIF file as a proxy target file.

[0693] The file control unit 43 executes the following addition process when, for example, a user or an application requests the generation of a proxy file and the addition process execution condition described in Fig. 69 is satisfied. The addition process is executed on an already existing proxy file (existing proxy file) as the addition process target. Here, the existing proxy file is assumed to be a proxy file of the second or third type of the first associative collection file.

[0694] In step S381, the file control unit 43 acquires, for example, an HEIF file (all or part) that already exists in a specified directory or media as a proxy target file, and a proxy thumbnail image of the main image stored in the proxy target file as an additional thumbnail image, and processing proceeds to step S382.

[0695] The file control unit 43, for example, obtains a copy of the main thumbnail image of the main image stored in the proxy target file as an additional thumbnail image. The file control unit 43 also generates the additional thumbnail image by, for example, thinning out the main image or main thumbnail image stored in the proxy target file, converting the signal format of the main image or main thumbnail image, or re-encoding the image.

[0696] In step S382, the file control unit 43 acquires, for example, uuid as identification information of the main image corresponding to the added thumbnail image generated in step S381, and the process proceeds to step S383.

[0697] In step S383, the file control unit 43 adds the additional thumbnail image as a proxy item to the mdat box of the existing proxy file to be added, and the process proceeds to step S384.

[0698] In step S384, the file control unit 43 obtains the meta box by reading it from the proxy file to be added, and the process proceeds to step S385.

[0699] In step S385, the file control unit 43 obtains the remaining capacity in the free box of the proxy file to be added, and confirms that the remaining capacity is sufficient for adding metadata for the added thumbnail image and for adding association information that associates the item ID of the added thumbnail image with the uuid as specific information of the main image corresponding to the added thumbnail image, and then the processing proceeds to step S386.

[0700] In step S385, if the remaining capacity of the free box is insufficient for addition, the same process as that described in step S224 of FIG. 53 can be performed.

[0701] In step S386, file control unit 43 regenerates the meta box acquired in step S384 into a new meta box. The new meta box is generated by adding, to the meta box acquired in step S384, metadata of the added thumbnail image as a proxy item added to the mdat box of the existing proxy file to be added, and adding association information that associates the item ID of the added thumbnail image with the uuid as identification information of the main image corresponding to the added thumbnail image, to the association information storage box of the meta box.

[0702] The file control unit 43 rewrites the meta box of the proxy file to be added with the new meta box, and the process proceeds from step S386 to step S387.

[0703] In step S387, the file control unit 43 generates a new free box whose size is reduced by the amount of increase in the size of the new meta box relative to the meta box before rewriting. Furthermore, the file control unit 43 rewrites the free box of the proxy file to be added with the new free box, and the adding process ends.

[0704] <Description of the computer to which this technology is applied>

[0705] Next, the series of processes of the above-mentioned file control unit 43 and other blocks constituting the signal processing unit 13 (FIG. 1) can be performed by hardware or software. When the series of processes are performed by software, a program constituting the software is installed in a computer or the like.

[0706] FIG. 75 is a block diagram showing an example of the configuration of an embodiment of a computer in which a program for executing the series of processes described above is installed.

[0707] The program can be recorded in advance on the hard disk 905 or ROM 903 as a recording medium built into the computer.

[0708] Alternatively, the program can be stored (recorded) in a removable recording medium 911 driven by the drive 909. Such a removable recording medium 911 can be provided as a so-called package software. Here, examples of the removable recording medium 911 include a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, and a semiconductor memory.

[0709] The program can be installed in the computer from the removable recording medium 911 as described above, or can be downloaded to the computer via a communication network or a broadcasting network and installed in the built-in hard disk 905. That is, the program can be transferred to the computer wirelessly from a download site via an artificial satellite for digital satellite broadcasting, or transferred to the computer by wire via a network such as a LAN (Local Area Network) or the Internet.

[0710] The computer includes a CPU (Central Processing Unit) 902 , to which an input / output interface 910 is connected via a bus 901 .

[0711] When a user inputs a command by operating an input unit 907 via an input / output interface 910, the CPU 902 executes a program stored in a read only memory (ROM) 903 in accordance with the command. Alternatively, the CPU 902 loads a program stored in a hard disk 905 into a random access memory (RAM) 904 and executes the program.

[0712] As a result, the CPU 902 performs processing according to the above-mentioned flowchart or processing performed by the configuration of the above-mentioned block diagram. Then, the CPU 902 outputs the processing result from the output unit 906 via the input / output interface 910, or transmits it from the communication unit 908, or further records it on the hard disk 905, as necessary.

[0713] The input unit 907 is composed of a keyboard, a mouse, a microphone, etc. The output unit 906 is composed of an LCD (Liquid Crystal Display), a speaker, etc.

[0714] Here, in this specification, the processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing executed in parallel or individually (for example, parallel processing or processing by objects).

[0715] The program may be processed by one computer (processor), or may be distributed among multiple computers. Furthermore, the program may be transferred to a remote computer for execution.

[0716] Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.

[0717] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0718] For example, the present technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0719] Furthermore, each step described in the above flow chart can be executed by one device, or can be shared and executed by a plurality of devices.

[0720] Furthermore, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device, or can be shared and executed by multiple devices.

[0721] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0722] The present technology can have the following configurations.

[0723] <1> A file control unit is provided that generates a proxy file, which is another HEIF file that stores related images related to an image in a HEIF file. File processing device. <2> The file control unit generates, as the proxy file, a collection file that is a HEIF file in an image item format in which the related images are stored as items. <1> The file processing device according to claim 1 . <3> The file control unit generates the proxy file in which association information that associates the related image with specific information that identifies the image corresponding to the related image is stored. <2> The file processing device according to claim 1 . <4> The file control unit generates the proxy file in which the association information that associates an item ID that identifies the related image with the identification information is stored. <3> The file processing device according to claim 1 . <5> The file control unit generates the proxy file in which the association information is stored in a meta box or an mdat box. <4> The file processing device according to claim 1 . <6> The file control unit The specific information is stored as an item in an mdat box of the proxy file, The association information that associates the item ID that identifies the related image with the item ID that identifies the specific information stored in the mdat box is stored in the meta box. Generate the proxy file <3> The file processing device according to claim 1 . <7> The related image is an image having a different image quality or data amount from the image. <1> Or <6> 2. A file processing device according to claim 1 , <8> The related image is an image having the same content as the image but with a different pixel count or size, an image having the same content as the image but with a different signal format, or an image having the same content but with a different gradation from the image. <1> Or <6> 2. A file processing device according to claim 1 , <9> The HEIF file stores the image and a thumbnail image of the image, The related image is the thumbnail image stored in the HEIF file, an image encoded with a compression ratio different from that of the thumbnail image, or an image encoded with a codec different from that of the thumbnail image. <1> Or <6> 2. A file processing device according to claim 1 , <10> Generate proxy files, which are other HEIF files that contain related images related to the images in the HEIF file 23. A file processing method comprising: <11> File control section that generates proxy files, which are other HEIF files that store related images related to the images in the HEIF (High Efficiency Image File Format) file A program that makes a computer function. <12> A file control unit detects an image corresponding to a selected related image selected from related images stored in a proxy file that is another HEIF file in which related images related to an image in the HEIF file are stored. File processing device. <13> The proxy file is a HEIF file in the image item format in which the related image is stored as an item. <12> The file processing device according to claim 1 . <14> the proxy file stores association information that associates the related image with identification information that identifies the image corresponding to the related image; The file control unit detects the image specified by the specification information associated with the selected related image as the image corresponding to the selected related image. <13> The file processing device according to claim 1 . <15> The proxy file stores the association information that associates an item ID that identifies the related image with the identification information, The file control unit detects the image identified by the identification information associated with the item ID of the selected related image as the image corresponding to the selected related image. <14> The file processing device according to claim 1 . <16> The related image is an image having a different image quality or data amount from the image. <12> Or <15> 2. A file processing device according to claim 1 , <17> The related image is an image having the same content as the image but with a different pixel count or size, an image having the same content as the image but with a different signal format, or an image having the same content but with a different gradation from the image. <12> Or <15> 2. A file processing device according to claim 1 , <18> The HEIF file stores the image and a thumbnail image of the image, The related image is the thumbnail image stored in the HEIF file, an image encoded with a compression ratio different from that of the thumbnail image, or an image encoded with a codec different from that of the thumbnail image. <12> Or <15> 2. A file processing device according to claim 1 , <19> Detecting an image corresponding to a selected related image selected from related images stored in a proxy file, which is another HEIF file in which related images related to an image in the HEIF file are stored. 23. A file processing method comprising: <20> A file control unit that detects an image corresponding to a selected related image selected from related images stored in a proxy file that is another HEIF file in which related images related to an image in a HEIF file are stored. A program that makes a computer function. [Explanation of symbols]

[0724] 10 digital camera, 11 optical system, 13 signal processing section, 14 media, 15,16 interface, 17 button / key, 18 touch panel, 19 liquid crystal panel, 20 viewfinder, 21 interface, 41 optical system / image sensor control section, 42 encoding control section, 43 file control section, 44 media control section, 45 operation control section, 46 display control section, 47 UI control section, 901 bus, 902 CPU, 903 ROM, 904 RAM, 905 hard disk, 906 output section, 907 input section, 908 communication section, 909 drive, 910 input / output interface, 911 removable recording medium

Claims

1. A method for generating a HEIF file that conforms to the High Efficiency Image File Format (HEIF) and stores an image, Generate a proxy file, which is another HEIF file that stores related images related to the images in the generated multiple HEIF files. A file control unit is provided, The file control unit A collection file, which is a HEIF file in an image item format in which the related images are stored as items, is generated as the proxy file; generating the proxy file in which association information that associates the related image with specific information that identifies the image corresponding to the related image is stored; File processing device.

2. The file control unit generates the proxy file in which the association information that associates an item ID that identifies the related image with the identification information is stored. The file processing device according to claim 1 .

3. The file control unit generates the proxy file in which the association information is stored in a meta box or an mdat box. The file processing device according to claim 2 .

4. The file control unit The specific information is stored as an item in an mdat box of the proxy file, The association information that associates the item ID that identifies the related image with the item ID that identifies the specific information stored in the mdat box is stored in the meta box. Generate the proxy file The file processing device according to claim 1 .

5. The related image is an image having a different image quality or data amount from the image. The file processing device according to claim 1 .

6. The related image is an image having the same content as the image but with a different pixel count or size, an image having the same content as the image but with a different signal format, or an image having the same content but with a different gradation from the image. The file processing device according to claim 1 .

7. The HEIF file stores the image and a thumbnail image of the image, The related image is the thumbnail image stored in the HEIF file, an image encoded with a compression ratio different from that of the thumbnail image, or an image encoded with a codec different from that of the thumbnail image. The file processing device according to claim 1 .

8. A method for generating a HEIF file that conforms to HEIF (High Efficiency Image File Format) and stores an image; Generating a proxy file that is another HEIF file storing related images related to the images in the generated multiple HEIF files; generating a collection file, which is a HEIF file in an image item format, in which the related images are stored as items, as the proxy file; generating the proxy file in which association information that associates the related image with specific information that identifies the image corresponding to the related image is stored; A file processing method comprising:

9. A method for generating a HEIF file that is compliant with HEIF (High Efficiency Image File Format) and stores an image, Generate a proxy file, which is another HEIF file that stores related images related to the images in the generated multiple HEIF files. A file control unit is provided, The file control unit A collection file, which is a HEIF file in an image item format in which the related images are stored as items, is generated as the proxy file; generating the proxy file in which association information that associates the related image with specific information that identifies the image corresponding to the related image is stored; A program that enables a computer to function as a file processing device.

10. Identifying an image corresponding to a selected related image selected from a plurality of related images stored in a proxy file, which is another HEIF file that is compliant with HEIF (High Efficiency Image File Format) and that stores related images related to an image in the HEIF file and association information that associates the related images with identification information that identifies the images corresponding to the related images, by the identification information associated with the selected related image; Detecting the identified image as the image corresponding to the selected related image. A file control unit is provided, The proxy file is a HEIF file in an image item format in which the related image is stored as an item. File processing device.

11. The proxy file stores the association information that associates an item ID that identifies the related image with the identification information, The file control unit detects the image identified by the identification information associated with the item ID of the selected related image as the image corresponding to the selected related image. The file processing device according to claim 10.

12. The related image is an image having a different image quality or data amount from the image. The file processing device according to claim 10.

13. The related image is an image having the same content as the image but with a different pixel count or size, an image having the same content as the image but with a different signal format, or an image having the same content but with a different gradation from the image. The file processing device according to claim 10.

14. The HEIF file stores the image and a thumbnail image of the image, The related image is the thumbnail image stored in the HEIF file, an image encoded with a compression ratio different from that of the thumbnail image, or an image encoded with a codec different from that of the thumbnail image. The file processing device according to claim 10.

15. Identifying an image corresponding to a selected related image selected from a plurality of related images stored in a proxy file, which is another HEIF file that is compliant with HEIF (High Efficiency Image File Format) and contains related images related to an image in the HEIF file and association information that associates the related images with identification information that identifies the images corresponding to the related images, by the identification information associated with the selected related image; Detecting the identified image as the image corresponding to the selected related image; Including, The proxy file is a HEIF file in an image item format in which the related image is stored as an item. File processing methods.

16. Identifying an image corresponding to a selected related image selected from a plurality of related images stored in a proxy file, which is another HEIF file that is compliant with HEIF (High Efficiency Image File Format) and that stores related images related to an image in the HEIF file and association information that associates the related images with identification information that identifies the images corresponding to the related images, by the identification information associated with the selected related image; Detecting the identified image as the image corresponding to the selected related image. A file control unit is provided, The proxy file is a HEIF file in an image item format in which the related image is stored as an item. A program that enables a computer to function as a file processing device.

Images