File generation device, file generation method, file reproduction device, and file reproduction method

Abstract

The present technology relates to a file generation device, a file generation method, a file reproduction device, a file reproduction method and a program that can realize the association between an image stored in a file and external data outside the file. According to the present technology, a file control unit generates an associative High Efficiency Image File Format (HEIF) file, in which an image that conforms to HEIF in the associative High Efficiency Image File Format file is associated and stored with specific information of external data outside the specified HEIF file, and the external data is to be associated with the image. The file control unit also reproduces the associative HEIF file. The present technology can be applied to the generation or reproduction of HEIF files.

Description

Technical Field

[0001] The present technology relates to a file generation device, a file generation method, a file reproduction device, a file reproduction method and a program, and in particular to a file generation device, a file generation method, a file reproduction device, a file reproduction method and a program that are capable of associating an image stored in a file with external data outside the file, for example. Background Art

[0002] As a file format for efficiently storing images, there is High Efficiency Image File Format (HEIF).

[0003] Reference List

[0004] Non-patent literature

[0005] Non-patent document 1: ISO / IEC 23008-12: 2017, Information technology:--Highefficiency coding and media delivery in heterogeneous environments--Part 12:Image File Format Summary of the invention

[0006] Problems to be solved by the present invention

[0007] With respect to a HEIF file conforming to the High Efficiency Image File Format (HEIF), it would be convenient if an image stored in the HEIF file could be associated with external data outside the HEIF file.

[0008] The present technology has been made in view of such circumstances, and makes it possible to associate an image stored in a HEIF file with external data outside the HEIF file.

[0009] Solution to the problem

[0010] A file generating device or a first program according to the present technology is a file generating device including a file control unit, which generates an associative high-efficiency image file format (HEIF) file in which an image compliant with HEIF and specific information of external data outside the specified HEIF file are stored in association with each other, and the external data is associated with the image, or a program that causes a computer to function as such a file generating device.

[0011] The file generation method according to the present technology is a file generation method including generating an associative High Efficiency Image File Format (HEIF) file, in which an image compliant with HEIF and specific information of external data outside the specified HEIF file are stored in association with each other, and the external data is associated with the image.

[0012] In a file generation device, a file generation method, and a first program according to the present technology, an associative High Efficiency Image File Format (HEIF) file is generated. In the associative High Efficiency Image File Format file, an image compliant with HEIF and specific information of external data outside the specified HEIF file are stored in association with each other, and the external data is associated with the image.

[0013] A file reproduction device or a second program according to the present technology is a file reproduction device including a file control unit that reproduces an associative high-efficiency image file format (HEIF) file in which an image compliant with HEIF and specific information of external data outside the specified HEIF file are stored in association with each other, the external data being associated with the image, or a program that enables a computer to function as such a file reproduction device.

[0014] A file reproduction method according to the present technology is a file reproduction method including reproducing an associated High Efficiency Image File Format (HEIF) file, in which an image compliant with HEIF and specific information of external data outside the specified HEIF file are stored in association with each other, and the external data is associated with the image.

[0015] In a file reproduction device, a file reproduction method, and a second program according to the present technology, an associative high-efficiency image file format (HEIF) file is reproduced, in which an image compliant with HEIF and specific information of external data outside the specified HEIF file are stored in association with each other, and the external data is associated with the image.

[0016] Note that the file generating device and the file reproducing device may be independent devices, or internal blocks constituting one device.

[0017] Furthermore, the first program and the second program may be provided by being recorded on a recording medium or transmitted via a transmission medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a block diagram illustrating a configuration example of one embodiment of a digital camera to which the present technology is applied.

[0019] Figure 2is a diagram illustrating an example of the format of a JPEG file conforming to the Joint Photographic Experts Group (JPEG).

[0020] Figure 3 is a diagram illustrating an example of the ISO base media file format.

[0021] Figure 4 is a diagram illustrating an example of the format of a HEIF file conforming to HEIF.

[0022] Figure 5 is a diagram illustrating an example of the format of a HEIF file in the form of image items.

[0023] Figure 6 is a diagram illustrating an example of an iprp box.

[0024] Figure 7 is a diagram illustrating an example of the format of a HEIF file in the form of an image sequence.

[0025] Figure 8 is a diagram illustrating an example of a trak box.

[0026] Fig. 9 is a diagram illustrating an example of a general set file storing a main image and thumbnail images.

[0027] Fig.10 is a diagram illustrating an example of a first associative collection file.

[0028] Fig.11 is a diagram illustrating an example of a second associative collection file.

[0029] Fig.12 is a diagram illustrating an example of a third associative collection file.

[0030] Fig.13 is a diagram illustrating an example of a general sequence file storing a track of a main image and a track of thumbnail images of the main image.

[0031] Fig.14 is a diagram illustrating an example of an associative sequence file.

[0032] Fig.15 : is a flowchart illustrating an overview of an example of a generation process for generating an associative HEIF file.

[0033] Fig.16 : is a flowchart illustrating an overview of an example of a reproduction process of reproducing an associative HEIF file.

[0034] Fig.17 is a flowchart illustrating an example of a reproduction process of a reproduction set file.

[0035] Fig.18: is a flowchart illustrating an example of a process of reading a reproduction target image in step S32 .

[0036] Fig.19 : is a flowchart illustrating a first example of the process of acquiring the reproduction target item ID in step S31.

[0037] Fig. 20 : is a flowchart illustrating a second example of the process of acquiring the reproduction target item ID in step S31.

[0038] Fig.21 is a flowchart illustrating an example of a process of acquiring uuid of specific information of a RAW file which is a predetermined main image from a first association type collection file.

[0039] Fig. 22 : is a flowchart illustrating an example of a process of acquiring uuid of specific information of a RAW file which is a predetermined main image from a second association type collection file.

[0040] Fig.23 : is a flowchart illustrating an example of a process of acquiring uuid of specific information of a RAW file which is a predetermined main image from a third association type collection file.

[0041] Fig.24 : is a flowchart illustrating an example of a process of acquiring a list of item IDs of main images from a collection file.

[0042] Fig.25 : is a flowchart illustrating an example of a process of reproducing thumbnail images of (frames of) a main image for predetermined time point information from a sequence file.

[0043] Fig.26 : is a flowchart illustrating an example of a process of acquiring uuid, which is specific information of a RAW file of (a frame of) a predetermined main image, from a linked sequence file.

[0044] Fig. 27 : is a diagram illustrating an example of storing uuid in the RAW file of the main image when the RAW file is adopted as external data and an associative set file is generated.

[0045] Fig.28 : is a diagram illustrating an example of storing uuid in the RAW file of the main image in the case of adopting the RAW file as external data and generating a linked sequence file.

[0046] Fig.29 : is a diagram illustrating an example of storing uuid in the WAV file of the main image when a WAV file is adopted as external data and an associative set file is generated.

[0047] Fig.30: is a diagram illustrating an example of storing uuid in the WAV file of the main image in the case of adopting a WAV file as external data and generating an associated sequence file.

[0048] Fig.31 is a block diagram illustrating a configuration example of an embodiment of a computer to which the present technology is applied. DETAILED DESCRIPTION

[0049] <One embodiment of a digital camera to which the present technology is applied>

[0050] Figure 1 is a block diagram illustrating a configuration example of one embodiment of a digital camera to which the present technology is applied.

[0051] The digital camera 10 includes an optical system 11 , an image sensor 12 , a signal processing unit 13 , a medium 14 , interfaces 15 and 16 , buttons / keys 17 , a touch panel 18 , a liquid crystal panel 19 , a viewfinder 20 , an interface 21 , and the like.

[0052] The optical system 11 condenses light from a subject on the image sensor 12 .

[0053] The image sensor 12 receives light from the optical system 11 and performs imaging for photoelectric conversion to generate data of an image as an electric signal and supplies the data to the signal processing unit 13 .

[0054] The signal processing unit 13 includes an optical system / image sensor control unit 41 , an encoding control unit 42 , a file control unit 43 , a medium control unit 44 , an operation control unit 45 , a display control unit 46 , and a UI control unit 47 .

[0055] The optical system / image sensor control unit 41 controls the optical system 11 and the image sensor 12 , and supplies (data of) an image obtained by imaging performed according to the control to the encoding control unit 42 .

[0056] The encoding control unit 42 supplies the image from the optical system / image sensor control unit 41 to the display control unit 46, and, if necessary, encodes the image and supplies the image to the file control unit 43. Furthermore, the encoding control unit 42 decodes the image supplied from the file control unit 43 and supplies the image to the display control unit 46 if necessary.

[0057] The file control unit 43 generates a file storing the image supplied from the encoding control unit 42, and supplies the file to the medium control unit 44. In addition, the file control unit 43 reproduces the file supplied from the medium control unit 44, that is, reads data such as the image stored in the file. For example, the image read from the file is supplied from the file control unit 43 to the encoding control unit 42.

[0058] The medium control unit 44 controls file exchange with the medium 14 and the interfaces 15 and 16. For example, the medium control unit 44 causes a file from the file control unit 43 to be recorded on the medium 14 or to be transferred from the interfaces 15 and 16. In addition, the medium control unit 44 reproduces a file from the medium 14 or causes a file to be received by the interfaces 15 and 16 and supplied to the file control unit 43.

[0059] The operation control unit 45 supplies an operation signal corresponding to the operation to a necessary block according to the operation of the button / key 17 or the touch panel 18 by the user.

[0060] The display control unit 46 performs display control and the like to supply the image and the like supplied from the encoding control unit 42 to the liquid crystal panel 19 , the viewfinder 20 , and the interface 21 and display the image and the like.

[0061] The UI control unit 47 manages user interface (UI) control.

[0062] The medium 14 is, for example, a storage medium such as an SD card. The interface 15 is, for example, an interface of a local area network (LAN) such as WiFi (registered trademark) or Ethernet (registered trademark). The interface 16 is, for example, a universal serial bus (USB) interface. When a command or other information is input to the digital camera 10, the button / key 17 and the touch panel 18 are operated by the user. The touch panel 18 may be formed integrally with the liquid crystal panel 19. The liquid crystal panel 19 and the viewfinder 20 display an image supplied from the display control unit 46, etc. The interface 21 is an interface for transmitting at least an image, such as a high-definition multimedia interface (HDMI) (registered trademark) or a display port (DP).

[0063] In the digital camera 10 configured as described above, the encoding control unit 42 generates a YUV image (hereinafter also referred to as a main image) having, for example, the same resolution (number of pixels) as the resolution of the RAW image from the image in the RAW data obtained by imaging by the image sensor 12 (hereinafter also referred to as a RAW image). Moreover, the encoding control unit 42 generates, from the YUV main image, a YUV image (hereinafter also referred to as a screen nail image) having, for example, a resolution lower than the resolution of the main image for display on the liquid crystal panel 19 or on an external display as a first other image based on the main image, and generates, for example, a YUV image (hereinafter also referred to as a thumbnail image) having a resolution lower than the resolution of the screen thumbnail image for list display as a second other image based on the main image. The encoding control unit 42 supplies the screen thumbnail image to the liquid crystal panel 19 via the display control unit 46, for example, and displays the screen thumbnail image as a so-called through-the-lens image. As the thumbnail image, for example, an image having a size of 320 pixels or less on the long side can be used. The size (number of pixels) of the main image can be set to, for example, two hundred times or less the size (number of pixels) of a thumbnail image of a screen as a first other image based on the main image or a thumbnail image of a second other image based on the main image. Similarly, the size of a thumbnail image of a screen as a first other image based on the main image can be set to, for example, two hundred times or less the size of a thumbnail image of a second other image based on the main image. As a screen thumbnail image, for example, an image with a resolution of 4K or higher can be used. In addition, as a screen thumbnail image, for example, a 4K (QFHD) or FHD image can be used according to the user's selection. Moreover, an image with the same resolution can be used as the current image and the screen thumbnail image. In the case of using an image with the same resolution as the current image and the screen thumbnail image, both the current image and the screen thumbnail image can be stored in a HEIF file, or the current image can be stored in a HEIF file without storing the screen thumbnail image therein. In the case of storing the current image in a HEIF file without storing the screen thumbnail image therein, the current image can be resized and used as a screen thumbnail image.

[0064] Furthermore, the encoding control unit 42 encodes a main image, screen thumbnail image, and thumbnail image corresponding to the RAW image (generated from the RAW image) as necessary, and supplies the encoded image to the file control unit 43 together with the RAW image.

[0065] The file control unit 43 generates a RAW file storing a RAW image, and generates a HEIF file or a JPEG file storing a corresponding main image, a screen thumbnail image, and a thumbnail image (the main image, the screen thumbnail image, and the thumbnail image generated from the same RAW image), and supplies the HEIF file or the JPEG file to the medium control unit 44. The HEIF file is a file conforming to the High Efficiency Image File Format (HEIF), and the JPEG file is a file conforming to the Joint Photographic Experts Group (JPEG).

[0066] The medium control unit 44 causes the RAW file and the HEIF file or the JPEG file from the file control unit 43 to be recorded on the medium 14, or causes the RAW file and the HEIF file or the JPEG file to be transmitted from the interface 15 or 16.

[0067] For example, which one of the HEIF file and the JPEG file is to be generated in the file control unit 43 can be selected according to a user operation. Further, as will be described later, there are an image item form and an image sequence form as formats of the HEIF file. For example, which one of the image item form and the image sequence form is to be adopted can be selected according to a user operation. Moreover, the file control unit 43 can perform mutual conversion between the HEIF file and the JPEG file according to a user operation.

[0068] Moreover, when generating the HEIF file, the file control unit 43 can associate internal data (data stored in the HEIF file) in the HEIF file with external data outside the HEIF file (data not stored in the HEIF file) and specific information specifying the external data, and store the internal data and the specific information in the HEIF file in association with each other. The HEIF file in which the internal data and the specific information of the external data to be associated with the internal data are stored in association with each other is also referred to as an associated HEIF file. In the associated HEIF file, the internal data and the specific information can be stored in association with each other, for example, by storing association information associating the internal data and the specific information.

[0069] <JPEG File>

[0070] Figure 2 is a diagram showing an example of the format of a JPEG file conforming to the Joint Photographic Experts Group (JPEG).

[0071] The JPEG file is configured to store, for example, Exif metadata, a thumbnail image, Extensible Metadata Platform (XMP) (registered trademark) metadata, an MPF indicating the storage location (place) of the main image and a simplified display image, the main image, and a simplified display image. As the simplified display image, for example, a screen thumbnail image can be adopted.

[0072] <ISO Base Media File Format>

[0073] Figure 3 is a figure showing an example of the ISO Base Media File Format.

[0074] HEIF (ISO / IEC 23008-12) is a file format that conforms to the ISO Base Media File Format (ISO / IEC 14496-12), so HEIF files conform to the ISO Base Media File Format.

[0075] The ISO Base Media File Format includes units called boxes as containers for storing data and has a structure called a box structure.

[0076] A box includes a type (box type), actual data (data), etc. The type indicates the type of the actual data inside the box. As the actual data, reproducible media data such as images (still images, moving images), audio, or subtitles (subtitles), attribute names (field names), attribute values (field values) of (variables represented by) attribute names, and various other data can be adopted.

[0077] Moreover, a box can be adopted as the actual data. That is, a box can have a box as the actual data, thus realizing a hierarchical structure.

[0078] A basic media file that conforms to the ISO Base Media File Format can include a ftyp box, a moov box (MovieBox), a meta box (MetaBox), a mdat box (MediaDataBox), etc. In the ftyp box, identification information for identifying the file format is stored. The moov box can store trak boxes, etc. The meta box can store iinf boxes, iprp boxes, iref boxes, iloc boxes, etc. The mdat box can store media data (AV data) and any other data.

[0079] HEIF conforms to the above ISO Base Media File Format.

[0080] <HEIF File>

[0081] Figure 4 is a figure showing an example of the format of a HEIF file that conforms to HEIF.

[0082] HEIF files are roughly divided into an image item form and an image sequence form. Moreover, as the image item form, there are a single image form having only one item described later and an image set form having multiple items.

[0083] A HEIF file in the image item form includes a ftyp box, a meta box, and a mdat box.

[0084] A HEIF file in the form of an image sequence includes an ftyp box, a moov box, and an mdat box.

[0085] Note that a HEIF file may include not only one of a meta box and a moov box, but may include both a meta box and a moov box.

[0086] The ftyp box stores identification information for identifying a file format, which indicates, for example, whether the file is a HEIF file in an image item form or an image sequence form.

[0087] In the meta box and the moov box, metadata required for reproduction, management, etc. of the media data stored in the mdat box (for example, metadata such as the storage location of the media data, etc.) is stored.

[0088] Media data (AV data) and the like are stored in the mdat box.

[0089] In the digital camera 10, for example, it is possible to select which of an image item form HEIF file or an image sequence form HEIF file to generate according to a user's operation. In addition, in the case where an image is encoded and stored in the mdat of a HEIF file, only intra-frame coding is allowed in the image item form, while intra-frame coding and inter-frame coding are allowed in the image sequence form. Therefore, for example, in the case where priority is given to high-speed access to data stored in a HEIF file, it is possible to select the generation of an image item form HEIF file, and in the case where priority is given to reducing the size (data amount) of the HEIF file, it is possible to select the generation of an image sequence form HEIF file.

[0090] Figure 5 is a diagram illustrating an example of the format of a HEIF file in the form of image items.

[0091] In a HEIF file in the image item form, information indicating that the HEIF file is in the image item form (for example, mif1, etc.) is stored in the ftyp box (as an attribute value).

[0092] In the meta box, an iinf box, an iref box, an iprp box, and an iloc box are stored.

[0093] In the iinf box, the number of items (attribute names and attribute values) as media data (AV data) stored in the mdat box, etc. are stored (indicated). An item is data stored in the mdat box of a HEIF file in the form of an image item, for example, one (picture) image is an item. In this specification, regardless of whether the image is a still image or a moving image, one image is also called a frame. One frame is one item.

[0094] In the iref box, information indicating the relationship between items is stored. For example, in the mdat box, each of the corresponding main image, the corresponding screen thumbnail image, and the corresponding thumbnail image can be stored as an item. In the case where item I1 as the main image, item I2 as the screen thumbnail image, and item I3 as the thumbnail image are stored in the mdat box, information indicating that item I2 is a screen thumbnail image of the main image of item I1 and information indicating that item I3 is a thumbnail image of the main image of item I1 are stored in the iref box.

[0095] In the iprp box, information about the characteristics of the item is stored.

[0096] In the iloc box, information on the storage location of the item stored in the mdat box is stored.

[0097] In the mdat box (of a HEIF file) in the form of an image item, for example, a frame of an image as an item is stored. One or more items can be stored in the mdat box. In addition, the frame as an item can be encoded and stored in the mdat box. However, the encoding of the frame as an item stored in the mdat box in the form of an image item is limited to intra-frame encoding. As a coding scheme (codec) for encoding the frame as an item, HEVC, etc. can be used, for example.

[0098] Figure 6 It is a graphic Figure 5 Figure 1 shows an example of an iprp box in FIG.

[0099] In the iprp box, the ipco box and the ipma box about the characteristics of the item are stored. In the ipco box, the characteristics of the item stored in the mdat box, for example, codec information about the codec of the image as the item and image size information about the size of the image are stored. In the ipma box, the index (pointer) of the item stored in the mdat box to the characteristics stored in the ipco box is stored.

[0100] Figure 7 is a diagram illustrating an example of the format of a HEIF file in the form of an image sequence.

[0101] In a HEIF file in the form of an image sequence, information indicating that the HEIF file is in the form of an image sequence is stored in an ftyp box, such as msf1.

[0102] In the moov box, a trak box is stored. In the trak box, information about the track stored in the mdat box is stored.

[0103] A track includes an independent media data such as an image or audio that is reproduced according to a timeline. For example, a track includes one or more frames of an image as an elementary stream. With respect to tracks stored in an mdat box, multiple tracks can be reproduced simultaneously, for example, tracks of images and audio that are recorded simultaneously.

[0104] The media data of a track includes units called samples. A sample is the minimum unit (access unit) in the case of accessing the media data in a HEIF file. Therefore, the media data in a HEIF file cannot be accessed in units finer than a sample.

[0105] Regarding image media data, for example, one frame is one sample. Also, regarding audio media data, for example, one audio frame is one sample as defined in the standard of audio media data.

[0106] In the mdat box (of a HEIF file) in the form of an image sequence, the media data of a track is arranged in units called chunks. A chunk is a collection of one or more samples arranged at logically consecutive addresses.

[0107] In the case where a plurality of tracks as media data are stored in the mdat box, the plurality of tracks are interleaved and arranged in units of blocks.

[0108] In the moov box, trak boxes that respectively manage the tracks stored in the mdat box are stored.

[0109] In the mdat box in the form of an image sequence, one or more tracks including media data such as images or audio are stored. In the mdat box, frames of images constituting the track can be encoded and stored. In the encoding of the frames constituting the track stored in the mdat box in the form of an image sequence, a long group of pictures (GOP) can be adopted as the GOP, and both intra-frame encoding and inter-frame encoding can be adopted. As a codec for encoding the frames constituting the track, for example, HEVC or the like can be adopted.

[0110] Figure 8 is a diagram illustrating an example of a trak box.

[0111] In the trak box, a tkhd box and an mdia box may be stored. In the tkhd box, header information of the track managed by the trak box, such as the creation date and time of the track, is stored. In the mdia box, a minf box and the like are stored. In the minf box, a stbl box is stored. In the stbl box, a stsd box, a stsc box, a stsz box, and a stco box are stored, which store information for accessing samples and thus blocks of the track. In the stsd box, codec information about the codec of the track is stored. In the stsc box, a block size (the number of samples of one block) is stored. In the stsz box, a sample size is stored. In the stco box, a block offset, that is, an offset of the arrangement location of each block of the track stored in the mdat box, is stored.

[0112] Here, HEIF files in the form of image items are also referred to as collection files, and HEIF files in the form of image sequences are also referred to as sequence files. Moreover, associated HEIF files in the form of image items are also referred to as associated collection files, and associated HEIF files in the form of image sequences are also referred to as associated sequence files.

[0113] In the digital camera 10 , a HEIF file (including an associated HEIF file) storing a main image and a desired screen thumbnail image or one of desired thumbnail images, or both the screen thumbnail image and the thumbnail image may be generated.

[0114] <Collection File>

[0115] Fig. 9 FIG. 1 is a diagram showing an example of a general set file storing a main image and thumbnail images.

[0116] Here, generally, a collection file refers to a collection file in which internal data in the collection file is not associated with specific information of external data.

[0117] Now, assume that the frames (items) are encoded by HEVC and stored in the mdat box of the collection file.

[0118] In the ftyp box, as identification information for identifying the file format, heic indicating that the format is an image item form and the codec is HEVC is stored.

[0119] In the iinf box, the number of items stored in the mdat box is stored. Fig. 9In the mdat box, a total of four items (frames) - that is, the main image specified by item ID #1 (hereinafter also referred to as main image Item #1), main image Item #2, the thumbnail image specified by item ID #101 (hereinafter also referred to as thumbnail image Item #101), and thumbnail image Item #102 are stored. Therefore, the number of items is four. Note that thumbnail image Item #101 is a thumbnail image of main image Item #1, and thumbnail image Item #102 is a thumbnail image of main image Item #2.

[0120] Furthermore, in the iinf box, for example, an infe box is stored for each item stored in the mdat box. In the infe box, an item ID and an item type for specifying an item are registered. Fig. 9 In the image, there are Infe boxes for the main images Item#1 and Item#2 and the thumbnail images Item#101 and Item#102.

[0121] In the iref box, for example, a thmb box is stored as information for associating items stored in the mdat box. In the thmb box, a reference source and a reference destination as information for associating a main image with a thumbnail image of the main image are stored in association with each other. In the thmb box, the reference source represents the item ID of the main image, and the reference destination represents the item ID of the thumbnail image of the main image specified by the item ID of the reference source. Therefore, based on the reference destination corresponding to the reference source, the item ID of the thumbnail image of the main image specified by the item ID represented by the reference source can be identified. In addition, based on the reference source corresponding to the reference destination, the item ID of the main image of the thumbnail image specified by the item ID represented by the reference destination can be identified.

[0122] like Figure 6 As shown in , the ipco box and ipma box are stored in the iprp box. In the ipco box, as Figure 6 As shown in , the frame characteristics as items stored in the mdat box, for example, codec information about the codec and image size information about the size are stored. In the ipma box, as shown in Figure 6 As shown in , the indexes of the items stored in the mdat box to the characteristics stored in the ipco box are stored.

[0123] In the iloc box, such as Figure 6 As shown in , information about the storage location of items in the mdat box is stored. Fig. 9 In the iloc box, information indicating that the number of items is four is stored. Also, in the iloc box, the offsets and sizes of the storage positions of the main images Item#1 and Item#2 and the thumbnail images Item#101 and Item#102 stored in the mdat box are stored in correspondence with the item ID.

[0124] In the following, specific information describing internal data and external data is stored in association with each other. Fig. 9 An associated collection file in a normal collection file.

[0125] Fig.10 is a diagram illustrating an example of a first associative collection file.

[0126] Here, for example, it is assumed hereinafter that a RAW file storing a RAW image of a main image which is internal data in a HEIF file is adopted as external data to be associated with the main image.

[0127] In the first associative set file, associative information is stored that associates a main image as internal data with specific information of a RAW file as external data (RAW file storing a RAW image of the main image as internal data), so that the specific information of the main image and the RAW file are stored in association with each other. Furthermore, in the first associative set file, the associative information is stored in a meta box.

[0128] As specific information of a RAW file as external data, a file name of the RAW file, a universal unique identifier (uuid) issued to the RAW file, a uniform resource locator (URL), and other arbitrary information that can specify the RAW file can be employed.

[0129] Regarding the first associative collection file, an associative information storage box storing associative information is defined as a new box to be stored in a meta box, and is stored in the meta box. In the associative information storage box of the first associative collection file, for example, associative information corresponding to an item ID for specifying a main image and a uuid as specific information specifying a RAW file associated with the main image is stored. Moreover, in the associative information storage box, the number of main images associated with the RAW file (the number of main images) is stored. Since the number of main images stored in the associative information storage box is the number of main images associated with the RAW file, the number of main images is a value equal to or smaller than the number of main images stored in the mdat box.

[0130] exist Fig.10 In the example, the RAW file of the main image Item#1 has the uuid UUID#1, and the RAW file of the main image Item#2 has the uuid UUID#2. Now, assuming that the RAW file with the uuid UUID#i is called RAW file UUID#i, then in Fig.10In the associated information storage box, the associated information that item ID#1 of the main image Item#1 corresponds to the uuid of the RAW file UUID#1 and that item ID#2 of the main image Item#2 corresponds to the uuid of the RAW file UUID#2 is stored.

[0131] Fig.11 is a diagram illustrating an example of a second associative collection file.

[0132] Similar to the first associative set file, in the second associative set file, associative information associating the main image as internal data with the specific information of the RAW file as external data is stored, so the main image and the specific information of the RAW file are stored in association with each other. However, in the second associative set file, the associative information is stored in the mdat box.

[0133] Regarding the second associative set file, for example, associative information similar to that of the first associative set file is stored as an item in the mdat box. Fig.11 In the example, the associated information is stored in the mdat box as an item of item ID#201.

[0134] As described above, in the second associative collection file, since the associative information of item Item#201 is stored in the meta box, the information stored in the mdat box is different from Fig. 9 In the second associative collection file, metadata as associative information of item Item#201 is stored in the meta box.

[0135] Specifically, in the second associative collection file, the number of items stored in the iinf box and the iloc box is increased from Fig. 9 The four in the case of Item#201 are changed to five obtained by adding one to the four, that is, Item#201. Moreover, the infe box for the item Item#201 is added to the iinf box, and the offset and size of the storage position of the item Item#201 are added to the iloc box. In the infe box for the item Item#201, the item ID#201 of the item Item#201 and the item type identification data information (IDIF) indicating that the item Item#201 is the associated information are stored. IDIF is a newly defined attribute value (field value) indicating that the item is the associated information.

[0136] Fig.12 is a diagram illustrating an example of a third associative collection file.

[0137] In the third associative set file, the specific information of the RAW file as external data is stored as an item in the mdat box for each piece of specific information, and the associated information associating the main image as internal data with the specific information of the RAW file as external data is stored in the meta box, so the main image and the specific information of the RAW file are stored in association with each other. However, in the third associative set file, the associated information is information in which the item ID of the main image as an item corresponds to the item ID of the specific information (of the RAW file) as an item, and is stored in the cdsc box stored in the iref box in the meta box.

[0138] In the cdsc box, a reference source and a reference destination as information for associating an item as a main image with specific information of a RAW file of the main image may be corresponded and stored. In the cdsc box, the reference source indicates the item ID of the main image, and the reference destination indicates the item ID of specific information of an item of the RAW file of the main image specified by the item ID of the reference source.

[0139] exist Fig.12 In mdat, UUID#1 (which is the uuid of the specific information of the RAW file of the main image Item#1) is stored as item Item#201, and UUID#2 (which is the uuid of the specific information of the RAW file of the main image Item#2) is stored as item Item#202. Moreover, a cdsc box storing association information that corresponds item ID#1 of the main image Item#1 and item ID#201 of the specific information UUID#1 as a reference source and a reference destination is stored in the iref box, and a cdsc box storing association information that corresponds item ID#2 of the main image Item#2 and item ID#202 of the specific information UUID#2 as a reference source and a reference destination is stored in the iref box.

[0140] <sequence file>

[0141] Fig.13 is a diagram illustrating an example of a general sequence file storing a track of a main image and a track of thumbnail images of the main image.

[0142] Here, generally, a sequence refers to a sequence file in which internal data in the sequence file is not associated with specific information of external data.

[0143] Now, assume that the frames are encoded by HEVC and stored in the mdat box of the sequence file.

[0144] In the ftyp box, hevc indicating that the format is an image sequence form and the codec is HEVC is stored as identification information for identifying the file format.

[0145] In the moov box, such as Figure 7 As shown in , the trak boxes are stored to manage the tracks stored in the mdat box respectively. Fig.13 In the mdat box, a track of a main image specified by track ID #1 (hereinafter also referred to as track #1) and track #2 of a thumbnail image of the main image of track #1 are stored. Therefore, in the moov box, a trak box for managing track #1 and a trak box for managing track #2 are stored. The nth thumbnail image (frame) of track #2 (from the beginning) is a thumbnail image of the nth main image of track #1.

[0146] For example, a sequence file is useful in a case where main images of a plurality of frames are recorded as one track and thumbnail images of a plurality of frames are recorded as one track, and in a case where continuous shooting is performed, for example, frames are obtained by continuous shooting performed by the digital camera 10 .

[0147] In the tkhd box of the trak box that manages the main image track #1, the track ID #1 that specifies the track #1, the image size of the main image constituting the track #1, the rotation information indicating the orientation of the digital camera 10 when the main image was captured, and the creation date and time of the track #1 are stored. In the tkhd box of the trak box that manages the track #2 of the thumbnail images, the track ID #2 that specifies the track #2 and the creation date and time of the track #2 are stored.

[0148] In the trak box, in addition to Figure 7 In addition to the tkhd box and the mdia box shown in , a tref box may be stored. In the tref box, a track ID for specifying another track associated with the track managed by the trak box storing the tref box, information indicating the content of the track, and the like are stored. Fig.13 In the tref box, a tref box is set in the trak box that manages track #2. Then, in the tref box, information indicating that the other track related to track #2 is track #1 (track_ID=1) and that the data constituting track #2 is a thumbnail image (track #2 is a track of thumbnail images) (type=thmb) is stored.

[0149] In the trak box's mdia box, in addition to Figure 8In addition to the minf box shown in , an hdlr box may be stored. In the hdlr box, information indicating the type of data constituting the track managed by the trak box storing the hdlr box is stored. Information (pict) indicating that the data constituting track #1 is a picture (frame) is stored in the hdlr box stored in the trak box (stored in the mdia box) of track #1 managing the main image, and information indicating that the data constituting track #2 is a picture is stored in the hdlr box stored in the trak box of track #2 managing the thumbnail image.

[0150] minf box Figure 8 as shown in .

[0151] In the following, it will be described Fig.13 An associative sequence file that stores specific information of internal data and external data in association with each other in a general sequence file.

[0152] Fig.14 is a diagram illustrating an example of an associative sequence file.

[0153] In the associated sequence file, track #3 of the (elementary) stream (Meta ES) of the uuid of the specific information of the RAW file as external data is added to the mdat box, and a trak box managing track #3 is added to the moov box.

[0154] Here, track #1 is a time sequence of one frame or a plurality of frames of a main image arranged on a timeline, and track #3 is a time sequence of uuids of RAW files of respective frames of the main image arranged on a timeline.

[0155] The nth uuid (from the beginning) of track #3 is specific information of the RAW file of the nth frame of the main image of track #1. In addition, multiple tracks (data) stored in the mdat box can be synchronously reproduced according to the time point information on one timeline. Therefore, by storing the uuid (stream) of the RAW file of each frame of the main image of track #1 and track #3 constituting track #1 in the mdat box, the nth frame of the main image of track #1 and the uuid of the RAW file of (the frame of) the main image are stored in association with each other. In this case, it can be said that the uuid of the frame of the main image of track #1 and the RAW file of (the frame of) the main image can be associated with each other through the time point information on the timeline.

[0156] Note that it can be understood that the nth uuid (from the beginning) of track #3 is specific information of the RAW file of the nth frame of track #1, and the uuids of the main image (frame) constituting track #1 and the uuids constituting track #3 are associated with each other in the order of arrangement in the tracks.

[0157] In an associated sequence file, since the track #3 of the uuid of the RAW file is added to the mdat box, the trak box that manages track #3 is added to the moov box.

[0158] In the trak box that manages track #3 of the uuid of the RAW file, the tkhd box, the tref box, the mdia box, etc. are stored.

[0159] In the tkhd box of the trak box that manages track #3, the track ID #3 of track #3 and the creation date and time of track #3 are stored.

[0160] In the tref box of the trak box that manages track #3, the track IDs of other tracks related to track #3 specified by the trak box storing the tref box, information indicating the content of track #3, etc. are stored. Since the uuid configuring track #3 is specific information of the RAW file that is the main image of track #1 and track #3 is related to track #1, information indicating that the other track related to track #3 is track #1 (track_ID = 1) and track #3 is metadata (here specific information) (type = cdsc) is stored in the tref box of the trak box that manages Fig.14 track #3.

[0161] In the mdia box of the trak box that manages track #3, the hdlr box and the minf box are stored. In the trak box that manages track #3, information (meta) indicating that the data constituting track #3 is metadata (of the main image) is stored in the hdlr box, and the stsc box, the stsc box, the stsz box, and the stco box regarding track #3 are stored in the minf box.

[0162] <Generation and Reproduction of HEIF File>

[0163] Fig.15 is a flowchart outlining an example of the generation process for generating an associated HEIF file.

[0164] In the generation process, in step S11, the file control unit 43 generates a uuid that is specific information of the RAW file of the frame that is the main image, and the process proceeds to step S12.

[0165] In step S12, the file control unit 43 assigns the uuid generated in step S11 to the RAW file of the frame that is the main image, and the process proceeds to step S13.

[0166] In step S13 , the file control unit 43 generates an associated HEIF file in which the uuids of the frame of the main image and the RAW file of the frame are stored in the HEIF file in association with each other, and terminates the generation process.

[0167] Fig.16 : is a flowchart illustrating an overview of an example of a reproduction process of reproducing an associative HEIF file.

[0168] In the reproduction process, in step S21 , the file control unit 43 generates, for example, a handle list of handles for respectively identifying frames of a main image stored in a HEIF file saved in the medium 14 , and the process proceeds to step S22 .

[0169] Here, the handle of the frame of the main image includes the file name of the HEIF file in which the frame is stored. The handle of the frame (item) of the main image stored in the collection file also includes the item ID of the frame. The handle of the frame of the main image stored in the sequence file includes the time point information of the frame. Using the handle of the frame of the main image, the frame used for the handle can be uniquely identified (specified).

[0170] Note that the handle of the frame of the main image stored in the sequence file may include the track ID of the track containing the frame and the order of the frame in the track (where the frame is arranged), rather than the time point information of the frame.

[0171] Regardless of whether one track or a plurality of tracks including the frame of the main image is stored in the sequence file, the time point information of each frame is unique. Therefore, according to the time point information of the frame, even if a plurality of tracks are stored in the sequence file, the frame of the time point information included in the handle can be uniquely specified from the frames constituting the plurality of corresponding tracks. Therefore, in the case where the time point information of the frame of the main image is included in the handle of the frame, even if there is no track ID of the track where the frame exists, the frame corresponding to the time point information can be uniquely specified.

[0172] The handle list may be generated for all frames of the main image stored in the HEIF file saved in the medium 14, or may be generated only for frames limited under certain conditions, such as frames with a certain creation date and time.

[0173] After generating the handle list, the file control unit 43 accesses the HEIF file with reference to the handle list as needed.

[0174] In step S22, for example, the UI control unit 47 waits for the user to operate the digital camera 10, for example, to display the thumbnail image, and requests the file control unit 43 to display the thumbnail image. The file control unit 43 reads the thumbnail image (frame) of the frame of the main image identified by the handle of the handle list from the HEIF file in response to the display request of the thumbnail image from the UI control unit 47. Then, the file control unit 43 displays the thumbnail image on, for example, the liquid crystal panel 19 ( Figure 1 ) is displayed on the , and the processing proceeds from step S22 to step S23.

[0175] In step S23, for example, after waiting for the user to select a desired thumbnail (frame) from the list of thumbnail images, the UI control unit 47 requests the file control unit 43 for a main image corresponding to the thumbnail image selected by the user. In response to the request for the main image from the UI control unit 47, the file control unit 43 reads the main image from the HEIF file. The file control unit 43 can display the main image read from the HEIF file on the liquid crystal panel 19 as needed.

[0176] Alternatively, the UI control unit 47 requests the file control unit 43 to supply the uuid of the RAW file of the main image corresponding to the thumbnail image selected by the user. In response to the request for the uuid from the UI control unit 47, the file control unit 43 reads the uuid from the associated HEIF file. The file control unit 43 can access the RAW file specified by the uuid read from the associated HEIF file as needed.

[0177] Fig.17 is a flowchart illustrating an example of a reproduction process of a reproduction set file.

[0178] In step S31 , the file control unit 43 acquires the item ID of the reproduction target image which is the image (item) to be reproduced (hereinafter also referred to as the reproduction target item ID), and the process proceeds to step S32 .

[0179] When obtaining the reproduction target item ID, for example, a main image identified by any handle of the handle list, a thumbnail image of the main image, a thumbnail image selected by a user from a list of thumbnail images (hereinafter also referred to as a selected thumbnail image), a main image of the selected thumbnail image, etc. are set as the reproduction target image, and the item ID of the reproduction target image (reproduction target item ID) is obtained.

[0180] In step S32 , the file control unit 43 reads the reproduction target image according to the reproduction target item ID acquired in step S31 .

[0181] When reading a reproduction target image, the reproduction target image specified by the reproduction target item ID is read from the set file.

[0182] Fig.18 It is a graphic Fig.17 Flowchart of an example of a process of reading a reproduction target image in step S32 of FIG.

[0183] In step S41, the file control unit 43 collects the files ( Figures 9 to 12 )’s iloc box, and the processing proceeds to step S42.

[0184] In step S42, in the iloc box, the file control unit 43 reads the offset and size corresponding to the reproduction target item ID searched in step S41, and the process proceeds to step S43.

[0185] In step S43, the file control unit 43 reads the reproduction target image stored in the mdat box of the set file according to the offset and size corresponding to the reproduction target item ID, and the processing is terminated.

[0186] Fig.19 It is a graphic Fig.17 Flowchart of a first example of the process of acquiring the reproduction target item ID in step S31 of FIG.

[0187] Right now, Fig.19 An example is illustrated in which a thumbnail image is set as a reproduction target image and the item ID of the thumbnail image as the reproduction target image is acquired.

[0188] Note that in Fig.19 In the above description, it is assumed that, for example, the file control unit 43 recognizes the item ID of the main image of the thumbnail image from the handle as the reproduction target image.

[0189] In step S51, the file control unit 43 collects the files ( Figures 9 to 12 ) searches for a thmb box whose reference source matches the item ID of the main image in the thmb box in the iref box, and the processing proceeds to step S52.

[0190] In step S52, the file control unit 43 reads the reference destination whose reference source matches the item ID of the main image in the thmb box which has been searched in step S51 as the item ID of the thumbnail image as the reproduction target image, and the processing is terminated.

[0191] Fig. 20 It is a graphic Fig.17 Flowchart of a second example of the process of acquiring the reproduction target item ID in step S31 of FIG.

[0192] Right now, Fig. 20 An example is illustrated in which a main image is set as a reproduction target image and the item ID of the main image as the reproduction target image is acquired.

[0193] Note that in Fig. 20 In the example, it is assumed that the user selects a thumbnail image (selected thumbnail image) from the list of thumbnail images, and the file control unit 43 recognizes the item ID of the selected thumbnail image.

[0194] In step S61, the file control unit 43 collects the files ( Figures 9 to 12 ) searches the thmb box in the iref box of the image processing unit for a thmb box whose reference destination matches the item ID of the selected thumbnail image, and the processing proceeds to step S62.

[0195] In step S62, the file control unit 43 reads the reference source whose reference destination in the thmb box matches the item ID of the selected thumbnail image, which thmb box has been searched in step S61, as the item ID of the main image as the reproduction target image, and the processing is terminated.

[0196] Fig.21 The diagram is from Fig.10 A flowchart of an example of a process of acquiring a uuid of specific information of a RAW file which is a predetermined main image in a first associative set file.

[0197] Note that in Fig.21 In the example, it is assumed that the file control unit 43 identifies the item ID of a predetermined main image by using a handle list or the like.

[0198] In step S71, the file control unit 43 generates a first associative set file ( Fig.10 ) is searched for the item ID of the predetermined main image in the associated information in the associated information storage box, and the processing proceeds to step S72.

[0199] In step S72 , the file control unit 43 reads the uuid corresponding to the item ID of the predetermined main image searched in step S71 in the association information, and the process terminates.

[0200] The file control unit 43 can access the RAW file of the predetermined master image by using the uuid read as described above.

[0201] Fig. 22 The diagram is from Fig.11 A flowchart of an example of a process of acquiring a uuid of specific information of a RAW file which is a predetermined main image in a second associative set file.

[0202] Note that in Fig. 22In the example, it is assumed that the file control unit 43 identifies the item ID of a predetermined main image by using a handle list or the like.

[0203] In step S81, the file control unit 43 generates a second associative set file ( Fig.11 ) searches for an infe box having an item type IDIF indicating that the item is associated information among the infe boxes in the iinf box of the associated information, and the processing proceeds to step S82.

[0204] In step S82, the file control unit 43 reads the item ID as the associated information of the item from the infe box having the item type IDIF searched in step S81, and the process proceeds to step S83.

[0205] In step S83, the file control unit 43 searches the iloc box of the second associative collection file for the item ID of the associative information read in step S82, and the process proceeds to step S84.

[0206] In step S84 , in the iloc box, the file control unit 43 reads the offset and size corresponding to the item ID of the associated information searched in step S83 , and the process proceeds to step S85 .

[0207] In step S85, the file control unit 43 reads the associated information of the item stored in the mdat box of the second associative collection file according to the offset and size corresponding to the item ID of the associated information, which have been read in step S84, and the processing proceeds to step S86.

[0208] In step S86 , the file control unit 43 searches for the item ID of the predetermined main image in the associated information read in step S85 , and the process proceeds to step S87 .

[0209] In step S87 , the file control unit 43 reads the uuid corresponding to the item ID of the predetermined main image searched in step S86 in the association information, and the process terminates.

[0210] The file control unit 43 can access the RAW file of the predetermined master image by using the uuid read as described above.

[0211] Fig.23 The diagram is from Fig.12 A flowchart of an example of a process of acquiring a uuid of specific information of a RAW file which is a predetermined main image of a third association type set file.

[0212] Note that in Fig.23 In the example, it is assumed that the file control unit 43 identifies the item ID of a predetermined main image by using a handle list or the like.

[0213] In step S91, the file control unit 43 generates a third association type set file ( Fig.12 ) searches for a cdsc box in the iref box whose reference source matches the item ID of the predetermined main image, and the processing continues to step S92.

[0214] In step S92, the file control unit 43 reads the reference destination whose reference source in the cdsc box matches the item ID of the predetermined main image (the cdsc box has been searched in step S91) as the item ID of the specific information of the RAW file of the predetermined main image as the item, and the processing proceeds to step S93.

[0215] In step S93, the file control unit 43 searches the iloc box of the third associative set file for the item ID as specific information of the item, which item ID has been read in step S92, and the processing proceeds to step S94.

[0216] In step S94, in the iloc box, the file control unit 43 reads the offset and size corresponding to the item ID of the specific information, which has been searched in step S93, and the process proceeds to step S95.

[0217] In step S95, the file control unit 43 reads the uuid of the specific information of the RAW file of the predetermined main image stored in the mdat box of the third associative collection file according to the offset and size corresponding to the item ID of the specific information, which have been read in step S94, and the processing is terminated.

[0218] The file control unit 43 can access the RAW file of the predetermined master image by using the uuid read as described above.

[0219] Fig.24 : is a flowchart illustrating an example of a process of acquiring a list of item IDs of main images from a collection file.

[0220] For example, in the case of generating a handle list, a process of acquiring a list of item IDs of main images from a collection file is performed.

[0221] In step S101, the file control unit 43 selects the collection file ( Figures 9 to 12 ), the item ID is read from all infe boxes in the iinf box of the main image, the item ID is registered in the list of item IDs of the main image (hereinafter also referred to as the main image list), and the processing proceeds to step S102.

[0222] In step S102, the file control unit 43 reads the item ID as the reference destination from all the boxes in the iref box of the collection file, excludes the item ID from the main image list, and the process terminates.

[0223] After the above-described processing, the item ID registered in the main image list becomes the item ID of the main image.

[0224] Fig.25 is a flowchart illustrating an example of a process of reproducing a thumbnail image of (a frame of) a main image for predetermined point-in-time information from a sequence file.

[0225] Note that in Fig.25 In the description, for example, it is assumed that the file control unit 43 identifies the time point information (or order) of a predetermined main image by using a handle list or the like.

[0226] In step S111, the file control unit 43 generates a sequence file ( Fig.13 and Fig.14 ) searches the trak box in the moov box for information indicating that the data constituting the track is the trak box stored in the tref box of the thumbnail image - that is, the trak box of type thmb in the tref box serves as the trak box for managing the track of the thumbnail image of the main image for the predetermined time point information, and the processing proceeds to step S112.

[0227] In step S112, the file control unit 43 reads the track ID in the tkhd box in the trak box that has been searched in step S111 as the track ID of the track of the thumbnail image of the main image for the predetermined time point information, and the processing proceeds to step S113.

[0228] In step S113, the file control unit 43 reproduces the track having the track ID read in step S112, and obtains (frames of) thumbnail images for predetermined time point information (or order) from the track as thumbnail images of the main image for the predetermined time point information, and the processing is terminated.

[0229] Note that the process of reproducing images stored in a sequence file is similar to the process of reproducing moving images of an MP4 file.

[0230] Fig.26 : is a flowchart illustrating an example of a process of acquiring uuid of specific information of a RAW file (a frame of a predetermined main image) from a relational sequence file.

[0231] Note that in Fig.26 In the description, for example, it is assumed that the file control unit 43 identifies the time point information (or order) of a predetermined main image by using a handle list or the like.

[0232] In step S121, the file control unit 43 generates a sequence file ( Fig.14 ) searches the trak box in the moov box for information indicating that the data constituting the track is the specific information stored in the trak box in the tref box - that is, the trak box of type cdsc in the tref box as the trak box of the track managing the specific information, and the processing proceeds to step S122.

[0233] In step S122, the file control unit 43 reads the track ID in the tkhd box in the trak box which has been searched in step S121 as the track ID of the track of the specific information, and the processing proceeds to step S123.

[0234] In step S123 , the file control unit 43 acquires uuid which is specific information of time point information (or order) for a predetermined main image from the track having the track ID read in step S122 as the uuid of the RAW file of the predetermined main image, and the processing terminates.

[0235] The file control unit 43 can access the RAW file of the predetermined master image by using the uuid acquired as described above.

[0236] As described above, the file control unit 43 generates and reproduces an associative HEIF file in which a main image in a HEIF file conforming to HEIF and specific information of external data outside the designated HEIF file are stored in association with each other in the HEIF file, and the external data is to be associated with the main image. Therefore, the main image stored in the HEIF file can be associated with external data outside the HEIF file.

[0237] Furthermore, in the case of using UUID as specific information, even if the file name of the external data is changed, the association between the main image in the HEIF file and the external data whose file name has been changed can be maintained through UUID.

[0238] <Storage of specific information assigned to external data>

[0239] Fig. 27 : is a diagram illustrating an example of storing uuid in a RAW file when a RAW file of a main image is used as external data and an associative set file is generated.

[0240] Note that in Fig. 27 In the example, the first associative collection file is used as the associative collection file.

[0241] The RAW file has an area called a marker comment (MakerNote) as a partial area for storing additional information of Exif as metadata.

[0242] The file control unit 43 may store the uuid assigned to the RAW file in, for example, the MakerNote of the RAW file.

[0243] exist Fig. 27 In the associative set file, main images Item#1, Item#2, Item#3, and Item#4 are stored as four items, and RAW files #1, #2, #3, and #4 are generated which respectively store RAW images of main images Item#1, Item#2, Item#3, and Item#4. Then, UUID#i is assigned to RAW file #i, and as associative information associating main image Item#i with UUID#i of RAW file #i of main image Item#i, associative information associating item ID#i of main image Item#i and UUID#i of RAW file #i associated with main image Item#i, associative information associating item ID#i of main image Item#i is stored in an associative information storage box.

[0244] Fig.28 : is a diagram illustrating an example of storing uuid in a RAW file when a RAW file of a main image is used as external data and a linked sequence file is generated.

[0245] The file control unit 43 may also store the uuid assigned to the RAW file in the MakerNote of the RAW file when generating an associated sequence file, similar to Fig. 27 The situation of generating an associative collection file as shown in .

[0246] exist Fig.28 In the example, track #1 including main images #1, #2, #3 and #4 as four frames is stored in an associated sequence file, and RAW files #1, #2, #3 and #4 storing the RAW images of the main images #1, #2, #3 and #4, respectively, are generated. Then, UUID #i is assigned to RAW file #i, and track #3 arranged and configured so that UUID #i of RAW file #i has the same time point information as that of main image #i corresponding to RAW file #i is stored in the associated sequence file.

[0247] As described above, track #3 is configured so that the UUID #i of the RAW file #i is arranged to have the same time point information as the time point information of the main image #i corresponding to the RAW file #i, so that the i-th main image #i of track #1 and the i-th UUID #i of track #3 (i.e., the UUID #i of the RAW file #i of the main image #i) are associated with each other and stored in the associated sequence file.

[0248] In the above description, the RAW file of the main image is used as the external data; however, other data may be used as the external data. As the external data, for example, a file storing audio (sound) recorded together with the imaging of the main image may be used. As the file storing audio, for example, a WAV file in the WAV format, an MP4 file in the MP4 format, etc. may be used. In the following, it is assumed that, for example, a WAV file is used as the file storing audio.

[0249] Fig.29 is a diagram illustrating an example of storing uuid in a WAV file in the case where a WAV file of a main image (a WAV file storing audio recorded together with imaging of the main image) is adopted as external data and an associative collection file is generated.

[0250] Note that in Fig.29 In the example, the first associative collection file is used as the associative collection file.

[0251] The WAV file has an area called a list chunk as a part of the area where metadata is described.

[0252] The file control unit 43 may store the uuid assigned to the WAV file in, for example, a list block of the WAV file.

[0253] exist Fig.29 In the associative set file, main images Item#1, Item#2, Item#3, and Item#4 are stored as four items, and WAV files #1, #2, #3, and #4 of the main images Item#1, Item#2, Item#3, and Item#4 are generated. Then, UUID#i is assigned to WAV file #i, and as association information associating main image Item#i with UUID#i of WAV file #i of main image Item#i, association information associating item ID#i of designated main image Item#i with UUID#i of WAV file #i associated with main image Item#i is stored in an association information storage box.

[0254] Fig.30is a diagram illustrating an example of storing uuid in a WAV file in the case where a WAV file of a main image is adopted as external data and an associated sequence file is generated.

[0255] Even in the case of generating an associative sequence file, the file control unit 43 may store the uuid assigned to the WAV file in the list block of the WAV file, similar to Fig.29 The situation of generating an associative collection file as shown in .

[0256] exist Fig.30 In the example, track #1 including main images #1, #2, #3 and #4 as four frames is stored in the associated sequence file, and WAV files #1, #2, #3 and #4 of the main images #1, #2, #3 and #4 are generated. Then, UUID #i is assigned to WAV file #i, and track #3 arranged and configured so that UUID #i of RAW file #i has the same time point information as that of main image #i corresponding to RAW file #i is stored in the associated sequence file.

[0257] As described above, track #3 is configured so that the UUID #i of the RAW file #i is arranged to have the same time point information as the time point information of the main image #i corresponding to the RAW file #i, so that the i-th main image #i of track #1 and the i-th UUID #i of track #3 (i.e., the UUID #i of the WAV file #i of the main image #i) are associated with each other and stored in the associated sequence file.

[0258] Note that in addition to HEIF files, the present technology can also be applied to, for example, ISO base media files, MP4 files, Miaf files, etc. having a box structure other than HEIF files.

[0259] Furthermore, the present technology can be applied to, for example, a file storing an image (main image) and other images with reduced image resolution, a file not having a box structure, or the like.

[0260] Furthermore, in addition to the case where the external data is associated with the main image in the HEIF file, the present technology can also be applied to the case where the external data is associated with the screen thumbnail image or thumbnail image in the HEIF file.

[0261] Furthermore, the present technology can also be applied to a case where, for example, external data is associated with internal data other than an image such as a main image in a HEIF file.

[0262] <Description of Computer to Which the Present Technology is Applied>

[0263] Next, the signal processing unit 13 (such as the above-mentioned file control unit 43) is constituted Figure 1 ) can be executed by hardware or can be executed by software. In the case where the series of processes are executed by software, a program configuring the software is installed on a computer or the like.

[0264] Fig.31 is a block diagram illustrating a configuration example of an embodiment of a computer installed with a program for executing the above-described series of processes.

[0265] The program may be recorded in advance in the hard disk 905 or the ROM 903 which is a recording medium built in the computer.

[0266] Alternatively, the program may be stored (recorded) in a removable recording medium 911 driven by the drive 909. Such a removable recording medium 911 may be provided as so-called packaged software. Here, examples of the removable recording medium 911 include a floppy disk, a compact disk read only memory (CD-ROM), a magneto-optical (MO) disk, a digital versatile disk (DVD), a magnetic disk, a semiconductor memory, and the like.

[0267] Note that the program can be installed on the computer from the removable recording medium 911 as described above, and further, can be downloaded to the computer via a communication network or a broadcasting network and installed on the built-in hard disk 905. That is, for example, the program can be wirelessly transmitted to the computer from a download site via an artificial satellite used for digital satellite broadcasting, or can be transmitted to the computer by cable via a network such as a local area network (LAN) or the Internet.

[0268] The computer includes a central processing unit (CPU) 902 , and an input / output interface 910 is connected to the CPU 902 via a bus 901 .

[0269] When a command is input by a user via the input / output interface 910 according to an operation of the input unit 907 or the like, the CPU 902 executes a program stored in a read only memory (ROM) 903 according to the command. Alternatively, the CPU 902 loads a program stored in a hard disk 905 into a random access memory (RAM) 904 and executes the program.

[0270] Therefore, the CPU 902 performs the processing according to the above-mentioned flowchart or the processing performed by the configuration of the above-mentioned block diagram. Then, the CPU 902 outputs the processing result from the output unit 906 via the input / output interface 910 or transmits the processing result from the communication unit 908 as needed, or in addition, for example, records the processing result in the hard disk 905.

[0271] Note that the input unit 907 includes a keyboard, a mouse, a microphone, etc. Furthermore, the output unit 906 includes a liquid crystal display (LCD), a speaker, and the like.

[0272] Here, in this specification, the processing performed by a computer according to a program does not necessarily need to be performed in time series in the order described in the flowchart. That is, the processing performed by a computer according to a program also includes processing performed in parallel or individually (for example, parallel processing or processing by an object).

[0273] In addition, the program may be processed by one computer (processor) or may be subjected to distributed processing by a plurality of computers. Also, the program may be transmitted to a remote computer and executed.

[0274] Moreover, in this specification, a system refers to a collection of multiple constituent elements (devices, modules (components), etc.), and it does not matter whether all the constituent elements are in the same housing. Therefore, each of multiple devices housed in separate housings and connected via a network, and a device in which multiple modules are housed in one housing are all systems.

[0275] Note that the embodiments of the present technology are not limited to the above-described embodiments, and various modifications may be made without departing from the scope of the present technology.

[0276] For example, the present technology may adopt a configuration of cloud computing in which one function is shared and jointly processed by a plurality of devices via a network.

[0277] Furthermore, each step described in the above flowchart may be performed by one device, or may be shared and performed by a plurality of devices.

[0278] Furthermore, in the case where a plurality of processes are included in one step, the plurality of processes included in one step may be shared and executed by a plurality of devices in addition to being executed by one device.

[0279] Furthermore, the effects described in this specification are merely examples and not limitations, and there may be other effects.

[0280] Note that the present technology may also be configured as follows.

[0281] <1>

[0282] A file generating device comprising

[0283] A file control unit generates an associative High Efficiency Image File Format (HEIF) file, in which an image compliant with HEIF and specific information of external data outside the specified HEIF file are stored in association with each other, and the external data is associated with the image.

[0284] <2>

[0285] according to <1> The file generating device, wherein the file control unit generates an associative HEIF file, and associative information associating the image with specific information of the external data is stored in the associative HEIF file.

[0286] <3>

[0287] according to <2> The file generating device described above, wherein the file control unit stores association information in an associative HEIF file, wherein the item ID of the specified image corresponds to the specific information.

[0288] <4>

[0289] according to <3> The file generating device, wherein the file control unit stores the associated information in a meta box or an mdat box of an associated HEIF file.

[0290] <5>

[0291] according to <2> The file generating device, wherein the file control unit

[0292] Store specific information in the mdat box of the associated HEIF file, and

[0293] The associated information that corresponds the item ID of the specified image and the item ID of the specified information stored in the mdat box is stored in the meta box of the associated HEIF file.

[0294] <6>

[0295] according to <1> The file generating device described above, wherein the file control unit stores a track of specific information of external data associated with each frame of a track constituting an image in an mdat box of an associated HEIF file.

[0296] <7>

[0297] according to <6> The file generating device, wherein each frame constituting the track of the image and the specific information constituting the track of the specific information are associated with each other by time point information on the time line or the order of arrangement in the track.

[0298] <8>

[0299] according to <1> to <7> A file generating device as described in any one of the above, wherein the image and other images based on the image are stored in an associated HEIF file.

[0300] <9>

[0301] A file generation method, comprising:

[0302] An associative High Efficiency Image File Format (HEIF) file is generated, in which an image conforming to the HEIF file and specific information specifying external data outside the HEIF file are stored in association with each other, and the external data is associated with the image.

[0303] <10>

[0304] A program for causing a computer to operate as follows

[0305] A file control unit generates an associative High Efficiency Image File Format (HEIF) file, in which an image compliant with HEIF and specific information of external data outside the specified HEIF file are stored in association with each other, and the external data is associated with the image.

[0306] <11>

[0307] A file reproduction device, comprising

[0308] A file control unit reproduces an associative high-efficiency image file format (HEIF) file in which an image conforming to the HEIF and specific information specifying external data outside the HEIF file are stored in association with each other, and the external data is associated with the image.

[0309] <12>

[0310] according to <11> The file reproduction device, wherein the file control unit reads specific information of external data associated with a predetermined image from an associative HEIF file, and associated information data associating the image with the specific information of the external data is stored in the associative HEIF file.

[0311] <13>

[0312] according to <12> The file reproducing device, wherein association information that associates the item ID of the designated image with the specific information is stored in the association-type HEIF file, and

[0313] The file control unit reads specific information corresponding to the item ID of the predetermined image in the association information.

[0314] <14>

[0315] according to <13> The file reproducing device,

[0316] The associated information is stored in the meta box or mdat box of the associated HEIF file, and

[0317] The file control unit reads specific information corresponding to the item ID of the predetermined image in the associated information from the meta box or the mdat box.

[0318] <15>

[0319] according to <12> The file reproducing device,

[0320] where specific information is stored in the mdat box of the associated HEIF file, and

[0321] Association information that corresponds an item ID that specifies an image to an item ID that specifies specific information stored in the mdat box is stored in the meta box of the association-type HEIF file, and

[0322] The file control unit reads, from the mdat box, specific information specified by the item ID of the specific information corresponding to the item ID of the predetermined image among the associated information stored in the meta box.

[0323] <16>

[0324] according to <11> The file reproducing device,

[0325] A track in which specific information of external data associated with each frame constituting the track of the image is stored in an mdat box of the associated HEIF file, and

[0326] Each frame of the track constituting the image and the specific information constituting the track of the specific information are associated with each other by time point information on the time line or the order of arrangement in the track, and

[0327] The file control unit acquires time point information on the time line of a predetermined frame or specific information of the order of arrangement of the predetermined frames in the track from the track of specific information.

[0328] <17>

[0329] according to <11> to <16> A file reproduction device as described in any one of the above, wherein the image and other images based on the image are stored in an associated HEIF file.

[0330] <18>

[0331] A file reproduction method, comprising:

[0332] An associative High Efficiency Image File Format (HEIF) file is reproduced, in which an image conforming to the HEIF file and specific information specifying external data outside the HEIF file are stored in association with each other, the external data being associated with the image.

[0333] <19>

[0334] A program for causing a computer to operate as follows

[0335] A file control unit reproduces an associative high-efficiency image file format (HEIF) file in which an image conforming to the HEIF and specific information specifying external data outside the HEIF file are stored in association with each other, and the external data is associated with the image.

[0336] Reference numerals list

[0337] 10 Digital Camera

[0338] 11 Optical System

[0339] 13 Signal Processing Unit

[0340] 14 Medium

[0341] 15, 16 Interface

[0342] 17 buttons / keys

[0343] 18 Touch Panel

[0344] 19 LCD Panel

[0345] 20 Viewfinder

[0346] 21 Interface

[0347] 41 Optical system / image sensor control unit

[0348] 42 Encoding control unit

[0349] 43 File Control Unit

[0350] 44 Media Control Unit

[0351] 45 Operation control unit

[0352] 46 Display control unit

[0353] 47 UI control unit

[0354] 901 bus

[0355] 902 CPU

[0356] 903 ROM

[0357] 904RAM

[0358] 905 Hard Drive

[0359] 906 Output Unit

[0360] 907 Input Unit

[0361] 908 Communication Unit

[0362] 909 Drive

[0363] 910 Input / Output Interface

[0364] 911 Removable recording media

Claims

1. A file generating device, comprising: A file control unit generates an associated HEIF file conforming to a high-efficiency image file format HEIF, in which an image and specific information specifying external data outside the HEIF file are stored in association with each other, wherein: The external data is associated with the image and is a RAW file including the image encoded as a RAW image; The file control unit generates an associated HEIF file, and associated information associating the image with the specific information of the external data is stored in the associated HEIF file; The file control unit stores the specific information in the mdat box of the associated HEIF file; The file control unit stores the associated information that corresponds the item ID of the specified image and the item ID of the specified information stored in the mdat box in the meta box of the associated HEIF file; The file control unit stores the track of specific information of the external data associated with each frame constituting the track of the image in the mdat box of the associated HEIF file; and Each frame of the track constituting the image and the specific information of the track constituting the specific information are associated with each other by time point information on a timeline or an order of arrangement in the track.

2. The file generation device according to claim 1, wherein the file control unit stores the association information in the association-type HEIF file, in which the item ID of the image is specified to correspond to the specific information.

3. The file generation device according to claim 2, wherein the file control unit stores the association information in a meta box or an mdat box of the association-type HEIF file. 4 . The file generating device according to claim 1 , wherein the image and other images based on the image are stored in the associative HEIF file.

5. A file generation method, comprising: Generate an associated HEIF file conforming to the High Efficiency Image File Format (HEIF), in which an image and specific information specifying external data outside the HEIF file are stored in association with each other, wherein: The external data is associated with the image and is a RAW file including the image encoded as a RAW image; Generate an associated HEIF file, in which associated information associating the image with the specific information of the external data is stored; Storing the specific information in an mdat box of the associated HEIF file; storing the associated information that corresponds the item ID of the specified image and the item ID of the specific information stored in the mdat box in the meta box of the associated HEIF file; storing a track of specific information of the external data associated with each frame constituting a track of the image in an mdat box of the associated HEIF file; and Each frame of the track constituting the image and the specific information of the track constituting the specific information are associated with each other by time point information on a timeline or an order of arrangement in the track.

6. A program product for causing a computer to: A file control unit generates an associated HEIF file conforming to a high-efficiency image file format HEIF, in which an image and specific information specifying external data outside the HEIF file are stored in association with each other, wherein: The external data is associated with the image and is a RAW file including the image encoded as a RAW image; The file control unit generates an associated HEIF file, and associated information associating the image with the specific information of the external data is stored in the associated HEIF file; The file control unit stores the specific information in the mdat box of the associated HEIF file; The file control unit stores the associated information that corresponds the item ID of the specified image and the item ID of the specified information stored in the mdat box in the meta box of the associated HEIF file; The file control unit stores the track of specific information of the external data associated with each frame constituting the track of the image in the mdat box of the associated HEIF file; as well as Each frame of the track constituting the image and the specific information of the track constituting the specific information are associated with each other by time point information on a timeline or an order of arrangement in the track.

7. A file reproduction device, comprising: A file control unit is configured to reproduce an associated HEIF file conforming to a high-efficiency image file format (HEIF), in which an image and specific information specifying external data outside the HEIF file are stored in association with each other, wherein: The external data is associated with the image and is a RAW file including the image encoded as a RAW image; The file control unit reads specific information of the external data associated with a predetermined image from the associated HEIF file, and stores associated information associating the image with the specific information of the external data in the associated HEIF file; The specific information is stored in the mdat box of the associated HEIF file; The association information that corresponds the item ID of the specified image to the item ID of the specified information stored in the mdat box is stored in the meta box of the associated HEIF file; the file control unit reads, from the mdat box, specific information specified by the item ID of the specific information corresponding to the item ID of the predetermined image in the associated information stored in the meta box; A track of specific information of the external data associated with each frame constituting a track of the image is stored in an mdat box of the associated HEIF file; Each frame of the track constituting the image and the specific information constituting the track of the specific information are associated with each other by time point information on a timeline or an order of arrangement in the track; and The file control unit acquires the specific information of time point information on a time line of a predetermined frame or an order of arrangement of the predetermined frame in the track from the track of the specific information.

8. The file reproducing device according to claim 7, wherein the association information that associates the item ID specifying the image with the specific information is stored in the association-type HEIF file, and The file control unit reads the specific information corresponding to the item ID of the predetermined image in the association information.

9. The file reproducing device according to claim 8, wherein the associated information is stored in a meta box or an mdat box of the associated HEIF file, and The file control unit reads the specific information corresponding to the item ID of the predetermined image in the association information from the meta box or the mdat box.

10. The file reproducing device according to claim 7, wherein the image and other images based on the image are stored in the associated HEIF file.

11. A file reproduction method, comprising: Reproducing an associated HEIF file conforming to the High Efficiency Image File Format (HEIF), in which an image and specific information specifying external data outside the HEIF file are stored in association with each other, wherein: The external data is associated with the image and is a RAW file including the image encoded as a RAW image; Reading specific information of the external data associated with a predetermined image from the associated HEIF file, and storing associated information associating the image with the specific information of the external data in the associated HEIF file; The specific information is stored in the mdat box of the associated HEIF file; The association information that corresponds the item ID of the specified image to the item ID of the specified information stored in the mdat box is stored in the meta box of the associated HEIF file; reading, from the mdat box, specific information specified by the item ID of the specific information corresponding to the item ID of the predetermined image among the associated information stored in the meta box; A track of specific information of the external data associated with each frame constituting a track of the image is stored in an mdat box of the associated HEIF file; Each frame of the track constituting the image and the specific information constituting the track of the specific information are associated with each other by time point information on a timeline or an order of arrangement in the track; and The specific information of time point information on a time line of a predetermined frame or an order of arrangement of the predetermined frame in the track is acquired from the track of the specific information.

12. A program product for causing a computer to: A file control unit is configured to reproduce an associated HEIF file conforming to a high-efficiency image file format (HEIF), in which an image and specific information specifying external data outside the HEIF file are stored in association with each other, wherein: The external data is associated with the image and is a RAW file including the image encoded as a RAW image; The file control unit reads specific information of the external data associated with a predetermined image from the associated HEIF file, and stores associated information associating the image with the specific information of the external data in the associated HEIF file; The specific information is stored in the mdat box of the associated HEIF file; The association information that corresponds the item ID of the specified image to the item ID of the specified information stored in the mdat box is stored in the meta box of the associated HEIF file; the file control unit reads, from the mdat box, specific information specified by the item ID of the specific information corresponding to the item ID of the predetermined image in the associated information stored in the meta box; A track of specific information of the external data associated with each frame constituting a track of the image is stored in an mdat box of the associated HEIF file; Each frame constituting the track of the image and the specific information constituting the track of the specific information are associated with each other by time point information on a timeline or an order of arrangement in the track; as well as The file control unit acquires the specific information of time point information on a time line of a predetermined frame or an order of arrangement of the predetermined frame in the track from the track of the specific information.

Images