File processing device, file processing method and program

Abstract

The present technology relates to a file processing device, a file processing method, and a program that enable an image stored in a file to be associated with external data outside the file. The file control unit generates a high-efficiency image file format HEIF file, the HEIF file includes specified information, the specified information is used to specify the HEIF file external data associated with the image stored in the HEIF file, but the specified information has not yet been assigned to the external data, and the HEIF file stores relationship information related to the association between the image and the specified information. In addition, the file control unit writes the specified information stored in the HEIF file into the file storing the external data. The present technology can be applied to situations such as generating a HEIF file and then associating the image stored in the HEIF file with external data.

Description

Technical Field

[0001] The present technology relates to a file processing device, a file processing method, and a program, and in particular to a file processing device, a file processing method, and a program that enable association of, for example, an image stored in a file with external data outside the file. Background Art

[0002] As a file format for efficiently storing images, there is High Efficiency Image File Format (HEIF) (see Non-Patent Literature 1).

[0003] Reference List

[0004] Non-patent literature

[0005] Non-patent document 1: ISO / IEC 23008-12: 2017, Information technology--Highefficiency coding and media delivery in heterogeneous environments--Part 12:Image File Format Summary of the invention

[0006] Problems to be solved by the present invention

[0007] For a HEIF file conforming to the High Efficiency Image File Format (HEIF), it is convenient if an image stored in the HEIF file can be associated with external data outside the HEIF file.

[0008] The present technology has been made in view of such circumstances, and makes it possible to associate an image stored in a HEIF file with external data outside the HEIF file.

[0009] Solution to the problem

[0010] A first file processing device or program according to the present technology is a file processing device including a file control unit, or a program for causing a computer to be used as such a file processing device, wherein the file control unit generates a High Efficiency Image File Format (HEIF) file, the HEIF file storing relationship information related to an association between an image and specified information, the relationship information including specified information before being assigned to external data and the specified information specifying external data outside the HEIF file to be associated with the image stored in the HEIF file.

[0011] The first file processing method of the present technology is such a file processing method, including: generating a High Efficiency Image File Format (HEIF) file, the HEIF file storing relationship information related to the association between an image and specified information, the relationship information including the specified information before being assigned to the external data and the specified information specifying the external data outside the HEIF file to be associated with the image stored in the HEIF file.

[0012] In the first file processing device, file processing method and program of the present technology, a High Efficiency Image File Format (HEIF) file is generated, and the HEIF file stores relationship information related to the association between an image and specified information, which relationship information includes specified information before being assigned to external data and the specified information specifies external data outside the HEIF file to be associated with the image stored in the HEIF file.

[0013] A second file processing device or program according to the present technology is a file processing device including a file control unit, or a program for causing a computer to be used as such a file processing device, wherein the file control unit writes specified information stored in a high-efficiency image file format (HEIF) to a file storing external data, the HEIF file storing relationship information related to the association between the image and the specified information, the relationship information including the specified information before being assigned to the external data and the specified information specifying external data outside the HEIF file to be associated with the image stored in the HEIF file.

[0014] A second file processing method of the present technology is a file processing method comprising: writing specified information stored in a high-efficiency image file format (HEIF) into a file storing external data, the HEIF file storing relationship information related to the association between the image and the specified information, the relationship information including the specified information before being assigned to the external data and the specified information specifying external data outside the HEIF file to be associated with the image stored in the HEIF file.

[0015] In a second file processing device, a file processing method, and a program according to the present technology, designated information stored in a high-efficiency image file format (HEIF) is written into a file storing external data, and the HEIF file stores relationship information related to the association between the image and the designated information, the relationship information including the designated information before being assigned to the external data, and the designated information specifies external data outside the HEIF file to be associated with the image stored in the HEIF file.

[0016] Note that the first and second file processing devices may be independent devices, or may be internal blocks forming one device.

[0017] Furthermore, the first and second programs may be provided by being recorded on a recording medium or by being transmitted via a transmission medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a block diagram illustrating a configuration example of an embodiment of a digital camera to which the present technology is applied.

[0019] Figure 2 is a view illustrating an example of the format of a Joint Photographic Experts Group (JPEG) file conforming to JPEG.

[0020] Figure 3 is a view illustrating an example of the ISO base media file format.

[0021] Figure 4 is a view illustrating a format example of a HEIF file conforming to HEIF.

[0022] Figure 5 is a view illustrating a format example of a HEIF file of an image item format.

[0023] Figure 6 is a view illustrating an example of an iprp box.

[0024] Figure 7 is a view illustrating an example of the format of a HEIF file in an image sequence format.

[0025] Figure 8 is a view illustrating an example of a trak box.

[0026] Fig. 9 is a view illustrating an example of a normal collection file in which a main image and thumbnail images are stored.

[0027] Fig.10 is a view illustrating an example of a first association type collection file.

[0028] Fig.11 is a view illustrating an example of a second association type collection file.

[0029] Fig.12 is a view illustrating an example of a third association type collection file.

[0030] Fig.13 is a view illustrating an example of a normal sequence file in which a track of a main image and a track of thumbnail images of the main image are stored.

[0031] Fig.14 is a view illustrating an example of an association type sequence file.

[0032] Fig.15: is a flowchart for explaining an outline of an example of a generation process of generating an association type HEIF file.

[0033] Fig.16 : is a flowchart for explaining an outline of an example of a reproduction process of reproducing an association type HEIF file.

[0034] Fig.17 is a flowchart for explaining an example of a reproduction process of a reproduction set file.

[0035] Fig.18 : is a flowchart for explaining an example of a process of reading a reproduction target image in step S32.

[0036] Fig.19 : is a flowchart for explaining a first example of the process of acquiring the reproduction target item ID in step S31.

[0037] Fig. 20 is a flowchart for explaining a second example of the process of acquiring the reproduction target item ID in step S31.

[0038] Fig.21 is a flowchart for explaining an example of a process of acquiring uuid of designation information of a RAW file which is a predetermined main image from a first association type set file.

[0039] Fig. 22 is a flowchart for explaining an example of a process of acquiring uuid of designation information of a RAW file which is a predetermined master image from a second association type set file.

[0040] Fig.23 is a flowchart for explaining an example of a process of acquiring uuid of designation information of a RAW file which is a predetermined master image from a third association type set file.

[0041] Fig.24 is a flowchart for explaining an example of a process of acquiring an item ID list of a main image from a collection file.

[0042] Fig.25 is a flowchart for explaining an example of a process of reproducing thumbnail images of (frames of) a main image with respect to predetermined time information from a sequence file.

[0043] Fig.26 It is a flowchart for explaining an example of a process of acquiring uuid of designation information of a RAW file which is a predetermined main image (frame) from an association type sequence file.

[0044] Fig. 27 is a view illustrating an example of storing uuid in a RAW file in a case where a RAW file of a main image is adopted as external data and an association type collection file is generated.

[0045] Fig.28 is a view illustrating an example of storing uuid in a RAW file in the case of adopting a RAW file of a main image as external data and generating an association type sequence file.

[0046] Fig.29 is a view illustrating an example of storing uuid into a WAV file in a case where a WAV file of a main image is adopted as external data and an association type collection file is generated.

[0047] Fig.30 is a view illustrating an example of storing uuid into a WAV file in a case where a WAV file of a main image is adopted as external data and an associated type sequence file is generated.

[0048] Fig.31 It is a view for explaining the association between images as internal data and external data after a HEIF file is generated.

[0049] Fig.32 It is a view for explaining the outline of an example of the area securing method.

[0050] Fig.33 It is a view for explaining an example of an association process of associating external data and a main image as internal data in a first association type set file generated by an area ensuring method after generating the first association type set file.

[0051] Fig.34 It is a view for explaining an outline of an example of the pre-storage method.

[0052] Fig.35 is a view for explaining an example of an association process of associating external data and a main image as internal data in a first association type aggregation file generated by a pre-storage method after generating the first association type aggregation file.

[0053] Fig.36 is a view for explaining an outline of another example of the pre-storage method.

[0054] Fig.37 is a view for explaining an outline of still another example of the pre-storage method.

[0055] Fig.38 It is a view for explaining the association between available designation information and internal data and external data for each of the area securing method and the pre-storage method.

[0056] Fig.39It is a view for explaining an example of performing an association process of 1-to-N association on a first association type set file generated by the area securing method.

[0057] Fig.40 is a view for explaining an example of performing an association process of 1-to-N association on a first association type set file generated by a pre-storage method.

[0058] Fig.41 It is a view for explaining an example of an association process of performing N-to-1 association on a first association type set file generated by the area securing method.

[0059] Fig.42 is a view for explaining an example of performing an association process of N-to-1 association on a first association type set file generated by a pre-storage method.

[0060] Fig.43 1 is a view illustrating a first example of association between a main image as internal data and audio as external data in a case where an MP4 file is adopted as a file storing audio as external data.

[0061] Fig.44 is a view illustrating a second example of association between a main image as internal data and audio as external data in a case where an MP4 file is adopted as a file storing audio as external data.

[0062] Fig.45 is a view illustrating a third example of association between a main image as internal data and audio as external data in a case where an MP4 file is adopted as a file storing audio as external data.

[0063] Fig.46 is a view illustrating a fourth example of association between a main image as internal data and audio as external data in a case where an MP4 file is adopted as a file storing audio as external data.

[0064] Fig.47 is a view illustrating a fifth example of association between a main image as internal data and audio as external data in a case where an MP4 file is adopted as a file storing audio as external data.

[0065] Fig.48 is a view illustrating a sixth example of association between a main image as internal data and audio as external data in a case where an MP4 file is adopted as a file storing audio as external data.

[0066] Fig.49is a view for explaining a first example of a first association type set file generated by an area securing method and a state of the first association type set file after an association process.

[0067] Fig.50 is a view for explaining a second example of a first association type set file generated by the area securing method and a state of the first association type set file after the association process.

[0068] Fig.51 is a view for explaining a third example of a first association type set file generated by the area securing method and a state of the first association type set file after the association process.

[0069] Fig.52 is a view for explaining a first example of a second association type set file generated by an area securing method and a state of the second association type set file after an association process.

[0070] Fig.53 is a view for explaining a second example of a second association type set file generated by the area securing method and a state of the second association type set file after the association process.

[0071] Fig.54 is a view for explaining a first example of a third association type set file generated by the area securing method and a state of the third association type set file after the association process.

[0072] Fig.55 is a view for explaining a second example of a third association type set file generated by the area securing method and a state of the third association type set file after the association process.

[0073] Fig.56 It is a view for explaining a first example of an association type sequence file generated by the area securing method and a state of the association type sequence file after an association process.

[0074] Fig.57 It is a view for explaining a second example of an association type sequence file generated by the area securing method and a state of the association type sequence file after the association process.

[0075] Fig.58 is a flowchart for explaining an example of a process of generating an association type HEIF file by a region ensuring method.

[0076] Fig.59 : is a flowchart for explaining an example of association processing of an association type HEIF file generated by the area ensuring method.

[0077] Fig.60is a flowchart for explaining an example of an association process of a first association type set file generated by the area securing method.

[0078] Fig.61 is a flowchart for explaining another example of the association process of the first association type set file generated by the area securing method.

[0079] Fig.62 is a flowchart for explaining an example of a process of generating an associated type HEIF file by a pre-storage method.

[0080] Fig.63 is a flowchart for explaining an example of association processing of an association type HEIF file generated by a pre-storage method.

[0081] Fig.64 is a block diagram illustrating a configuration example of an embodiment of a computer to which the present technology is applied. DETAILED DESCRIPTION

[0082] <One embodiment of a digital camera to which the present technology is applied>

[0083] Figure 1 is a block diagram illustrating a configuration example of an embodiment of a digital camera to which the present technology is applied.

[0084] The digital camera 10 includes an optical system 11 , an image sensor 12 , a signal processing unit 13 , a medium 14 , interfaces 15 and 16 , buttons / keys 17 , a touch panel 18 , a liquid crystal panel 19 , a viewfinder 20 , an interface 21 , and the like.

[0085] The optical system 11 condenses light from a subject on the image sensor 12 .

[0086] The image sensor 12 receives light from the optical system 11 and performs imaging of photoelectric conversion to generate image data as an electric signal and supplies it to the signal processing unit 13 .

[0087] The signal processing unit 13 includes an optical system / image sensor control unit 41 , an encoding control unit 42 , a file control unit 43 , a medium control unit 44 , an operation control unit 45 , a display control unit 46 , and a UI control unit 47 .

[0088] The optical system / image sensor control unit 41 controls the optical system 11 and the image sensor 12 , and supplies (data of) an image obtained by imaging performed according to the control to the encoding control unit 42 .

[0089] The encoding control unit 42 supplies the image from the optical system / image sensor control unit 41 to the display control unit 46, and encodes the image as needed and supplies it to the file control unit 43. Furthermore, the encoding control unit 42 decodes the image supplied from the file control unit 43 as needed and supplies it to the display control unit 46.

[0090] The file control unit 43 generates a file storing the image supplied from the encoding control unit 42, and supplies it to the medium control unit 44. In addition, the file control unit 43 reproduces the file supplied from the medium control unit 44, that is, reads data such as the image stored in the file, etc. For example, the image read from the file is supplied from the file control unit 43 to the encoding control unit 42.

[0091] The medium control unit 44 controls file exchange between the medium 14 and the interfaces 15 and 16. For example, the medium control unit 44 causes a file from the file control unit 43 to be recorded on the medium 14 or transferred from the interfaces 15 and 16. In addition, the medium control unit 44 reads a file from the medium 14 or causes the interfaces 15 and 16 to receive a file and supplies it to the file control unit 43.

[0092] The operation control unit 45 supplies an operation signal corresponding to the operation to a necessary block according to the user's operation on the button / key 17 or the touch panel 18 .

[0093] The display control unit 46 performs display control and the like to supply the image and the like supplied from the encoding control unit 42 to the liquid crystal panel 19 , the viewfinder 20 , and the interface 21 for display.

[0094] The UI control unit 47 manages user interface (UI) control.

[0095] The medium 14 is, for example, a storage medium such as an SD card. The interface 15 is, for example, an interface of a local area network (LAN) such as Wi-Fi (registered trademark) or Ethernet (registered trademark). The interface 16 is, for example, a universal serial bus (USB) interface. When a command or other information is input to the digital camera 10, the button / key 17 and the touch panel 18 are operated by the user. The touch panel 18 can be formed integrally with the liquid crystal panel 19. The liquid crystal panel 19 and the viewfinder 20 display images supplied from the display control unit 46, etc. The interface 21 is an interface for transmitting at least an image, such as a high-definition multimedia interface (HDMI) (registered trademark) or a display port (DP).

[0096] In the digital camera 10 configured as described above, the optical system / image sensor control unit 41 generates, for example, a YUV image having the same resolution (number of pixels) as that of the RAW image as the main image of the HEIF file based on an image of the RAW data obtained by imaging by the image sensor 12 (hereinafter, also referred to as a RAW image), and supplies it to the encoding control unit 42.

[0097] The encoding control unit 42 generates, for example, a YUV image (hereinafter, also referred to as a screen nail image) having a lower resolution than the main image from the YUV main image for display as a first other image based on the main image on the liquid crystal display panel 19 or an external display, and generates, for example, a YUV image (hereinafter, also referred to as a thumbnail image) having a resolution lower than the resolution of the screen nail image for list display as a second other image based on the main image. For example, the encoding control unit 42 supplies the screen nail image to the liquid crystal panel 19 via the display control unit 46 to be displayed as a so-called through image. As the thumbnail image, for example, an image having a size of 320 pixels or less on the long side can be used. For example, the ratio of the size (number of pixels) between the main image and the screen nail image as the first other image based on the main image or the thumbnail image as the second other image based on the main image can be, for example, 200 times or less. Similarly, the ratio of the size between the main image and the screen nail image as the first other image based on the main image and the thumbnail image as the second other image based on the main image can also be 200 times or less. As a screen nail image, for example, an image with a resolution of 4K or higher can be used. In addition, as a screen nail image, for example, a 4K (QFHD) or FHD image can be used according to the user's selection. Moreover, an image with the same resolution can be used as the main image and the screen nail image. In the case of using an image with the same resolution as the main image and the screen nail image, both the main image and the screen nail image can be stored in a HEIF file, or the main image can be stored without storing the screen nail image. In the case where the main image is stored in a HEIF file without storing the screen nail image, the size of the main image can be adjusted and used as a screen nail image.

[0098] In addition, the encoding control unit 42 encodes the main image, the screen nail image, and the thumbnail image corresponding to the RAW image (the main image, the screen nail image, and the thumbnail image generated from the same RAW image) when necessary, and supplies them to the file control unit 43 together with the RAW image.

[0099] The file control unit 43 generates a RAW file storing a RAW image, and generates a HEIF file, a JPEG file, etc. storing corresponding main images, screen nail images, and thumbnail images (main images, screen nail images, and thumbnail images generated from the same RAW image), and supplies them to the medium control unit 44. The HEIF file is a file conforming to the High-Efficiency Image File Format (HEIF), and the JPEG file is a file conforming to the Joint Photographic Experts Group (JPEG).

[0100] The medium control unit 44 records the RAW file and the HEIF file or JPEG file from the file control unit 43 on the medium 14, or causes the RAW file and the HEIF file or JPEG file to be transmitted from the interface 15 or 16.

[0101] In the file control unit 43, for example, it is possible to select which of the HEIF file and the JPEG file to generate according to a user operation. In addition, although the HEIF file includes an image item format and an image sequence format described later, it is possible to select which of the image item format and the image sequence format to adopt according to a user operation. Moreover, the file control unit 43 can perform mutual conversion between the HEIF file and the JPEG file according to a user operation.

[0102] Moreover, when generating a HEIF file, the file control unit 43 can associate internal data in the HEIF file (data stored in the HEIF file, which is in the HEIF file and is to be associated with external data (data not stored in the HEIF file)) with specified information specifying the external data, and store it in the HEIF file. A HEIF file in which the internal data and the specified information of the external data to be associated with the internal data are stored in association with each other is also referred to as an associated type HEIF file. The associated type HEIF file can store the internal data and the specified information in association with each other, for example, by storing association information associating the internal data and the specified information.

[0103] <JPEG file>

[0104] Figure 2 is a view showing an example of the format of a Joint Photographic Experts Group (JPEG) file conforming to JPEG.

[0105] The JPEG file is configured by storing, for example, Exif metadata, a thumbnail image, and Extensible Metadata Platform (XMP) (registered trademark) metadata, MPF indicating the storage location (position) of the main image and the simple display image, and the main image and the simple display image. As the simple display image, for example, a screen nail image can be adopted.

[0106] <ISO Base Media File Format>

[0107] Figure 3 A view showing an example of the ISO base media file format.

[0108] HEIF (ISO / IEC 23008-12) is a file format that conforms to the ISO base media file format (ISO / IEC 14496-12). Therefore, HEIF files conform to the ISO base media file format.

[0109] The ISO base media file format is formed with units called boxes as containers for storing data and has a structure called a box structure.

[0110] Boxes include a type (box type), actual data (data), etc. The type indicates the type of actual data in the box. As actual data, it is possible to adopt reproducible media data such as images (still images or moving images), audio, and subtitles, attribute names (field names), and attribute values (field values) of (variables represented by) attribute names, as well as various other data.

[0111] Moreover, a box can be adopted as actual data. That is, a box can have a box as actual data, which enables a hierarchical structure.

[0112] A basic media file that conforms to the ISO base media file format can have an ftyp box, a moov box (MovieBox), a meta box (MetaBox), an mdat box (MediaDataBox), etc. The ftyp box stores identification information for identifying the file format. The moov box can store trak boxes, etc. The meta box can store iinf boxes, iprp boxes, iref boxes, iloc boxes, etc. The mdat box can store media data (AV data) and any other data.

[0113] HEIF conforms to the above ISO base media file format.

[0114] <HEIF file>

[0115] Figure 4 A view showing an example of the format of a HEIF file that conforms to HEIF.

[0116] HEIF files are roughly divided into an image item format and an image sequence format. Moreover, the image item format includes a single image format with only one item and an image collection format with multiple items, which will be described later.

[0117] HEIF files in the image item format have an ftyp box, a meta box, and an mdat box.

[0118] HEIF files in image sequence format have ftyp box, moov box and mdat box.

[0119] Note that HEIF files can also have both a meta box and a moov box, not just one or the other.

[0120] The ftyp box stores identification information for identifying the format of a file, such as, for example, the fact that the file is a HEIF file in the image item format or the image sequence format.

[0121] The meta box and the moov box store metadata necessary for reproduction, management, etc. of the media data stored in the mdat box, for example, metadata such as the storage location of the media data.

[0122] The mdat box stores media data (AV data) and the like.

[0123] For example, in the digital camera 10, which HEIF file of the HEIF file in the image item format and the image sequence format is to be generated can be selected according to the user's operation. In addition, when the image is encoded and stored in the mdat box of the HEIF file, the image item format only allows intra coding, while the image sequence format allows intra coding and inter coding. Therefore, for example, in the case where priority is given to high-speed access to data stored in the HEIF file, it is possible to choose to generate a HEIF file in the image item format. And in the case where priority is given to reducing the size (data amount) of the HEIF file, it is possible to choose to generate a HEIF file in the image sequence format.

[0124] Figure 5 is a view illustrating an example of the format of a HEIF file of an image item format.

[0125] The HEIF file in the image item format stores information indicating that the HEIF file is in the image item format, such as mif1, etc. (as an attribute value) in the ftyp box.

[0126] The meta box stores the iinf box, iref box, iprp box, and iloc box.

[0127] The iinf box stores (indicates) the number of items (attribute names and attribute values) such as media data (AV data) stored in the mdat box. The item is one data stored in the mdat box of the HEIF file in the image item format, for example, one image (screen) is an item. In this specification, whether it is a still image or a moving image, one image is also called a frame. One frame is one item.

[0128] The iref box stores information indicating the relationship between items. For example, in the mdat box, each of the corresponding main image, screen nail image, and thumbnail image may be stored as an item. In the case where the mdat box stores item I1 as the main image, item I2 as the screen nail image, and item I3 as the thumbnail image, the iref box stores information indicating that item I2 is the screen nail image as the main image of item I1 and information indicating that item I3 is the thumbnail image as the main image of item I1.

[0129] The iprp box stores information about the properties of an item.

[0130] The iloc box stores information about the storage location of the items stored in the mdat box.

[0131] The mdat box (of a HEIF file) in the image item format stores, for example, frames of an image as items. In the mdat box, one or more items can be stored. In addition, the frames as items can be encoded and stored in the mdat box. However, the encoding of the frames as items stored in the mdat box in the image item format is limited to intra-frame encoding. As a coding method (codec) for encoding frames as items, for example, HEVC, etc. can be used.

[0132] Figure 6 It is a graphic Figure 5 A view of an example of an iprp box.

[0133] The iprp box stores the ipco box and the ipma box related to the attributes of the item. The ipco box stores the attributes of the item stored in the mdat box, for example, codec information about the codec of the image as the item and image size information about the size. The ipma box stores the index (pointer) to the property stored in the ipco box of the item stored in the mdat box.

[0134] Figure 7 is a view illustrating an example of the format of a HEIF file in an image sequence format.

[0135] A HEIF file in an image sequence format stores information indicating that the HEIF file is in an image sequence format, such as msf1, in an ftyp box.

[0136] The moov box stores the trak box. The trak box stores information about the track stored in the mdat box.

[0137] A track includes a piece of independent media data, such as an image or audio, to be reproduced according to a timeline. For example, a track includes an image that will become one or more frames of an elementary stream. For tracks stored in an mdat box, multiple tracks can be reproduced simultaneously, such as individual tracks of images and audio recorded simultaneously.

[0138] The media data of a track is formed in units called samples. A sample is the minimum unit (access unit) when accessing media data in a HEIF file. Therefore, the media data in a HEIF file cannot be accessed in units finer than a sample.

[0139] For example, for image media data, one frame or the like is one sample. Also, for audio media data, for example, one audio frame or the like defined in the standard of audio media data is one sample.

[0140] In the mdat box (of the HEIF file) of the image sequence format, the media data of the track is arranged in units called blocks. A block is a collection of one or more samples arranged at logically consecutive addresses.

[0141] In the case where a plurality of tracks as media data are stored in the mdat box, the plurality of tracks are interleaved and arranged in units of blocks.

[0142] As described above, in the mdat box of the image sequence format, one or more tracks including media data such as images and audio are stored.

[0143] In the mdat box, the image frames constituting the track can be encoded and stored. For the encoding of the frames constituting the track stored in the mdat box of the image sequence format, a long group of pictures (GOP) can be used as the GOP, and either intra-frame encoding or inter-frame encoding can be used. As a codec for encoding the frames constituting the track, for example, HEVC can be used.

[0144] Figure 8 is a view illustrating an example of a trak box.

[0145] The trak box can store a tkhd box and an mdia box. The tkhd box stores header information of the track, such as the creation date and time of the pipeline managed by the trak box. The mdia box stores a minf box, etc. The minf box stores a stbl box. The stbl box stores a stsd box, a stsc box, a stsz box, and a stco box, which store samples of the track and thus store information for accessing blocks. The stsd box stores codec information about the codec of the track. The stsc box stores a block size (the number of samples of one block). The stsz box stores a sample size. The stco box stores a block offset, that is, an offset of the arrangement position of each block of the track stored in the mdat box.

[0146] Here, the HEIF file in the image item format is also referred to as a collection file, and the HEIF file in the image sequence format is also referred to as a sequence file. Moreover, the associated type HEIF file in the image item format is also referred to as an associated type collection file, and the associated type HEIF file in the image sequence format is also referred to as an associated type sequence file.

[0147] In the digital camera 10, it is possible to generate a HEIF file (including an associated type HEIF file) that stores a main image, and also stores one or both of necessary screen nail images and thumbnail images.

[0148] <Collection File>

[0149] Fig. 9 is a view illustrating an example of a normal collection file in which a main image and thumbnail images are stored.

[0150] Here, a normal collection file refers to a collection file in which internal data in the collection file is not associated with designated information of external data.

[0151] Now, assume that the frame (item) is encoded by HEVC and stored in the mdat box of the collection file.

[0152] The ftyp box stores heic indicating that the format is the image item format and the codec is HEVC, as identification information for identifying the file format.

[0153] The iinf box stores the number of items stored in the mdat box (number of items). Fig. 9 In the mdat box, a total of four items (frames) are stored: a main image specified by item ID #1 (hereinafter, also referred to as main image Item #1); main image Item #2; a thumbnail image specified by item ID #101 (hereinafter, also referred to as thumbnail image Item #101); and thumbnail image Item #102. Therefore, the number of items is 4. Note that thumbnail image Item #101 is a thumbnail image of main image Item #1, and thumbnail image Item #102 is a thumbnail image of main image Item #2.

[0154] Furthermore, the iinf box also stores an infe box for each item stored in the mdat box. In the infe box, an item ID and an item type for specifying an item are registered. Fig. 9 In the image, there is an infe box for each of the main images Item#1 and Item#2 and the thumbnail images Item#101 and Item#102.

[0155] The iref box stores, for example, a thmb box as information for associating items stored in the mdat box. In the thmb box, a reference source and a reference destination, which are information for associating a main image with a thumbnail image of the main image, are associated with each other and stored. In the thmb box, the reference source represents the item ID of the main image, and the reference destination represents the item ID of the thumbnail image of the main image specified by the item ID of the reference source. Therefore, based on the reference destination associated with the reference source, the item ID of the thumbnail image of the main image specified by the item ID represented by the reference source can be identified. In addition, based on the reference source associated with the reference destination, the item ID of the main image of the thumbnail image specified by the item ID represented by the reference destination can be identified.

[0156] like Figure 6 As described in , the ipco box and ipma box are stored in the iprp box. Figure 6 As described in , the ipco box stores the characteristics of the frame as items stored in the mdat box, for example, codec information about the codec and image size information about the size. Figure 6 As described in, the ipma box stores references to the properties stored in the ipco box for the items stored in the mdat box.

[0157] like Figure 6 As described in , the iloc box stores information about the storage locations of items in the mdat box. Fig. 9 In the iloc box, the number of items is stored as 4. Also, in the iloc box, the offset and size to each storage location of the main image Item#1 and Item#2 and thumbnail image items Item#101 and Item#102 IDs stored in the mdat box are stored in association with the item IDs.

[0158] In the following, it will be described Fig. 9 An associative collection file that stores designated information of internal data and external data in association with each other in a normal collection file.

[0159] Fig.10 is a view illustrating an example of a first association type collection file.

[0160] Here, it is assumed that as external data to be associated with a main image as internal data in a HEIF file, for example, a RAW image (RAW file in which) the main image is adopted (stored).

[0161] In the first association type set file, by storing association information associating the main image as internal data with the designated information of the RAW file (RAW image stored in) storing the RAW image as external data, the main image and the designated information of the RAW file storing the RAW image are stored in association with each other. Also, in the first association type set file, the association information is stored in a meta box.

[0162] As specifying information of a RAW file that stores a RAW image as external data, in addition to the file name of the RAW file, the universally unique identifier (uuid) and uniform resource locator (URL) published for the RAW file, it is possible to adopt any information that can specify the RAW file (the RAW image stored therein).

[0163] For the first association type set file, as a new box to be stored in the meta box, an association information storage box storing association information is defined and stored in the meta box. The association information storage box of the first association type set file stores association information in which, for example, an item ID for specifying a main image is associated with a uuid as designation information specifying a RAW file (RAW image stored in) to be associated with the main image. Moreover, the association information storage box stores the number of main images (the number of main images) to be associated with the RAW file (RAW image stored in). The number of main images stored in the association information storage box is the number of main images associated with the RAW file, and therefore is a value equal to or less than the number of main images stored in the mdat box.

[0164] exist Fig.10 In the example, the uuid of the RAW file of the main image Item#1 (the uuid of the RAW image associated with the main image Item#1) is UUID#1, and the uuid of the RAW file of the main image Item#2 is UUID#2. Now, assuming that the RAW file whose uuid is UUID#i is described as RAW file UUID#i, in Fig.10 In the associated information storage box, associated information is stored, in which item ID#1 of the main image Item#1 is associated with the uuid of the RAW file UUID#1, and item ID#2 of the main image Item#2 is associated with the uuid of the RAW file UUID#2.

[0165] Fig.11 is a view illustrating an example of a second association type collection file.

[0166] In the second association type set file, similar to the first association set file, by storing association information associating the main image as internal data with the designated information of the RAW file as external data, the main image and the designated information of the RAW file are stored in association with each other. However, in the second association type set file, the association information is stored in the mdat box.

[0167] For the second associative collection file, for example, associative information similar to that of the first associative collection file is stored as an item in the mdat box. Fig.11 In the example, the associated information is stored in the mdat box as an item with item ID #201.

[0168] As described above, in the second association type set file, in response to storing the association information as item Item#201 in the mdat box, the information stored in the meta box is the same as Fig. 9 In the second associative collection file, metadata as associative information of item Item#201 is stored in the meta box.

[0169] Specifically, in the second association type collection file, the number of items stored in the iinf box and the iloc box is 5, which is from Fig. 9 In the case of 4 plus 1 of item Item#201, it is obtained. Moreover, the infe box for item Item#201 is added to the iinf box, and the offset and size of the storage position of item Item#201 are added to the iloc box. The infe box for item Item#201 stores the item ID#201 of item Item#201 and item type identification data information (IDIF) indicating that item Item#201 is associated information. IDIF is a newly defined attribute value (field value) indicating that the item is associated information.

[0170] Fig.12 is a view illustrating an example of a third association type collection file.

[0171] In the third association type set file, by storing the designated information of the RAW file as external data in the mdat box as an item of each designated information, and storing the associated information associating the main image as internal data with the designated information of the RAW file as external data in the meta box, the main image and the designated information of the RAW file are stored in association with each other. However, in the third association type set file, the associated information is information in which the item ID of the main image as an item is associated with the item ID of the designated information (of the RAW file) as an item, and is stored in the cdsc box stored in the iref box in the meta box.

[0172] The cdsc box can store a reference source and a reference destination as information for associating a main image with items of respective pieces of designated information in a RAW file as the main image. In the cdsc box, the reference source represents the item ID of the main image, and the reference destination represents the item ID of the designated information of the item of the RAW file as the main image designated by the item ID of the reference source.

[0173] exist Fig.12 , UUID#1 which is the uuid of the designated information of the RAW file of the main image Item#1 is stored in the mdat box as item Item#201, and UUID#2 which is the uuid of the designated information of the RAW file of the main image Item#2 is stored in the mdat box as item Item#202. Moreover, the iref box stores a cdsc box storing association information in which the item ID#1 of the main image Item#1 and the item ID#201 of the designated information UUID#1 are associated with each other as a reference source and a reference destination, respectively, and the iref box stores a cdsc box storing association information in which the item ID#2 of the main image Item#2 and the item ID#202 of the designated information UUID#2 are associated with each other as a reference source and a reference destination, respectively.

[0174] <sequence file>

[0175] Fig.13 is a view illustrating an example of a normal sequence file in which a track of a main image and a track of thumbnail images of the main image are stored.

[0176] Here, a normal sequence refers to a sequence file in which internal data in the sequence file is not associated with designated information of external data.

[0177] Now, assume that the frames are encoded by HEVC and stored in the mdat box of the sequence file.

[0178] The ftyp box stores hevc indicating that the format is an image sequence format and the codec is HEVC as identification information for identifying the file format.

[0179] like Figure 7 As described in , the moov box stores and manages the trak boxes of each track stored in the mdat box. Fig.13 In the mdat box, the track of the main image specified by the track ID #1 (hereinafter, also referred to as track #1) and the track #2 of the thumbnail image of the main image specified by the track #1 are stored. Therefore, in the moov box, the trak box managing the track #1 and the trak box managing the track #2 are stored. The nth thumbnail (frame) of the track #2 (from the head) is the thumbnail image of the nth main image of the track #1.

[0180] For example, in the case where continuous shooting is performed by the digital camera 10 , the sequence file is useful in the case of recording main images and thumbnail images of a plurality of frames obtained by continuous shooting as one track or the like.

[0181] The tkhd box of the trak box of track #1 managing the main image stores: track ID #1 specifying track #1; the image size of the main image constituting track #1; rotation information indicating the orientation of the digital camera 10 when the main image was captured; and the creation date and time of track #1. In the tkhd box of the trak box of track #2 managing the thumbnail images, track ID #2 specifying track #2 and the creation date and time of track #2 are stored.

[0182] In the trak box, in addition to Figure 7 In addition to the tkhd box and the mdia box described in , a tref box may be stored. The tref box stores a track ID for specifying another track associated with the track managed by the trak box in which the tref box is stored, information indicating the content of the track, and the like. Fig.13 In the example, the tref box is provided in the trak box managing track #2. Then, the tref box stores information indicating that another track associated with track #2 is track #1 (track_ID=1) and that data constituting track #2 is a thumbnail image (track #2 is a track of thumbnail images) (type=thmb).

[0183] In the trak box's mdia box, in addition to Figure 8 In addition to the minf box described in , an hdlr box may be stored. The hdlr box stores information indicating the type of data constituting the track managed by the trak box in which the hdlr box is stored. In the hdlr box stored in the trak box (stored in the mdia box) of track #1 managing the main image, information (pict) indicating that the data constituting track #1 is a picture (frame) is stored. In the hdlr box stored in the trak box of track #2 managing the thumbnail image, information indicating that the data constituting track #2 is a picture is stored.

[0184] minf box Figure 8 As described in.

[0185] In the following, it will be described Fig.13 An association type sequence file that stores designated information of internal data and external data in association with each other in a normal sequence file.

[0186] Fig.14 is a view illustrating an example of an association type sequence file.

[0187] In the associated type sequence file, track #3 of the (elementary) stream (Meta ES) of the uuid of the designated information of the RAW file as external data is added to the mdat box, and the trak box managing the track #3 is added to the moov box.

[0188] Here, track #1 is a time sequence of one or more frames of the main image aligned on the timeline, and track #3 is a time sequence of uuids of the RAW files of each frame of the main image aligned on the timeline.

[0189] The nth uuid (starting from the head) of track #3 is the designated information of the RAW file of the frame of the nth main image of track #1. In addition, multiple tracks (data) stored in the mdat box can be synchronously reproduced according to the time information on a timeline. Therefore, by storing the uuid (stream) of the RAW file of track #1 of the main image and track #3 of each frame of the main image constituting track #1 in the mdat box, the frame of the nth main image of track #1 and the uuid of the RAW file of the main image (frame) are stored in association with each other. In this case, it can be said that the uuid of the frame of the main image of track #1 and the RAW file of the main image (frame) are associated with each other through the time information on the timeline.

[0190] Note that the nth uuid (starting from the head) of track #3 is the designated information of the RAW file of the nth frame of track #1, and it can also be understood that the main image (frame) constituting track #1 and the uuids constituting track #3 are associated with each other according to the arrangement order in the track.

[0191] In the associated type sequence file, in response to track #3 of the uuid of the RAW file being added to the mdat box, a trak box managing track #3 is added to the moov box.

[0192] In the trak box of track #3 that manages the uuid of the RAW file, a tkhd box, a tref box, an mdia box, and the like are stored.

[0193] In the tkhd box of the trak box managing track #3, track ID #3 specifying track #3 and the creation date and time of track #3 are stored.

[0194] The tref box of the trak box managing track #3 stores a track ID for specifying another track associated with track #3 managed by the trak box in which the tref box is stored, information indicating the contents of track #3, etc. The uuid constituting track #3 is the designation information of the RAW file of the main image constituting track #1. Since track #3 is associated with track #1, Fig.14The tref box that manages the trak box of track #3 stores information indicating that another track associated with track #3 is track #1 (track_ID = 1) and that track #3 is a track of metadata (here, specified information) (type = cdsc).

[0195] In the mdia box that manages the trak box of track #3, an hdlr box and a minf box are stored. In the trak box that manages track #3, the hdlr box stores information indicating that the data constituting track #3 is metadata (of the main image), and the minf box stores an stsc box, an stsc box, an stsz box, and an stco box for track #3.

[0196] <Generation and Reproduction of HEIF Files>

[0197] Fig.15 It is a flowchart outlining an example of a generation process for explaining the generation of an associated type HEIF file.

[0198] In the generation process, in step S11, the file control unit 43 generates a uuid as specified information for the RAW file (RAW image) of the frame of the main image, and the process advances to step S12.

[0199] In step S12, the file control unit 43 assigns the uuid generated in step S11 to the RAW file (RAW image) of the frame of the main image, and the process advances to step S13.

[0200] In step S13, the file control unit 43 generates an associated type HEIF file in which the frame of the main image and the uuid of the RAW file of that frame are stored in association with each other in the HEIF file, and the generation process ends.

[0201] Fig.16 It is a flowchart outlining an example of a reproduction process for explaining the reproduction of an associated type HEIF file.

[0202] In the reproduction process, in step S21, the file control unit 43 generates, for example, a handle list for identifying handles of each frame of the main image stored in the HEIF file stored in the medium 14, and the process advances to step S22.

[0203] Here, the handle of the frame of the main image includes the file name of the HEIF file in which the frame is stored. Moreover, the handle of the frame (item) of the main image stored in the collection file also includes the item ID of the frame. Moreover, the handle of the frame of the main image stored in the sequence file also includes the time information of the frame. Based on the handle of the frame of the main image, the frame corresponding to the handle can be uniquely identified (specified).

[0204] Note that the handle of the frame of the main image stored in the sequence file may include the track ID of the track containing the frame and the order of the frame in the track (what number the frame is), but not the time information of the frame.

[0205] Regardless of whether one or more tracks including the frame of the main image are stored in the sequence file, the time information of each frame is unique. Therefore, based on the time information of the frame, even if multiple tracks are stored in the sequence file, the frame of the time information included in the handle can be uniquely specified from the frames constituting each of the multiple tracks. Therefore, in the case where the time information of the frame is included in the handle of the frame of the main image, even if there is no track ID of the track where the frame is located, the frame corresponding to the time information can be uniquely specified.

[0206] The handle list may be generated for all frames of the main image stored in the HEIF file stored in the medium 14, or may be generated only for frames that are reduced under specific conditions (such as frames of a specific creation date and time).

[0207] After generating the handle list, the file control unit 43 accesses the HEIF file with reference to the handle list when necessary.

[0208] In step S22, for example, after waiting for the user to operate the digital camera 10 to display the thumbnail image, the UI control unit 47 requests the file control unit 43 to display the thumbnail image. In response to the thumbnail image display request from the UI control unit 47, the file control unit 43 reads the thumbnail image (frame) of the frame of the main image identified by the handle of the handle list from the HEIF file. Then, the file control unit 43, for example, causes the liquid crystal panel 19 ( Figure 1 ) displays a list of thumbnail images read from the HEIF file, and processing proceeds from step S22 to step S23.

[0209] In step S23, for example, after waiting for the user to select a desired thumbnail (frame) in the thumbnail image list, the UI control unit 47 requests the file control unit 43 to transmit a main image corresponding to the thumbnail image selected by the user. The file control unit 43 reads the main image from the HEIF file in response to the request for the main image from the UI control unit 47. When necessary, the file control unit 43 can cause the liquid crystal panel 19 to display the main image read from the HEIF file.

[0210] Alternatively, the UI control unit 47 requests the uuid of the RAW file of the main image corresponding to the thumbnail image selected by the user from the file control unit 43. In response to the uuid request from the UI control unit 47, the file control unit 43 reads the uuid from the associated type HEIF file. If necessary, the file control unit 43 can access the RAW file specified by the uuid read from the associated type HEIF file.

[0211] Fig.17 is a flowchart for explaining an example of a reproduction process of a reproduction set file.

[0212] In step S31 , the file control unit 43 acquires the item ID of the reproduction target image which is the image (item) to be reproduced (hereinafter, also referred to as the reproduction target item ID), and the process proceeds to step S32 .

[0213] In acquiring the reproduction target item ID, the item ID of the reproduction target image (reproduction target item ID) is acquired by taking as the reproduction target image a main image identified by, for example, any handle in a handle list, a thumbnail image of the main image, a thumbnail image selected by a user from a list of thumbnail images (hereinafter, also referred to as a selected thumbnail image), a main image of a selected thumbnail image, and the like.

[0214] In step S32 , the file control unit 43 reads the reproduction target image according to the reproduction target item ID acquired in step S31 .

[0215] In reading of the reproduction target image, the reproduction target image specified by the reproduction target item ID is read from the set file.

[0216] Fig.18 It is used to explain Fig.17 Flowchart of an example of a process of reading a reproduction target image in step S32 of FIG.

[0217] In step S41, the file control unit 43 collects files ( Figures 9 to 12 )'s iloc box, and the processing proceeds to step S42.

[0218] In step S42, the file control unit 43 reads the offset and size associated with the reproduction target item ID searched in step S41 in the iloc box, and the process proceeds to step S43.

[0219] In step S43, the file control unit 43 reads the reproduction target image stored in the mdat box of the set file according to the offset and size associated with the reproduction target item ID, and the processing ends.

[0220] Fig.19is used to explain Fig.17 Flowchart of a first example of the process of acquiring the reproduction target item ID in step S31 of FIG.

[0221] Right now, Fig.19 An example of acquiring the item ID of a thumbnail image as a reproduction target image is illustrated.

[0222] Note that in Fig.19 , it is assumed that the file control unit 43 has recognized the item ID of the main image which is the thumbnail image of the reproduction target image from the handle, for example.

[0223] In step S51, the file control unit 43 collects the files ( Figures 9 to 12 ) in the thmb box of the iref box searches for the thmb box whose reference source matches the item ID of the main image, and the process proceeds to step S52.

[0224] In step S52, as the item ID of the thumbnail which is the reproduction target image, the file control unit 43 reads the reference destination in the thmb box searched in step S51 and having a reference source matching the item ID of the main image, and the processing ends.

[0225] Fig. 20 is used to explain Fig.17 Flowchart of a second example of the process of acquiring the reproduction target item ID in step S31 of FIG.

[0226] Right now, Fig. 20 An example of acquiring the item ID of the main image as the reproduction target image is illustrated.

[0227] Note that in Fig. 20 In the description, for example, it is assumed that the user has selected a thumbnail image (selected thumbnail image) from the list of thumbnail images, and the file control unit 43 has recognized the item ID of the selected thumbnail image.

[0228] In step S61, the file control unit 43 collects the files ( Figures 9 to 12 )'s thmb box in the iref box is searched for a thmb box whose reference destination matches the item ID of the selected thumbnail, and the processing proceeds to step S62.

[0229] In step S62, the file control unit 43 reads the reference source in the thmb box searched in step S61 as the item ID of the main image of the reproduction target image and has a reference destination matching the item ID of the selected thumbnail image, and the processing ends.

[0230] Fig.21is used to explain Fig.10 A flowchart of an example of a process of acquiring uuid of designation information of a RAW file which is a predetermined main image in a first association type collection file.

[0231] Note that in Fig.21 In the example, it is assumed that the file control unit 43 has already recognized the item ID of a predetermined main image through a handle list or the like.

[0232] In step S71, the file control unit 43 collects files of the first association type ( Fig.10 ) is searched for the item ID of the predetermined main image in the associated information in the associated information storage box, and the processing proceeds to step S72.

[0233] In step S72 , the file control unit 43 reads the uuid associated with the item ID of the predetermined main image searched in step S71 in the association information, and the process ends.

[0234] The file control unit 43 can access the RAW file of the predetermined master image through the uuid read as described above.

[0235] Fig. 22 is used to explain Fig.11 A flowchart of an example of a process of acquiring uuid of designation information of a RAW file which is a predetermined main image in a second association type set file.

[0236] Note that in Fig. 22 In the example, it is assumed that the file control unit 43 has already recognized the item ID of a predetermined main image through a handle list or the like.

[0237] In step S81, the file control unit 43 generates a second association type set file ( Fig.11 ) searches for an infe box of item type IDIF indicating that the item is associated information, and the processing proceeds to step S82.

[0238] In step S82, the file control unit 43 reads the item ID of the associated information as an item from the infe box of the item type IDIF searched in step S81, and the process proceeds to step S83.

[0239] In step S83, the file control unit 43 searches the iloc box of the second association type set file for the item ID of the association information read in step S82, and the process proceeds to step S84.

[0240] In step S84 , the file control unit 43 reads the offset and size associated with the item ID of the associated information searched in step S83 in the iloc box, and the process proceeds to step S85 .

[0241] In step S85, the file control unit 43 reads association information as an item stored in the mdat box of the second association type set file based on the offset and size associated with the item ID of the association information read in step S84, and the process proceeds to step S86.

[0242] In step S86 , the file control unit 43 searches for the item ID of the predetermined main image from the associated information read in step S85 , and the process proceeds to step S87 .

[0243] In step S87 , the file control unit 43 reads the uuid associated with the item ID of the predetermined main image searched in step S86 in the association information, and the process ends.

[0244] The file control unit 43 can access the RAW file of the predetermined master image through the uuid read as described above.

[0245] Fig.23 is used to explain Fig.12 A flowchart of an example of a process of acquiring uuid of designation information of a RAW file which is a predetermined main image in a third association type set file.

[0246] Note that in Fig.23 In the example, it is assumed that the file control unit 43 has already recognized the item ID of a predetermined main image through a handle list or the like.

[0247] In step S91, the file control unit 43 generates a third association type set file ( Fig.12 )'s cdsc box in the iref box is searched for a cdsc box whose reference source matches the item ID of the predetermined main image, and the processing continues to step S92.

[0248] In step S92, as the item ID of the designating information of the RAW file of the predetermined main image as the item, the file control unit 43 reads the reference destination in the cdsc box searched in step S91 and the reference source having the item ID matching the predetermined main image, and the processing proceeds to step S93.

[0249] In step S93, the file control unit 43 searches the iloc box of the third association type set file for the item ID which is the specification information of the item read in step S92, and the process proceeds to step S94.

[0250] In step S94 , the file control unit 43 reads the offset and size associated with the item ID of the designated information searched in step S93 in the iloc box, and the process proceeds to step S95 .

[0251] In step S95, the file control unit 43 reads uuid as designated information of the RAW file of a predetermined main image stored in the mdat box of the third association type set file according to the offset and size associated with the item ID of the designated information read in step S94, and the processing ends.

[0252] The file control unit 43 can access the RAW file of the predetermined master image through the uuid read as described above.

[0253] Fig.24 is a flowchart for explaining an example of a process of acquiring an item ID list of a main image from a collection file.

[0254] For example, in the case of generating a handle list or the like, a process of acquiring an item ID list of a main image from a collection file is performed.

[0255] In step S101, the file control unit 43 reads the item ID ( Figures 9 to 12 ), and is registered in the item ID list of the main image (hereinafter, also referred to as the main image list), and the processing proceeds to step S102.

[0256] In step S102, the file control unit 43 reads the item ID as the reference destination from all the boxes in the iref box of the collection file, excludes the item ID from the main image list, and the process ends.

[0257] After the above processing, the item ID registered in the main image list becomes the item ID of the main image.

[0258] Fig.25 is a flowchart for explaining an example of a process of reproducing thumbnail images of (frames of) a main image with respect to predetermined time information from a sequence file.

[0259] Note that in Fig.25 In the example, it is assumed that the file control unit 43 has already recognized the time information (or order) of the predetermined main image through the handle list or the like.

[0260] In step S111, the file control unit 43 generates a track box as a track for managing thumbnail images of the main image with respect to predetermined time information in the sequence file ( Fig.13 and 14 ) searches the trak box in the moov box of the image processing unit for a trak box in which information indicating that data constituting the track is a thumbnail image is stored in the tref box, that is, a trak box in which the type in the tref box is thmb, and the processing proceeds to step S112.

[0261] In step S112, the file control unit 43 reads the track ID in the tkhd box in the trak box searched in step S111 as the track ID of the track of the thumbnail image of the main image with respect to the predetermined time information, and the processing proceeds to step S113.

[0262] In step S113, the file control unit 43 reproduces the track having the track ID read in step S112, and obtains the thumbnail image (frame) corresponding to the predetermined time information (or order) from the track as the thumbnail image of the main file corresponding to the predetermined time information, and the processing ends.

[0263] Note that the processing type of reproducing a track of images stored in a sequence file is similar to the processing of reproducing a moving image of an MP4 file.

[0264] Fig.26 It is a flowchart for explaining an example of a process of acquiring uuid of designation information of a RAW file which is a predetermined main image (frame) from an association type sequence file.

[0265] Note that in Fig.26 In the example, it is assumed that the file control unit 43 has already recognized the time information (or order) of the predetermined main image through the handle list or the like.

[0266] In step S121, the file control unit 43 associates the type sequence file ( Fig.14 )'s trak box in the moov box is searched for a trak box in which information indicating that data constituting the track is designated information is stored in the tref box, that is, a trak box in which the type in the tref box is cdsc, as the trak box for managing the track of the designated information, and the processing proceeds to step S122.

[0267] In step S122, the file control unit 43 reads the track ID in the tkhd box in the trak box searched in step S121 as the track ID of the track of the designated information, and the processing proceeds to step S123.

[0268] In step S123, from the track having the track ID read in step S122, the file control unit 43 acquires uuid as designation information relative to time information (or order) of the predetermined main image as the uuid of the RAW file of the predetermined main image, and the processing ends.

[0269] The file control unit 43 can access the RAW file of the predetermined main image through the uuid acquired as described above.

[0270] As described above, the file control unit 43 generates and reproduces an association type HEIF file that stores the main image in the HEIF file and the designation information of the external data to be associated with the main image outside the HEIF file in association with each other in the HEIF file conforming to the HEIF. Therefore, the main image stored in the HEIF file can be associated with the external data outside the HEIF file.

[0271] Furthermore, in the case of using uuid as the designation information, even if the file name of the external data is changed, the association between the main image in the HEIF file and the external data whose file name is changed can be maintained by the uuid.

[0272] <Storage of specified information assigned to external data>

[0273] Fig. 27 It is a view illustrating an example of storing uuid in a RAW file in a case where a RAW file of a main image is adopted as (a file storing) external data and an association type collection file is generated.

[0274] Note that in Fig. 27 In the example, the first association type set file is used as the association type set file.

[0275] The RAW file has an area called a marker comment (MakerNote) as a partial area for storing additional information of Exif as metadata.

[0276] The file control unit 43 may store the uuid assigned to the RAW file (RAW image) into, for example, a mark comment of the RAW file.

[0277] exist Fig. 27 , four items of main image Item#1, Item#2, Item#3, and Item#4 are stored in the association type set file, and RAW files #1, #2, #3, and #4 storing the RAW images of the main images Item#1, Item#2, Item#3, and Item#4 are generated. Then, UUID#i is assigned to RAW file #i (RAW image), and the association information storage box stores association information in which the item ID#i of the main image Item#i is designated to be associated with the UUID#i of the RAW file #i to be associated with the main image Item#i, as association information associating the main image Item#i with the UUID#i of the RAW file #i of the main image Item#i.

[0278] Fig.28 is a view illustrating an example of storing uuid in a RAW file in the case of adopting a RAW file of a main image as external data and generating an association type sequence file.

[0279] Also in the case of generating an associated type sequence file, with Fig. 27 Similar to the case of generating an associated type set file described in , the file control unit 43 can store the uuid assigned to the RAW file in the tag comment of the RAW file.

[0280] exist Fig.28 In the example, track #1 including main images #1, #2, #3 and #4 as four frames is stored in an associated type sequence file, and RAW files #1, #2, #3 and #4 storing RAW images of the main images #1, #2, #3 and #4 are generated. Then, UUID #i is assigned to RAW file #i, and the associated type sequence file stores track #3, which is configured so that UUID #i of RAW file #i is arranged to have the same time information as main image #i corresponding to RAW file #i (RAW image).

[0281] As described above, when track #3 is configured so that the UUID #i of RAW file #i is arranged to have the same time information as the main image #i corresponding to the RAW file #i, the i-th main image #i of track #1 and the i-th UUID #i of track #3 (i.e., the UUID #i of the RAW file #i of the main image #i) are associated and stored in the association type sequence file.

[0282] In the above description, the RAW file (RAW image) of the main image is used as the external data, but other data may be used as the external data. As the external data, for example, audio (sound) recorded together with the imaging of the main image may be used. As the file storing the audio, for example, a WAV file in the WAV format, an MP4 file in the MP4 format, etc. may be used. In the following, for example, it is assumed that a WAV file is used as the file storing the audio.

[0283] Fig.29 is a view illustrating an example of storing uuid in a WAV file in a case where a WAV file of a main image is adopted as (a file storing) external data and an association type collection file is generated.

[0284] Note that in Fig.29 In the example, the first association type set file is used as the association type set file.

[0285] The WAV file has an area called a list block as a partial area of ​​the area in which metadata is described.

[0286] The file control unit 43 may store the uuid assigned to the WAV file (audio) in, for example, a list block of the WAV file.

[0287] exist Fig.29 In the association type set file, main images Item#1, Item#2, Item#3, and Item#4 are stored as four items, and WAV files #1, #2, #3, and #4 of the main images Item#1, Item#2, Item#3, and Item#4 are generated. Then, UUID#i is assigned to WAV file #i (audio), and the association information storage box stores association information in which the item ID#i of the main image Item#i is specified to be associated with the UUID#i of the WAV file #i to be associated with the main image Item#i, as association information for associating the main image Item#i with the UUID#i of the WAV file #i of the main image Item#i.

[0288] Fig.30 is a view illustrating an example of storing uuid into a WAV file in a case where a WAV file of a main image is adopted as external data and an associated type sequence file is generated.

[0289] Also in the case of generating an associated type sequence file, with Fig.29 Similar to the case of generating an associated type set file described in , the file control unit 43 can store the uuid assigned to the WAV file in the list block of the WAV file.

[0290] exist Fig.30 In the example, track #1 including main images #1, #2, #3 and #4 as four frames is stored in an associated type sequence file, and WAV files #1, #2, #3 and #4 of the main images #1, #2, #3 and #4 are generated. Then, UUID #i is assigned to WAV file #i, and the associated type sequence file stores track #3, which is configured so that UUID #i of WAV file #i is arranged to have the same time information as the main image #i corresponding to WAV file #i.

[0291] As described above, when track #3 is configured with the UUID #i of the WAV file #i (which UUID #i is arranged to have the same time information as the main image #i corresponding to the WAV file #i), the i-th main image #i of track #1 and the i-th UUID #i of track #3 (i.e., the UUID #i of the WAV file #i of the main image #i) are associated and stored in the association type sequence file.

[0292] Note that in addition to HEIF files, the present technology can also be applied to, for example, ISO base media files having a box structure other than HEIF files, MP4 files, Miaf files, and the like.

[0293] Furthermore, the present technology can be applied to, for example, a file or the like that stores an image (main image) having no box structure and another image in which the resolution of the image is lowered.

[0294] Moreover, in addition to the case where the external data is associated with the main image in the HEIF file, the present technology can also be applied to the case where the external data is associated with the screen nail image or thumbnail image in the HEIF file.

[0295] Furthermore, the present technology can be applied to a case where, for example, external data is associated with internal data other than an image such as a main image in a HEIF file.

[0296] <Association between images as internal data and external data after HEIF file generation>

[0297] Fig.31 It is a view for explaining the association between images as internal data and external data after a HEIF file is generated.

[0298] When generating a HEIF file, in the case where there is external data to be associated with an image (main image) as internal data stored in the HEIF file, an associated type HEIF file can be generated by storing designated information of the image as internal data and the external data in association with each other in the HEIF file.

[0299] However, in the case where no external data is expected to be associated with an image as internal data when a HEIF file is generated, but external data is generated after the HEIF file is generated and it is expected that the image as internal data is associated with the external data, it is necessary to store relationship information related to the association between the image as internal data and the specified information of the external data in the HEIF file after the HEIF file is generated to generate an association type HEIF file. In addition, even in the case where external data exists when a HEIF file is generated, it is expected that the image as internal data is associated with the existing external data after the HEIF file is generated. In this case as well, it is necessary to store the relationship information in the HEIF file after the HEIF file is generated to generate an association type HEIF file.

[0300] Note that, as a case where external data is generated after the HEIF file is generated and it is desired to associate the image as internal data with the external data, for example, there is a case where it is desired to add annotations to the image captured by the digital camera 10 or the like through audio or text as external data. As a case where it is desired to associate the image as internal data with the existing external data after the HEIF file is generated, for example, there is a case where it is desired to add background music (BGM) as external data to the image captured by the digital camera 10 or the like.

[0301] Here, as described above, the relationship information is information related to the association between the image as internal data and the specified information of the external data. As the relationship information, for example, there is the association information ( Fig.10 ). Moreover, as the relationship information, for example, there is association information as stored in the second association type collection file ( Fig.11 ), an infe box for storing the item Item#201 in the iinf box of the meta box, an offset of the storage position of the item Item#201 stored in the iloc box of the meta box, etc. In addition, as the relationship information, for example, there is designation information as an item ( Fig.12 ), and association information in which the item ID of the main image as the item stored in the cdsc box of the iref box of the meta box is associated with the item ID of the designated information as the item. Also, as the relationship information, for example, there is track #3 ( Fig.14 ), and a trak box of track #3 which is stored in the moov box and manages designated information.

[0302] As described above, in the case where an image as internal data is associated with external data after generating a HEIF file, it is necessary to store the relationship information in the HEIF file. In this case, when no measures are taken to store the relationship information, the offset of the data stored in the mdat box can be shifted when the relationship information is stored in the HEIF file.

[0303] When moving the offset of data stored in the mdat box, it is necessary to calculate the offset shift amount for each data stored in the mdat box and rewrite the iloc box with the offset reflecting the shift amount, which increases the load of the association processing of associating the image as internal data with the external data.

[0304] As a method for preventing the offset shift of data stored in the mdat box and the corresponding increase in the load of the association processing as described above, for example, there is a method of generating a HEIF file in which a reserved area for storing relationship information related to the association between internal data and specified information of specified external data is ensured as an association type HEIF file (hereinafter, also referred to as an area ensuring method), and a method of generating a HEIF file that stores relationship information including specified information as an association type HEIF file before being assigned to external data (hereinafter, also referred to as a pre-storage method).

[0305] In the case where external data is generated after the HEIF file is generated, when information (such as a URL or a file name) generated when the external data (file) is generated is used as the designated information, the designated information cannot be obtained when the HEIF file is generated. Therefore, it is possible to adopt an area ensuring method for generating an association type HEIF file in which a reserved area for storing relationship information is ensured so that relationship information including any designated information can be stored in the future. As the designated information generated when the external data is generated, in addition to the URL or the file name, for example, there is a hash value generated using the external data as an input, a track ID (track number) of a track when the external data is data of a track stored in an MP4 file, and the like.

[0306] In addition, when information that can be generated before external data is generated (such as UUID) is used as designated information, it is possible to adopt a pre-storage method of generating the designated information in advance (before being assigned to external data) when generating a HEIF file, and generate an associated type HEIF file that stores relationship information including the designated information. It can also be said that the designated information before being assigned to the external data is the designated information that was not assigned to the external data when the associated type HEIF file was generated. Note that in the case where the pre-storage method can be adopted, the area ensuring method can also be adopted.

[0307] By generating an associated type HEIF file using the area ensuring method or the pre-storage method, it is possible to prevent the offset of the data stored in the mdat box from shifting when the associated processing is performed after the HEIF file is generated. Therefore, the load of the associated processing can be prevented from increasing, and an image as internal data can be easily associated with external data.

[0308] Hereinafter, the area securing method and the pre-storage method will be described.

[0309] Fig.32 It is a view for explaining the outline of an example of the area securing method.

[0310] Note that in Fig.32 In the example, it is assumed that the first association type collection file is generated by the region ensuring method.

[0311] The file control unit 43 generates Fig.10 The situation is similar for the first association type collection file. Fig.32 In the example, a first association type set file is generated, in which a main image of a frame and a thumbnail image of the main image are stored in the mdat box. The item IDs of the main image and the thumbnail image are item ID#1 and item ID#1001, respectively.

[0312] However, in the area securing method, in the associated information storage box, the associated information is not stored, but a padding area is secured as a reserved area for writing (storing (overwriting)) the associated information as relationship information in the future.

[0313] Note that in Fig.32 In the method, in addition to the reserved area, the associated information storage box is also provided with the number of associated information. The number of associated information indicates the number of associated information (here, pieces of information associating a main image of one frame with a piece of specified information) stored in the associated information storage box. In the area ensuring method, the number of associated information is set to 0 as an initial value when the first association type set file is generated, and is updated (overwritten) with the number of pieces of associated information stored in the associated information storage box according to writing the associated information into the reserved area of ​​the associated information storage box.

[0314] Fig.33 It is a view for explaining an example of an association process of associating external data and a main image as internal data in a first association type set file generated by an area ensuring method after generating the first association type set file.

[0315] Note that in Fig.33 In the generation Fig.32 An association process is then performed after the first association type collection file in which the main image (main image Item#1) of item ID#1 in the first association type collection file is associated with (the audio stored in) the WAV file as external data.

[0316] In the association process, for the WAV file to be associated with the main image Item#1 in the first association type set file, the file control unit 43 acquires (recognizes) the file name (DSC00001.WAV) as the designation information of (the audio stored in) the WAV file, and generates association information in which the file name is associated with the item ID#1 of the main image Item#1. Then, the file control unit 43 overwrites (a part of) the padding area in the association information storage box with the association information generated for the WAV file, and updates the association information quantity to a value obtained by incrementing the current value by 1 (here, 1).

[0317] Fig.34 It is a view for explaining an outline of an example of the pre-storage method.

[0318] Note that in Fig.32 In the example, it is assumed that a first association type collection file is generated by a pre-storage method.

[0319] The file control unit 43 generates Fig.10 The situation is similar for the first association type collection file. Fig.34 In, with Fig.32 Similarly, a first association type set file is generated, in which a frame of main image Item#1 and thumbnail image Item#1001 of the main image Item#1 are stored in the mdat box.

[0320] However, in this case, the external data to be associated with the main image Item#1 in the first association type collection file is not determined. Therefore, for (all or some of) the main image Item#1 in the first association type collection file, the file control unit 43 generates an association type HEIF file storing relationship information including designation information before being assigned to the external data and for designating the external data to be associated with the main image Item#1 in the future.

[0321] Specifically, the file control unit 43 generates in advance UUID#1 as the uuid of the designation information that designates the external data to be associated with the main image Item#1 in the future, that is, generates UUID#1 in a state where the external data to which UUID#1 is assigned is not determined, and generates association information in which UUID#1 is associated with the item ID#1 of the main image Item#1. Then, the file control unit 43 generates a first association type set file provided with an association information storage box that stores association information in which UUID#1 is associated with the item ID#1 of the main image Item#1, and the number of association information indicating the number of pieces of association information.

[0322] Note that in Fig.34 In the example, since one piece of associated information of the main image Item#1 and UUID#1 is stored in the associated information storage box, the number of associated information is 1.

[0323] Fig.35 is a view for explaining an example of an association process of associating external data and a main image as internal data in a first association type aggregation file generated by a pre-storage method after generating the first association type aggregation file.

[0324] Note that in Fig.35 In the example above, it is assumed that an association process is performed, where Fig.34 After the first association type collection file in , the main image Item#1 in the first association type collection file is associated with the WAV file (audio) as external data.

[0325] In the association process, the file control unit 43 assigns UUID#1, which is pre-generated for the main image Item#1 in the first association type set file and is used as designation information of external data to be associated with the main image Item#1, to the WAV file (audio) to be associated with the main image Item#1, and the file control unit 43 stores (writes) the UUID#1 assigned to the WAV file into, for example, the list block of the WAV file. As a result, the assignment of UUID#1 to the WAV file is maintained (saved).

[0326] It is noted that, in the pre-storage method, regarding the association information, assuming that the fact that the designated information included in the association information has been assigned and has not been assigned to the external data is defined as valid and invalid, respectively, then all the association information including the designated information generated in advance when the first association type set file is generated is invalid. Then, when the designated information that has not been assigned to the external data is assigned to the external data, the association information including the designated information will be valid. In the pre-storage method, information indicating whether the association information is valid or invalid can also be stored in the association information storage box.

[0327] Fig.36 is a view for explaining an outline of another example of the pre-storage method.

[0328] Note that in Fig.36 In, with Fig.34 Similarly, it is assumed that the first association type collection file is generated by a pre-storage method.

[0329] The file control unit 43 generates Fig.10 The situation is similar for the first association type collection file. Fig.36 In the example, a first association type set file is generated, in which four frames of main images Item#1 to Item#4 and thumbnail images Item#1 to Item#4 of four frames of main images Item#1 to Item#4 are stored in the mdat box.

[0330] However, when generating the first association type collection file, the external data to be associated with the main image Item#i is not determined. Therefore, for each main image Item#i in the first association type collection file, the file control unit 43 pre-generates UUID#i as designation information, which is used to designate the external data to be associated with the main image Item#i in the future before being assigned to the external data, and generates association information in which UUID#i is associated with the item ID#i of the main image Item#i. Then, the file control unit 43 generates a first association type collection file provided with an association information storage box, which stores association information in which UUID#i is associated with the item ID#i of the main image Item#i, and the number of association information indicating the number of pieces of association information. Fig.36 In the associated information storage box, four pieces of associated information are stored according to the number of main images Item#1 to Item#4, and the number of associated information is 4.

[0331] In the association process of associating the main images Item#1 to Item#4 in the first association type set file with the WAV files #1 to #4 as external data respectively after the first association type set file is generated, the file control unit 43 assigns UUID#i which is generated in advance for the main image Item#i in the first association type set file and is used as the designation information of the WAV file to be associated with the main image Item#i, and the file control unit 43 stores the UUID#i assigned to the WAV file#i into, for example, the list block of the WAV file#i. As a result, the assignment of the UUID#i to the WAV file#i is maintained.

[0332] Fig.37 is a view for explaining an outline of still another example of the pre-storage method.

[0333] Note that in Fig.37 In , it is assumed that the association type sequence file is generated by the pre-storage method.

[0334] The file control unit 43 generates Fig.14 The situation is similar for the associated type sequence files.

[0335] exist Fig.37 In the example, the mdat box stores track#1 including a main image of four frames and track#2 including thumbnail images of the main image of four frames, and the associated type set file stored in the moov box is generated in the trak box that manages tracks#1 and#2.

[0336] However, in Fig.37In the example, the external data to be associated with the main image of the four frames is not determined when the associated type sequence file is generated. Then, for each of the main images of the four frames in the associated type sequence file, UUID#i is pre-generated as designation information, which is used to designate the external data to be associated with the main image in the future before being assigned to the external data, and the track #3 including UUID#i is stored in the mdat box, and the associated type sequence file stored in the moov box is generated in the trak box that manages track #3. Fig.37 In the example, UUID#i is used as the designated information of the external data to be associated with the i-th main image among the main images of the four frames. It is noted that in the association type sequence file, multiple tracks including the main image (main image tracks) and multiple tracks including designated information (designated information tracks) can be stored in the mdat box. In the case where multiple main image tracks and multiple designated information tracks are stored in the mdat box of the association type sequence file, the association type sequence file stores information for associating the main image track and the designated information track having the designated information before being assigned to the external data to be associated with the main image of the main image track. The information for associating the main image track with the designated information track can be stored in, for example, the moov box of the association type sequence file. This point is similarly applicable to the case where the association processing is performed after the generation of the association type sequence file generated by the area ensuring method.

[0337] In the association process of associating the i-th main image of track #1 in the association type sequence file with the WAV file #i (audio) as external data after the association type sequence file is generated, the file control unit 43 assigns UUID #i, which is generated in advance for the i-th main image of track #1 in the association type sequence file and serves as designation information of the WAV file to be associated with the i-th main image, to the WAV file #i to be associated with the i-th main image, and the file control unit 43 stores the UUID #i assigned to the WAV file #i in, for example, the list block of the WAV file #i. As a result, the assignment of the UUID #i to the WAV file #i is maintained.

[0338] Fig.38 It is a view for explaining the association between available designation information and internal data and external data for each of the area securing method and the pre-storage method.

[0339] As for the designation information that can be used, in the area securing method, since any information can be written into the reserved area, any designation information can be adopted. For example, as the designation information, it is possible to adopt a hash value generated using external data as an input, and information to be assigned to the external data after the external data is generated (URL, file name, track ID of the MP4 file storing the external data, etc.).

[0340] However, in the case of adopting designated information (such as UUID) that needs to be written to a file of external data (a file in which the external data is stored), the file of external data is limited to files in a format (that can maintain the assignment of the designated information) in which the separately generated designated information can be written (stored).

[0341] In the case where designation information (such as a URL) of a file that does not need to be written in external data is adopted, any external data (file) can be adopted.

[0342] In the pre-storage method, the specified information is generated in advance (before the association between the internal data and the external data). Therefore, the files of the external data associated with the internal data are limited to the files (RAW files, WAV files, MP4 files, HEIF files, etc.) in which the pre-generated specified information can be written.

[0343] Furthermore, it is impossible to employ specified information that can be generated after determining external data associated with internal data (such as a hash value generated using external data as an input).

[0344] Note that in any of the area ensuring method and the pre-storage method, the external data may be data generated before the associated type HEIF file is generated, or may be data generated after the associated type HEIF file is generated.

[0345] For a 1-to-N association that associates one piece of internal data with one or more pieces of external data, in the area securing method, in the case where different information is adopted as the designation information for each piece of external data (such as a hash value with the external data as an input), the data amount of the designation information increases in proportion to the number of pieces of external data to be associated with one piece of internal data. Therefore, the number of pieces of external data to be associated with one piece of internal data is limited according to the capacity of the reserved area.

[0346] In the pre-storage method, since the designation information generated in advance for one piece of internal data is written into the file in which each piece of external data to be associated with one piece of internal data is stored, the number of pieces of external data associated with one piece of internal data is not limited.

[0347] For an N-to-1 association that associates one or more internal data with a piece of external data, in the area ensuring method, since the data amount of the specified information increases proportionally with the number of internal data to be associated with a piece of external data, the number of internal data associated with a piece of external data is limited according to the capacity of the reserved area.

[0348] In the pre-storage method, when the number of specified information that can be written into a file storing a piece of external data is limited, the number of internal data to be associated with the piece of external data is limited according to the number of specified information that can be written into the file storing the piece of external data.

[0349] Fig.39 It is a view for explaining an example of performing an association process of 1-to-N association on a first association type set file generated by the area securing method.

[0350] The file control unit 43 generates a file in which, for example, the area securing method is used to secure Fig.32 The first association type of the fill area described in the collection file.

[0351] exist Fig.39 In, with Fig.32 Similarly, a first association type set file is generated, in which a frame of main image Item#1 and thumbnail image Item#1001 of the main image Item#1 are stored in the mdat box.

[0352] In the case where there is a request for a 1-to-2 association process of associating the main image Item#1 in the first association type collection file with two WAV files #1 and #2 (audio stored therein) as external data after the first association type collection file is generated, in the association process, for each of the WAV files #1 and #2 to be associated with the main image Item#1 in the first association type collection file, the file control unit 43 acquires the file name as designated information, and generates association information in which the file name is associated with the item ID#1 of the main image Item#1. The file names of the WAV files #1 and #2 are DSC00001.WAV and DSC00002.WAV, respectively. Therefore, as the association information, generated information in which the item ID#1 and the file name DSC00001.WAV are associated with each other and information in which the item ID#1 and the file name DSC00002.WAV are associated with each other are generated. Then, the file control unit 43 overwrites the padding area in the associated information storage box with the associated information generated for the WAV files #1 and #2, and updates the number of associated information to a value (here, 2) obtained by incrementing the current value by the number of pieces of associated information overwritten on the padding area by 2. As a result, the main image Item #1 can be associated with the WAV files #1 and #2.

[0353] For example, in the case where a different file name is adopted as the designation information for each WAV file as external data, the data amount of the designation information (i.e., the association information in which the designation information and the item ID#1 of the main image Item#1 are associated with each other) increases in proportion to the number of WAV files to be associated with the main image Item#1 as a piece of internal data. Therefore, the number of WAV files (external data) to be associated with the main image Item#1 as a piece of internal data is limited according to the capacity of the padding area as the reserved area.

[0354] Fig.40 is a view for explaining an example of performing an association process of 1-to-N association on a first association type set file generated by a pre-storage method.

[0355] The file control unit 43 generates, for example, Fig.34 The first association type collection file described in .

[0356] exist Fig.40 In, with Fig.34 Similarly, a first association type set file is generated, in which a frame of main image Item#1 and thumbnail image Item#1001 of the main image Item#1 are stored in the mdat box. Fig.40 In, with Fig.34 Similarly, UUID#1, which is designation information for designating external data to be associated with main image Item#1 in the future, is generated in advance, and association information in which UUID#1 is associated with item ID#1 of main image Item#1 is generated. Then, a first association type set file is generated that is provided with an association information storage box (the association information storage box stores association information in which UUID#1 is associated with item ID#1 of main image Item#1) and an association information quantity indicating the number of pieces of association information.

[0357] In the case where there is a request for a 1-to-2 association process of associating the main image Item#1 in the first association type set file with two WAV files #1, #2 as external data after the first association type is generated, in the association process, the file control unit 43 assigns UUID#1, which is pre-generated for the main image Item#1 in the first association type set file and serves as designation information of the external data to be associated with the main image Item#1, to the WAV files #1 and #2 to be associated with the main image Item#1, and the file control unit 43 stores (writes) the UUID#1 assigned to the WAV files #1 and #2 in the list block of each of the WAV files #1 and #2. As a result, the main image Item#1 can be associated with (the audio stored in) the WAV files #1 and #2.

[0358] As described above, for the first association type collection file generated by the pre-storage method, by writing UUID#1 which is the designated information pre-generated for the main image Item#1 into a WAV file which is external data, and by maintaining the assignment of UUID#1 to the WAV file, any number of WAV files which are external data can be associated with the main image Item#1.

[0359] Fig.41 It is a view for explaining an example of an association process of performing N-to-1 association on a first association type set file generated by the area securing method.

[0360] The file control unit 43 generates a file in which, for example, the area securing method is used to secure Fig.32 The first association type of the fill area described in the collection file.

[0361] exist Fig.41 In the example, a first association type set file is generated, in which the mdat box stores the main images Item#1 and Item#2 of two frames and the thumbnail images Item#1001 and Item#1002 of the main images Item#1 and Item#2.

[0362] In the case where there is a request for a 2-to-1 association process of associating each of main images Item#1 and Item#2 in the first association type collection file with (audio stored in) one WAV file #1 as external data after the first association type collection file is generated, in the association process, the file control unit 43 obtains the file name as designated information for associating WAV file #1 with main images Item#1 and Item#2 in the first association type collection file, and generates association information in which the file name is associated with each of item ID#1 of main image Item#1 and item ID#2 of main image Item#2. In the case where the file name of WAV file #1 is DSC00001.WAV, a total of two pieces of association information are generated: association information in which item ID#1 and file name DSC00001.WAV are associated with each other; and association information in which item ID#2 and file name DSC00001.WAV are associated with each other. Then, the file control unit 43 overwrites the padding area in the association information storage box with the two pieces of association information generated for WAV file #1, and updates the number of association information to a value (here, 2) obtained by incrementing the current value by the number of pieces of association information overwritten on the padding area by 2. As a result, the main image Item #1 and (the audio stored in) WAV file #1 can be associated with each other, and the main image Item #2 and WAV file #1 can be associated with each other.

[0363] For the association type HEIF file generated by the area ensuring method, the data amount of the association information in which the item ID of the main image is associated with the designated information of the WAV file #i increases in proportion to the number of main images associated with one WAV file #1 as external data. Therefore, the number of main images to be associated with one WAV file #1 as external data is limited according to the capacity of the padding area as the reserved area.

[0364] Fig.42 is a view for explaining an example of performing an association process of N-to-1 association on a first association type set file generated by a pre-storage method.

[0365] The file control unit 43 generates, for example, Fig.34 The first association type collection file described in .

[0366] exist Fig.42 In the example, a first association type set file is generated, wherein the mdat box stores the main images Item#1 and Item#2 of two frames and the thumbnail images Item#1001 and Item#1002 of the main images Item#1 and Item#2. Fig.42 In, with Fig.34 Similarly, UUID#1 and UUID#2 are generated in advance as designation information for designating external data to be associated with respective main images Item#1 and Item#2 in the future, and two pieces of association information are generated: association information in which UUID#1 is associated with item ID#1 of main image Item#1; and association information in which UUID#2 is associated with item ID#2 of main image Item#2. Then, a first association type set file is generated that is provided with an association information storage box that stores two pieces of association information and an association information quantity indicating the number of pieces of association information.

[0367] In the case where there is a request for 2-to-1 association processing of associating each of main image Item#1 and Item#2 in the first association type collection file with one WAV file #1 as external data in addition to generating the first association type collection file, in the association processing, the file control unit 43 assigns UUID#1 and UUID#2, which are pre-generated for the main image Item#1 in the first association type collection file and used as designation information of external data to be associated with the main image Item#1, to the WAV file #1 (audio) to be associated with the main image Item#1 and iterm#2, and the UUID#2 which are pre-generated for the main image Item#2 and used as designation information of external data to be associated with the main image Item#2, and the file control unit 43 stores the UUID#1 and UUID#2 assigned to the WAV file #1 in the list block of the WAV file #1. As a result, each of the main image Item#1 and Item#2 can be associated with the WAV file #1.

[0368] As described above, for the set file of the first association type generated by the pre-storage method, the uuid as the designated information pre-generated for the main image is written into the WAV file (in which the audio is stored) as the external data. Therefore, in the case where the number of designated information that can be written to one WAV file is limited, the number of main images to be associated with (the audio stored in) one WAV file is limited according to the number of designated information that can be written to one WAV file.

[0369] <Using MP4 file as a file for storing external data>

[0370] Fig.43 1 is a view illustrating a first example of association between a main image as internal data and audio as external data in a case where an MP4 file is adopted as a file storing audio as external data.

[0371] Here, one (a series of) audio is stored in a WAV file. In an MP4 file including a moov box, an mdat box, etc., one audio in multiple tracks can be used as one track, that is, multiple audios are stored together. Therefore, in the case where audio as external data is stored in an MP4 file, when a main image stored in a HEIF file that can store one or more images (frames) is associated with multiple audios as external data, by storing the multiple audios as external data in an MP4 file with multiple tracks, a one-to-one correspondence can be achieved in which one file of the HEIF file is associated with one file of the MP4 file in units of files.

[0372] In the case where a main image as internal data is associated with multiple audios as external data, it is convenient to jointly process the multiple audios by storing the multiple audios as external data in an MP4 file. For example, in the case where a main image in an association type HEIF file is associated with multiple audios, when the multiple audios are stored in a WAV file, an equal number of WAV files as the multiple audios are required. As a result, in the case of transmitting a main image and multiple audios that have been associated, it is necessary to transmit an association type HEIF file storing the main image and multiple WAV files storing the multiple audios.

[0373] In the case of associating a main image in an association type HEIF file with multiple audios, when the multiple audios are stored in an MP4 file, the multiple audios can be stored in one MP4 file. As a result, in the case of transmitting a main image and multiple associated audios, it is only necessary to transmit one file of the association type HEIF file storing the main image and one file of the MP4 file storing the audio clips.

[0374] Note that as the plurality of pieces of audio as external data, for example, a plurality of pieces of audio having the same content in different languages ​​or the like may be employed.

[0375] Fig.43 An example of association between a main image stored in a first association type set file and an audio stored in an MP4 file is illustrated.

[0376] The first association type set file is generated, for example, by a pre-stored method. Fig.43 In the first association type set file, one (one frame) main image Item#1 and the thumbnail image Item#1001 of the main image Item#1 are stored in the mdat box. Fig.43 In, with Fig.34 Similarly, UUID#1, which is designation information for specifying external data to be associated with main image Item#1 in the future, is generated in advance, and associated information in which UUID#1 is associated with item ID#1 of main image Item#1 is generated, and the associated information and the number of associated information indicating the number of pieces of associated information are stored in an associated information storage box.

[0377] In the MP4 file storing an audio #1 associated with the main image Item#1, the track #1 of the audio #1 is stored in the mdat box, and the trak box managing the track #1 is stored in the moov box. Here, the trak box having the tkhd box in which the track ID is set to i is the trak box managing the track #i.

[0378] In the case where there is a request for association processing to associate main image Item#1 in the first association type collection file with audio #1 in the MP4 file after the first association type collection file is generated, in the association processing, the file control unit 43 assigns UUID#1 to the audio #1 to be associated with the main image Item#1, which UUID#1 is pre-generated for the main image Item#1 in the first association type collection file and is used for designation information of external data to be associated with the main image Item#1, and the file control unit 43 stores (writes) the UUID#1 assigned to the audio #1 in the MP4 file storing the audio #1.

[0379] That is, the file control unit 43 generates track #2 of UUID #1 for the MP4 file and stores it in the mdat box, and generates a trak box for managing track #2 and stores it in the moov box.

[0380] The trak box for managing track #2 has a tkhd box and a tref box. In the tkhd box of the trak box for managing track #2, the track ID 2 for specifying the track #2 to be managed is set.

[0381] Furthermore, in the tref box of the trak box managing track #2, it is set that another track associated with track #2 is track #1 (track_ID=1) and track #2 is a track (type=cdsc) of metadata (here, designation information).

[0382] As described above, by storing UUID#1 as designation information generated in advance for main image Item#1 in the MP4 file storing audio#1, main image Item#1 and audio#1 can be associated with each other.

[0383] Fig.44 is a view illustrating a second example of association between a main image as internal data and audio as external data in a case where an MP4 file is adopted as a file storing audio as external data.

[0384] Fig.44 An example of association between four main images stored in a first association type set file and four pieces of audio stored in an MP4 file is illustrated.

[0385] The first association type set file is generated, for example, by a pre-stored method. Fig.44 In the first association type set file, four main images Item#1 to Item#4 and thumbnails Item#1001 to Item#1004 of the main images Item#1 to Item#4 are stored in the mdat box. Fig.44 In, with Fig.34Similarly, UUID#i is generated in advance as designation information for designating external data to be associated with main image Item#i (where i=1, 2, 3, 4) in the future, and four pieces of associated information in which UUID#i is associated with item ID#i of main image Item#i are generated. Then, an associated information example indicating the number of pieces of four pieces of associated information is stored in the associated information storage box.

[0386] In the MP4 file storing the audio #i to be associated with the main image Item #i, the track #2i-1 of each of the four segments of audio #i is stored in the mdat box (i=1, 2, 3, 4), and the trak box managing the track #2i-1 (the trak box of the tkhd box with the track ID set to 2i-1) is stored in the moov box.

[0387] In the case where there is a request for association processing to associate the main image Item#i in the first association type collection file with the audio #i in the MP4 file after the first association type collection file is generated, when processing in the association, the file control unit 43 assigns UUID#i to the audio #i to be associated with the main image Item#i, which UUID#i is pre-generated for the main image Item#i in the first association type collection file and is used as designation information of external data to be associated with the main image Item#i, and the file control unit 43 stores the UUID#i assigned to the audio #i in the MP4 file storing the audio #i.

[0388] That is, the file control unit 43 generates the track #2i of the UUID #i for the MP4 file and stores it in the mdat box, and generates the trak box for managing the track #2i and stores it in the moov box.

[0389] The trak box for managing track #2i has a tkhd box and a tref box. In the tkhd box of the trak box for managing track #2i, a track ID 2i for specifying the track #2i to be managed is set.

[0390] Furthermore, in the tref box of the trak box managing track #2i, it is set that another track associated with track #2i is track #2i-1 (track_ID=2i-1), and track #2i is a track of metadata (here, designation information) (type=cdsc).

[0391] As described above, by storing UUID#i which is designation information generated in advance for main image Item#1 in the MP4 file storing audio#i, main image Item#i and audio#i can be associated with each other.

[0392] Fig.45is a view illustrating a third example of association between a main image as internal data and audio as external data in a case where an MP4 file is adopted as a file storing audio as external data.

[0393] Fig.45 An example of association between four main images stored in an association type sequence file and four pieces of audio stored in an MP4 file is illustrated.

[0394] The association type sequence file is generated, for example, by a pre-storage method. Fig.45 In the associated type sequence file, track #1 including four main images and track #2 including thumbnail images of the corresponding four main images are stored in the mdat box.

[0395] Moreover, in Fig.45 In, with Fig.37 Similarly, UUIDs #1, #2, #3, and #4, which are designation information for designating external data to be respectively associated with the first to fourth main images of track #1 in the future, are generated in advance, and track #3 is generated in which UUIDs #1 to #4 are arranged in the same order as the corresponding four main images. Then, track #3 including UUIDs #1 to #4 is stored in the mdat box.

[0396] In addition, Fig.45 In the associated type sequence file, a trak box managing each of tracks #1 to #3 stored in the mdat box is stored in the moov box. The trak box managing each of tracks #1 to #3 includes a tkhd box in which the track ID of the track to be managed by the trak box is set.

[0397] Furthermore, the trak box managing tracks #2 and #3 also has a tref box. In the tref box of the trak box managing track #2, it is set that another track associated with track #2 is track #1 (track_ID=1), and track #2 is a track of thumbnail images (type=thmb). In the tref box of the trak box managing track #3, it is set that another track associated with track #3 is track #1 (track_ID=1), and track #3 is a track of metadata (specification information here) (type=cdsc).

[0398] And Fig.44 The MP4 files storing four pieces of audio #1 to #4 respectively associated with the four main images in the association type sequence file are configured similarly.

[0399] In the case where there is a request for association processing of each of the four main images of track #1 in the associated type sequence file with each of the four audio segments #1 to #4 in the MP4 file after an associated type sequence file is generated, in the association processing, the file control unit 43 assigns UUIDs #1 to #4 to the four audio segments #1 to #4 to be respectively associated with the four main images, which UUIDs are pre-generated for the corresponding four main images of track #1 in the associated sequence file and are used as designation information of external data, and the file control unit 43 stores the UUIDs #1 to #4 respectively assigned to the audio segments #1 to #4 in the MP4 file storing the audio segments #1 to #4.

[0400] That is, the file control unit 43 generates the track #2i of the UUID #i for the MP4 file and stores it in the mdat box, and generates the trak box for managing the track #2i and stores it in the moov box.

[0401] The trak box for managing track #2i has a tkhd box and a tref box. In the tkhd box of the trak box for managing track #2i, a track ID 2i for specifying the track #2i to be managed is set.

[0402] Furthermore, in the tref box of the trak box managing track #2i, it is set that another track associated with track #2i is track #2i-1 (track_ID=2i-1), and track #2i is a metadata (here, designation information) track (type=cdsc).

[0403] As described above, by storing the pre-generated uuid as designation information for associating four main images of track #1 in a type sequence file into an MP4 file storing audio segments #1 to #4, the four main images can be associated with audio segments #1 to #4.

[0404] Fig.46 is a view illustrating a fourth example of association between a main image as internal data and audio as external data in a case where an MP4 file is adopted as a file storing audio as external data.

[0405] Fig.46 The diagram illustrates an example of association between one main image stored in a first association type set file and four audio tracks stored in an MP4 file.

[0406] The first association type set file is generated, for example, by a pre-storage method and is associated with Fig.43 The situation is configured similarly.

[0407] An MP4 file storing four audios #1 to #4 associated with a main image in an association type sequence file Fig.44 The situation is configured similarly.

[0408] In the case where there is a request for association processing to associate main image Item#1 in the first association type collection file with audio segments #1 to #4 in the MP4 file after the first association type collection file is generated, in the association processing, the file control unit 43 assigns UUID#1 to the audio segments #1 to #4 to be associated with the main image Item#1, which UUID#1 is pre-generated for the main image Item#1 in the first association type collection file and is used as designation information of external data to be associated with the main image Item#1, and the file control unit 43 stores the UUID#1 assigned to the audio segments #1 to #4 in the MP4 file storing the audio segments #1 to #4.

[0409] That is, for MP4 files, the file control unit 43 generates tracks #2, #4, #6 and #8 of UUID #1 assigned to audio segments #1 to #4 respectively and stores them in the mdat box, and generates trak boxes that manage tracks #2, #4, #6 and #8 respectively and stores them in the moov box.

[0410] The trak box for managing track #2i (i=1, 2, 3, 4) has a tkhd box and a tref box. In the tkhd box of the trak box for managing track #2i, a track ID 2i for specifying the track #2i to be managed is set.

[0411] Furthermore, in the tref box of the trak box managing track #2i, it is set that another track associated with track #2i is track #2i-1 (track_ID=2i-1), and track #2i is a track (type=cdsc) of metadata (here, designation information).

[0412] As described above, by storing UUID#1 as designation information generated in advance for main image Item#1 in the MP4 file storing audio clips #1 to #4, one main image Item#1 can be associated with four clips as a plurality of audio clips #1 to #4.

[0413] exist Fig.46 , although audio clips #1 to #4 are audio clips with the same content, audio #1 is Japanese audio, audio #2 is English audio, audio #3 is French audio, and audio #4 is Chinese audio. Fig.46 By association processing, it is possible to associate the main image Item#1 with four audios #1 to #4 having the same content but different languages.

[0414] Fig.47is a view illustrating a fifth example of association between a main image as internal data and audio as external data in a case where an MP4 file is adopted as a file storing audio as external data.

[0415] Fig.47 An example of association between one main image stored in a first association type set file and four pieces of audio stored in an MP4 file is illustrated.

[0416] The first association type set file is generated, for example, by a pre-storage method and is associated with Fig.43 The situation is configured similarly.

[0417] The MP4 files storing the four audios #1 to #4 associated with one main image in the association type sequence file are Fig.44 However, in Fig.47 In the MP4 file, track #1 is set as the main track in the trak box of track #1 that manages audio #1, and tracks #3, #5 and #7 are set as secondary tracks in the trak box of tracks #3, #5 and #7 that manage audio clips #2 to #4, whose main track is track #1.

[0418] In the case where there is a request for association processing to associate main image Item#1 in the first association type collection file with audio segments #1 to #4 in the MP4 file after the first association type collection file is generated, in the association processing, the file control unit 43 assigns UUID#1 to main audio #1 among the audio segments #1 to #4 to be associated with the main image Item#1, which UUID#1 is pre-generated for the main image Item#1 in the first association type collection file and is used as designation information of external data to be associated with the main image Item#1, and the file control unit 43 stores UUID#1 assigned to audio #1 in the MP4 file storing audio segments #1 to #4.

[0419] That is, for the MP4 file, the file control unit 43 generates track #2 assigned to UUID #1 of primary audio #1 and stores in the mdat box, and generates a trak box that manages track #2 and stores in the moov box.

[0420] The trak box for managing track #2 has a tkhd box and a tref box. In the tkhd box of the trak box for managing track #2, the track ID 2 for specifying the track #2 to be managed is set.

[0421] Furthermore, in the tref box of the trak box managing track #2, it is set that another track associated with track #2 is track #1 (track_ID=1), and track #2 is a track (type=cdsc) of metadata (specification information here).

[0422] As described above, by storing UUID#1 as designation information generated in advance for main image Item#1 in the MP4 file storing audio clips #1 to #4, one main image Item#1 can be associated with four clips as a plurality of audio clips #1 to #4.

[0423] Here, in Fig.47 In the example, since UUID#1 is not assigned to the secondary audio #2 to #4 stored in the MP4 file, each of the main image Item#1 and the secondary audio clips #2 to #4 is not directly associated with each other.

[0424] However, in the MP4 file, as described above, the audio clips #2 to #4 are set as the secondary audio of the audio #1 of track #1 as the primary audio. In addition, according to the setting of the MP4 file, each of the main image Item #1 and the secondary audio #2 to #4 is indirectly associated with each other through the association between the main image Item #1 and the primary audio #1.

[0425] Fig.48 is a view illustrating a sixth example of association between a main image as internal data and audio as external data in a case where an MP4 file is adopted as a file storing audio as external data.

[0426] Fig.48 An example of association between one main image stored in a first association type set file and four pieces of audio stored in an MP4 file is illustrated.

[0427] The first association type set file is generated, for example, by a pre-storage method and is associated with Fig.43 However, in Fig.48 In the first association type collection file, in addition to the association information in which the main image Item#1 (item ID) is associated with UUID#1, the association information in which the main image Item#1 is associated with UUID#2 which is different from UUID#1 is also stored in the association information storage box, and the number of association information is not 1 but 2.

[0428] The MP4 files storing the four audios #1 to #4 associated with the two main images in the association type sequence file are Fig.47 The situation is configured similarly.

[0429] However, in Fig.47 In the MP4 file of , in the trak box of tracks #1 and #3 that manage the corresponding audio clips #1 and #2 separately, tracks #1 and #3 are set as primary tracks. In addition, in the trak box of track #5 that manages audio #3, it is set that track #5 is a secondary track, which is primary track #1, and in the trak box of track #7 that manages audio #4, it is set that track #7 is a secondary track, which is primary track track #3.

[0430] In the case where there is a request for association processing to associate main image Item#1 in the first association type collection file with audio segments #1 and #2 in the MP4 file and to associate main image Item#1 with audio segments #3 and #4 in the MP4 file after the first association type collection file is generated, in the association processing, the file control unit 43 assigns UUID#1 to main audio #1 of audio segments #1 and #2 to be associated with main image Item#1, which UUID#1 is pre-generated for main image Item#1 in the first association type collection file and is used as designation information of external data to be associated with main image Item#1, and the file control unit 43 stores UUID#1 assigned to audio #1 in the MP4 file storing audio segments #1 and #2.

[0431] Moreover, the file control unit 43 assigns UUID#2 to the main audio #3 of the audio clips #3 and #4 to be associated with the main image Item#2, which UUID#2 is pre-generated for the main image Item#1 in the first association type collection file and is used as designation information of the external data to be associated with the main image Item#1, and the file control unit 43 stores the UUID#2 assigned to the audio #3 into the MP4 file storing the audio clips #3 and #4.

[0432] That is, for the MP4 file, the file control unit 43 generates track #2 assigned to UUID #1 of primary audio #1 and stores in the mdat box, and generates a trak box that manages track #2 and stores in the moov box.

[0433] The trak box for managing track #2 has a tkhd box and a tref box. In the tkhd box of the trak box for managing track #2, the track ID 2 for specifying the track #2 to be managed is set.

[0434] Furthermore, in the tref box of the trak box managing track #2, it is set that another track associated with track #2 is track #1 (track_ID=1), and track #2 is a track (type=cdsc) of metadata (specification information here).

[0435] For the MP4 file, the file control unit 43 generates track #4 assigned to UUID #2 of primary audio #3 and stores in the mdat box, and generates a trak box for managing track #4 and stores in the moov box.

[0436] The trak box for managing track #4 has a tkhd box and a tref box. In the tkhd box of the trak box for managing track #4, the track ID 4 for specifying the track #4 to be managed is set.

[0437] Furthermore, in the tref box of the trak box managing track #4, it is set that another track associated with track #4 is track #3 (track_ID=3), and track #4 is a track of metadata (type=cdsc).

[0438] As described above, by storing UUID #1 and #2 as multiple different designated information pre-generated for the main image Item #1 in the MP4 file storing audio segments #1 to #4, it is possible to associate two audio segments #1 and #2 and two audio segments #3 and #4 with the main image Item #1, that is, to associate the main image Item #1 with the four audio segments #1 to #4.

[0439] Here, in Fig.48 In Fig.47 , UUID#1 is not assigned to secondary audio #2, and primary image Item#1 and secondary audio #2 are not directly associated with each other. However, by setting an MP4 file indicating that audio #2 is a secondary audio with audio #1 as primary audio, primary image Item#1 and secondary audio #2 are indirectly associated via the association between primary image Item#1 and primary audio #1. Similarly, UUID#2 is not assigned to secondary audio #4, and primary image Item#1 and secondary audio #4 are not directly associated with each other. However, by setting an MP4 file indicating that audio #4 is a secondary audio with audio #3 as primary audio, primary image Item#1 and secondary audio #4 are indirectly associated via the association between primary image Item#3 and primary audio #1.

[0440] <Association type HEIF file generated by the region assurance method and association type HEIF file after association processing>

[0441] Fig.49 is a view for explaining a first example of a first association type set file generated by an area securing method and a state of the first association type set file after an association process.

[0442] Note that, hereinafter, description of storing media data such as images and tracks of the media data into the mdat box in generation of an associated type HEIF file will be appropriately omitted.

[0443] When generating a first association type collection file by the area ensuring method, for example, the file control unit 43 generates an association information storage box having the number of association information set to 0 and having a blank area (filling area) as a reserved area, and generates a first association type collection file in which the association information storage box is stored in a meta box.

[0444] Whether or not valid associated information is stored (set) in the associated information storage box can be identified by the associated information quantity.

[0445] Furthermore, the association information storage box may be provided with a flag indicating whether valid association information is stored in the association information storage box, and the flag allows identification of whether valid association information is stored in the association information storage box.

[0446] In the following, it is assumed that whether or not valid association information is stored in the association information storage box is identified based on the amount of association information, for example.

[0447] In the case where there is a request for association processing of associating main image Item#1 in the first association type collection file with external data after the first association type collection file is generated, in the association processing, the file control unit 43 generates UUID#1 as designation information of the external data to be associated with main image Item#1, and generates association information in which UUID#1 is associated with item ID#1 of main image Item#1.

[0448] Furthermore, the file control unit 43 overwrites a blank area as a reserved area in the associated information storage box with the associated information, and updates the associated information quantity to a value obtained by incrementing the current value by 1 (here, 1).

[0449] exist Fig.49 In the first association type set file, writing the association information into the association information storage box may be performed within the capacity range of the blank area in the association information storage box.

[0450] As described above, in the association processing performed after the first association type set file is generated, by generating a first association type set file in which an association information storage box having a reserved area is stored in a meta box, and writing (overwriting) the association information in which UUID#1, which is designated information of external data, is associated with item ID#1 of the main image Item#1 into the reserved area in the association information storage box, it is possible to later associate internal data with external data while preventing the offset of the data already stored in the mdat box from being shifted.

[0451] Fig.50 is a view for explaining a second example of a first association type set file generated by the area securing method and a state of the first association type set file after the association process.

[0452] When generating the first association type collection file by the area ensuring method, the file control unit 43 generates, for example, a free box having a blank area (filling area) as a reserved area, and generates the first association type collection file in which the free box is stored in the meta box. The free box is a box that can store any data, and in the present embodiment, the free box has a blank area as a reserved area.

[0453] In the case where there is a request for association processing of associating main image Item#1 in the first association type collection file with external data after the first association type collection file is generated, in the association processing, the file control unit 43 generates UUID#1 as designation information of the external data to be associated with main image Item#1, and generates association information in which UUID#1 is associated with item ID#1 of main image Item#1.

[0454] Furthermore, the file control unit 43 generates an associating information number in which the number of generated pieces of associating information is set, and generates an associating information storage box storing the associating information number and the associating information.

[0455] Then, the file control unit 43 overwrites the associated information storage box into the meta box by using the reserved area of ​​the free box, and reduces the capacity (size) of the free box by the capacity of the associated information storage box.

[0456] exist Fig.50 In the first association type set file, the writing of the association information storage box can be performed within the capacity range of the free box when the first association type set file is generated.

[0457] As described above, in the association processing performed after the first association type set file is generated, by generating a first association type set file in which a free box with a reserved area is stored in a meta box, and writing an association information storage box storing association information into the reserved area using the reserved area of ​​the free box, in which UUID#1 as designated information of external data is associated with item ID#1 of the main image Item#1, it is possible to later associate internal data with external data while preventing the offset of the data already stored in the mdat box from being shifted.

[0458] Fig.51 is a view for explaining a third example of a first association type set file generated by the area securing method and a state of the first association type set file after the association process.

[0459] exist Fig.51 In addition to generating a first association type set file (in which the free boxes are not stored in the meta box, but in the file hierarchy between the meta box and the mdat box), the same operations as Fig.50 The situation is handled similarly.

[0460] Also in Fig.51 In the case of Fig.50 Similar to the case of , it is possible to later associate internal data with external data while preventing the offset of data already stored in the mdat box from being shifted.

[0461] Fig.52 is a view for explaining a first example of a second association type set file generated by an area securing method and a state of the second association type set file after an association process.

[0462] When generating a first association type collection file by the area ensuring method, the file control unit 43 generates a second association type collection file having the number of association information set to 0, for example, and in which an area for association information with a blank area as a reserved area is stored as an item in the mdat box.

[0463] Whether the area used as the associated information of the item has valid associated information can be identified by the amount of associated information. In addition, in the area used as the associated information of the item, a flag indicating whether the area used as the associated information of the item has valid associated information can be provided, and the flag enables identification of whether the area used as the associated information of the item has valid associated information.

[0464] Hereinafter, it is assumed that whether or not an area for association information as an item has valid association information is identified by, for example, the amount of association information.

[0465] In the case where there is a request for association processing of associating main image Item#1 in the second association type collection file with external data after the second association type collection file is generated, in the association processing, the file control unit 43 generates UUID#1 as designated information of the external data to be associated with the main image Item#1, and generates association information in which UUID#1 is associated with item ID#1 of the main image Item#1.

[0466] Furthermore, the file control unit 43 overwrites the blank area of ​​the reserved area as the area for the associated information as the item with the associated information, and updates the associated information quantity to a value obtained by incrementing the current value by 1 (here, 1).

[0467] exist Fig.52 In the second association type set file, the association information can be written into the area for the association information within the capacity range of the blank area in the area for the association information.

[0468] As described above, in the association processing performed after the second association type set file is generated, by generating the second association type set file (in which an area for association information having a reserved area is stored as an item in the meta box, and association information in which UUID#1, which is designated information of external data, and item ID#1 of the main image Item#1 are associated with each other is written in the reserved area in the area for association information), it is possible to later associate internal data with external data while preventing the data already stored in the mdat box from being offset.

[0469] Fig.53 is a view for explaining a second example of a second association type set file generated by the area securing method and a state of the second association type set file after the association process.

[0470] When generating the second association type set file by the area ensuring method, for example, the file control unit 43 generates a free box, and generates the second association type set file, wherein the free box is stored in the file hierarchy between the meta box and the mdat box.

[0471] Note that since the association information is not stored in the mdat box immediately after it is generated, Fig.53 The collection file generated in does not have the form of a second association type collection file in which association information is stored as an item in the mdat box. However, the collection file will have the form of a second association type collection file when association processing is performed in the future, so it is called a second association type collection file for convenience.

[0472] In the case where there is a request for association processing of associating main image Item#1 in the second association type collection file with external data after the second association type collection file is generated, in the association processing, the file control unit 43 generates UUID#1 as designated information of the external data to be associated with the main image Item#1, and generates association information in which UUID#1 is associated with item ID#1 of the main image Item#1.

[0473] Furthermore, the file control unit 43 generates the number of associated information in which the number of pieces of generated associated information is set, and stores the number of associated information and the associated information as one item in a form added after the last item of the mdat box.

[0474] Furthermore, the file control unit 43 adds metadata about the number of associated information and the associated information (metadata to be stored in an iinf box, an iloc box, or the like) as items added to the mdat box to the meta box.

[0475] Adding metadata to the meta box increases the capacity (data amount) of the meta box, but the reserved area of ​​the free box is used to write data corresponding to the increase in capacity. Therefore, it can be considered to write metadata about the number of associated information and the associated information as items added to the mdat box to the reserved area of ​​the free box.

[0476] In the above-mentioned relationship information related to the association between the main image Item#1 and the external data, that is, in the association information in which UUID#1 is associated with the item ID#1 of the main image Item#1, the amount of associated information and the metadata added to the meta box, the file control unit 43 reduces the capacity of the free box by the amount of data of the metadata added to the meta box.

[0477] Here, in Fig.53 In the example, since the number of associated information and the associated information as items are stored (written) in a form added after the last item of the mdat box, the reserved area of ​​the free box is not consumed. Therefore, the capacity of the free box is not changed (reduced) by the number of associated information and the associated information as items added after the last item of the mdat box.

[0478] exist Fig.53 In the second association type collection file, writing of the relationship information may be performed within a range where the capacity of metadata of the meta box to be added to the relationship information does not exceed the capacity of the free box when generating the second association type collection file.

[0479] As described above, in the association processing performed after the second association type set file is generated, by generating a second association type set file storing a free box having a reserved area, storing the association information in which UUID#1 used as designated information for external data is associated with item ID#1 of main image Item#1 as the last item of the mdat box, and adding metadata related to the association information to the meta box by using the reserved area of ​​the free box, it becomes possible to later associate internal data with external data while preventing the offset of the data already stored in the mdat box from being shifted.

[0480] Note that in Fig.53 In , free boxes can be provided not in the file hierarchy but in the meta box.

[0481] Fig.54is a view for explaining a first example of a third association type set file generated by the area securing method and a state of the third association type set file after the association process.

[0482] For example, when generating the third association type set file by the area ensuring method, the file control unit 43 generates the third association type set file in which each of one or more pieces of information serving as the designation information is stored as an item in the mdat box.

[0483] The area (specified information area) used as the designated information stored in the mdat box is a reserved area for storing (future) designated information, and can be filled with invalid values, for example. For example, when using uuid as the designated information, all zeros can be used as invalid values.

[0484] When generating the third association type set file, for example, the file control unit 43 generates an invalid value as the uuid and writes the invalid value as an entry in the mdat box, thereby ensuring a reserved area for storing the designated information.

[0485] Note that, when the third association type collection file is generated, metadata about information (invalid value) serving as designation information of an item is also generated and stored in the meta box.

[0486] Examples of metadata related to the information used as the designated information as the item include: item ID of the information used as the designated information; offset; size; associated information associating the main image as the internal data with the designated information of the external data, that is, associated information in which the item ID of the main image and the item ID of the information used as the designated information as the item are associated as a reference source and a reference destination, respectively (stored in Fig.12 ); etc.

[0487] Furthermore, information serving as designation information as items may be stored in the mdat box by a number equal to or smaller than the number of main images (the number of items) as internal data of the third association type set file, for example.

[0488] In the case where there is a request for association processing to associate main image Item#1 in the third association type collection file with external data after the third association type collection file is generated, in the association processing, the file control unit 43 generates UUID#1 as valid designation information of the external data to be associated with main image Item#1, and writes (overwrites an invalid value) UUID#1 into (an area of) information serving as designation information of an item to be associated with item ID#1 of main image Item#1 through metadata in a meta box.

[0489] exist Fig.54In the third association type collection file, writing of valid designation information can be performed within the range of the number of pieces of designation information used as stored in the mdat box when the third association type collection file is generated.

[0490] As described above, in the association processing performed after the third association type set file is generated, by generating a third association type set file in which a reserved area for storing designated information is ensured in the mdat box (i.e., a third association type set file in which information used as designated information is stored in the mdat box), and writing UUID#1 as the designated information to be assigned to external data in the information used as the designated information associated with item ID#1 of the main image Item#1, it is possible to later associate internal data with external data while preventing the offset of the data already stored in the mdat box from being shifted.

[0491] Note that when generating the third association type set file, for example, valid designation information (designation information of a valid value) such as uuid may be stored in the mdat box instead of information used as designation information (invalid value).

[0492] The generation of the third association type set file in which the effective designation information is stored in the mdat box is the generation of the third association type set file that stores the designation information before being assigned to the external data, and thus is the generation of the third association type set file by the pre-storage method.

[0493] In this case, for the designation information, for example, by providing a flag indicating whether the designation information is assigned to the external data, the flag enables identification of whether the designation information is assigned or not assigned.

[0494] Fig.55 is a view for explaining a second example of a third association type set file generated by the area securing method and a state of the third association type set file after the association process.

[0495] When generating the third association type set file by the area ensuring method, for example, the file control unit 43 generates a free box, and generates the third association type set file in which the free box is stored in the file hierarchy between the meta box and the mdat box.

[0496] Note that since the specified information as an item is not stored in the mdat box immediately after generation, and therefore metadata about the specified information as an item is not stored in the meta box, Fig.55The collection file generated in does not have the form of a third association type collection file in which the specified information is stored as an item in the mdat box. However, this collection file will have the form of a third association type collection file when the association process is performed in the future, so for convenience, it is called a third association type collection file.

[0497] In the case where there is a request for association processing of associating main image Item#1 in the third association type collection file with external data after the third association type collection file is generated, in the association processing, the file control unit 43 generates UUID#1 as designation information of the external data to be associated with the main image Item#1, and stores UUID#1 as an item in the form of being added after the last item of the mdat box.

[0498] Furthermore, the file control unit 43 adds metadata (metadata to be stored in an iinf box, an iref box, an iloc box, or the like) regarding UUID#1 as an item added to the mdat box to the meta box.

[0499] Adding metadata to the meta box increases the capacity of the meta box, but the reserved area of ​​the free box is used to write data corresponding to the increase in capacity. Therefore, it can be considered to write metadata about the specified information as an item added to the mdat box to the reserved area of ​​the free box.

[0500] In the relationship information related to the association between the main image Item#1 and the external data, that is, among UUID#1 as the item and the metadata associated with UUID#1, the file control unit 43 reduces the capacity of the free box by the data amount of the metadata associated with UUID#1.

[0501] Here, in Fig.55 In, with Fig.53 Similar to the case of , since UUID#1 as an item is stored in the form of being added after the last item of the mdat box, the reserved area of ​​the free box is not consumed. Therefore, UUID#1 as an item added after the last item of the mdat box does not change the capacity of the free box.

[0502] exist Fig.55 In the third association type collection file in the embodiment, when the third association type collection file is generated, writing of the relationship information can be performed within a range in which the capacity of metadata regarding the specified information as the item in the relationship information does not exceed the capacity of the free box.

[0503] As described above, in the association processing performed after the third association type set file is generated, by generating the third association type set file storing the free box, storing UUID#1 of the designated information used as the external data as the last item of the mdat box, and adding metadata related to the designated information to the meta box by using the reserved area of ​​the free box, it is possible to associate the internal data with the external data later while preventing the offset of the data already stored in the mdat box from being shifted.

[0504] Note that in Fig.53 In , free boxes can be provided not in the file hierarchy but in the meta box.

[0505] Fig.56 It is a view for explaining a first example of an association type sequence file generated by the area securing method and a state of the association type sequence file after an association process.

[0506] When generating an association type sequence file by the area ensuring method, the file control unit 43 generates an association type sequence file in which, for example, a track of information serving as designation information is stored in an mdat box.

[0507] The area of ​​the track of information used as designated information stored in the mdat box (area for designated information) is a reserved area where the designated information is stored (in the future), and can be filled with invalid values, for example. For example, in the case of using uuid as designated information, all zeros can be used as invalid values.

[0508] When generating an association type set file, an invalid value is generated as a uuid, for example, by the file control unit 43, and a track with an invalid value is written into an mdat box, thereby ensuring a reserved area for storing designated information.

[0509] Note that, when the association type sequence is generated, metadata for managing the track of information (invalid value) serving as the designation information is also generated and stored in (the trak box of) the meta box.

[0510] Examples of metadata for managing the track of information used as the designation information include: track ID of the track of information used as the designation information; offset; size; and information used as the designation information (to be stored in the management Fig.14 The track ID of another track (here, the track of the main image) associated with the track of the designated information described in the trak box of track #3; and so on.

[0511] Furthermore, the track of information serving as specifying information may be constituted by, for example, information of specifying information of a number equal to the number (number of frames) of main images serving as internal data of an association type sequence file.

[0512] In the case where there is a request for association processing to associate main image Item#1 in the association type sequence file with external data after an association type sequence file is generated, in the association processing, the file control unit 43 generates UUID#1 as valid designation information of external data to be associated with main image Item#1, and writes UUID#1 (overwrites an invalid value) to (an area of) information used as designation information to be associated with main image Item#1 through metadata in a meta box and time information on a timeline in a track of information used as designation information.

[0513] exist Fig.56 In an associated type sequence file, writing of valid designation information can be performed within the range of the number of pieces of information constituting a track of information used as designation information stored in an mdat box when generating the associated type sequence file.

[0514] As described above, in the association processing performed after the association type sequence file is generated, by generating an association type sequence file in which a reserved area for storing designated information is ensured in the mdat box (i.e., an association type sequence file of a track in which information serving as designated information is stored in the mdat box), and writing UUID#1 serving as designated information to be assigned to external data into the information serving as designated information associated with the main image Item#1, it is possible to later associate internal data with external data while preventing the offset of the data already stored in the mdat box from being shifted.

[0515] Note that, when generating an association type sequence file, the mdat box may store a track of valid designation information such as uuid, for example, instead of information (invalid value) used as designation information.

[0516] The generation of an association type sequence file in which tracks of effective designation information are stored in an mdat box is the generation of an association type sequence file that stores designation information before being assigned to external data, and thus is the generation of an association type sequence file by a pre-storage method.

[0517] In this case, for the designation information, for example, by providing a flag indicating whether the designation information is assigned to the external data, the flag enables identification of whether the designation information is assigned or not assigned.

[0518] Fig.57 It is a view for explaining a second example of an association type sequence file generated by the area securing method and a state of the association type sequence file after the association process.

[0519] When generating an association type sequence file by the area securing method, for example, the file control unit 43 generates a free box, and generates an association type sequence file in which the free box is stored in a file hierarchy structure between a meta box and an mdat box.

[0520] Note that since the track of the designated information is not stored in the mdat box immediately after generation, and therefore metadata for managing the track of the designated information is not stored in the moov box, Fig.57 The sequence file generated in does not have the form of an association type sequence file in which the track of the designated information is stored in the mdat box. However, the sequence file will have the form of an association type collection file when the association process is performed in the future, so for convenience, it is called an association type sequence file.

[0521] In the case where there is a request for an association process of associating the main image Item#1 in the association type sequence file with external data after the association type sequence file is generated, in the association process, the file control unit 43 stores a track (hereinafter also referred to as an additional information track) including one or more pieces of information serving as designated information in the form of adding a track after the last track of the mdat box. As the information serving as designated information constituting the additional information track, for example, Fig.56 The situation is similar for invalid values.

[0522] Also, the file control unit 43 generates a trak box to store metadata and manage an additional information track added to the mdat box, and adds the trak box to the meta box.

[0523] When generating a trak box for managing an additional information track, a trak box storing the track ID of the additional information track is generated (management reference Fig.14 The trak box of track #3 of the designated information described in the specification), the offset, size, track ID, etc. of another track (here, the track of the main image) associated with the additional information track.

[0524] The file control unit 43 generates UUID#1 as valid designation information of external data to be associated with the main image Item#1, and overwrites (overwrites an invalid value with) UUID#1 in the additional information track to serve as designation information associated with the main image Item#1 through metadata in the meta box and time information on the timeline.

[0525] Adding a trak box (storing metadata) for managing an additional information track to a meta box increases the capacity of the meta box, but uses a reserved area of ​​a free box to write data corresponding to the capacity increase. Therefore, it may be considered to write a trak box for managing an additional information track to a reserved area of ​​a free box.

[0526] In the above-mentioned relationship information related to the association between the main image Item#1 and the external data (i.e., between the additional information track and the track box managing the additional information track), the file control unit 43 reduces the capacity of the free box by the data amount of the trak box managing the additional information track.

[0527] Here, in Fig.57 In the example, since the additional information track is stored in the form of being added after the last track of the mdat box, the reserved area of ​​the free box is not consumed.

[0528] exist Fig.57 In the associated type sequence file, the writing of the relationship information can be performed within the range where the data amount of the trak box of the additional information track in the management relationship information does not exceed the capacity of the free box when the associated type sequence file is generated.

[0529] As described above, in the association processing performed after the association type sequence file is generated, by generating an association type sequence file storing a free box, storing the additional information track as the last track of the mdat box and adding a trak box that manages the additional information track to the meta box by using the reserved area of ​​the free box, and overwriting the information used as the designation information constituting the additional information track with UUID#1 which is the designation information of the external data, it is possible to associate internal data with external data later while preventing the offset of the data already stored in the mdat box from being shifted.

[0530] Note that in Fig.57 In , free boxes can be provided not in the file hierarchy but in the meta box.

[0531] In addition, as an additional information track, it is possible to generate a track including a number of pieces of information equal to the number of main images to be associated with external data when generating the additional information track, or to generate a track including a number of pieces of information equal to the number of tracks constituting the main image when generating the additional information track for the first time.

[0532] However, in the case where a track including a number of pieces of information equal to the number of main images to be associated with external data is generated as an additional information track when generating an additional information track, the additional information track needs to be added each time the association processing is performed, which complicates the management of the additional information track, the association between the specified information written in the additional information track and the main image, and the like.

[0533] When a track including a number of pieces of information equal to the number of tracks constituting the main image is generated as an additional information track when the additional information track is generated for the first time, in the subsequent association processing, it is only necessary to overwrite the information used as the designated information associated with the main image with valid designated information of external data in the additional information track.

[0534] Furthermore, when generating an association type sequence file, for example, the mdat box may store an additional information track including valid designation information such as uuid, instead of information (invalid value) serving as the designation information.

[0535] The generation of an association type sequence file storing an additional information track including valid designation information in an mdat box is the generation of an association type sequence file storing designation information before being assigned to external data, and thus is the generation of an association type sequence file by a pre-storage method.

[0536] In this case, for the designation information, for example, by providing a flag indicating whether the designation information is assigned to the external data, the flag enables identification of whether the designation information is assigned or not assigned.

[0537] Fig.58 is a flowchart for explaining an example of a process of generating an association type HEIF file by a region ensuring method.

[0538] In step S211 , the file control unit 43 determines a required capacity (hereinafter, also referred to as required capacity) as the capacity of the reserve area, and the process proceeds to step S212 .

[0539] Here, in the case of generating a box with a reserved area (such as a free box), at least 8 bytes are required in the box for storing the size and type. Therefore, the size of the box is equal to or greater than the value obtained by adding 8 bytes to the required capacity of the reserved area.

[0540] For example, in the case of generating a HEIF file storing a free box, Fig.53 When the associated information or the specified information is added as an item to the mdat box in the associated processing shown in 55, it is necessary to add metadata about the associated information or the specified information as an item to the meta box according to the addition of the associated information or the specified information as an item. In this case, the required capacity is determined to be a value equal to or larger than the data amount of the metadata to be added to the meta box.

[0541] Furthermore, for example, in the case of generating a HEIF file storing a free box, Fig.57When the additional information track is added to the mdat box in the associated process shown, it is necessary to add metadata for managing the additional information track to the moov box according to the addition of the additional information track. In this case, the required capacity is determined to be equal to or greater than the value of the data amount of the metadata to be added to the moov box.

[0542] In addition, the required capacity can be determined based on one or more of the following: the data volume of a specified piece of information; the number of main images stored in the HEIF file (the number of main images that can be associated with external data); the number of external data that can be associated with the main image; and so on.

[0543] The required capacity increases as the amount of data of the specified information increases. For example, the number of primary images that can be associated with external data can be determined within the range of the number of primary images in the HEIF file. The required capacity increases as the number of primary images that can be associated with external data increases. The number of pieces of external data that can be associated with a primary image can be determined to be any number.

[0544] For example, in the case where information (such as uuid) that enables the same value to be assigned to multiple pieces of external data is adopted as the designation information, an increase or decrease in the number of pieces of external data that can be associated with one main image does not affect the required capacity. For example, in the case where designation information that is different for each piece of external data (such as a hash value or file name with external data as input) is adopted as the designation information, the required capacity increases as the number of external data that can be associated with the main image increases.

[0545] For example, the required capacity may be determined according to the product of the data amount of one piece of specified information and the number of main images that can be associated with the external data.

[0546] The data volume of a specified information can be determined by the type of the specified information, for example, whether the specified information is a URL, a uuid, a hash value, etc.

[0547] For example, according to the remaining capacity of the medium storing the HEIF file, as the remaining capacity increases, the number of main images associated with the external data can be determined to be a larger value. In addition, for example, by determining the maximum number of main images that can be stored in one HEIF file, according to the maximum number, as the maximum number is larger, the number of main images with which the external data can be associated can be determined to be a larger value. Moreover, for example, by evaluating the main images stored in the HEIF file through the file control unit 43, according to the number of main images with good evaluations, it can be determined that the number of main images that can be associated with the external data is a larger value as the number of main images with good evaluations increases. In the evaluation of the main image, for example, by obtaining information about the image quality of the main image (such as S / N and clarity), and a main image with S / N, clarity, etc. equal to or greater than a threshold value can be determined as a main image with a good evaluation. In addition, the number of main images that can be associated with the external data can be determined based on multiple factors (such as the remaining capacity of the above-mentioned medium).

[0548] In step S212, the file control unit 43 generates a reference Figures 49 to 57 The associated type HEIF file described, wherein a reserved area with the required capacity is ensured, and the processing ends.

[0549] Fig.59 : is a flowchart for explaining an example of association processing of an association type HEIF file generated by the area ensuring method.

[0550] In step S221 , the file control section 43 acquires designation information of external data, and the process proceeds to step S222 .

[0551] For example, in the case where the external data is data stored in a single-track MP4 file or WAV file, uuid, a hash value of the MP4 file or WAV file storing the external data (or a hash value of the external data itself), a URL, etc. may be used as designation information.

[0552] Furthermore, for example, in the case where the external data is a multi-track MP4 file, it is possible to adopt a set of URLs of the MP4 file storing the external data and the track ID of the track of the external data, a set of URLs of the MP4 file storing the external data and time information on the timeline of the external data, a set of URLs of the MP4 file storing the external data and a hash value of the track of the external data, etc. as the designation information. The uuid, hash value, and URL are acquired by generation. The track ID and time information are acquired by referring to the MP4 file storing the external data.

[0553] In step S222, by using a reserved area secured in advance, the file control unit 43 writes relationship information including designated information and related to the association between the main image and the external data (the designated information) into an association type HEIF file generated by the area securing method, thereby generating an association type HEIF file storing the relationship information, and the processing proceeds to step S223.

[0554] In step S223, the file control unit 43 determines whether it is necessary to write (store) the designated information in (the file of) the external data.

[0555] In the case where it is determined in step S223 that the designated information does not need to be written to the external data, for example, in the case where the designated information is information that can designate the external data without writing the external data (such as a hash value generated using the external data or the like as an input), the process skips step S224 and ends. That is, in the case where it is not necessary to write the designated information to the external data, the association between the main image and the external data is completed by writing the relationship information to the association type HEIF file.

[0556] Furthermore, in the case where it is determined in step S233 that it is necessary to write the designated information into the external data, for example, in the case where the designated information is information that can designate the external data by writing the external data (such as uuid), the processing proceeds to step S224.

[0557] In step S224, the file control unit 43 writes (stores) the designated information acquired in step S221 into the file storing the external data, and the processing ends. That is, in the case where it is necessary to write the designated information into the external data, the association between the main image and the external data is completed by writing the relationship information into the association type HEIF file and writing the designated information into the file storing the external data.

[0558] Note that the file storing the external data to be associated with the main image stored in the association type HEIF file generated by the area ensuring method may be generated before the association process is performed, or may be generated in parallel with the execution of the association process. In the case where the file storing the external data is generated in parallel with the execution of the association process, for example, in step S224, a file storing the specified information together with the external data is generated.

[0559] Fig.60 is a flowchart for explaining an example of an association process of a first association type set file generated by the area securing method.

[0560] For example, in Fig.60 In the assumption that Fig.51The first association type set file shown in , which provides a free box in a meta box, is set as a target of association processing, and uses designated information that does not require writing of external data.

[0561] In step S231 , the file control unit 43 acquires designation information of external data, and the process proceeds to step S232 .

[0562] For example, the file control unit 43 generates a hash value by performing calculation with SHA-256 by using a file storing external data as an input, and acquires the hash value as the designation information of the external data.

[0563] In step S232, the file control unit 43 generates the Fig.51 The meta box is obtained by reading the first association type set file, and the processing proceeds to step S233.

[0564] In step S233, the file control unit 43 obtains Fig.51 and confirming that the remaining capacity is sufficient to add the relationship information related to the association between the main image and the external data (ie, Fig.51 In the example, it is sufficient to add an associated information storage box storing the associated information quantity and the associated information), and the processing proceeds to step S234.

[0565] Note that in step S233, in the event that the remaining capacity of the reserve area is insufficient to add relationship information related to the association between the main image and the external data, this fact is displayed on the liquid crystal panel 19, for example, and the processing ends.

[0566] In step S234, the file control unit 43 uses the external data designation information and the meta box information to generate Fig.51 The associated information storage box storing the number of associated information and the associated information in the associated information storage box is further generated, and a new meta box storing the associated information storage box is further generated.

[0567] Furthermore, the file control unit 43 rewrites the meta box of the first association type set file with the new meta box, and the process proceeds from step S234 to step S235.

[0568] In step S235, the file control unit 43 generates a new free box with respect to the meta box before rewriting, and the size of the new free box decreases as the size (data amount) of the new meta box increases.

[0569] Furthermore, the file control unit 43 overwrites the free box of the first association type set file with the new free box, and the process ends.

[0570] Fig.61 is a flowchart for explaining another example of the association process of the first association type set file generated by the area securing method.

[0571] For example, in Fig.61 In, with Fig.60 Similarly, assuming that Fig.51 In the example shown in FIG. 1 , a first association type set file providing a free box in a meta box is set as a target of association processing. Furthermore, in Fig.61 In the example, it is assumed that the specified information that needs to be written to the external data is used.

[0572] In step S241, the file control unit 43 acquires the designation information of the external data, and the process proceeds to step S242. For example, the file control unit 43 generates uuid, and acquires uuid as the designation information of the external data.

[0573] In step S242 , the file control unit 43 writes (stores) uuid as the specification information acquired in step S221 into the file storing the external data, and the process proceeds to step S243 .

[0574] For example, when the file storing the external data is a WAV file, the file control unit 43 writes the uuid into the LIST block of the WAV file, as shown in FIG. Fig.29 In addition, for example, in the case where the file storing the external data is an MP4 file, the track of the uuid is written into the mdat box of the MP4 file, as described in reference Fig.43 etc. as described.

[0575] In steps S243 to S246, respectively, Fig.61 Similar processing is performed from steps S232 to S235 in , and the processing ends.

[0576] Fig.62 is a flowchart for explaining an example of a process of generating an associated type HEIF file by a pre-storage method.

[0577] In step S251 , the file control unit 43 acquires designation information of each main image stored in the association type HEIF file, and the process proceeds to step S252 .

[0578] For example, by generating, the file control unit 43 acquires uuid as the designation information of each main image stored in the association type HEIF file.

[0579] In step S252, the file control unit 43 generates a HEIF file of the associated type storing the relationship information (with Figures 10 to 12Or an associated type HEIF file similar to an associated type HEIF file of 14, in which the main image and the external data are obviously associated with each other), the relationship information includes the designated information obtained in step S251 before being assigned to the external data and is related to the association between the main image and the external data (the designated information), and the processing ends.

[0580] Fig.63 is a flowchart for explaining an example of association processing of an association type HEIF file generated by a pre-storage method.

[0581] In step S261 , the file control unit 43 acquires, for example, uuid as designation information associated with a main image desired to be associated with external data from the association type HEIF file generated by the pre-storage method, and the process proceeds to step S262 .

[0582] For example, in the case where the associated type HEIF file generated by the pre-storage method is a first associated type set file, obtaining the specified information associated with the predetermined main image from the first associated type set file may be performed, such as Fig.21 as described in the flow chart.

[0583] In step S262 , the file control unit 43 assigns the specification information acquired in step S261 to the external data, that is, determines the specification information of the external data as the specification information acquired in step S261 , and the process proceeds to step S263 .

[0584] In step S263 , the file control unit 43 writes (stores) uuid as the designation information acquired in step S221 into the file storing the external data, and the process ends.

[0585] For example, when the file storing the external data is a WAV file, the file control unit 43 writes the uuid into the LIST block of the WAV file, as shown in FIG. Fig.29 In addition, for example, when the file storing external data is an MP4 file, the track of the uuid is written into the mdat box of the MP4 file, as shown in the reference Fig.43 etc.

[0586] In the association process for the association type HEIF file generated by the pre-storage method, the association between the main image and the external data is completed by writing the specified information into the file storing the external data.

[0587] <Description of Computer to Which the Present Technology is Applied>

[0588] Next, the above-mentioned file control unit 43 and another signal processing unit 13 ( Figure 1) can be executed by hardware or software. In the case where the series of processes are executed by software, a program forming the software is installed in a computer or the like.

[0589] Fig.64 : is a block diagram illustrating a configuration example of an embodiment of a computer to be installed with a program for executing the series of processes described above.

[0590] The program can be recorded in advance on the hard disk 905 or the ROM 903 which is a recording medium built in the computer.

[0591] Alternatively, the program may be stored (recorded) in a removable recording medium 911 driven by the drive 909. Such a removable recording medium 911 may be provided as so-called packaged software. Here, examples of the removable recording medium 911 include, for example, a floppy disk, a compact disk read-only memory (CD-ROM), a magneto-optical (MO) disk, a digital versatile disk (DVD), a magnetic disk, a semiconductor memory, and the like.

[0592] Note that, as described above, the program can be installed in the computer from the removable recording medium 911, or can also be downloaded to the computer via a communication network or a broadcasting network and installed in the built-in hard disk 905. That is, for example, the program can be wirelessly transmitted from a download site to the computer via an artificial satellite used for digital satellite broadcasting, or can be transmitted to the computer LAN by a cable via a network such as a local area network (LAN) and the Internet.

[0593] The computer incorporates a central processing unit (CPU) 902 , and an input / output interface 910 is connected to the CPU 902 via a bus 901 .

[0594] When a user inputs a command by operating the input unit 907 or the like via the input / output interface 910, in response thereto, the CPU 902 executes a program stored in a read only memory (ROM) 903. Alternatively, the CPU 902 loads a program stored in a hard disk 905 into a random access memory (RAM) 904 and executes the program.

[0595] Therefore, the CPU 902 performs processing according to the above-mentioned flowchart or the processing performed by the configuration of the above-mentioned block diagram. Then, when necessary, the CPU 902 outputs the processing result from the output unit 906 via the input / output interface 910 or transmits it from the communication unit 908, and further records it on the hard disk 905 or the like.

[0596] Note that the input unit 907 includes a keyboard, a mouse, a microphone, etc. Furthermore, the output unit 906 includes a liquid crystal display (LCD), a speaker, and the like.

[0597] Here, in this specification, the processing performed by the computer according to the program is not necessarily performed in chronological order according to the order described in the flowchart. That is, the processing performed by the computer according to the program includes processing performed in parallel or individually (for example, parallel processing or processing by target).

[0598] In addition, the program may be processed by one computer (processor), or may be distributed and processed by a plurality of computers. Also, the program may be transmitted to a remote computer for execution.

[0599] Moreover, in this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network and a single device with multiple modules housed in one housing are both systems.

[0600] Note that the embodiments of the present technology are not limited to the above-described embodiments, and various modifications may be made without departing from the scope of the present technology.

[0601] For example, the present technology may have a cloud computing configuration in which one function is shared and collaboratively processed by multiple devices via a network.

[0602] In addition, each step described in the above flowchart may be performed by one device, and may also be shared and performed by a plurality of devices.

[0603] Furthermore, in the case where one step includes a plurality of processes, the plurality of processes included in one step may be executed by one device, and may also be shared and executed by a plurality of devices.

[0604] Furthermore, the effects described in this specification are merely examples and not limitations, and other effects may exist.

[0605] Note that the present technology can have the following configurations.

[0606] <1>

[0607] A file processing device, comprising:

[0608] A file control unit, the file control unit is configured to generate a high-efficiency image file format HEIF file, the HEIF file stores relationship information related to the association between an image and specified information, the relationship information includes the specified information before being assigned to the external data, and the specified information specifies the external data outside the HEIF file to be associated with the image stored in the HEIF file.

[0609] <2>

[0610] according to <1> The file processing device, wherein:

[0611] The file control unit generates a HEIF file storing association information that associates the image with designated information of the external data.

[0612] <3>

[0613] according to <2> The file processing device, wherein:

[0614] The file control unit generates a HEIF file storing associated information in which an item ID of a designated image is associated with the designated information.

[0615] <4>

[0616] according to <3> The file processing device, wherein:

[0617] The file control unit generates a HEIF file in which associated information is stored in a meta box or an mdat box.

[0618] <5>

[0619] according to <2> The file processing device, wherein:

[0620] The file control unit generates a HEIF file in which designated information is stored in an mdat box and associated information is stored in a meta box, the associated information associating an item ID of a designated image with an item ID of designated information stored in the mdat box.

[0621] <6>

[0622] according to <1> The file processing device, wherein:

[0623] The file control unit generates a HEIF file in which an mdat box stores a track of designated information of external data to be associated with a frame constituting a track of an image.

[0624] <7>

[0625] according to <1> to <6> The file processing device according to any one of claims , wherein:

[0626] The file control unit generates designation information for each image stored in the HEIF file, and generates a HEIF file storing relationship information including the designation information.

[0627] <8>

[0628] A file processing method, comprising:

[0629] A High Efficiency Image File Format (HEIF) file is generated, the HEIF file stores relationship information related to the association between an image and designated information, the relationship information includes designated information before being assigned to external data, and the designated information specifies external data outside the HEIF file to be associated with the image stored in the HEIF file.

[0630] <9>

[0631] A program for causing a computer to function as:

[0632] A High Efficiency Image File Format (HEIF) file is generated, the HEIF file stores relationship information related to the association between an image and designated information, the relationship information includes designated information before being assigned to external data, and the designated information specifies external data outside the HEIF file to be associated with the image stored in the HEIF file.

[0633] <10>

[0634] A file processing device, comprising:

[0635] A file control unit, the file control unit is configured to write specified information stored in a high-efficiency image file format HEIF file into a file storing external data, the HEIF file storing relationship information related to the association between the image and the specified information, the relationship information including the specified information before being assigned to the external data and the specified information specifying external data outside the HEIF file to be associated with the image stored in the HEIF file.

[0636] <11>

[0637] according to <10> The file processing device, wherein:

[0638] The file control unit writes the designated information stored in the HEIF file into the file storing the external data, and the HEIF file stores the association information associating the image with the designated information of the external data.

[0639] <12>

[0640] according to <11> The file processing device, wherein:

[0641] The file control unit writes the designated information stored in the HEIF file into the file storing the external data, the HEIF file storing the associated information in which the item ID of the designated image is associated with the designated information.

[0642] <13>

[0643] according to <12> The file processing device, wherein:

[0644] The file control unit writes the specified information stored in the HEIF file into the file storing the external data, and the associated information in the HEIF file is stored in the meta box or the mdat box.

[0645] <14>

[0646] according to <11> The file processing device, wherein:

[0647] The file control unit writes the specified information stored in the HEIF file into the file storing external data. The specified information in the HEIF file is stored in the mdat box and the associated information is stored in the meta box. The associated information associates the item ID of the specified image with the item ID of the specified information stored in the mdat box.

[0648] <15>

[0649] according to <10> The file processing device, wherein:

[0650] The file control unit writes designated information stored in a HEIF file in which an mdat box stores a track of designated information of external data to be associated with a frame constituting a track of an image, into a file storing external data.

[0651] <16>

[0652] according to <10> to <15> The file processing device according to any one of claims , wherein:

[0653] The file control unit writes the designated information stored in the HEIF file into the file storing the external data, the HEIF file storing relationship information including the designated information generated for each image stored in the HEIF file.

[0654] <17>

[0655] A file processing method, comprising:

[0656] The specified information stored in a high-efficiency image file format HEIF file is written into a file storing external data, the HEIF file storing relationship information related to the association between the image and the specified information, the relationship information including the specified information before being assigned to the external data and the specified information specifying external data outside the HEIF file to be associated with the image stored in the HEIF file.

[0657] <18>

[0658] A program for causing a computer to function as:

[0659] A file control unit, the file control unit is configured to write specified information stored in a high-efficiency image file format HEIF file into a file storing external data, the HEIF file storing relationship information related to the association between the image and the specified information, the relationship information including the specified information before being assigned to the external data and the specified information specifying external data outside the HEIF file to be associated with the image stored in the HEIF file.

[0660] Reference numerals list

[0661] 10 Digital Camera

[0662] 11 Optical System

[0663] 13 Signal Processing Unit

[0664] 14 Medium

[0665] 15, 16 Interface

[0666] 17 buttons / keys

[0667] 18 Touch Panel

[0668] 19 LCD Panel

[0669] 20 Viewfinder

[0670] 21 Interface

[0671] 41 Optical system / image sensor control unit

[0672] 42 Encoding control unit

[0673] 43 File Control Unit

[0674] 44 Media Control Unit

[0675] 45 Operation control unit

[0676] 46 Display control unit

[0677] 47 UI control unit

[0678] 901 Bus

[0679] 902 CPU

[0680] 903 ROM

[0681] 904 RAM

[0682] 905 Hard Drive

[0683] 906 Output Unit

[0684] 907 Input Unit

[0685] 908 Communication Unit

[0686] 909 Drive

[0687] 910 Input / Output Interface

[0688] 911 Removable recording media

Claims

1. A file processing device, comprising: a file control unit configured to generate a High Efficiency Image File Format (HEIF) file, the HEIF file storing relationship information related to an association between an image and designated information, the relationship information including designated information that is not assigned to external data when the HEIF file is generated and the designated information is reserved for specifying external data outside the HEIF file to be associated with the image stored in the HEIF file, Among them, the file control unit generates a HEIF file with an associated information storage box, which stores associated information, which associates the image with specified information of external data, and the associated information storage box has an associated information quantity, which indicates whether valid associated information is stored in the associated information storage box.

2. The file processing device according to claim 1, wherein: The file control unit generates a HEIF file storing associated information in which an item ID of a designated image is associated with the designated information.

3. The file processing device according to claim 2, wherein: The file control unit generates a HEIF file in which associated information is stored in a meta box or an mdat box.

4. The file processing device according to claim 1, wherein: The file control unit generates a HEIF file in which designated information is stored in an mdat box and associated information is stored in a meta box, the associated information associating an item ID of a designated image with an item ID of designated information stored in the mdat box.

5. The file processing device according to claim 1, wherein: The file control unit generates a HEIF file in which an mdat box stores a track of designated information of external data to be associated with a frame constituting a track of an image.

6. The file processing device according to claim 1, wherein: The file control unit generates designation information for each image stored in the HEIF file, and generates a HEIF file storing relationship information including the designation information.

7. A file processing method, comprising: generating a High Efficiency Image File Format (HEIF) file, the HEIF file storing relationship information related to association between an image and designated information, the relationship information including designated information that is not assigned to external data when the HEIF file is generated and the designated information is reserved for specifying external data outside the HEIF file to be associated with the image stored in the HEIF file, Among them, a HEIF file with an associated information storage box is generated, the associated information storage box stores associated information, the associated information associates the image with specified information of external data, and the associated information storage box has an associated information quantity, which indicates whether valid associated information is stored in the associated information storage box.

8. A program product for causing a computer to function as: a file control unit configured to generate a High Efficiency Image File Format (HEIF) file, the HEIF file storing relationship information related to an association between an image and designated information, the relationship information including designated information that is not assigned to external data when the HEIF file is generated and the designated information is reserved for specifying external data outside the HEIF file to be associated with the image stored in the HEIF file, in, The file control unit generates a HEIF file having an association information storage box, which stores association information that associates an image with specified information of external data. The association information storage box has an association information quantity that indicates whether valid association information is stored in the association information storage box.

9. A file processing device, comprising: a file control unit configured to write designated information stored in a high-efficiency image file format HEIF file into a file storing external data, the HEIF file storing relationship information related to an association between an image and the designated information, the relationship information including designated information that is not assigned to the external data when the HEIF file is generated and the designated information is reserved for designating external data outside the HEIF file to be associated with the image stored in the HEIF file, Among them, the HEIF file has an associated information storage box, which stores associated information, which associates the image with specified information of external data. The associated information storage box has an associated information quantity, which indicates whether valid associated information is stored in the associated information storage box.

10. The file processing device according to claim 9, wherein: The file control unit writes the designated information stored in the HEIF file into the file storing the external data, the HEIF file storing the associated information in which the item ID of the designated image is associated with the designated information.

11. The file processing device according to claim 10, wherein: The file control unit writes the specified information stored in the HEIF file into the file storing the external data, and the associated information in the HEIF file is stored in the meta box or the mdat box.

12. The file processing device according to claim 9, wherein: The file control unit writes the specified information stored in the HEIF file into the file storing external data. The specified information in the HEIF file is stored in the mdat box and the associated information is stored in the meta box. The associated information associates the item ID of the specified image with the item ID of the specified information stored in the mdat box.

13. The file processing device according to claim 9, wherein: The file control unit writes the designated information stored in the HEIF file in which the mdat box stores the track of designated information of the external data to be associated with the frames of the track constituting the image, into the file storing the external data.

14. The file processing device according to claim 9, wherein: The file control unit writes the designated information stored in the HEIF file into the file storing the external data, the HEIF file storing relationship information including the designated information generated for each image stored in the HEIF file.

15. A file processing method, comprising: writing designated information stored in a high-efficiency image file format HEIF file into a file storing external data, the HEIF file storing relationship information related to an association between an image and the designated information, the relationship information including designated information that is not assigned to the external data when the HEIF file is generated and the designated information is reserved for designating external data outside the HEIF file to be associated with the image stored in the HEIF file, Among them, the HEIF file has an associated information storage box, which stores associated information, which associates the image with specified information of external data. The associated information storage box has an associated information quantity, which indicates whether valid associated information is stored in the associated information storage box.

16. A program product for causing a computer to: a file control unit configured to write designated information stored in a high-efficiency image file format HEIF file into a file storing external data, the HEIF file storing relationship information related to an association between an image and the designated information, the relationship information including designated information that is not assigned to the external data when the HEIF file is generated and the designated information is reserved for designating external data outside the HEIF file to be associated with the image stored in the HEIF file, in, The HEIF file has an associated information storage box, which stores associated information that associates the image with specified information of external data. The associated information storage box has an associated information number that indicates whether valid associated information is stored in the associated information storage box.

Images