File processing device, file processing method, and program

The file processing device and method allow HEIF files to associate images with external data by storing specific information in the mdat box and meta box, ensuring robust data linkage and reducing processing load through area reservation and pre-storage methods.

JP7589683B2Active Publication Date: 2024-11-26SONY GROUP CORP
4 Cites 0 Cited by

Patent Information

Application Number
JP2021527602
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-06-18
Filing Date
2020-06-05
Publication Date
2024-11-26
Estimated Expiration
2040-06-05

AI Technical Summary

Technical Problem

HEIF files lack the capability to associate images stored within them with external data outside the file format.

Method used

A file processing device and method that identifies external data to be associated with images in a HEIF file, generating a HEIF file with specific information stored in an mdat box and association information in a meta box to link the image with the external data.

Benefits of technology

Enables the association of images in HEIF files with external data, maintaining the association even if the external data's file name changes, and reduces the processing load by using area reservation and pre-storage methods to manage offset shifts.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007589683000001
    Figure 0007589683000001
  • Figure 0007589683000002
    Figure 0007589683000002
  • Figure 0007589683000003
    Figure 0007589683000003
Patent Text Reader

Abstract

The present technology relates to a file processing device, a file processing method, and a program for enabling an image stored in a file to be associated with external data outside of the file. A file control unit generates a high-efficiency image file format (HEIF) file that includes specification information which is for specifying HEIF file external data associated with an image stored in the HEIF file but which has yet to be assigned to external data and that stores relation information related to the association between the image and the specification information. In addition, the file control unit writes the specification information stored in the HEIF file into a file in which the external data is stored. The present technology can be applied to, for example, a situation where an HEIF file is generated and the image stored in the HEIF file and the external data are subsequently associated with each other.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present technology relates to a file processing device, a file processing method, and a program, and in particular to a file processing device, a file processing method, and a program that enable, for example, an image stored in a file to be associated with external data outside the file. [Background technology]

[0002] High Efficiency Image File Format (HEIF) is a file format for efficiently storing images (see Non-Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Non-Patent Document 1] ISO / IEC 23008-12:2017, Information technology -- High efficiency coding and media delivery in heterogeneous environments -- Part 12: Image File Format Summary of the Invention [Problem to be solved by the invention]

[0004] For HEIF files that comply with the High Efficiency Image File Format (HEIF), it would be useful if images stored in the HEIF file could be associated with external data outside the HEIF file.

[0005] The present technology was developed in light of these circumstances, and makes it possible to associate images stored in a HEIF file with external data outside the HEIF file. [Means for solving the problem]

[0006] A first file processing device or program of the present technology includes: a file control unit that generates a HEIF file including identification information before being assigned to external data, the identification information identifying external data outside a High Efficiency Image File Format (HEIF) file to be associated with an image stored in the HEIF file, and in which relationship information relating to the association between the image and the identification information is stored. The file control unit generates the HEIF file in which specific information of the external data is stored in an mdat box, and association information that associates the image with the specific information stored in the mdat box is stored in a meta box. A file processing device, or a program for causing a computer to function as such a file processing device.

[0007] A first file processing method of the present technology includes generating a HEIF file including identification information before being assigned to the external data, which identifies external data outside a High Efficiency Image File Format (HEIF) file to be associated with an image stored in the HEIF file, and storing association information relating to the association between the image and the identification information. generating the HEIF file in which specific information of the external data is stored in an mdat box and association information associating the image with the specific information stored in the mdat box is stored in a meta box; A file processing method comprising the steps of:

[0008] In the first file processing device, file processing method, and program of the present technology, a High Efficiency Image File Format (HEIF) file is generated, which includes specific information before being assigned to external data that identifies external data outside the HEIF file to be associated with an image stored in the HEIF file, and in which relationship information related to the association between the image and the specific information is stored. In addition, specific information of the external data is stored in an mdat box, and the HEIF file is generated in which association information that associates the image with the specific information stored in the mdat box is stored in a meta box.

[0009] The second file processing device or program of the present technology is a file processing device that supports HEIF (High Efficiency Image Format) a file control unit that writes the specific information stored in the HEIF file, which includes specific information before being assigned to the external data, to a file in which the external data is stored, the specific information being stored in the HEIF file in which relationship information related to the association between the image and the specific information is stored. The file control unit writes the specific information stored in the HEIF file, in which specific information of the external data is stored in an mdat box and association information associating the image with the specific information stored in the mdat box is stored in a meta box, to a file in which the external data is stored. A file processing device, or a program for causing a computer to function as such a file processing device.

[0010] A second file processing method of the present technology includes identifying external data outside a High Efficiency Image File Format (HEIF) file to be associated with an image stored in the HEIF file, and writing the identifying information stored in the HEIF file, which includes identifying information before being assigned to the external data and in which relationship information relating to the association between the image and the identifying information is stored, to a file in which the external data is stored. and writing the specific information stored in the HEIF file, in which specific information of the external data is stored in an mdat box and association information associating the image with the specific information stored in the mdat box is stored in a meta box, to a file in which the external data is stored. A file processing method comprising the steps of:

[0011] A second file processing device, file processing method, and program of the present technology include specific information before being assigned to external data that identifies external data outside a High Efficiency Image File Format (HEIF) file to be associated with an image stored in the HEIF file, and the specific information stored in the HEIF file, in which relationship information related to the association between the image and the specific information is stored, is written to a file in which the external data is stored. In addition, specific information of the external data is stored in an mdat box, and association information associating the image with the specific information stored in the mdat box is stored in a meta box, and the specific information stored in the HEIF file is written to the file in which the external data is stored.

[0012] The first and second file processing devices may be independent devices, or may be internal blocks constituting a single device.

[0013] In addition, the first and second programs can be provided by being recorded on a recording medium or transmitted via a transmission medium. [Brief description of the drawings]

[0014] [Figure 1] 1 is a block diagram showing an example of the configuration of an embodiment of a digital camera to which the present technology is applied. [Diagram 2] FIG. 1 is a diagram showing an example of a JPEG file format that complies with JPEG (Joint Photographic Experts Group). [Diagram 3] FIG. 1 is a diagram illustrating an example of an ISO base media file format. [Figure 4] FIG. 2 is a diagram showing an example of a format of a HEIF file that complies with HEIF. [Diagram 5] FIG. 1 is a diagram showing an example of the format of a HEIF file in image item format. [Figure 6] FIG. 13 is a diagram showing an example of an iprp box. [Figure 7] FIG. 1 is a diagram showing an example of the format of a HEIF file in image sequence format. [Figure 8] FIG. 1 is a diagram showing an example of a trak box. [Figure 9] FIG. 13 is a diagram showing an example of a normal collection file in which main images and thumbnail images are stored. [Figure 10] FIG. 13 is a diagram showing an example of a first association-type collection file. [Figure 11] FIG. 13 is a diagram showing an example of a second association-type collection file. [Figure 12] FIG. 13 is a diagram showing an example of a third association-type collection file. [Figure 13] FIG. 13 is a diagram showing an example of a normal sequence file in which a track of main images and a track of thumbnail images of the main images are stored. [Figure 14] FIG. 13 is a diagram showing an example of an associated sequence file. [Figure 15] 11 is a flowchart outlining an example of a generation process for generating an associated HEIF file. [Figure 16]11 is a flowchart outlining an example of a playback process for playing an associated HEIF file. [Figure 17] 11 is a flowchart illustrating an example of a reproduction process for reproducing a collection file. [Figure 18] 13 is a flowchart illustrating an example of a process for reading an image to be reproduced in step S32. [Figure 19] 13 is a flowchart illustrating a first example of a process for obtaining a playback target item ID in step S31. [Figure 20] 13 is a flowchart illustrating a second example of the process of acquiring a playback target item ID in step S31. [Figure 21] 13 is a flowchart illustrating an example of a process for acquiring a uuid as identification information of a RAW file of a predetermined main image from a first associative collection file. [Figure 22] 13 is a flowchart illustrating an example of a process for acquiring a uuid as identification information of a RAW file of a predetermined main image from a second associative collection file. [Diagram 23] 13 is a flowchart illustrating an example of a process for acquiring a uuid as identification information of a RAW file of a predetermined main image from a third associative collection file. [Figure 24] 13 is a flowchart illustrating an example of a process for acquiring a list of item IDs of main images from a collection file. [Diagram 25] 11 is a flowchart illustrating an example of a process for reproducing a thumbnail image of a main image (a frame) corresponding to predetermined time information from a sequence file. [Figure 26] 13 is a flowchart illustrating an example of a process for acquiring a uuid as identification information of a RAW file of a predetermined main image (frame) from an associative sequence file. [Figure 27] FIG. 13 is a diagram showing an example of storing a uuid in a RAW file when a RAW file of a main image is used as external data and an associated collection file is generated. [Figure 28] FIG. 13 is a diagram showing an example of storing a uuid in a RAW file when a RAW file of a main image is used as external data and an associated sequence file is generated. [Figure 29] FIG. 13 is a diagram showing an example of storing a uuid in a WAV file when a WAV file of a main image is used as external data and an associated collection file is generated. [Diagram 30] 13 is a diagram showing an example of storing a uuid in a WAV file when a WAV file of a main image is used as external data and an associated sequence file is generated. FIG. [Diagram 31] 1 is a diagram illustrating the association between an image as internal data and external data after a HEIF file is generated. [Diagram 32] FIG. 13 is a diagram for explaining an overview of an example of an area allocation method. [Diagram 33] This figure explains an example of an association process that associates a main image as internal data in a first associative collection file generated by an area allocation method with external data after the first associative collection file is generated. [Diagram 34] FIG. 1 is a diagram outlining an example of a pre-storing method. [Diagram 35] This figure explains an example of an association process that associates a main image as internal data in a first associative collection file generated by a pre-storage method with external data after the first associative collection file is generated. [Diagram 36] FIG. 13 is a diagram outlining another example of a pre-storing method. [Figure 37] FIG. 13 is a diagram outlining yet another example of a pre-storing method. [Figure 38] 1A and 1B are diagrams illustrating available specific information and associations between internal data and external data for each of the area allocation method and pre-storage method. [Figure 39] 13 is a diagram illustrating an example of an association process that performs one-to-N association for a first association-type collection file generated by the area allocation method. FIG. [Diagram 40] FIG. 13 is a diagram illustrating an example of an association process that performs 1-to-N association for a first association-type collection file generated by a pre-storing method. [Diagram 41] 13 is a diagram illustrating an example of an association process that performs N-to-1 association for a first association-type collection file generated by the area allocation method. FIG. [Diagram 42] FIG. 13 is a diagram illustrating an example of an association process that performs N-to-1 association for a first association-type collection file generated by a pre-storing method. [Diagram 43] FIG. 13 is a diagram showing a first example of association between a main image as internal data and audio as external data when an MP4 file is used as a file for storing audio as external data. [Diagram 44] FIG. 13 is a diagram showing a second example of association between a main image as internal data and audio as external data when an MP4 file is used as the file for storing audio as external data. [Diagram 45] FIG. 13 is a diagram showing a third example of association between a main image as internal data and audio as external data when an MP4 file is used as the file for storing audio as external data. [Figure 46] FIG. 13 is a diagram showing a fourth example of association between a main image as internal data and audio as external data when an MP4 file is used as the file for storing audio as external data. [Figure 47] FIG. 13 is a diagram showing a fifth example of association between a main image as internal data and audio as external data when an MP4 file is used as the file for storing audio as external data. [Figure 48] FIG. 13 is a diagram showing a sixth example of association between a main image as internal data and audio as external data when an MP4 file is used as the file for storing audio as external data. [Figure 49]11 is a diagram illustrating a first example of a first associative collection file generated by an area allocation method and a state of the first associative collection file after an associating process. FIG. [Figure 50] 13 is a diagram illustrating a second example of a first associative collection file generated by the area allocation method and a state of the first associative collection file after an associating process. FIG. [Figure 51] FIG. 13 is a diagram illustrating a third example of a first associative collection file generated by the area allocation method and a state of the first associative collection file after the associating process. [Figure 52] 13 is a diagram illustrating a first example of a second associative collection file generated by an area allocation method and a state of the second associative collection file after an associating process. FIG. [Figure 53] 13 is a diagram illustrating a second example of a second associative collection file generated by the area allocation method and the state of the second associative collection file after the associating process. FIG. [Figure 54] 13 is a diagram illustrating a first example of a third associative collection file generated by an area allocation method and a state of the third associative collection file after an associating process. FIG. [Figure 55] FIG. 13 is a diagram illustrating a second example of a third associative collection file generated by the area allocation method and a state of the third associative collection file after associating processing. [Figure 56] 1 is a diagram illustrating a first example of an associative sequence file generated by an area allocation method and a state of the associative sequence file after associating processing. FIG. [Figure 57] 13 is a diagram illustrating a second example of an associative sequence file generated by the area allocation method and the state of the associative sequence file after the associative process. FIG. [Figure 58] 13 is a flowchart illustrating an example of a process for generating an associated HEIF file using an area allocation method. [Figure 59] 13 is a flowchart illustrating an example of an association process for an associative HEIF file generated by the area allocation method. [Figure 60] 13 is a flowchart illustrating an example of an association process for a first association-type collection file generated by an area allocation method. [Figure 61] 13 is a flowchart illustrating another example of the association process for the first association-type collection file generated by the area allocation method. [Figure 62] 11 is a flowchart illustrating an example of a process for generating an associated HEIF file using a pre-storing method. [Figure 63] 11 is a flowchart illustrating an example of an association process for an associated HEIF file created using the pre-storage method. [Figure 64] 1 is a block diagram showing an example of the configuration of an embodiment of a computer to which the present technology is applied. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] <One embodiment of a digital camera to which the present technology is applied>

[0016] FIG. 1 is a block diagram showing an example of the configuration of an embodiment of a digital camera to which the present technology is applied.

[0017] The digital camera 10 has an optical system 11, an image sensor 12, a signal processing unit 13, a medium 14, interfaces 15 and 16, buttons / keys 17, a touch panel 18, a liquid crystal panel 19, a viewfinder 20, and an interface 21, etc.

[0018] The optical system 11 collects light from a subject onto the image sensor 12 .

[0019] The image sensor 12 receives light from the optical system 11 and performs photoelectric conversion to capture an image, thereby generating image data as an electrical signal, and supplies the data to a signal processing unit 13 .

[0020] The signal processing unit 13 has an optical system / image sensor control unit 41, an encoding control unit 42, a file control unit 43, a media control unit 44, an operation control unit 45, a display control unit 46, and a UI control unit 47.

[0021] The optical system / image sensor control unit 41 controls the optical system 11 and the image sensor 12, and supplies the image (data) obtained by imaging performed under that control to the encoding control unit .

[0022] The encoding control unit 42 supplies the image from the optical system / image sensor control unit 41 to the display control unit 46, and also encodes the image as necessary and supplies it to the file control unit 43. The encoding control unit 42 also decodes the image supplied from the file control unit 43 as necessary and supplies it to the display control unit 46.

[0023] The file control unit 43 generates a file storing the image supplied from the encoding control unit 42, and supplies the file to the media control unit 44. The file control unit 43 also plays the file supplied from the media control unit 44, i.e., reads data such as images stored in the file. For example, an image read from a file is supplied from the file control unit 43 to the encoding control unit 42.

[0024] The media control unit 44 controls the exchange of files between the medium 14 and the interfaces 15 and 16. For example, the media control unit 44 causes a file from the file control unit 43 to be recorded on the medium 14 or transmitted from the interfaces 15 and 16. The media control unit 44 also reads a file from the medium 14 or causes the interfaces 15 and 16 to receive a file and supply it to the file control unit 43.

[0025] In response to a user's operation of the buttons / keys 17 or the touch panel 18, the operation control unit 45 supplies an operation signal corresponding to the operation to a necessary block.

[0026] The display control unit 46 performs display control and the like to supply the images and the like supplied from the encoding control unit 42 to the liquid crystal panel 19, the viewfinder 20, and the interface 21 for display.

[0027] The UI control unit 47 is in charge of UI (User Interface) control.

[0028] The medium 14 is, for example, a storage medium such as an SD card. The interface 15 is, for example, a LAN (Local Area Network) interface such as WiFi (registered trademark) or Ethernet (registered trademark). The interface 16 is, for example, a USB (Universal Serial Bus) interface. The buttons / keys 17 and the touch panel 18 are operated by a user when inputting commands or other information to the digital camera 10. The touch panel 18 can be configured integrally with a liquid crystal panel 19. The liquid crystal panel 19 and the viewfinder 20 display images and the like supplied from the display control unit 46. The interface 21 is an interface that transmits at least images, such as HDMI (High-Definition Multimedia Interface) (registered trademark) or DP (Display Port).

[0029] In the digital camera 10 configured as described above, the optical system / image sensor control unit 41 generates, from the RAW data image (hereinafter also referred to as the RAW image) obtained by imaging with the image sensor 12, for example, a YUV image having the same resolution (number of pixels) as the RAW image, as the main image of the HEIF file, and supplies this to the encoding control unit 42.

[0030] The encoding control unit 42 generates, from the YUV main image, a YUV image (hereinafter also referred to as a screen nail image) having a lower resolution than the main image as a first other image based on the main image for display on the liquid crystal panel 19 or an external display, and generates, for example, a YUV image (hereinafter also referred to as a thumbnail image) having a lower resolution than the screen nail image as a second other image based on the main image for displaying a list. The encoding control unit 42 supplies, for example, the screen nail image to the liquid crystal panel 19 via the display control unit 46, and displays it as a so-called through image. As the thumbnail image, for example, an image with a long side of 320 pixels or less can be used. The ratio of the size (number of pixels) between the main image and the screen nail image as the first other image based on the main image, or the thumbnail image as the second other image based on the main image can be, for example, 200 times or less. Similarly, the ratio of the size between the screen nail image as the first other image based on the main image and the thumbnail image as the second other image based on the main image can be 200 times or less. As the screen nail image, for example, an image with a resolution of 4K or more can be adopted. Furthermore, as the screen nail image, for example, an image with a resolution of 4K (QFHD) or FHD can be adopted depending on the user's selection. Furthermore, images with the same resolution can be adopted as the main image and the screen nail image. When images with the same resolution are adopted as the main image and the screen nail image, the HEIF file can store both the main image and the screen nail image, or the main image can be stored without storing the screen nail image. When the main image is stored without storing the screen nail image in the HEIF file, the main image can be resized and used as the screen nail image.

[0031] In addition, the encoding control unit 42 encodes the main image, screennail image, and thumbnail image corresponding to the RAW image (the main image, screennail image, and thumbnail image generated from the same RAW image) as necessary, and supplies them to the file control unit 43 together with the RAW image.

[0032] The file control unit 43 generates a RAW file in which a RAW image is stored, and also generates a HEIF file or a JPEG file in which a corresponding main image, screennail image, and thumbnail image (a main image, screennail image, and thumbnail image generated from the same RAW image) are stored, and supplies the files to the media control unit 44. A HEIF file is a file that conforms to HEIF (High Efficiency Image File Format), and a JPEG file is a file that conforms to JPEG (Joint Photographic Experts Group).

[0033] The media control unit 44 records the RAW file and the HEIF file or JPEG file from the file control unit 43 on the media 14 or transmits them from the interface 15 or 16.

[0034] In the file control unit 43, whether to generate a HEIF file or a JPEG file can be selected according to, for example, a user operation. Also, as described later, HEIF files come in an image item format and an image sequence format, and whether to adopt the image item format or the image sequence format can be selected according to, for example, a user operation. Furthermore, the file control unit 43 can perform mutual conversion between a HEIF file and a JPEG file according to a user operation.

[0035] Furthermore, in generating a HEIF file, the file control unit 43 can associate internal data (data stored in the HEIF file) in the HEIF file that is associated with external data outside the HEIF file (data not stored in the HEIF file) with specific information that identifies the external data and store it in the HEIF file. A HEIF file in which internal data and specific information of external data associated with the internal data are stored in association with each other is also called an associated HEIF file. In an associated HEIF file, for example, the internal data and specific information can be stored in association with each other by storing association information that associates the internal data and the specific information.

[0036] <JPEGファイル>

[0037] FIG. 2 is a diagram showing an example of a JPEG file format that complies with JPEG (Joint Photographic Experts Group).

[0038] A JPEG file is configured by storing, for example, Exif metadata, a thumbnail image, XMP (Extensible Metadata Platform) (registered trademark) metadata, an MPF ​​indicating the storage location (position) of the main image and the simplified display image, etc., a main image, and an image for simplified display. For example, a screen nail image can be used as the image for simplified display.

[0039] <ISOベースメディアファイルフォーマット>

[0040] FIG. 3 is a diagram showing an example of the ISO base media file format.

[0041] HEIF (ISO / IEC 23008-12) is a file format that conforms to the ISO Base Media File Format (ISO / IEC 14496-12), and therefore HEIF files conform to the ISO Base Media File Format.

[0042] The ISO base media file format is composed of units called boxes, which serve as containers for storing data, and has a structure called a box structure.

[0043] A box has a type (box type) and actual data (data). The type indicates the type of actual data in the box. Actual data can be playable media data such as images (still images, videos), audio, subtitles, etc., attribute names (field names) and attribute values ​​(field values) of the attribute names (variables represented by the attribute names), and various other data.

[0044] Furthermore, a box can be used as actual data, that is, a box can have another box as actual data, which allows a hierarchical structure to be created.

[0045] A base media file that complies with the ISO base media file format can have an ftyp box, a moov box (MovieBox), a meta box (MetaBox), an mdat box (MediaDataBox), and the like. The ftyp box stores identification information that identifies the file format. The moov box can store a trak box, and the like. The meta box can store an iinf box, an iprp box, an iref box, an iloc box, and the like. The mdat box can store media data (AV data) and any other data.

[0046] HEIF complies with the above ISO Base Media File Formats.

[0047] <HEIFファイル>

[0048] FIG. 4 is a diagram showing an example of a format of a HEIF file that complies with HEIF.

[0049] HEIF files are broadly divided into image item and image sequence formats, with the image item format further divided into single image formats that contain only one item, and image collection formats that contain multiple items, as described below.

[0050] A HEIF file in the image item format has an ftyp box, a meta box, and an mdat box.

[0051] A HEIF file in image sequence format has an ftyp box, a moov box, and an mdat box.

[0052] Note that a HEIF file can have both a meta box and a moov box, not just one of them.

[0053] The ftyp box stores identification information for identifying the file format, for example, that the file is a HEIF file in image item format or image sequence format.

[0054] The meta box and moov box store metadata necessary for playback and management of the media data stored in the mdat box, such as the storage location of the media data.

[0055] The mdat box stores media data (AV data) and the like.

[0056] In the digital camera 10, which HEIF file to generate, the image item format or the image sequence format, can be selected according to, for example, a user's operation. Also, when encoding and storing an image in the mdat box of the HEIF file, only intra-encoding is permitted for the image item format, and intra-encoding and inter-encoding are permitted for the image sequence format. Therefore, for example, when high-speed access to data stored in the HEIF file is prioritized, generation of a HEIF file in the image item format can be selected, and when reducing the size (amount of data) of the HEIF file is prioritized, generation of a HEIF file in the image sequence format can be selected.

[0057] FIG. 5 is a diagram showing an example of the format of a HEIF file in image item format.

[0058] In a HEIF file in the image item format, information indicating that it is a HEIF file in the image item format, such as mif1, is stored (as an attribute value) in the ftyp box.

[0059] The meta box stores an iinf box, an iref box, an iprp box, and an iloc box.

[0060] The iinf box stores the number of items (attribute names and attribute values ​​representing) that are media data (AV data) stored in the mdat box. An item is a piece of data stored in the mdat box of a HEIF file in image item format; for example, one image (screen) is an item. In this specification, one image, whether a still image or a video, is also called a frame. One frame is one item.

[0061] The iref box stores information indicating the relationship between items. For example, the mdat box can store the corresponding main image, screennail image, and thumbnail image as items. When the mdat box stores item I1 as the main image, item I2 as the screennail image, and item I3 as the thumbnail image, the iref box stores information indicating that item I2 is the screennail image of the main image as item I1, and information indicating that item I3 is the thumbnail image of the main image as item I1.

[0062] The iprp box stores information about the properties of an item.

[0063] The iloc box stores information about the storage location of the items stored in the mdat box.

[0064] In the mdat box of the image item format (HEIF file), for example, a frame of an image is stored as an item. One or more items can be stored in the mdat box. Also, a frame as an item can be encoded and stored in the mdat box. However, encoding of a frame as an item stored in the mdat box of the image item format is limited to intra encoding. As an encoding method (codec) for encoding a frame as an item, for example, HEVC or the like can be adopted.

[0065] FIG. 6 is a diagram showing an example of the iprp box of FIG.

[0066] The iprp box stores an ipco box and an ipma box related to item properties. The ipco box stores the properties of the item stored in the mdat box, such as codec information related to the codec of the image as the item and image size information related to the size. The ipma box stores an index (pointer) to the properties stored in the ipco box of the item stored in the mdat box.

[0067] FIG. 7 is a diagram showing an example of the format of a HEIF file in image sequence format.

[0068] In a HEIF file in the image sequence format, information indicating that it is a HEIF file in the image sequence format, such as msf1, is stored in the ftyp box.

[0069] The moov box stores a trak box, which stores information about the track stored in the mdat box.

[0070] A track consists of one independent piece of media data, such as images and audio, that is played according to a timeline. For example, a track consists of one or more frames of images that form an elementary stream. For tracks stored in an mdat box, multiple tracks, for example, image and audio tracks recorded at the same time, can be played simultaneously.

[0071] The media data of a track is composed of units called samples. A sample is the smallest unit (access unit) for accessing media data in a HEIF file. Therefore, it is not possible to access media data in a HEIF file in units smaller than a sample.

[0072] For image media data, one sample is, for example, one frame, etc. For audio media data, one sample is, for example, one audio frame or the like defined by the standard of the audio media data.

[0073] In the mdat box of the image sequence format (HEIF file), the media data of a track is arranged in units called chunks. A chunk is a collection of one or more samples arranged at logically consecutive addresses.

[0074] When multiple tracks of media data are stored in the mdat box, the multiple tracks are arranged in an interleaved manner in chunk units.

[0075] As described above, an mdat box in the image sequence format stores one or more tracks made up of media data such as images and audio.

[0076] In the mdat box, the frames of the images that make up the track can be encoded and stored. To encode the frames that make up the track stored in the mdat box in the image sequence format, long GOP can be used as the GOP (Group of Picture), and both intra-encoding and inter-encoding can be used. As a codec for encoding the frames that make up the track, for example, HEVC can be used.

[0077] FIG. 8 is a diagram showing an example of a trak box.

[0078] The trak box can store a tkhd box and an mdia box. The tkhd box stores header information of the track managed by the trak box, such as the creation date and time of the track. The mdia box stores the minf box and the like. The minf box stores the stbl box. The stbl box stores the stsd box, stsc box, stsz box, and stco box, which store the track samples and, in turn, information for accessing the chunks. The stsd box stores codec information related to the codec of the track. The stsc box stores the chunk size (the number of samples in one chunk). The stsz box stores the sample size. The stco box stores the chunk offset, i.e., the offset of the placement position of each chunk of the track stored in the mdat box.

[0079] Here, a HEIF file in the image item format is also called a collection file, and a HEIF file in the image sequence format is also called a sequence file. Furthermore, an associated HEIF file in the image item format is also called an associated collection file, and an associated HEIF file in the image sequence format is also called an associated sequence file.

[0080] The digital camera 10 can generate a HEIF file (including an associated HEIF file) that stores the main image and also any required screennail and / or thumbnail images.

[0081] <Collection file>

[0082] FIG. 9 is a diagram showing an example of a normal collection file in which main images and thumbnail images are stored.

[0083] Here, a normal collection file refers to a collection file in which internal data within the collection file is not associated with specific information of external data.

[0084] Now, it is assumed that frames (items) are encoded in HEVC and stored in the mdat box of the collection file.

[0085] In the ftyp box, heic, which indicates that the file format is an image item format and that the codec is HEVC, is stored as identification information for identifying the file format.

[0086] The number of items (item count) stored in the mdat box is stored in the iinf box. In Fig. 9, a total of four items (frames) are stored in the mdat box: the main image (hereinafter also referred to as main image Item#1) identified by item ID#1, the main image Item#2, the thumbnail image (hereinafter also referred to as thumbnail image Item#101) identified by item ID#101, and the thumbnail image Item#102. Therefore, the number of items is four. Note that the thumbnail image Item#101 is the thumbnail image of the main image Item#1, and the thumbnail image Item#102 is the thumbnail image of the main image Item#2.

[0087] The iinf box further stores an infe box for each item stored in the mdat box. The infe box stores an item ID that identifies the item and an item type. In FIG. 9, there are infe boxes for the main images Item#1 and Item#2, and the thumbnail images Item#101 and Item#102.

[0088] In the iref box, for example, a thmb box is stored as information relating items stored in the mdat box. In the thmb box, a reference source and a reference destination are stored in association with each other as information relating a main image and a thumbnail image of the main image. In the thmb box, the reference source represents the item ID of the main image, and the reference destination represents the item ID of the thumbnail image of the main image specified by the item ID of the reference source. Therefore, according to the reference destination associated with the reference source, it is possible to recognize the item ID of the thumbnail image of the main image specified by the item ID represented by the reference source. Also, according to the reference source associated with the reference destination, it is possible to recognize the item ID of the main image of the thumbnail image specified by the item ID represented by the reference destination.

[0089] The iprp box stores an ipco box and an ipma box, as described in Fig. 6. The ipco box stores properties of a frame as an item stored in the mdat box, such as codec information related to the codec and image size information related to the size, as described in Fig. 6. The ipma box stores an index to the properties stored in the ipco box of the item stored in the mdat box, as described in Fig. 6.

[0090] The iloc box stores information about the storage location of an item in the mdat box, as described in Fig. 6. In Fig. 9, the iloc box stores that the number of items is 4. Furthermore, the iloc box stores the offsets to the storage locations and sizes of the main images Item#1 and Item#2 and the thumbnail images Item#101 and Item#102 stored in the mdat box, in association with the item IDs.

[0091] Hereinafter, an associated collection file in which internal data and specific information of external data are stored in association with each other in the normal collection file of FIG. 9 will be described.

[0092] FIG. 10 is a diagram showing an example of the first association-type collection file.

[0093] Hereinafter, the external data to be associated with a main image as internal data within a HEIF file will be, for example, a RAW image of the main image (or a RAW file in which the RAW image is stored).

[0094] In the first associative collection file, by storing association information that associates the main image as internal data with specific information of the RAW file in which the RAW image is stored (the RAW image stored in the RAW file), the main image and specific information of the RAW file in which the RAW image is stored are stored in association with each other. Furthermore, in the first associative collection file, the association information is stored in a meta box.

[0095] The identifying information for a RAW file in which a RAW image as external data is stored can include the file name of the RAW file, a universally unique identifier (uuid) issued to the RAW file, a uniform resource locator (URL), or any other information that can identify the RAW file (the RAW image stored in it).

[0096] For the first association type collection file, an association information storage box in which association information is stored is defined as a new box stored in the meta box, and is stored in the meta box. The association information storage box of the first association type collection file stores association information in which, for example, an item ID that identifies a main image and a uuid as identification information that identifies a RAW file (in which a RAW image is stored) associated with the main image are associated. Furthermore, the association information storage box stores the number of main images (the number of main images) associated with the RAW file (the RAW image stored in the RAW file). The number of main images stored in the association information storage box is the number of main images associated with the RAW file, and is therefore equal to or less than the number of main images stored in the mdat box.

[0097] In Fig. 10, the uuid of the RAW file of the main image Item#1 (the uuid of the RAW image associated with the main image Item#1) is UUID#1, and the uuid of the RAW file of the main image Item#2 is UUID#2. If a RAW file with a uuid of UUID#i is described as a RAW file UUID#i, then in Fig. 10, the item ID#1 of the main image Item#1 is associated with the uuid of the RAW file UUID#1, and the item ID#2 of the main image Item#2 is associated with the uuid of the RAW file UUID#2, and association information is stored in the association information storage box.

[0098] FIG. 11 is a diagram showing an example of the second association-type collection file.

[0099] In the second associative collection file, similarly to the first associative collection file, the main image as internal data is stored in association information that associates the main image with the specific information of the RAW file as external data, and the main image and the specific information of the RAW file are stored in association with each other. However, in the second associative collection file, the association information is stored in the mdat box.

[0100] For the second association collection file, for example, the same association information as that of the first association collection file is stored as an item in the mdat box. In Fig. 11, the association information is stored in the mdat box as an item of item ID #201.

[0101] As described above, in the second associative collection file, the association information for the item Item#201 is stored in the mdat box, and accordingly the information stored in the meta box differs from that in the normal collection file of Fig. 9. In the second associative collection file, metadata of the association information for the item Item#201 is stored in the meta box.

[0102] Specifically, in the second associative collection file, the number of items stored in the iinf box and the iloc box changes from 4 in the case of FIG. 9 to 5, which is 4 plus 1 for item Item#201. Furthermore, an infe box for item Item#201 is added to the iinf box, and an offset to the storage location and size of item Item#201 are added to the iloc box. The infe box for item Item#201 stores item ID#201 of item Item#201 and an item type IDIF (identifying data info) indicating that item Item#201 is associative information. IDIF is a newly defined attribute value (field value) indicating that the item is associative information.

[0103] FIG. 12 is a diagram showing an example of the third association-type collection file.

[0104] In the third associative collection file, the specific information of the RAW file as external data is stored as an item for each specific information in the mdat box, and the association information that associates the main image as internal data with the specific information of the RAW file as external data is stored in the meta box, so that the main image and the specific information of the RAW file are stored in association with each other. However, in the third associative collection file, the association information is information that associates the item ID of the main image as an item with the item ID of the specific information (of the RAW file) as an item, and is stored in the cdsc box stored in the iref box in the meta box.

[0105] In the cdsc box, a reference source and a reference destination can be stored in association with each other as information relating items as a main image and specific information of a RAW file of the main image. In the cdsc box, the reference source indicates the item ID of the main image, and the reference destination indicates the item ID of specific information as an item of the RAW file of the main image specified by the item ID of the reference source.

[0106] 12, UUID#1, which is the uuid as the identifying information of the RAW file of the main image Item#1, is stored in the mdat box as item Item#201, and UUID#2, which is the uuid as the identifying information of the RAW file of the main image Item#2, is stored in the mdat box as item Item#202. Furthermore, a cdsc box storing association information that associates the item ID#1 of the main image Item#1 with the item ID#201 of the identifying information UUID#1 as the reference source and the reference destination, respectively, is stored in the iref box, and a cdsc box storing association information that associates the item ID#2 of the main image Item#2 with the item ID#202 of the identifying information UUID#2 as the reference source and the reference destination, respectively, is stored in the iref box.

[0107] <Sequence file>

[0108] FIG. 13 is a diagram showing an example of a normal sequence file in which a track of a main image and a track of thumbnail images of the main image are stored.

[0109] Here, a normal sequence refers to a sequence file in which internal data within the sequence file is not associated with specific information of external data.

[0110] Now, let us assume that frames are encoded in HEVC and stored in the mdat box of the sequence file.

[0111] In the ftyp box, hevc, which indicates that the file format is an image sequence format and that the codec is HEVC, is stored as identification information for identifying the file format.

[0112] As described in Fig. 7, the moov box stores a trak box that manages each track stored in the mdat box. In Fig. 13, the track of the main image identified by track ID #1 (hereinafter also referred to as track #1) and track #2 of the thumbnail image of the main image in track #1 are stored in the mdat box. Therefore, the moov box stores a trak box that manages track #1 and a trak box that manages track #2. The (frame of) n-th thumbnail image (from the beginning) of track #2 is the thumbnail image of the n-th main image in track #1.

[0113] A sequence file is useful, for example, when continuous shooting is performed with the digital camera 10, in which case the main images and thumbnail images of a plurality of frames obtained by the continuous shooting are each recorded as one track.

[0114] The tkhd box of the trak box that manages track #1 of the main images stores track ID #1 that identifies track #1, the image size of the main images that make up track #1, rotation information that indicates the orientation of the digital camera 10 when the main images were captured, and the creation date and time of track #1. The tkhd box of the trak box that manages track #2 of the thumbnail images stores track ID #2 that identifies track #2 and the creation date and time of track #2.

[0115] In addition to the tkhd box and mdia box described in Fig. 7, the trak box can store a tref box. The tref box stores a track ID that identifies other tracks related to the track managed by the trak box in which the tref box is stored, and information that indicates the contents of the track. In Fig. 13, a tref box is provided in a trak box that manages track #2. The tref box stores information that indicates that the other track related to track #2 is track #1 (track_ID=1) and that the data that constitutes track #2 is a thumbnail image (track #2 is a thumbnail image track) (type=thmb).

[0116] In addition to the MINF box described in Fig. 8, the mdia box of the trak box can store an hdlr box. The hdlr box stores information indicating the type of data that constitutes the track managed by the trak box in which the hdlr box is stored. The hdlr box stored in the trak box that manages track #1 of the main image (in the mdia box in which it is stored) stores information (pict) indicating that the data that constitutes track #1 is a picture (frame), and the hdlr box stored in the trak box that manages track #2 of the thumbnail images stores information indicating that the data that constitutes track #2 is a picture.

[0117] The minf box is as explained in FIG.

[0118] Hereinafter, an associated sequence file in which internal data and specific information of external data are stored in association with each other in the normal sequence file of FIG. 13 will be described.

[0119] FIG. 14 is a diagram showing an example of an associated sequence file.

[0120] In an associated sequence file, track #3 of the (elementary) stream (Meta ES) of the uuid as specific information of the RAW file as external data is added to the mdat box, and the trak box that manages track #3 is added to the moov box.

[0121] Here, track #1 is a time series of one or more frames of the main image lined up on the timeline, and track #3 is a time series of the uuids of the RAW files of each frame of the main image lined up on the timeline.

[0122] The nth uuid (from the beginning) of track #3 is specific information of the RAW file of the nth frame of the main image of track #1. In addition, (data of) a plurality of tracks stored in the mdat box can be played back in synchronization according to time information on one timeline. Therefore, by storing track #1 of the main image and track #3 of the uuid (stream) of the RAW file of each frame of the main image constituting track #1 in the mdat box, the nth frame of the main image of track #1 and the uuid of the RAW file of the main image (frame) are stored in association with each other. In this case, it can be said that the frame of the main image of track #1 and the uuid of the RAW file of the main image (frame) are associated with each other by time information on the timeline.

[0123] In addition, the nth uuid (from the beginning) of track #3 is specific information for the RAW file of the nth frame of track #1, and the main images (frames) that make up track #1 and the uuids that make up track #3 can also be considered to be related by the order in which they are arranged on the tracks.

[0124] In an associative sequence file, when track #3 of the uuid of the RAW file is added to the mdat box, the trak box that manages that track #3 is added to the moov box.

[0125] The trak box that manages track #3 of the uuid of the RAW file stores a tkhd box, a tref box, an mdia box, and the like.

[0126] The tkhd box of the trak box that manages track #3 stores track ID #3 that identifies track #3 and the creation date and time of track #3.

[0127] The tref box of the trak box managing track #3 stores a track ID that identifies other tracks related to track #3 managed by the trak box in which the tref box is stored, and information that indicates the contents of track #3, etc. The uuid that constitutes track #3 is specific information of the RAW file of the main image that constitutes track #1, and since track #3 is related to track #1, the tref box of the trak box managing track #3 in Fig. 14 stores information that indicates that the other track related to track #3 is track #1 (track_ID=1) and that track #3 is a track of metadata (specific information here) (type=cdsc).

[0128] In the mdia box of the trak box that manages Track #3, an hdlr box and a minf box are stored. In the trak box that manages Track #3, the hdlr box stores information indicating that the data constituting Track #3 is metadata (of the main image), and the minf box stores an stsc box, an stsc box, an stsz box, and an stco box for Track #3.

[0129] <Generation and Playback of HEIF Files>

[0130] Figure 15 is a flowchart for explaining an outline of an example of a generation process for generating an associated HEIF file.

[0131] In the generation process, in step S11, the file control unit 43 generates a uuid as specific information of the RAW file (RAW image) of the frame of the main image, and the process proceeds to step S12.

[0132] In step S12, the file control unit 43 assigns the uuid generated in step S11 to the RAW file (RAW image) of the frame of the main image, and the process proceeds to step S13.

[0133] In step S13, the file control unit 43 generates an associated HEIF file in which the frame of the main image and the uuid of the RAW file of the frame are associated and stored in the HEIF file, and ends the generation process.

[0134] Figure 16 is a flowchart for explaining an outline of an example of a playback process for playing back an associated HEIF file.

[0135] In the playback process, in step S21, the file control unit 43 generates, for example, a handle list of handles for identifying individual frames of the main image stored in the HEIF file stored in the medium 14, and the process proceeds to step S22.

[0136] Here, the handle of the main image frame includes the file name of the HEIF file in which the frame is stored. The handle of the main image frame (item) stored in the collection file further includes the item ID of the frame. The handle of the main image frame stored in the sequence file further includes the time information of the frame. The handle of the main image frame allows the frame for that handle to be uniquely identified (specified).

[0137] In addition, the handle of a main image frame stored in a sequence file can include the track ID of the track that contains the frame and the order of the frame in that track (the number of the frame) instead of the frame's time information.

[0138] Whether the sequence file contains one or multiple tracks made up of frames of the main image, the time information of each frame is unique. Therefore, even if multiple tracks are stored in the sequence file, the frame of the time information included in the handle can be uniquely identified from the frames that make up each of the multiple tracks, according to the frame time information. Therefore, if the handle of the main image frame contains the time information of that frame, the frame for the time information can be uniquely identified even if there is no track ID of the track in which the frame exists.

[0139] The handle list can be generated for all frames of the main image stored in the HEIF file stored on media 14, or it can be generated for only frames narrowed down by specific criteria, such as frames with a specific creation date and time.

[0140] After the file control unit 43 generates the handling list, the HEIF file is accessed by referring to the handling list as necessary.

[0141] In step S22, for example, upon waiting for the user to operate the digital camera 10 to display thumbnail images, the UI control unit 47 requests the file control unit 43 to display thumbnail images. In response to the request to display thumbnail images from the UI control unit 47, the file control unit 43 reads out from the HEIF file (a frame of) a thumbnail image of the main image frame identified by the handle in the handle list. The file control unit 43 then causes a list of thumbnail images read out from the HEIF file to be displayed, for example, on the liquid crystal panel 19 (FIG. 1), and the process proceeds from step S22 to step S23.

[0142] In step S23, for example, after the user selects a desired thumbnail (frame) from a list of thumbnail images, the UI control unit 47 requests the file control unit 43 for a main image corresponding to the thumbnail image selected by the user. In response to the request for the main image from the UI control unit 47, the file control unit 43 reads the main image from the HEIF file. The file control unit 43 can display the main image read from the HEIF file on the liquid crystal panel 19 as necessary.

[0143] Alternatively, the UI control unit 47 requests the uuid of the RAW file of the main image corresponding to the thumbnail image selected by the user from the file control unit 43. In response to the request for the uuid from the UI control unit 47, the file control unit 43 reads the uuid from the associated HEIF file. The file control unit 43 can access the RAW file identified by the uuid read from the associated HEIF file as necessary.

[0144] FIG. 17 is a flowchart illustrating an example of a playback process for playing back a collection file.

[0145] In step S31, the file control unit 43 acquires the item ID of the image to be reproduced (hereinafter, also referred to as the reproduction target item ID) which is the image (item) to be reproduced, and the process proceeds to step S32.

[0146] When obtaining the playback target item ID, for example, a main image identified by an arbitrary handle in the handle list, a thumbnail image of that main image, a thumbnail image selected by the user from a list of thumbnail images (hereinafter also referred to as a selected thumbnail image), or the main image of the selected thumbnail image is taken as the playback target image, and the item ID (playback target item ID) of that playback target image is obtained.

[0147] In step S32, the file control unit 43 reads out the image to be played back in accordance with the playback target item ID acquired in step S31.

[0148] In reading the image to be played back, the image to be played back specified by the playback target item ID is read out from the collection file.

[0149] FIG. 18 is a flowchart illustrating an example of the process of reading the image to be reproduced in step S32 of FIG.

[0150] In step S41, the file control unit 43 searches the iloc box of the collection file (FIGS. 9 to 12) for the playback target item ID, and the process proceeds to step S42.

[0151] In step S42, the file control unit 43 reads out the offset and size associated with the playback target item ID found in the iloc box in step S41, and the process proceeds to step S43.

[0152] In step S43, the file control unit 43 reads out the image to be played back stored in the mdat box of the collection file in accordance with the offset and size associated with the playback target item ID, and the process ends.

[0153] FIG. 19 is a flowchart illustrating a first example of the process of acquiring the playback target item ID in step S31 of FIG.

[0154] That is, FIG. 19 shows an example of acquiring the item ID of a thumbnail image which is an image to be played back.

[0155] In FIG. 19, it is assumed that the file control unit 43 recognizes the item ID of the main image of the thumbnail image as the image to be played back from the handle, for example.

[0156] In step S51, the file control unit 43 searches for a thmb box whose reference source matches the item ID of the main image from among the thmb boxes in the iref box of the collection file (FIGS. 9 to 12), and the process proceeds to step S52.

[0157] In step S52, file control unit 43 reads out the reference in the thmb box whose reference source matches the item ID of the main image found in step S51 as the item ID of the thumbnail image to be played back, and then the process ends.

[0158] FIG. 20 is a flowchart illustrating a second example of the process of acquiring the playback target item ID in step S31 of FIG.

[0159] That is, FIG. 20 shows an example of acquiring the item ID of the main image, which is the image to be reproduced, with the main image being the image to be reproduced.

[0160] In FIG. 20, for example, it is assumed that the user selects a thumbnail image (selected thumbnail image) from a list of thumbnail images, and file control unit 43 recognizes the item ID of the selected thumbnail image.

[0161] In step S61, the file control unit 43 searches the thmb boxes in the iref box of the collection file (FIGS. 9 to 12) for a thmb box whose reference matches the item ID of the selected thumbnail image, and the process proceeds to step S62.

[0162] In step S62, the file control unit 43 reads out the reference source in the thmb box whose reference destination matches the item ID of the selected thumbnail image found in step S61 as the item ID of the main image to be played back, and the process ends.

[0163] FIG. 21 is a flowchart illustrating an example of a process for acquiring the uuid as identification information of a RAW file of a predetermined main image from the first associative collection file of FIG.

[0164] In FIG. 21, for example, it is assumed that the file control unit 43 recognizes the item ID of a predetermined main image by a handle list or the like.

[0165] In step S71, the file control unit 43 searches for the item ID of a predetermined main image from the association information in the association information storage box of the first association-type collection file (FIG. 10), and the process proceeds to step S72.

[0166] In step S72, the file control unit 43 reads out the uuid associated with the item ID of the predetermined main image searched for in step S71 from the association information, and the process ends.

[0167] The file control unit 43 can access the RAW file of a predetermined main image using the uuid read out as described above.

[0168] FIG. 22 is a flowchart illustrating an example of a process for acquiring the uuid as identifying information of a RAW file of a predetermined main image from the second associative collection file of FIG.

[0169] In FIG. 22, for example, it is assumed that the file control unit 43 recognizes the item ID of a predetermined main image by a handle list or the like.

[0170] In step S81, the file control unit 43 searches the infe box in the iinf box of the second associative collection file (FIG. 11) for an infe box of item type IDIF indicating that the item is association information, and the process proceeds to step S82.

[0171] In step S82, the file control unit 43 reads out the item ID of the association information as an item from the infe box of the item type IDIF searched for in step S81, and the process proceeds to step S83.

[0172] In step S83, the file control unit 43 searches the iloc box of the second associative collection file for the item ID of the association information read out in step S82, and the process proceeds to step S84.

[0173] In step S84, the file control unit 43 reads out the offset and size associated with the item ID of the association information searched for in step S83 in the iloc box, and the process proceeds to step S85.

[0174] In step S85, the file control unit 43 reads out the association information as an item stored in the mdat box of the second association type collection file according to the offset and size associated with the item ID of the association information read in step S84, and the process proceeds to step S86.

[0175] In step S86, the file control unit 43 searches for the item ID of a predetermined main image from the association information read out in step S85, and the process proceeds to step S87.

[0176] In step S87, the file control unit 43 reads out the uuid associated with the item ID of the predetermined main image searched for in step S86 from the association information, and the process ends.

[0177] The file control unit 43 can access the RAW file of a predetermined main image using the uuid read out as described above.

[0178] FIG. 23 is a flowchart illustrating an example of a process for acquiring the uuid as identifying information of a RAW file of a predetermined main image from the third associative collection file of FIG.

[0179] In FIG. 23, for example, it is assumed that the file control unit 43 recognizes the item ID of a predetermined main image by a handle list or the like.

[0180] In step S91, the file control unit 43 searches for a cdsc box whose reference source matches the item ID of a predetermined main image from the cdsc boxes in the iref box of the third associative collection file (FIG. 12), and the process proceeds to step S92.

[0181] In step S92, the file control unit 43 reads out the reference in the cdsc box searched for in step S91, whose reference source matches the item ID of the specified main image, as the item ID of the specific information of the RAW file of the specified main image as an item, and the processing proceeds to step S93.

[0182] In step S93, the file control unit 43 searches the iloc box of the third associative collection file for the item ID of the specific information as the item read out in step S92, and the process proceeds to step S94.

[0183] In step S94, the file control unit 43 reads out the offset and size associated with the item ID of the specific information searched for in step S93 in the iloc box, and the process proceeds to step S95.

[0184] In step S95, the file control unit 43 reads out the uuid as specific information of the RAW file of a specified main image stored in the mdat box of the third associative collection file, according to the offset and size associated with the item ID of the specific information read in step S94, and the processing ends.

[0185] The file control unit 43 can access the RAW file of a predetermined main image using the uuid read out as described above.

[0186] FIG. 24 is a flowchart illustrating an example of a process for acquiring a list of item IDs of main images from a collection file.

[0187] The process of acquiring the list of item IDs of main images from the collection file is performed, for example, when generating a handle list.

[0188] In step S101, the file control unit 43 reads out item IDs from all infe boxes in the iinf box of the collection file (Figures 9 to 12) and registers them in a list of item IDs of main images (hereinafter also referred to as the main image list), and the process proceeds to step S102.

[0189] In step S102, the file control unit 43 reads out the item IDs that are referenced from all boxes in the iref box of the collection file, and removes them from the main image list, and the process ends.

[0190] After the above processing, the item ID registered in the main image list becomes the item ID of the main image.

[0191] FIG. 25 is a flowchart illustrating an example of a process for reproducing a thumbnail image of a main image (a frame thereof) corresponding to predetermined time information from a sequence file.

[0192] In FIG. 25, for example, it is assumed that the file control unit 43 recognizes the time information (or order) of a predetermined main image by a handle list or the like.

[0193] In step S111, the file control unit 43 searches the trak boxes in the moov box of the sequence file (Figures 13 and 14) for a trak box in which information indicating that the data constituting the track is a thumbnail image is stored in the tref box, that is, a trak box in which the type in the tref box is thmb, as the trak box that manages the track of the thumbnail image of the main image for the specified time information, and the processing proceeds to step S112.

[0194] In step S112, the file control unit 43 reads out the track ID in the tkhd box in the trak box searched for in step S111 as the track ID of the track of the thumbnail image of the main image for the specified time information, and the process proceeds to step S113.

[0195] In step S113, the file control unit 43 plays the track with the track ID read out in step S112, and obtains from that track a thumbnail image (frame) for the specified time information (or sequence) as a thumbnail image of the main image for the specified time information, and then the processing ends.

[0196] The process of playing back image tracks stored in a sequence file is similar to the process of playing back video from an MP4 file.

[0197] FIG. 26 is a flowchart illustrating an example of a process for acquiring the uuid as identification information of the RAW file of a predetermined main image (frame) from an associative sequence file.

[0198] In FIG. 26, for example, it is assumed that the file control unit 43 recognizes the time information (or order) of a predetermined main image by a handle list or the like.

[0199] In step S121, the file control unit 43 searches the trak boxes in the moov box of the associative sequence file (Figure 14) for a trak box in which information indicating that the data constituting a track is specific information is stored in the tref box, that is, a trak box in which the type in the tref box is cdsc, as the trak box that manages the track of specific information, and the processing proceeds to step S122.

[0200] In step S122, the file control unit 43 reads out the track ID in the tkhd box in the trak box searched for in step S121 as the track ID of the specific information track, and the process proceeds to step S123.

[0201] In step S123, the file control unit 43 obtains the uuid as specific information for the time information (or order) of the specified main image from the track of the track ID read out in step S122, as the uuid of the RAW file of the specified main image, and the processing ends.

[0202] The file control unit 43 can access the RAW file of a predetermined main image using the uuid acquired as described above.

[0203] As described above, the file control unit 43 generates and plays an associated HEIF file in which a HEIF file conforming to HEIF is stored in association with a main image within the HEIF file and specific information identifying external data outside the HEIF file that is associated with the main image, so that the main image stored in the HEIF file can be associated with external data outside the HEIF file.

[0204] Furthermore, when a uuid is used as the identification information, even if the file name of the external data is changed, the uuid can maintain the association between the main image in the HEIF file and the external data after the file name is changed.

[0205] <Storage of specific information assigned to external data>

[0206] FIG. 27 is a diagram showing an example of storing uuid in a RAW file when a RAW file of a main image is used as external data (a file in which the external data is stored) and an associative collection file is generated.

[0207] In FIG. 27, the first associative collection file is used as the associative collection file.

[0208] A RAW file has an area called a MakerNote as part of an area for storing Exif attribute information as metadata.

[0209] The file control unit 43 can store the uuid assigned to a RAW file (RAW image) in, for example, a marker note of the RAW file.

[0210] 27, main images Item#1, Item#2, Item#3, and Item#4 are stored as four items in the association collection file, and RAW files #1, #2, #3, and #4 are generated in which RAW images of the main images Item#1, Item#2, Item#3, and Item#4 are stored. A UUID#i is assigned to the RAW file #i (RAW image), and as association information associating the main image Item#i with the UUID#i of the RAW file #i of the main image Item#i, association information in which an item ID#i identifying the main image Item#i is associated with the UUID#i of the RAW file #i associated with the main image Item#i is stored in an association information storage box.

[0211] FIG. 28 is a diagram showing an example of storing uuid in a RAW file when a RAW file of a main image is used as external data and an associated sequence file is generated.

[0212] When generating an associated sequence file, the file control unit 43 can store the uuid assigned to the RAW file in the marker note of the RAW file, just as when generating the associated collection file described in Figure 27.

[0213] 28, track #1 consisting of four frames of main images #1, #2, #3, #4 is stored in the associative sequence file, and RAW files #1, #2, #3, #4 are generated in which RAW images of main images #1, #2, #3, #4 are stored. UUID #i is assigned to RAW file #i, and track #3 is stored in the associative sequence file, in which UUID #i of RAW file #i is arranged to have the same time information as main image #i corresponding to RAW file #i (RAW image).

[0214] As described above, track #3 is constructed by arranging the UUID #i of RAW file #i so that it has the same time information as the main image #i corresponding to RAW file #i, and the i-th main image #i of track #1 and the i-th UUID #i of track #3, i.e., the UUID #i of the RAW file #i of main image #i, are associated and stored in an associated sequence file.

[0215] In the above, a RAW file (RAW image) of the main image is used as the external data, but other data can be used as the external data. For example, audio (sound) recorded together with capturing the main image can be used as the external data. For example, a WAV file in WAV format or an MP4 file in MP4 format can be used as the file storing the audio. In the following, for example, a WAV file will be used as the file storing the audio.

[0216] FIG. 29 is a diagram showing an example of storing uuid in a WAV file when a WAV file of a main image is adopted as external data (a file in which the external data is stored) and an associative collection file is generated.

[0217] In FIG. 29, the first associative collection file is used as the associative collection file.

[0218] A WAV file has an area called a List chunk as a part of the area in which metadata is written.

[0219] The file control unit 43 can store the uuid assigned to a WAV file (audio) in, for example, a List chunk of the WAV file.

[0220] 29, main images Item#1, Item#2, Item#3, and Item#4 are stored as four items in the associated collection file, and WAV files #1, #2, #3, and #4 for the main images Item#1, Item#2, Item#3, and Item#4 are generated. A UUID#i is assigned to the WAV file #i (audio), and as association information associating the main image Item#i with the UUID#i of the WAV file #i of the main image Item#i, association information in which the item ID#i identifying the main image Item#i corresponds to the UUID#i of the WAV file #i associated with the main image Item#i is stored in the association information storage box.

[0221] FIG. 30 is a diagram showing an example of storing uuid in a WAV file when a WAV file of a main image is used as external data and an associated sequence file is generated.

[0222] When generating an associative sequence file, the file control unit 43 can store the uuid assigned to a WAV file in the List chunk of the WAV file, just as when generating an associative collection file as described with reference to FIG. 29.

[0223] In Fig. 30, track #1 consisting of four frames of main images #1, #2, #3, #4 is stored in the associative sequence file, and WAV files #1, #2, #3, #4 of main images #1, #2, #3, #4 are generated. UUID #i is assigned to WAV file #i, and track #3 is stored in the associative sequence file, which is configured by arranging the UUID #i of WAV file #i so as to have the same time information as the main image #i corresponding to WAV file #i.

[0224] As described above, track #3 is constructed by arranging the UUID #i of WAV file #i so that it has the same time information as the main image #i corresponding to WAV file #i, and the i-th main image #i of track #1 and the i-th UUID #i of track #3, i.e., the UUID #i of WAV file #i of main image #i, are associated and stored in the associated sequence file.

[0225] In addition to HEIF files, this technology can also be applied to files other than HEIF files that have a box structure, such as ISO-based media files, MP4 files, and Miaf files.

[0226] Furthermore, the present technology can also be applied to, for example, a file that does not have a box structure and stores an image (main image) and another image that is the main image with a lowered resolution.

[0227] Furthermore, this technology can be applied to associating external data with the main image in a HEIF file, as well as with screennail and thumbnail images in a HEIF file.

[0228] In addition, the present technology can also be applied, for example, when associating external data with internal data other than an image such as a main image in a HEIF file.

[0229] <Association between an image as internal data after generation of a HEIF file and external data>

[0230] FIG. 31 is a diagram for explaining the association between an image as internal data after generation of a HEIF file and external data.

[0231] When generating a HEIF file, if there is external data to be associated with an image (main image) as internal data stored in the HEIF file, an associated-type HEIF file can be generated by associating the image as internal data with specific information of the external data and storing it in the HEIF file.

[0232] However, at the time of generating a HEIF file, if there was no external data to be associated with the image as internal data, but after generation of the HEIF file, external data is generated and it is desired to associate the image as internal data with the external data, it is necessary to store relationship information related to the association between the image as internal data and specific information of the external data in the HEIF file after generation of the HEIF file to generate an associated-type HEIF file. Also, even when external data exists at the time of generating a HEIF file, there may be a case where it becomes desired to associate the image as internal data with the already existing external data after generation of the HEIF file. In this case as well, it is necessary to store the relationship information in the HEIF file after generation of the HEIF file to generate an associated-type HEIF file.

[0233] After the HEIF file is generated, external data is generated, and an image serving as internal data may be associated with the external data, for example, when a comment in the form of audio or text is added as external data to an image captured by the digital camera 10. After the HEIF file is generated, an image serving as internal data may be associated with existing external data, for example, when background music (BGM) is added as external data to an image captured by the digital camera 10.

[0234] Here, as described above, the relationship information is information related to the association between an image as internal data and specific information of external data. For example, the relationship information is the relationship information stored in the association information storage box of the first associative collection file (FIG. 10). Further, the relationship information is, for example, the association information as the item Item#201 stored in the mdat box of the second associative collection file (FIG. 11), the infe box for the item Item#201 stored in the iinf box of the meta box, and the offset to the storage location of the item Item#201 stored in the iloc box of the meta box. Further, the relationship information is, for example, the specific information as an item stored in the mdat box of the third associative collection file (FIG. 12), and the association information that associates the item ID of the main image as an item stored in the cdsc box of the iref box of the meta box with the item ID of the specific information as an item. Furthermore, the related information includes, for example, track #3 of specific information stored in the mdat box of the associated sequence file (Figure 14), and a trak box that manages track #3 of specific information stored in the moov box.

[0235] As described above, when associating an image as internal data with external data after generating a HEIF file, it is necessary to store the relationship information in the HEIF file. In this case, if no measures are taken to store the relationship information, the offset of the data stored in the mdat box may shift when storing the relationship information in the HEIF file.

[0236] If the offset of the data stored in the mdat box is shifted, the amount of offset shift must be calculated for each piece of data stored in the mdat box, and the iloc box must be rewritten with an offset that reflects that shift, which increases the load of the association process that associates the image as internal data with external data.

[0237] Methods for preventing the offset shift of the data stored in the mdat box as described above, and ultimately the load of the association process from becoming too large, include, for example, a method of generating an HEIF file as an associated HEIF file in which a reserved area is secured in which relationship information relating to the association between internal data and specific information identifying external data is stored (hereinafter also referred to as the area allocation method), and a method of generating an HEIF file as an associated HEIF file in which relationship information including specific information before it is assigned to external data is stored (hereinafter also referred to as the pre-storage method).

[0238] In the case where external data is generated after the generation of a HEIF file, when using information generated when generating the external data (file), such as a URL or file name, as identifying information, it is not possible to obtain the identifying information when the HEIF file is generated, so that it is possible to employ an area allocation method for generating an associated HEIF file in which a reserved area for storing related information is secured so that related information including any identifying information can be stored in the future. Examples of identifying information generated when the external data is generated include, in addition to the URL and file name, a hash value generated using the external data as input, and, if the external data is data of a track stored in an MP4 file, the track ID (track number) of that track.

[0239] Furthermore, when using information that can be generated before the generation of external data, such as a UUID, as the identifying information, a pre-storage method can be adopted in which the identifying information before being assigned to the external data is generated in advance (before being assigned to the external data) when the HEIF file is generated, and an associated HEIF file is generated in which related information including the identifying information is stored. It can also be said that the identifying information before being assigned to the external data is identifying information that is not assigned to the external data at the time the associated HEIF file is generated. In addition, in cases where the pre-storage method can be adopted, the area allocation method can also be adopted.

[0240] By generating an associated HEIF file using the area allocation method or pre-storage method, it is possible to prevent the offset of the data stored in the mdat box from shifting when performing association processing after the HEIF file is generated. As a result, it is possible to prevent the load of the association processing from becoming too large and to easily associate images as internal data with external data.

[0241] The area allocation method and the pre-storage method will be described below.

[0242] FIG. 32 is a diagram for explaining an outline of an example of an area allocation method.

[0243] In FIG. 32, the first associative collection file is generated by the area allocation method.

[0244] The file control unit 43 generates a first association-type collection file similar to that in Fig. 10. In Fig. 32, a first association-type collection file is generated in which one frame of a main image and a thumbnail image of the main image are stored in the mdat box. The item IDs of the main image and the thumbnail image are item ID#1 and item ID#1001, respectively.

[0245] However, in the area allocation method, the association information storage box does not store the association information, but instead allocates a padding area as a reserve area for writing (storing (overwriting)) the association information as related information in the future.

[0246] In Fig. 32, the associating information storage box is provided with a number of pieces of associating information in addition to a reserved area. The number of pieces of associating information (here, each piece of information associating one frame of a main image with one piece of specific information) stored in the associating information storage box. In the area allocation method, when the first associating collection file is generated, the number of pieces of associating information is set to 0 as an initial value, and is updated (rewritten) to the number of pieces of associating information stored in the associating information storage box in response to writing of the associating information to the reserved area of ​​the associating information storage box.

[0247] Figure 33 is a diagram illustrating an example of an association process that associates a main image as internal data in a first associative collection file generated by an area allocation method with external data after the first associative collection file is generated.

[0248] In Figure 33, after generating the first associative collection file of Figure 32, an association process is performed to associate the main image (main image Item#1) of item ID#1 in the first associative collection file with a WAV file (audio stored in it) as external data.

[0249] In the association process, the file control unit 43 acquires (recognizes) the file name (DSC00001.WAV) as identifying information for the WAV file (the audio stored in the WAV file) to be associated with the main image Item#1 in the first association-type collection file, and generates association information that associates the file name with the item ID#1 of the main image Item#1.The file control unit 43 then overwrites (a part of) the padding area in the association information storage box with the association information generated for the WAV file, and updates the number of pieces of association information to a value incremented by 1 from the current value (here, 1).

[0250] FIG. 34 is a diagram outlining an example of the pre-storing method.

[0251] In FIG. 32, it is assumed that the first associative collection file is generated by the pre-storing method.

[0252] The file control unit 43 generates a first association-type collection file similar to that in Fig. 10. In Fig. 34, similar to Fig. 32, a first association-type collection file is generated in which one frame of a main image Item#1 and a thumbnail image Item#1001 of the main image Item#1 are stored in the mdat box.

[0253] However, in this case, the external data to be associated with the main image Item#1 in the first associative collection file has not yet been determined. Therefore, the file control unit 43 generates an associative HEIF file that stores relationship information including identification information before it is assigned to the external data, which identifies the external data to be associated with the main image Item#1 in the future for (all or some of) the main image Item#1 in the first associative collection file.

[0254] Specifically, the file control unit 43 generates in advance UUID#1, which is a UUID serving as specific information for identifying external data to be associated with the main image Item#1 in the future, i.e., generates the UUID#1 in a state in which the external data to which the UUID#1 is to be assigned has not yet been determined, and generates association information that associates the UUID#1 with the item ID#1 of the main image Item#1.The file control unit 43 then generates a first association-type collection file provided with an association information storage box that stores the association information that associates the UUID#1 with the item ID#1 of the main image Item#1, and the number of pieces of association information that indicates the number of pieces of association information.

[0255] In FIG. 34, one piece of association information that associates the main image Item#1 with the UUID#1 is stored in the association information storage box, so the number of pieces of association information is one.

[0256] Figure 35 is a diagram illustrating an example of an association process in which a main image as internal data in a first associative collection file generated by the pre-storage method is associated with external data after the first associative collection file is generated.

[0257] In Figure 35, after the first associative collection file of Figure 34 is generated, an association process is performed to associate the main image Item#1 in the first associative collection file with a WAV file (audio) as external data.

[0258] In the association process, the file control unit 43 assigns UUID#1, which is generated in advance for the main image Item#1 in the first association-type collection file and serves as specific information for the external data associated with the main image Item#1, to the WAV file (audio) associated with the main image Item#1, and stores (writes) the UUID#1 assigned to the WAV file in, for example, the List chunk of the WAV file. This maintains (stores) the assignment of UUID#1 to the WAV file.

[0259] In addition, if the specific information included in the association information is assigned to the external data and is not assigned, it means that the association information is valid and invalid, respectively. In the pre-storage method, when the first association type collection file is generated, all the association information including the specific information generated in advance is invalid. Then, the specific information not assigned to the external data is assigned to the external data, and the association information including the specific information becomes valid. In the pre-storage method, information indicating whether the association information is valid or invalid can be further stored in the association information storage box.

[0260] FIG. 36 is a diagram outlining another example of the pre-storing method.

[0261] In FIG. 36, the first associative collection file is generated by the pre-storing method, similarly to FIG.

[0262] The file control unit 43 generates a first association-type collection file similar to that in Fig. 10. In Fig. 36, a first association-type collection file is generated in which four frames of main images Item#1 to Item#4 and thumbnail images Item#1001 to Item#1004 of the four frames of main images Item#1 to Item#4 are stored in the mdat box.

[0263] However, when the first associative collection file is generated, the external data to be associated with the main image Item#i is not yet determined. Therefore, the file control unit 43 generates in advance, for each main image Item#i in the first associative collection file, a UUID#i as identification information before being assigned to the external data, which identifies the external data to be associated with the main image Item#i in the future, and generates association information in which the UUID#i is associated with the item ID#i of the main image Item#i. Then, the file control unit 43 generates the first associative collection file provided with an association information storage box in which association information in which the UUID#i is associated with the item ID#i of the main image Item#i and the number of pieces of association information representing the number of pieces of association information are stored. In FIG. 36, four pieces of association information are stored in the association information storage box according to the number of main images Item#1 to Item#4, and the number of pieces of association information is four.

[0264] In the association process for associating the main images Item#1 to Item#4 in the first associative collection file with the WAV files #1 to #4 as external data, the file control unit 43 assigns the UUID#i, which was generated in advance for the main image Item#i in the first associative collection file, as specific information for the WAV file associated with the main image Item#i, to the WAV file #i associated with the main image Item#i, and stores the UUID#i assigned to the WAV file #i in, for example, the List chunk of the WAV file #i. This maintains the assignment of the UUID#i to the WAV file #i.

[0265] FIG. 37 is a diagram outlining yet another example of the pre-storing method.

[0266] In FIG. 37, it is assumed that an associated sequence file is generated by the pre-storage method.

[0267] The file control unit 43 generates an associated sequence file similar to that in FIG.

[0268] In Figure 37, track #1 consisting of four frames of main images and track #2 consisting of thumbnail images of those four frames of main images are stored in the mdat box, and an associated collection file is generated in which trak boxes that manage tracks #1 and #2 are stored in the moov box.

[0269] However, in FIG. 37, the external data associated with the four main images is not determined when the associated sequence file is generated. Then, for each of the four main images in the associated sequence file, a UUID#i is generated in advance as specific information before being assigned to the external data, which specifies the external data to be associated with the main image in the future, and a track#3 consisting of the UUID#i is stored in the mdat box, and an associated sequence file is generated in which a trak box for managing the track#3 is stored in the moov box. In FIG. 37, UUID#i is adopted as specific information of the external data associated with the i-th main image of the four main images. Note that in the associated sequence file, the mdat box can store multiple tracks consisting of main images (main image tracks) and multiple tracks consisting of specific information (specific information tracks). In the associative sequence file, when multiple main image tracks and multiple specific information tracks are stored in the mdat box, information that associates the main image track with the specific information track having the specific information before being assigned to the external data associated with the main image of the main image track is stored in the associative sequence file. The information that associates the main image track with the specific information track can be stored in the moov box of the associative sequence file, for example. This is also the case when performing the association process after the generation of the associative sequence file generated by the area allocation method.

[0270] In the association process for associating the i-th main image of track #1 in the association sequence file with the WAV file #i (audio) as external data after the association sequence file is generated, the file control unit 43 assigns the UUID#i, which is generated in advance for the i-th main image of track #1 in the association sequence file as specific information for the WAV file associated with the i-th main image, to the WAV file #i associated with the i-th main image, and stores the UUID#i assigned to the WAV file #i in, for example, the List chunk of the WAV file #i. This maintains the assignment of UUID#i to the WAV file #i.

[0271] FIG. 38 is a diagram for explaining usable specific information and association between internal data and external data for each of the area allocation method and prestorage method.

[0272] Regarding the available specific information, in the area allocation method, any information can be written in the reserved area, so any specific information can be used. For example, a hash value generated by inputting external data, or information assigned to the external data after the external data is generated (such as a URL, a file name, or the track ID of the MP4 file in which the external data is stored) can be used as specific information.

[0273] However, when using specific information such as UUID that needs to be written to an external data file (a file in which external data is stored), the external data file is limited to a file format that can write (store) separately generated specific information (that can maintain the allocation of specific information).

[0274] When specific information that does not need to be written to a file of external data, such as a URL, is employed, any external data (file) can be employed.

[0275] On the other hand, in the pre-storage method, specific information is generated in advance (before the internal data and external data are associated), so the external data files to be associated with the internal data are limited to files that can write the specific information generated in advance (RAW files, WAV files, MP4 files, HEIF files, etc.).

[0276] Furthermore, it is not possible to employ specific information (such as a hash value generated using external data as input) that can be generated after the external data to be associated with the internal data has been determined.

[0277] In both the space allocation method and the pre-storage method, the external data may be data generated before the associated HEIF file is generated, or data generated after the associated HEIF file is generated.

[0278] Regarding 1-to-N associations that associate one internal datum with one or more external datums, in the area allocation method, when different information is used for each external datum as the specific information, such as a hash value that uses the external datum as input, the amount of data in the specific information increases in proportion to the number of external datums that are associated with one internal datum, and therefore the number of external datums that can be associated with one internal datum is limited according to the capacity of the reserved area.

[0279] On the other hand, in the pre-storage method, specific information generated in advance for a single piece of internal data is written to the file in which each piece of external data to be associated with that piece of internal data is stored, so there is no limit to the number of external data to be associated with a single piece of internal data.

[0280] For N-to-1 associations that associate one or more internal data with one external data, in the area allocation method, the amount of data of specific information increases in proportion to the number of internal data associated with one external data, so the number of external data associated with one external data is limited according to the capacity of the reserved area.

[0281] On the other hand, in the pre-storage method, when the number of specific information pieces that can be written to a file in which a single piece of external data is stored is limited, the number of internal data pieces to be associated with that single piece of external data is limited according to the number of specific information pieces that can be written to a file in which a single piece of external data is stored.

[0282] FIG. 39 is a diagram for explaining an example of an association process for performing one-to-N association for the first association-type collection file generated by the area allocation method.

[0283] The file control unit 43 generates a first associative collection file in which the padding area described with reference to FIG. 32 is secured by the area securing method, for example.

[0284] In FIG. 39, similarly to FIG. 32, a first associative collection file is generated in which one frame of a main image Item#1 and a thumbnail image Item#1001 of that main image Item#1 are stored in the mdat box.

[0285] After the first associative collection file is generated, when a one-to-two associative process is requested to associate the main image Item#1 in the first associative collection file with two WAV files #1 and #2 (audio stored in them) as external data, the file control unit 43 acquires file names as specific information for each of the WAV files #1 and #2 to be associated with the main image Item#1 in the first associative collection file, and generates associative information that associates the file names with the item ID#1 of the main image Item#1. The file names of the WAV files #1 and #2 are DSC00001.WAV and DSC00002.WAV, respectively, and therefore, as the associative information, information that associates the item ID#1 with the file name DSC00001.WAV and information that associates the item ID#1 with the file name DSC00002.WAV are generated. Then, the file control unit 43 overwrites the associated information generated for the WAV files #1 and #2 in the padding area in the associated information storage box, and updates the number of pieces of associated information from the current value to a value (here, 2) incremented by the number of pieces of associated information overwritten in the padding area, that is, 2. This makes it possible to associate the main image Item#1 with the WAV files #1 and #2.

[0286] For example, if a different file name is adopted as the specific information for each WAV file as external data, the amount of data for the specific information, and ultimately the association information that matches the specific information with item ID#1 of main image Item#1, increases in proportion to the number of WAV files associated with main image Item#1 as a single piece of internal data.Therefore, the number of WAV files (external data) associated with main image Item#1 as a single piece of internal data is limited according to the capacity of the padding area as a reserved area.

[0287] FIG. 40 is a diagram for explaining an example of an association process for performing 1:N association for the first association-type collection file generated by the pre-storing method.

[0288] The file control unit 43 generates, for example, the first association-type collection file described with reference to FIG. 34 by the pre-storing method.

[0289] In Fig. 40, a first association-type collection file is generated in which one frame of the main image Item#1 and a thumbnail image Item#1001 of the main image Item#1 are stored in an mdat box, similar to Fig. 34. Furthermore, in Fig. 40, a UUID#1 is generated in advance as specific information for identifying external data to be associated with the main image Item#1 in the future, similar to Fig. 34, and association information is generated in which the UUID#1 is associated with the item ID#1 of the main image Item#1. Then, a first association-type collection file is generated in which an association information storage box is provided in which association information in which the UUID#1 is associated with the item ID#1 of the main image Item#1 and the number of pieces of association information representing the number of pieces of association information are stored.

[0290] After the first associative collection file is generated, if a one-to-two associating process is requested to associate the main image Item#1 in the first associative collection file with two WAV files #1 and #2 as external data, the file control unit 43 assigns UUID#1, which is generated in advance for the main image Item#1 in the first associative collection file, as specific information of the external data associated with the main image Item#1, to the WAV files #1 and #2 associated with the main image Item#1, and stores (writes) the UUID#1 assigned to the WAV files #1 and #2 in the List chunks of the WAV files #1 and #2. This allows the main image Item#1 to be associated with the WAV files #1 and #2 (the audio stored in them).

[0291] As described above, for the first associated collection file generated by the pre-storage method, by writing UUID#1 as specific information generated in advance for the main image Item#1 to the WAV file as external data, and maintaining the assignment of UUID#1 to the WAV file, any number of WAV files as external data can be associated with the main image Item#1.

[0292] FIG. 41 is a diagram for explaining an example of an association process for performing N-to-1 association for the first association-type collection file generated by the area allocation method.

[0293] The file control unit 43 generates a first associative collection file in which the padding area described with reference to FIG. 32 is secured by the area securing method, for example.

[0294] In Figure 41, a first associative collection file has been generated in which two frames of main images Item#1 and Item#2 and thumbnail images Item#1001 and Item#1002 of the main images Item#1 and Item#2 are stored in the mdat box.

[0295] After the first associative collection file is generated, if a two-to-one associative process is requested to associate each of the main images Item#1 and Item#2 in the first associative collection file with one WAV file#1 (audio stored in it) as external data, the file control unit 43 acquires a file name as specific information for the WAV file#1 to be associated with the main images Item#1 and Item#2 in the first associative collection file, and generates associative information that associates the file name with the item ID#1 of the main image Item#1 and the item ID#2 of the main image Item#2. If the file name of the WAV file#1 is DSC00001.WAV, two pieces of associative information are generated in total: one piece of associative information that associates the item ID#1 with the file name DSC00001.WAV, and one piece of associative information that associates the item ID#2 with the file name DSC00001.WAV. Then, the file control unit 43 overwrites the two pieces of association information generated for WAV file #1 into the padding area in the association information storage box, and updates the number of pieces of association information from the current value to a value (here, 2) incremented by the number of pieces of association information overwritten in the padding area, that is, 2. This makes it possible to associate the main image Item#1 with (the audio stored in) WAV file #1, and also to associate the main image Item#2 with WAV file #1.

[0296] For associated HEIF files generated using the area allocation method, the amount of data in the association information that matches the item ID of the main image with the specific information of WAV file #i increases in proportion to the number of main images associated with a single WAV file #1 as external data, so the number of main images associated with a single WAV file #1 as external data is limited according to the capacity of the padding area as a reserved area.

[0297] FIG. 42 is a diagram for explaining an example of an association process for performing N-to-1 association for the first association-type collection file generated by the pre-storing method.

[0298] The file control unit 43 generates, for example, the first association-type collection file described with reference to FIG. 34 by the pre-storing method.

[0299] In Fig. 42, a first association type collection file is generated in which two frames of main images Item#1 and Item#2 and thumbnail images Item#1001 and Item#1002 of the main images Item#1 and Item#2 are stored in an mdat box. Furthermore, in Fig. 42, similar to Fig. 34, UUID#1 and UUID#2 are generated in advance as specific information for identifying external data to be associated with the main images Item#1 and Item#2, respectively, and two pieces of association information are generated: association information that associates UUID#1 with the item ID#1 of the main image Item#1, and association information that associates UUID#2 with the item ID#2 of the main image Item#2. Then, a first association type collection file is generated in which an association information storage box is provided in which the two pieces of association information and the number of pieces of association information representing the number of the two pieces of association information are stored.

[0300] After the first associative collection file is generated, if a two-to-one associative process is requested to associate each of the main images Item#1 and Item#2 in the first associative collection file with one WAV file#1 as external data, the file control unit 43 assigns UUID#1, which is generated in advance for the main image Item#1 in the first associative collection file as identifying information for the external data associated with the main image Item#1, and UUID#2, which is generated in advance for the main image Item#2 as identifying information for the external data associated with the main image Item#2, to the WAV file#1 (audio) associated with the main image Item#1 and iterm#2, and stores the UUID#1 and UUID#2 assigned to the WAV file#1 in the List chunk of the WAV file#1. This allows the main images Item#1 and Item#2 to be associated with the WAV file#1.

[0301] As described above, for the first associated collection file generated by the pre-storage method, a uuid is written as specific information generated in advance for the main image into the WAV file (in which audio is stored) as external data, so if the amount of specific information that can be written into one WAV file is limited, the number of main images that can be associated with that one WAV file (the audio stored in it) is limited according to the amount of specific information that can be written into one WAV file.

[0302] <When using MP4 files as files to store external data>

[0303] FIG. 43 is a diagram showing a first example of association between a main image as internal data and audio as external data in a case where an MP4 file is used as a file for storing audio as external data.

[0304] Here, in a WAV file, one (series) of audio is stored. In contrast, in an MP4 file consisting of a moov box, an mdat box, etc., one audio can be treated as one track, making it a multi-track, that is, multiple audios can be stored together. Therefore, when storing audio as external data in an MP4 file, when associating a main image stored in a HEIF file capable of storing one or more images (frames) with multiple audios as external data, storing the multiple audios as external data in the MP4 file in multiple tracks can be used to realize a one-to-one correspondence relationship, in other words, a correspondence between one HEIF file and one MP4 file on a file-by-file basis.

[0305] When associating a main image as internal data with multiple audios as external data, storing the multiple audios as external data in an MP4 file makes it convenient to handle the multiple audios together. For example, when associating a main image in an associated HEIF file with multiple audios, if the audios are stored in a WAV file, a number of WAV files equal to the number of audios is required. As a result, when transmitting an associated main image and multiple audios, it is necessary to transmit an associated HEIF file in which the main image is stored and multiple WAV files in which the audios are stored.

[0306] In contrast, when associating a main image and multiple audios in an associated HEIF file, if the audios are stored in an MP4 file, the multiple audios can be stored in one MP4 file. As a result, when transmitting an associated main image and multiple audios, it is sufficient to transmit one associated HEIF file in which the main image is stored, and one MP4 file in which the audios are stored.

[0307] The multiple voices serving as external data may be, for example, voices in different languages ​​with the same content.

[0308] FIG. 43 shows an example of an association between one main image stored in a first associative collection file and one audio stored in an MP4 file.

[0309] The first association type collection file is generated, for example, by a pre-storage method. In the first association type collection file in Fig. 43, one (one frame) main image Item#1 and a thumbnail image Item#1001 of the main image Item#1 are stored in the mdat box. Furthermore, in Fig. 43, similar to Fig. 34, UUID#1 is generated in advance as specific information for identifying external data to be associated with the main image Item#1 in the future, association information that associates the UUID#1 with the item ID#1 of the main image Item#1 is generated, and the association information and the number of pieces of association information representing the number of pieces of association information are stored in the association information storage box.

[0310] On the other hand, in an MP4 file in which one audio #1 associated with a main image Item#1 is stored, track #1 of audio #1 is stored in the mdat box, and a trak box that manages track #1 is stored in the moov box. Here, the trak box having a tkhd box with track ID set to i is the trak box that manages track #i.

[0311] If, after the generation of a first associative collection file, an association process is requested to associate the main image Item#1 in the first associative collection file with the audio #1 in the MP4 file, in the association process, the file control unit 43 assigns UUID#1, which was generated in advance for the main image Item#1 in the first associative collection file as specific information of external data to be associated with the main image Item#1, to the audio #1 to be associated with the main image Item#1, and stores (writes) the UUID#1 assigned to the audio #1 in the MP4 file in which the audio #1 is stored.

[0312] That is, the file control unit 43 generates track #2 of UUID #1 for the MP4 file and stores it in the mdat box, and also generates a trak box for managing track #2 and stores it in the moov box.

[0313] The trak box that manages track #2 has a tkhd box and a tref box. In the tkhd box of the trak box that manages track #2, the track ID that identifies the track #2 to be managed is set to 2.

[0314] Furthermore, in the tref box of the trak box that manages track #2, it is set that the other track associated with track #2 is track #1 (track_ID=1), and that track #2 is a metadata (here, specific information) track (type=cdsc).

[0315] As described above, by storing UUID#1, which is pre-generated specific information for the main image Item#1, in the MP4 file in which audio #1 is stored, the main image Item#1 and audio #1 can be associated with each other.

[0316] FIG. 44 is a diagram showing a second example of association between a main image as internal data and audio as external data in a case where an MP4 file is used as a file for storing audio as external data.

[0317] FIG. 44 shows an example of association between four main images stored in a first associative collection file and four audios stored in an MP4 file.

[0318] The first association type collection file is generated, for example, by a pre-storage method. In the first association type collection file in FIG. 44, four main images Item#1 to Item#4 and thumbnail images Item#1001 to Item#1004 of the main images Item#1 to Item#4 are stored in the mdat box. Furthermore, in FIG. 44, similar to FIG. 34, UUID#i is generated in advance as specific information for identifying external data to be associated with the main image Item#i (here, i=1, 2, 3, 4) in the future, and four pieces of association information are generated by associating UUID#i with the item ID#i of the main image Item#i. Then, the number of pieces of association information representing the number of the four pieces of association information is stored in the association information storage box.

[0319] On the other hand, in the MP4 file in which audio #i to be associated with main image Item#i is stored, track #2i-1 for each of the four audio #i (i=1, 2, 3, 4) is stored in the mdat box, and a trak box that manages track #2i-1 (a trak box having a tkhd box with track ID set to 2i-1) is stored in the moov box.

[0320] If, after the generation of a first associative collection file, an association process is requested to associate the main image Item#i in the first associative collection file with the audio #i in the MP4 file, in the association process, the file control unit 43 assigns a UUID#i, which was generated in advance for the main image Item#i in the first associative collection file as specific information of external data to be associated with the main image Item#i, to the audio #i to be associated with the main image Item#i, and stores the UUID#i assigned to the audio #i in the MP4 file in which the audio #i is stored.

[0321] That is, the file control unit 43 generates track #2i of UUID #i for the MP4 file and stores it in the mdat box, and also generates a trak box for managing track #2i and stores it in the moov box.

[0322] The trak box that manages track #2i has a tkhd box and a tref box. In the tkhd box of the trak box that manages track #2i, the track ID that identifies the track #2i to be managed is set to 2i.

[0323] Furthermore, in the tref box of the trak box that manages track #2i, it is set that the other track associated with track #2i is track #2i-1 (track_ID=2i-1), and that track #2i is a metadata (here, specific information) track (type=cdsc).

[0324] As described above, by storing UUID#i, which is generated in advance as specific information for the main image Item#1, in the MP4 file in which the audio #i is stored, the main image Item#i and the audio #i can be associated with each other.

[0325] FIG. 45 is a diagram showing a third example of association between a main image as internal data and audio as external data in a case where an MP4 file is used as a file for storing audio as external data.

[0326] FIG. 45 shows an example of association between four main images stored in an associative sequence file and four audios stored in an MP4 file.

[0327] The associative sequence file is generated, for example, by a pre-storage method. In the associative sequence file of Fig. 45, track #1 consisting of four main images and track #2 consisting of thumbnail images of each of the four main images are stored in the mdat box.

[0328] Furthermore, in Fig. 45, similarly to Fig. 37, UUIDs #1, #2, #3, and #4 are generated in advance as specific information for identifying external data to be associated with the first to fourth main images of track #1 in the future, and track #3 is generated in which the UUIDs #1 to #4 are arranged in the same order as the corresponding four main images. Then, track #3 consisting of UUIDs #1 to #4 is stored in the mdat box.

[0329] 45, the moov box stores trak boxes that manage tracks #1 to #3 stored in the mdat box. The trak boxes that manage tracks #1 to #3 each have a tkhd box in which the track ID of the track that the trak box manages is set.

[0330] The trak boxes that manage tracks #2 and #3 each further have a tref box. In the tref box of the trak box that manages track #2, it is set that another track related to track #2 is track #1 (track_ID=1) and that track #2 is a thumbnail image track (type=thmb). In the tref box of the trak box that manages track #3, it is set that another track related to track #3 is track #1 (track_ID=1) and that track #3 is a metadata (here, specific information) track (type=cdsc).

[0331] On the other hand, the MP4 file in which the four audios #1 to #4 associated with the four main images in the associative sequence file are stored is configured in the same manner as in the case of FIG.

[0332] If, after the generation of an associated sequence file, an association process is requested to associate each of the four main images of track #1 in the associated sequence file with each of the four audios #1 to #4 in the MP4 file, in the association process, the file control unit 43 assigns UUIDs #1 to #4, which were generated in advance for each of the four main images of track #1 in the associated sequence file as specific information of external data, to each of the four audios #1 to #4 to be associated with the four main images, and stores the UUIDs #1 to #4 assigned to audios #1 to #4, respectively, in the MP4 file in which audios #1 to #4 are stored.

[0333] That is, the file control unit 43 generates track #2i of UUID #i for the MP4 file and stores it in the mdat box, and also generates a trak box for managing track #2i and stores it in the moov box.

[0334] The trak box that manages track #2i has a tkhd box and a tref box. In the tkhd box of the trak box that manages track #2i, the track ID that identifies the track #2i to be managed is set to 2i.

[0335] Furthermore, in the tref box of the trak box that manages track #2i, it is set that the other track associated with track #2i is track #2i-1 (track_ID=2i-1), and that track #2i is a metadata (here, specific information) track (type=cdsc).

[0336] As described above, by storing the UUID, which is pre-generated specific information for the four main images of track #1 in the associated sequence file, in the MP4 file in which audio #1 to #4 are stored, the four main images can be associated with audio #1 to #4.

[0337] FIG. 46 is a diagram showing a fourth example of association between a main image as internal data and audio as external data in a case where an MP4 file is used as a file for storing audio as external data.

[0338] FIG. 46 shows an example of association between one main image stored in a first associative collection file and four pieces of audio stored in an MP4 file.

[0339] The first associative collection file is generated, for example, by the pre-storage method and is configured in the same manner as in FIG.

[0340] The MP4 file in which four audios #1 to #4 to be associated with one main image in an associative sequence file are stored is configured in the same manner as in the case of FIG.

[0341] If, after the generation of a first association-type collection file, an association process is requested to associate the main image Item#1 in the first association-type collection file with audio #1 to #4 in the MP4 file, in the association process, the file control unit 43 assigns UUID#1, which was generated in advance for the main image Item#1 in the first association-type collection file as specific information of external data to be associated with the main image Item#1, to audio #1 to #4 to be associated with the main image Item#1, and stores the UUID#1 assigned to audio #1 to #4 in the MP4 file in which audio #1 to #4 is stored.

[0342] That is, the file control unit 43 generates tracks #2, #4, #6, and #8 of UUID #1 assigned to each of audio #1 to #4 for the MP4 file and stores them in the mdat box, and also generates trak boxes to manage tracks #2, #4, #6, and #8 respectively and stores them in the moov box.

[0343] The trak box that manages track #2i (i=1, 2, 3, 4) has a tkhd box and a tref box. In the tkhd box of the trak box that manages track #2i, the track ID that identifies the track #2i to be managed is set to 2i.

[0344] Furthermore, in the tref box of the trak box that manages track #2i, it is set that the other track associated with track #2i is track #2i-1 (track_ID=2i-1), and that track #2i is a metadata (here, specific information) track (type=cdsc).

[0345] As described above, by storing UUID#1, which is specific information generated in advance for the main image Item#1, in the MP4 file in which audios #1 to #4 are stored, it is possible to associate one main image Item#1 with multiple audios #1 to #4.

[0346] In Fig. 46, audio #1 to #4 are audio with the same content, but audio #1 is a Japanese audio, audio #2 is an English audio, audio #3 is a French audio, and audio #4 is a Chinese audio. Therefore, according to the association process in Fig. 46, it is possible to associate main image Item#1 with four audios #1 to #4 that have the same content but are in different languages.

[0347] FIG. 47 is a diagram showing a fifth example of association between a main image as internal data and audio as external data in a case where an MP4 file is used as a file for storing audio as external data.

[0348] FIG. 47 shows an example of association between one main image stored in a first associative collection file and four pieces of audio stored in an MP4 file.

[0349] The first associative collection file is generated, for example, by the pre-storage method and is configured in the same manner as in FIG.

[0350] An MP4 file in which four audios #1 to #4 to be associated with one main image in an associated sequence file are stored is configured in the same manner as in Fig. 44. However, in the MP4 file in Fig. 47, the trak box managing track #1 of audio #1 is set to track #1 as the primary track, and the trak boxes managing tracks #3, #5, and #7 of audios #2 to #4 are set to tracks #3, #5, and #7 as secondary tracks with track #1 as the primary.

[0351] If, after the generation of a first association-type collection file, an association process is requested to associate the main image Item#1 in the first association-type collection file with the audio #1 to #4 in the MP4 file, in the association process, the file control unit 43 assigns UUID#1, which was generated in advance for the main image Item#1 in the first association-type collection file as specific information of external data to be associated with the main image Item#1, to the primary audio #1 of the audio #1 to #4 to be associated with the main image Item#1, and stores the UUID#1 assigned to audio #1 in the MP4 file in which the audio #1 to #4 are stored.

[0352] That is, for the MP4 file, the file control unit 43 generates track #2 of UUID #1 assigned to primary audio #1 and stores it in the mdat box, and also generates a trak box to manage track #2 and stores it in the moov box.

[0353] The trak box that manages track #2 has a tkhd box and a tref box. In the tkhd box of the trak box that manages track #2, the track ID that identifies the track #2 to be managed is set to 2.

[0354] Furthermore, in the tref box of the trak box that manages track #2, it is set that the other track associated with track #2 is track #1 (track_ID=1), and that track #2 is a metadata (here, specific information) track (type=cdsc).

[0355] As described above, by storing UUID#1, which is specific information generated in advance for the main image Item#1, in the MP4 file in which audios #1 to #4 are stored, it is possible to associate one main image Item#1 with multiple audios #1 to #4.

[0356] Here, in Figure 47, UUID #1 is not assigned to the secondary audio #2 to #4 stored in the MP4 file, so there is no direct association, so to speak, between the main image Item #1 and each of the secondary audio #2 to #4.

[0357] However, as described above, in the MP4 file, audio #2 to #4 (tracks) are set to be secondary audio (tracks) with audio #1 on track #1 as the primary audio, and as a result of this MP4 file setting, the main image Item #1 and the secondary audio #2 to #4 are indirectly associated, so to speak, through the association between the main image Item #1 and the primary audio #1.

[0358] FIG. 48 is a diagram showing a sixth example of association between a main image as internal data and audio as external data in a case where an MP4 file is used as a file for storing audio as external data.

[0359] FIG. 48 shows an example of association between one main image stored in a first associative collection file and four pieces of audio stored in an MP4 file.

[0360] The first association type collection file is generated, for example, by a pre-storage method, and is configured similarly to the case of Fig. 43. However, in the first association type collection file of Fig. 48, in addition to the association information associating the main image Item#1 (its item ID) with UUID#1, association information associating the main image Item#1 with UUID#2 different from UUID#1 is stored in the association information storage box, and the number of pieces of association information is two instead of one.

[0361] The MP4 file in which four audios #1 to #4 associated with two main images in an associative sequence file are stored is configured in the same manner as in the case of FIG.

[0362] However, in the MP4 file of Fig. 47, the trak boxes that manage tracks #1 and #3 of audio #1 and #2, respectively, have tracks #1 and #3 set as primary tracks, while the trak box that manages track #5 of audio #3 has track #5 set as secondary track with track #1 as primary, and the trak box that manages track #7 of audio #4 has track #7 set as secondary track with track #3 as primary.

[0363] If, after the generation of a first association-type collection file, an association process is requested to associate the main image Item#1 in the first association-type collection file with audio #1 and #2 in the MP4 file and to associate the main image Item#1 with audio #3 and #4 in the MP4 file, in the association process, the file control unit 43 assigns UUID#1, which was generated in advance for the main image Item#1 in the first association-type collection file as specific information of external data to be associated with the main image Item#1, to the primary audio #1 of audio #1 and #2 to be associated with the main image Item#1, and stores the UUID#1 assigned to audio #1 in the MP4 file in which audio #1 and #2 are stored.

[0364] Furthermore, the file control unit 43 assigns UUID#2, which was generated in advance for the main image Item#1 in the first association-type collection file and serves as identifying information for external data to be associated with the main image Item#1, to the primary audio #3 of audio #3 and #4 to be associated with the main image Item#2, and stores the UUID#2 assigned to audio #3 in the MP4 file in which audio #3 and #4 are stored.

[0365] That is, for the MP4 file, the file control unit 43 generates track #2 of UUID #1 assigned to primary audio #1 and stores it in the mdat box, and also generates a trak box to manage track #2 and stores it in the moov box.

[0366] The trak box that manages track #2 has a tkhd box and a tref box. In the tkhd box of the trak box that manages track #2, the track ID that identifies the track #2 to be managed is set to 2.

[0367] Furthermore, in the tref box of the trak box that manages track #2, it is set that the other track associated with track #2 is track #1 (track_ID=1), and that track #2 is a metadata (here, specific information) track (type=cdsc).

[0368] The file control unit 43 generates track #4 of UUID #2 assigned to primary audio #3 for the MP4 file and stores it in the mdat box, and also generates a trak box for managing track #4 and stores it in the moov box.

[0369] The trak box that manages track #4 has a tkhd box and a tref box. In the tkhd box of the trak box that manages track #4, the track ID that identifies the managed track #4 is set to 4.

[0370] Furthermore, the tref box of the trak box that manages track #4 is set to indicate that the other track associated with track #4 is track #3 (track_ID=3), and that track #4 is a metadata track (type=cdsc).

[0371] As described above, by storing UUIDs #1 and #2 as multiple different identifying information generated in advance for the main image Item #1 in the MP4 file in which audios #1 to #4 are stored, it is possible to associate the two audios #1 and #2 with the main image Item #1, as well as the two audios #3 and #4, i.e., it is possible to associate the main image Item #1 with the four audios #1 to #4.

[0372] Here, in Fig. 48, as described in Fig. 47, UUID#1 is not assigned to secondary audio #2, and secondary audio #2 is not directly associated with primary image Item#1, but audio #2 is indirectly associated with primary image Item#1 through association with primary image Item#1 and primary audio #1 due to the MP4 file setting that audio #2 is secondary audio with audio #1 as primary. Similarly, UUID#2 is not assigned to secondary audio #4, and secondary audio #4 is not directly associated with primary image Item#1, but secondary audio #4 is indirectly associated with primary image Item#1 through association with primary image Item#1 and primary audio #3 due to the MP4 file setting that audio #4 is secondary audio with audio #3 as primary.

[0373] <Associated HEIF file generated by the space allocation method and associated HEIF file after association processing>

[0374] FIG. 49 is a diagram illustrating a first example of a first associative collection file generated by the area allocation method and a state of the first associative collection file after the associating process.

[0375] In the following, the description of how media data such as images and media data tracks are stored in the mdat box when generating an associated HEIF file will be omitted as appropriate.

[0376] In generating a first association-type collection file using the area allocation method, the file control unit 43 generates an association information storage box in which the number of association information items is set to 0 and which has an empty area (padding area) as a reserved area, and generates a first association-type collection file in which the association information storage box is stored in a meta box.

[0377] Whether or not valid association information is stored (set) in the association information storage box can be recognized based on the number of pieces of association information.

[0378] Also, a flag indicating whether valid association information is stored in the association information storage box can be provided in the association information storage box, and the flag allows the user to recognize whether valid association information is stored in the association information storage box.

[0379] In the following, for example, it is assumed that whether or not valid association information is stored in the association information storage box is recognized based on the number of pieces of association information.

[0380] If, after the generation of the first associative collection file, an association process is requested to associate the main image Item#1 in the first associative collection file with external data, in the association process, the file control unit 43 generates a UUID#1 as specific information for the external data to be associated with the main image Item#1, and generates association information that corresponds the UUID#1 to the item ID#1 of the main image Item#1.

[0381] Furthermore, file control unit 43 overwrites the association information in a free area serving as a reserved area in the association information storage box, and updates the number of pieces of association information to a value obtained by incrementing the current value by 1 (here, 1).

[0382] In the first association-type collection file in FIG. 49, the association information can be written to the association information storage box within the capacity of the free space in the association information storage box.

[0383] As described above, an association information storage box with a reserved area generates a first association type collection file stored in a meta box, and in the association process performed after the generation of the first association type collection file, association information that matches UUID#1 as identifying information for the external data with item ID#1 for the main image Item#1 is written (overwritten) into the reserved area in the association information storage box, thereby making it possible to associate internal data with external data later while preventing the offset of data already stored in the mdat box from shifting.

[0384] FIG. 50 is a diagram illustrating a second example of a first associative collection file generated by the area allocation method and a state of the first associative collection file after the associating process.

[0385] In generating the first association type collection file by the area allocation method, the file control unit 43 generates, for example, a free box having an empty area (padding area) as a reserved area, and generates the first association type collection file in which the free box is stored in a meta box. A free box is a box in which any data can be stored, and in this embodiment, the free box has an empty area as a reserved area.

[0386] If, after the generation of the first associative collection file, an association process is requested to associate the main image Item#1 in the first associative collection file with external data, in the association process, the file control unit 43 generates a UUID#1 as specific information for the external data to be associated with the main image Item#1, and generates association information that corresponds the UUID#1 to the item ID#1 of the main image Item#1.

[0387] Furthermore, the file control unit 43 generates an association information number that sets the number of pieces of generated association information, and generates an association information storage box that stores the number of pieces of association information and the association information.

[0388] Then, the file control unit 43 overwrites the association information storage box in the meta box using the reserved area of ​​the free box, and reduces the capacity (size) of the free box by the capacity of the association information storage box.

[0389] In the first association type collection file of FIG. 50, writing to the association information storage box can be performed within the capacity of the free box at the time the first association type collection file is created.

[0390] As described above, a free box with a reserved area generates a first association type collection file stored in a meta box, and in the association process performed after the generation of the first association type collection file, an association information storage box storing association information that matches UUID#1 as specific information of the external data with item ID#1 of the main image Item#1 is written using the reserved area of ​​the free box, thereby making it possible to associate internal data with external data later while preventing the offset of data already stored in the mdat box from shifting.

[0391] FIG. 51 is a diagram illustrating a third example of a first associative collection file generated by the area allocation method and a state of the first associative collection file after the associating process.

[0392] In Figure 51, the same processing is performed as in Figure 50, except that a first associative collection file is generated in which the free box is stored in the file hierarchy between the meta box and the mdat box, rather than in the meta box.

[0393] In the case of FIG. 51, as in the case of FIG. 50, it is possible to prevent the offset of data already stored in the mdat box from shifting, and to associate internal data with external data later.

[0394] FIG. 52 is a diagram illustrating a first example of a second associative collection file generated by the area allocation method and a state of the second associative collection file after the associating process.

[0395] In generating a first association-type collection file using the area allocation method, the file control unit 43 generates a second association-type collection file in which the number of association information items is set to 0, for example, and an area for association information having an empty area as a reserved area is stored in the mdat box as an item.

[0396] Whether the area for associating information as an item has valid associating information can be recognized based on the number of pieces of associating information. Also, a flag indicating whether the area for associating information as an item has valid associating information can be provided in the area for associating information as an item, and whether the area for associating information as an item has valid associating information can be recognized based on the flag.

[0397] In the following, for example, it is assumed that whether or not an area for associating information as an item has valid associating information is recognized based on the number of pieces of associating information.

[0398] If, after the second associative collection file is generated, an association process is requested to associate the main image Item#1 in the second associative collection file with external data, in the association process, the file control unit 43 generates a UUID#1 as specific information for the external data to be associated with the main image Item#1, and generates association information that corresponds the UUID#1 to the item ID#1 of the main image Item#1.

[0399] Furthermore, the file control unit 43 overwrites the association information in the free area serving as a reserved area in the area for association information as an item, and updates the number of pieces of association information to a value incremented by 1 from the current value (here, 1).

[0400] In the second associative collection file in FIG. 52, associative information can be written to the area for associative information within the capacity of the free space in the area for associative information.

[0401] As described above, a second association type collection file is generated in which an area for association information having a reserved area is stored as an item in the meta box, and in the association process performed after the generation of the second association type collection file, association information that matches UUID#1 as specific information for the external data with item ID#1 of the main image Item#1 is written into the reserved area of ​​the area for association information, thereby making it possible to associate internal data and external data later while preventing the offset of data already stored in the mdat box from shifting.

[0402] FIG. 53 is a diagram illustrating a second example of a second associative collection file generated by the area allocation method and a state of the second associative collection file after the associating process.

[0403] In generating a second associative collection file using the area allocation method, the file control unit 43, for example, generates a free box, and generates a second associative collection file in which the free box is stored in the file hierarchy between the meta box and the mdat box.

[0404] Note that immediately after generation, the collection file generated in Figure 53 does not have the form of a second associated collection file in which association information is stored as an item in the mdat box, because no association information is stored in the mdat box. However, in the future, as the association process is performed, the file will have the form of a second associated collection file, so for convenience, it will be referred to as the second associated collection file.

[0405] If, after the second associative collection file is generated, an association process is requested to associate the main image Item#1 in the second associative collection file with external data, in the association process, the file control unit 43 generates a UUID#1 as specific information for the external data to be associated with the main image Item#1, and generates association information that corresponds the UUID#1 to the item ID#1 of the main image Item#1.

[0406] In addition, the file control unit 43 generates a number of pieces of association information that is set to the number of pieces of generated association information, and stores the number of pieces of association information and the association information as one item by adding it after the last item in the mdat box.

[0407] Furthermore, the file control unit 43 adds the number of pieces of association information as items added to the mdat box and metadata related to the association information (metadata to be stored in the iinf box, iloc box, etc.) to the meta box.

[0408] Adding metadata to the meta box increases the capacity (amount of data) of the meta box, but the increased capacity of the data is written using the reserved area of ​​the free box. Therefore, the number of association information items added to the mdat box and the metadata related to the association information can be considered to be written to the reserved area of ​​the free box.

[0409] The file control unit 43 reduces the capacity of the free box by the amount of data of the metadata added to the meta box, including the relationship information relating to the association between the main image Item#1 and external data, i.e., the association information that associates UUID#1 with item ID#1 of the main image Item#1, the number of association information, and the metadata added to the meta box.

[0410] Here, in Fig. 53, the number of association information items and the association information are stored (written) in a form that is added after the last item in the mdat box, so the reserved area of ​​the free box is not consumed. Therefore, the capacity of the free box does not change (decrease) depending on the number of association information items and the association information added after the last item in the mdat box.

[0411] In the second associative collection file of Figure 53, relationship information can be written to the extent that the capacity of the metadata added to the meta box of the relationship information does not exceed the capacity of the free box at the time the second associative collection file is generated.

[0412] As described above, a second associated collection file is generated in which a free box with a reserved area is stored, and in the association process performed after the generation of the second associated collection file, association information that matches UUID#1 as specific information for the external data with item ID#1 of the main image Item#1 is stored as the last item in the mdat box, and metadata related to that association information is added to the meta box using the reserved area of ​​the free box, thereby making it possible to associate internal data and external data later while preventing the offset of data already stored in the mdat box from shifting.

[0413] In FIG. 53, the free box can be provided in the meta box, not in the file hierarchy.

[0414] FIG. 54 is a diagram illustrating a first example of a third associative collection file generated by the area allocation method and a state of the third associative collection file after associating processing.

[0415] In generating the third association-type collection file using the area allocation method, the file control unit 43 generates the third association-type collection file in which, for example, one or more pieces of specific information are each stored as an item in an mdat box.

[0416] The area of ​​the information that becomes the specific information stored in the mdat box (area for specific information) is a reserved area where specific information will be stored (in the future), and can be padded with an invalid value, for example. For example, if a UUID is used as the specific information, all zeros can be used as an invalid value.

[0417] In generating the third associated collection file, the file control unit 43 generates, for example, an invalid value as a uuid, and writes the invalid value as an item into the mdat box, thereby securing a reserved area in which specific information is stored.

[0418] When the third associative collection file is generated, metadata regarding information (invalid values) that is specific information for an item is also generated and stored in the meta box.

[0419] Metadata related to information that serves as specific information for an item includes the item ID of the information that serves as specific information, offset, size, association information that associates the main image as internal data with the specific information of the external data, i.e., association information that associates the item ID of the main image with the item ID of the information that serves as specific information for an item as the reference source and reference destination, respectively (association information stored in the iref box described in Figure 12).

[0420] Furthermore, the information that is the specific information of an item can be stored in the mdat box in a number equal to or less than the number of main images (number of items) as internal data that the third associative collection file has.

[0421] If, after the generation of the third associative collection file, an association process is requested to associate the main image Item#1 in the third associative collection file with external data, in the association process the file control unit 43 generates UUID#1 as valid identifying information for the external data to be associated with the main image Item#1, and writes the UUID#1 to (an area of) information that serves as identifying information for the item associated with item ID#1 of the main image Item#1 by the metadata in the meta box (overwriting the invalid value).

[0422] In the third associative collection file of Figure 54, valid specific information can be written within the range of the number of pieces of information that become specific information stored in the mdat box when the third associative collection file is generated.

[0423] As described above, a third association type collection file is generated in which a reserved area for storing specific information is secured in the mdat box, i.e., a third association type collection file in which information that becomes specific information is stored in the mdat box.In the association process performed after the generation of the third association type collection file, UUID#1 as specific information to be assigned to external data is written to information that becomes specific information associated with item ID#1 of main image Item#1, thereby making it possible to associate internal data and external data later while preventing the offset of data already stored in the mdat box from shifting.

[0424] In addition, when generating the third associated collection file, valid specific information (specific information of a valid value), such as a uuid, can be stored in the mdat box instead of specific information (invalid value).

[0425] The generation of the third associative collection file in which valid specific information is stored in the mdat box is the generation of the third associative collection file in which specific information is stored before it is assigned to external data, and is therefore the generation of the third associative collection file using the pre-storage method.

[0426] In this case, for the specific information, for example, a flag is provided to indicate whether the specific information has been assigned to external data or is unassigned, and the flag can be used to recognize whether the specific information has been assigned or is unassigned.

[0427] FIG. 55 is a diagram illustrating a second example of a third associative collection file generated by the area allocation method and a state of the third associative collection file after the associating process.

[0428] In generating a third associative collection file using the area allocation method, the file control unit 43, for example, generates a free box, and generates a third associative collection file in which the free box is stored in the file hierarchy between the meta box and the mdat box.

[0429] In addition, immediately after generation, the collection file generated in Figure 55 does not have specific information as items stored in the mdat box, and therefore metadata regarding the specific information as items is not stored in the meta box, so it does not take the form of a third associated collection file in which specific information is stored as items in the mdat box. However, in the future, as the association process is performed, it will take the form of a third associated collection file, so for convenience, it will be referred to as the third associated collection file.

[0430] If, after the generation of the third associative collection file, an association process is requested to associate the main image Item#1 in the third associative collection file with external data, in the association process the file control unit 43 generates UUID#1 as specific information for the external data to be associated with the main image Item#1, and stores the UUID#1 as a single item by adding it after the last item in the mdat box.

[0431] Furthermore, the file control unit 43 adds metadata (metadata to be stored in the iinf box, iref box, iloc box, etc.) relating to UUID#1 as the item added to the mdat box to the meta box.

[0432] Adding metadata to the meta box increases the capacity of the meta box, but the data for the increased capacity is written using the reserved area of ​​the free box. Therefore, metadata related to specific information as an item added to the mdat box can be considered to be written to the reserved area of ​​the free box.

[0433] The file control unit 43 reduces the capacity of the free box by the amount of data of the relationship information relating to the association between the above main image Item#1 and external data, i.e., UUID#1 as an item and the metadata relating to that UUID#1.

[0434] Here, in Fig. 55, as in Fig. 53, UUID#1 as an item is stored by adding it after the last item in the mdat box, so it does not consume the reserved area of ​​the free box. Therefore, the capacity of the free box does not change due to UUID#1 as an item added after the last item in the mdat box.

[0435] In the third associative collection file of Figure 55, relationship information can be written to the extent that the capacity of metadata relating to specific information as an item of the relationship information does not exceed the capacity of the free box at the time the third associative collection file is generated.

[0436] As described above, a third associated collection file in which a free box is stored is generated, and in the association process performed after the generation of the third associated collection file, UUID#1 as specific information of the external data is stored as the last item in the mdat box, and metadata regarding that specific information is added to the meta box using the reserved area of ​​the free box, thereby making it possible to associate the internal data and external data later while preventing the offset of the data already stored in the mdat box from shifting.

[0437] In FIG. 53, the free box can be provided in the meta box, not in the file hierarchy.

[0438] FIG. 56 is a diagram for explaining a first example of an associative sequence file generated by the area allocation method and the state of the associative sequence file after the associative process.

[0439] In generating an association-type sequence file using the area allocation method, the file control unit 43 generates an association-type sequence file in which, for example, a track of information serving as specific information is stored in an mdat box.

[0440] The area of ​​the track of information that becomes the specific information stored in the mdat box (area for specific information) is a reserved area where specific information will be stored (in the future), and can be padded with an invalid value, for example. For example, if a UUID is used as the specific information, all zeros can be used as an invalid value.

[0441] In generating an associated collection file, the file control unit 43 generates, for example, an invalid value as a uuid, and writes the track of the invalid value to the mdat box, thereby securing a reserved area in which specific information is stored.

[0442] When an associated sequence is generated, metadata for managing tracks of specific information (invalid values) is also generated and stored in the meta box (in the trak box).

[0443] The metadata that manages the track of the information that constitutes the specific information includes the track ID of the track of the information that constitutes the specific information, the offset, size, and the track ID of another track (here, the track of the main image) related to the track of the information that constitutes the specific information (information stored in the trak box that manages track #3 of the specific information described in Figure 14).

[0444] Furthermore, the track of information serving as specific information can be configured, for example, with information serving as specific information in a number equal to the number of main images (number of frames) as internal data contained in the associative sequence file.

[0445] If, after the generation of an associative sequence file, an association process is requested to associate the main image Item#1 in the associative sequence file with external data, in the association process, the file control unit 43 generates a UUID#1 as valid identifying information for the external data to be associated with the main image Item#1, and writes the UUID#1 to the metadata in the meta box and to the information (area) that is the identifying information associated with the main image Item#1 by the time information on the timeline in the track of the identifying information (overwriting the invalid value).

[0446] In the associative sequence file of Figure 56, valid specific information can be written within the range of the number of pieces of information that make up the track of information that becomes specific information stored in the mdat box when the associative sequence file is generated.

[0447] As described above, an associated sequence file is generated in which a reserved area for storing specific information is secured in the mdat box, i.e., an associated sequence file in which tracks of information that becomes specific information are stored in the mdat box.In the association process that is performed after the generation of the associated sequence file, UUID#1 as specific information to be assigned to external data is written to information that becomes specific information associated with the main image Item#1, thereby making it possible to associate the internal data and external data later while preventing the offset of data already stored in the mdat box from shifting.

[0448] When generating an associated sequence file, the mdat box can store a track of valid specific information, such as a uuid, instead of specific information (invalid value).

[0449] The generation of an associative sequence file in which a track of valid specific information is stored in the mdat box is the generation of an associative sequence file in which specific information is stored before it is assigned to external data, and is therefore the generation of an associative sequence file using the pre-storage method.

[0450] In this case, for the specific information, for example, a flag is provided to indicate whether the specific information has been assigned to external data or is unassigned, and the flag can be used to recognize whether the specific information has been assigned or is unassigned.

[0451] FIG. 57 is a diagram for explaining a second example of an associative sequence file generated by the area allocation method and the state of the associative sequence file after the associative process.

[0452] In generating an associative sequence file using the area allocation method, the file control unit 43 generates, for example, a free box, and generates an associative sequence file in which the free box is stored in the file hierarchy between the meta box and the mdat box.

[0453] In addition, immediately after generation, the sequence file generated in Figure 57 does not have tracks of specific information stored in the mdat box, and therefore the metadata managing the tracks of specific information is not stored in the moov box, so it does not take the form of an associated sequence file in which tracks of specific information are stored in the mdat box. However, in the future, when the association process is performed, it will take the form of an associated collection file, so for convenience it will be called an associated sequence file.

[0454] After the generation of the associative sequence file, if an associating process is requested to associate the main image Item#1 in the associative sequence file with external data, in the associating process, the file control unit 43 stores a track consisting of one or more pieces of information that serve as specific information (hereinafter also referred to as additional information track) by adding it to the end of the last track of the mdat box. For example, the information that serves as specific information that constitutes the additional information track can be an invalid value similar to that in the case of FIG. 56.

[0455] Furthermore, the file control unit 43 generates a trak box that stores metadata for managing the additional information track added to the mdat box, and adds it to the meta box.

[0456] When generating a trak box for managing an additional information track, a trak box (a trak box for managing track #3 of specific information described in Figure 14) is generated that stores the track ID of the additional information track, its offset, size, and the track IDs of other tracks related to the additional information track (here, the track of the main image).

[0457] The file control unit 43 generates UUID#1 as valid identifying information for external data to be associated with the main image Item#1, and uses that UUID#1 to rewrite (overwrite with invalid values) the metadata in the meta box and the information in the additional information track that serves as identifying information associated with the main image Item#1 by the time information on the timeline.

[0458] By adding a trak box (storing metadata) that manages additional information tracks to the meta box, the capacity of the meta box increases, but the data for the increased capacity is written using the reserved area of ​​the free box. Therefore, the trak box that manages the additional information tracks can be considered to be written to the reserved area of ​​the free box.

[0459] The file control unit 43 reduces the capacity of the free box by the amount of data in the trak box that manages the additional information track, out of the relationship information relating to the association between the main image Item#1 and external data described above, i.e., the additional information track and the trak box that manages the additional information track.

[0460] Here, in FIG. 57, the additional information track is stored by adding it after the last track of the mdat box, so it does not consume the reserved area of ​​the free box.

[0461] In the associative sequence file of Figure 57, relationship information can be written to the extent that the amount of data in the trak box that manages the additional information track does not exceed the capacity of the free box at the time the associative sequence file is generated.

[0462] As described above, an associated sequence file in which a free box is stored is generated, and in the association process performed after the generation of the associated sequence file, the additional information track is stored as the last track of the mdat box, and a trak box that manages the additional information track is added to the meta box using the reserved area of ​​the free box.In addition, by rewriting the information that becomes the specific information that constitutes the additional information track with UUID#1 as specific information of the external data, it is possible to associate the internal data and external data later while preventing the offset of the data already stored in the mdat box from shifting.

[0463] In FIG. 57, the free box can be provided in the meta box, not in the file hierarchy.

[0464] In addition, when generating the additional information track, a track consisting of a number of pieces of information equal to the number of main images to be associated with external data can be generated, or when initially generating the additional information track, a track consisting of a number of pieces of information equal to the number of main images that make up the main image track can be generated.

[0465] However, when generating an additional information track, if a track consisting of a number of pieces of information equal to the number of main images to be associated with external data is generated as the additional information track, an additional information track must be added each time the association process is performed, which complicates the management of the additional information track and the association of specific information written in the additional information track with the main images.

[0466] When the additional information track is first generated, a track consisting of a number of pieces of information equal to the number of main images constituting the main image track is generated as the additional information track. In the subsequent association process, it is sufficient to simply rewrite the specific information associated with the main image in the additional information track with valid specific information from the external data.

[0467] Furthermore, when generating an associative sequence file, an additional information track consisting of valid specific information such as a uuid can be stored in the mdat box instead of specific information (invalid values).

[0468] The generation of an associative sequence file in which an additional information track consisting of valid specific information is stored in the mdat box is the generation of an associative sequence file in which specific information is stored before it is assigned to external data, and is therefore the generation of an associative sequence file using the pre-storage method.

[0469] In this case, for the specific information, for example, a flag is provided to indicate whether the specific information has been assigned to external data or is unassigned, and the flag can be used to recognize whether the specific information has been assigned or is unassigned.

[0470] FIG. 58 is a flowchart illustrating an example of a process for generating an associative HEIF file using the area allocation method.

[0471] In step S211, the file control unit 43 determines a required capacity (hereinafter also referred to as a required capacity) as the capacity of the reserved area, and proceeds to step S212.

[0472] Here, when creating a box with a reserved area such as a free box, the box requires at least 8 bytes to store its size and type, so the size of the box will be at least the required capacity of the reserved area plus 8 bytes.

[0473] For example, when generating a HEIF file that stores a free box, when adding association information or specific information as an item to the mdat box in the association process as shown in Figure 53 or Figure 55, metadata related to the association information or specific information as an item needs to be added to the meta box along with the addition of the association information or specific information as an item. In this case, the required capacity is determined to be a value equal to or greater than the amount of data of the metadata to be added to the meta box.

[0474] Also, for example, when generating a HEIF file that stores a free box, when an additional information track is added to the mdat box in the association process as shown in Fig. 57, metadata for managing the additional information track needs to be added to the moov box along with the addition of the additional information track. In this case, the required capacity is determined to be a value equal to or greater than the amount of data of the metadata to be added to the moov box.

[0475] The required capacity can also be determined based on one or more of the following: the amount of data for one specific piece of information, the number of main images stored in the HEIF file (the number of main images that can be associated with external data), the number of external data that can be associated with the main images, etc.

[0476] The larger the data amount of the specific information, the larger the required capacity. The number of main images that can be associated with external data can be determined, for example, within the range of the number of main images in the HEIF file. The larger the number of main images that can be associated with external data, the larger the required capacity. The number of external data that can be associated with a main image can be determined to be any number.

[0477] When the specific information is information that can assign the same value to multiple external data, such as UUID, an increase or decrease in the number of external data that can be associated with one main image does not affect the required capacity. When the specific information is information that differs for each external data, such as a hash value or file name that uses external data as input, the required capacity increases as the number of external data that can be associated with the main image increases.

[0478] The required capacity can be determined, for example, according to the product of the data amount of one piece of specific information and the number of main images with which external data can be associated.

[0479] The data amount of one piece of specific information can be determined depending on the type of specific information, for example, whether the specific information is a URL, a uuid, a hash value, or the like.

[0480] The number of main images to which external data can be associated can be determined, for example, according to the remaining capacity of the medium storing the HEIF file, so that the larger the remaining capacity, the larger the value. In addition, the number of main images to which external data can be associated can be determined, for example, according to a maximum number of main images that can be stored in one HEIF file, and the larger the maximum number, the larger the value. In addition, the number of main images to which external data can be associated can be determined, for example, by evaluating the main images stored in the HEIF file in the file control unit 43, so that the larger the number of main images with good evaluations, the larger the value. In the evaluation of the main images, for example, information regarding the image quality of the main images, such as S / N and sharpness, can be obtained, and a main image with S / N, sharpness, or the like equal to or greater than a threshold value can be determined as a main image with good evaluation. In addition, the number of main images to which external data can be associated can be determined according to multiple factors, such as the remaining capacity of the medium described above.

[0481] In step S212, the file control unit 43 generates the associative HEIF file described with reference to Figures 49 to 57 in which a reserve area of ​​the required capacity has been secured, and then ends the process.

[0482] FIG. 59 is a flowchart illustrating an example of an association process for an associative HEIF file generated using the area allocation method.

[0483] In step S221, file control unit 43 acquires specific information of the external data, and the process proceeds to step S222.

[0484] For example, if the external data is data stored in a single-track MP4 file or WAV file, the specific information may include a UUID, a hash value of the MP4 file or WAV file in which the external data is stored (or the hash value of the external data itself), a URL, etc.

[0485] Also, for example, when the external data is a multi-track MP4 file, the specific information may be a set of the URL of the MP4 file in which the external data is stored and the track ID of the track of the external data, a set of the URL of the MP4 file in which the external data is stored and the time information on the timeline of the external data, a set of the URL of the MP4 file in which the external data is stored and the hash value of the track of the external data, etc. The uuid, hash value, and URL are obtained by generation. The track ID and time information are obtained by referring to the MP4 file in which the external data is stored.

[0486] In step S222, the file control unit 43 writes relationship information, including specific information, relating to the association between the main image and the external data (specific information), into the reserved area secured in advance in the associative HEIF file generated by the area allocation method, thereby generating an associative HEIF file in which the relationship information is stored, and processing proceeds to step S223.

[0487] In step S223, file control unit 43 determines whether or not it is necessary to write the specific information to the external data (the file in which the external data is stored).

[0488] If it is determined in step S223 that there is no need to write the identifying information to the external data, for example if the identifying information is information that can identify the external data without writing it to the external data, such as a hash value generated using the external data as input, the process skips step S224 and ends. In other words, if there is no need to write the identifying information to the external data, the association between the main image and the external data is completed by writing the relationship information to the associated HEIF file.

[0489] Also, if it is determined in step S233 that it is necessary to write the specific information to the external data, for example, if the specific information is information such as a uuid that can identify the external data by writing it to the external data, processing proceeds to step S224.

[0490] In step S224, the file control unit 43 writes (stores) the identifying information acquired in step S221 in the file in which the external data is stored, and the process ends. That is, if it is necessary to write identifying information to the external data, the association between the main image and the external data is completed by writing the relationship information to the associative HEIF file and writing the identifying information to the file in which the external data is stored.

[0491] The file storing the external data associated with the main image stored in the associated HEIF file generated by the area allocation method may be generated before the association process is performed, or may be generated in parallel with the execution of the association process. When the file storing the external data is generated in parallel with the execution of the association process, for example, in step S224, a file storing the specific information is generated together with the external data.

[0492] FIG. 60 is a flowchart illustrating an example of an association process for a first association-type collection file generated by the area allocation method.

[0493] In Figure 60, for example, the first association-type collection file shown in Figure 51, in which the free box is provided in the meta box, is the target of the association process, and specific information that does not need to be written to external data is used.

[0494] In step S231, file control unit 43 acquires specific information of the external data, and the process proceeds to step S232.

[0495] For example, the file control unit 43 receives a file in which external data is stored as an input, calculates a hash value using SHA-256, and obtains the hash value as identifying information of the external data.

[0496] In step S232, the file control unit 43 obtains the meta box by reading it from the first associative collection file of FIG. 51 generated by the area allocation method, and the process proceeds to step S233.

[0497] In step S233, the file control unit 43 obtains the remaining capacity of the reserved area in the free box of the first associative collection file in Figure 51, and confirms that the remaining capacity is sufficient to add relationship information related to the association between the main image and external data, i.e., to add an association information storage box storing the number of association information in Figure 51 and the association information, and the processing proceeds to step S234.

[0498] If the remaining capacity of the reserve area is insufficient to add the relationship information relating to the association between the main image and the external data in step S233, for example, a message to that effect is displayed on the liquid crystal panel 19, and the process ends.

[0499] In step S234, the file control unit 43 uses the specific information of the external data and the information of the meta box to generate an association information storage box that stores the number of association information pieces and the association information in Figure 51, and further generates a new meta box that stores the association information storage box.

[0500] Furthermore, the file control unit 43 rewrites the meta box of the first association-type collection file with the new meta box, and the process proceeds from step S234 to step S235.

[0501] In step S235, file control unit 43 generates a new free box whose size is reduced by the increase in size (amount of data) of the new meta box relative to the meta box before the rewrite.

[0502] Furthermore, the file control unit 43 rewrites the free box of the first association-type collection file with the new free box, and the process ends.

[0503] FIG. 61 is a flowchart illustrating another example of the association process for the first association-type collection file generated by the area allocation method.

[0504] In Fig. 61, for example, the first association type collection file shown in Fig. 51 in which the free box is provided in the meta box is set as the target of the association process, similarly to Fig. 60. Furthermore, in Fig. 61, specific information that needs to be written to the external data is used.

[0505] In step S241, the file control unit 43 acquires the identification information of the external data, and the process proceeds to step S242. For example, the file control unit 43 generates a uuid and acquires the uuid as the identification information of the external data.

[0506] In step S242, file control unit 43 writes (stores) the uuid as the identification information acquired in step S221 in the file in which the external data is stored, and the process proceeds to step S243.

[0507] For example, if the file in which the external data is stored is a WAV file, the file control unit 43 writes the uuid in the LIST chunk of the WAV file, as described with reference to Fig. 29 etc. Also, for example, if the file in which the external data is stored is an MP4 file, the file control unit 43 writes the track of the uuid in the mdat box of the MP4 file, as described with reference to Fig. 43 etc.

[0508] In steps S243 to S246, the same processes as in steps S232 to S235 in FIG. 61 are carried out, respectively, and then the process ends.

[0509] FIG. 62 is a flowchart illustrating an example of a process for generating an associated HEIF file using the pre-storing method.

[0510] In step S251, the file control unit 43 acquires specific information for each main image stored in the associative HEIF file, and the process proceeds to step S252.

[0511] For example, the file control unit 43 acquires a uuid as identification information by generating it for each main image stored in the associated HEIF file.

[0512] In step S252, the file control unit 43 generates an associative HEIF file (which appears to be similar to the associative HEIF file in Figures 10 to 12 or Figure 14 in which the main image and external data are associated) that includes the specific information acquired in step S251 before it was assigned to the external data and stores relationship information related to the association between the main image and the external data (the specific information), and then terminates the processing.

[0513] FIG. 63 is a flowchart illustrating an example of an association process for an associated HEIF file generated by the pre-storing method.

[0514] In step S261, the file control unit 43 acquires, for example, a uuid as specific information associated with the main image to be associated with external data from the associated HEIF file generated by the pre-storage method, and the process proceeds to step S262.

[0515] For example, if the associative HEIF file generated by the pre-storage method is a first associative collection file, obtaining specific information associated with a given main image from the first associative collection file can be performed as described in the flowchart of FIG. 21.

[0516] In step S262, the file control unit 43 assigns the identification information acquired in step S261 to the external data, that is, determines the identification information of the external data to be the identification information acquired in step S261, and proceeds to step S263.

[0517] In step S263, file control unit 43 writes (stores) the uuid as the specific information acquired in step S221 in the file in which the external data is stored, and the process ends.

[0518] For example, if the file in which the external data is stored is a WAV file, the file control unit 43 writes the uuid in the LIST chunk of the WAV file, as described with reference to Fig. 29 etc. Also, for example, if the file in which the external data is stored is an MP4 file, the file control unit 43 writes the track of the uuid in the mdat box of the MP4 file, as described with reference to Fig. 43 etc.

[0519] In the association process for associated HEIF files generated using the pre-storage method, the association between the main image and the external data is completed by writing specific information to the file in which the external data is stored.

[0520] <Description of the computer to which this technology is applied>

[0521] Next, the series of processes of the above-mentioned file control unit 43 and other blocks constituting the signal processing unit 13 (FIG. 1) can be performed by hardware or software. When the series of processes are performed by software, a program constituting the software is installed in a computer or the like.

[0522] FIG. 64 is a block diagram showing an example of the configuration of an embodiment of a computer in which a program for executing the series of processes described above is installed.

[0523] The program can be recorded in advance on the hard disk 905 or ROM 903 as a recording medium built into the computer.

[0524] Alternatively, the program can be stored (recorded) in a removable recording medium 911 driven by the drive 909. Such a removable recording medium 911 can be provided as a so-called package software. Here, examples of the removable recording medium 911 include a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, and a semiconductor memory.

[0525] The program can be installed in the computer from the removable recording medium 911 as described above, or can be downloaded to the computer via a communication network or a broadcasting network and installed in the built-in hard disk 905. That is, the program can be transferred to the computer wirelessly from a download site via an artificial satellite for digital satellite broadcasting, or transferred to the computer by wire via a network such as a LAN (Local Area Network) or the Internet.

[0526] The computer includes a CPU (Central Processing Unit) 902 , to which an input / output interface 910 is connected via a bus 901 .

[0527] When a user inputs a command by operating an input unit 907 via an input / output interface 910, the CPU 902 executes a program stored in a read only memory (ROM) 903 in accordance with the command. Alternatively, the CPU 902 loads a program stored in a hard disk 905 into a random access memory (RAM) 904 and executes the program.

[0528] As a result, the CPU 902 performs processing according to the above-mentioned flowchart or processing performed by the configuration of the above-mentioned block diagram. Then, the CPU 902 outputs the processing result from the output unit 906 via the input / output interface 910, or transmits it from the communication unit 908, or further records it on the hard disk 905, as necessary.

[0529] The input unit 907 is composed of a keyboard, a mouse, a microphone, etc. The output unit 906 is composed of an LCD (Liquid Crystal Display), a speaker, etc.

[0530] Here, in this specification, the processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing executed in parallel or individually (for example, parallel processing or processing by objects).

[0531] The program may be processed by one computer (processor), or may be distributed among multiple computers. Furthermore, the program may be transferred to a remote computer for execution.

[0532] Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.

[0533] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0534] For example, the present technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0535] Furthermore, each step described in the above flow chart can be executed by one device, or can be shared and executed by a plurality of devices.

[0536] Furthermore, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device, or can be shared and executed by multiple devices.

[0537] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0538] The present technology can have the following configurations.

[0539] <1> a file control unit that generates a HEIF file that includes specific information before being assigned to the external data, identifies external data outside the HEIF file to be associated with an image stored in the HEIF file, and stores relationship information related to the association between the image and the specific information; File processing device. <2> The file control unit generates the HEIF file in which association information that associates the image with specific information of the external data is stored. <1> The file processing device according to claim 1 . <3> The file control unit generates the HEIF file in which the association information that associates an item ID that identifies the image with the identification information is stored. <2> The file processing device according to claim 1 . <4> The file control unit generates the HEIF file in which the association information is stored in a meta box or an mdat box. <3> The file processing device according to claim 1 . <5> The file control unit generates the HEIF file in which the specific information is stored in an mdat box, and the association information that associates an item ID that identifies the image with an item ID that identifies the specific information stored in the mdat box is stored in a meta box. <2> The file processing device according to claim 1 . <6> The file control unit generates the HEIF file in which a track of specific information of the external data associated with the frames constituting the track of the image is stored in an mdat box. <1> The file processing device according to claim 1 . <7> The file control unit generates the identification information for each image stored in the HEIF file, and generates the HEIF file in which related information including the identification information is stored. <1> Or <6> 2. A file processing device according to claim 1 , <8> Identifying external data outside a High Efficiency Image File Format (HEIF) file to be associated with an image stored in the HEIF file, and generating the HEIF file including specific information before being assigned to the external data and storing relationship information related to the association between the image and the specific information. 23. A file processing method comprising: <9> A file control unit that generates a HEIF file including specific information before being assigned to the external data, which identifies external data outside a High Efficiency Image File Format (HEIF) file to be associated with an image stored in the HEIF file, and in which relationship information relating to the association between the image and the specific information is stored. A program that makes a computer function. <10> a file control unit that identifies external data outside a High Efficiency Image File Format (HEIF) file to be associated with an image stored in the HEIF file, the specific information including the specific information before being assigned to the external data, and that writes the specific information stored in the HEIF file, in which relationship information related to the association between the image and the specific information is stored, to a file in which the external data is stored; File processing device. <11> The file control unit writes the specific information stored in the HEIF file, in which association information that associates the image with the specific information of the external data is stored, to a file in which the external data is stored. <10> The file processing device according to claim 1 . <12> The file control unit writes the specific information stored in the HEIF file, in which the association information that associates the item ID that identifies the image with the specific information is stored, to a file in which the external data is stored. <11> The file processing device according to claim 1 . <13> The file control unit writes the specific information stored in the HEIF file in which the association information is stored in a meta box or an mdat box to a file in which the external data is stored. <12> The file processing device according to claim 1 . <14> The file control unit writes the specific information stored in the HEIF file, in which the specific information is stored in an mdat box and the association information, which associates an item ID that identifies the image with an item ID that identifies the specific information stored in the mdat box, is stored in a meta box, to a file in which the external data is stored. <11> The file processing device according to claim 1 . <15> The file control unit writes the specific information stored in the HEIF file in which a track of specific information of the external data associated with frames constituting the track of the image is stored in an mdat box to a file in which the external data is stored. <10> The file processing device according to claim 1 . <16> The file control unit writes the specific information stored in the HEIF file in which the relationship information including the specific information generated for each image stored in the HEIF file is stored to a file in which the external data is stored. <10> Or <15> 2. A file processing device according to claim 1 , <17> A method of identifying external data outside a High Efficiency Image File Format (HEIF) file to be associated with an image stored in the HEIF file, the method including identifying information before being assigned to the external data, and writing the identifying information stored in the HEIF file in which relationship information related to the association between the image and the identifying information is stored to a file in which the external data is stored 23. A file processing method comprising: <18> a file control unit that writes specific information stored in a HEIF file, which includes specific information before being assigned to the external data and which identifies external data outside the HEIF file to be associated with an image stored in the HEIF file, and in which relationship information relating to the association between the image and the specific information is stored, to a file in which the external data is stored A program that makes a computer function. [Explanation of symbols]

[0540] 10 digital camera, 11 optical system, 13 signal processing section, 14 media, 15,16 interface, 17 button / key, 18 touch panel, 19 liquid crystal panel, 20 viewfinder, 21 interface, 41 optical system / image sensor control section, 42 encoding control section, 43 file control section, 44 media control section, 45 operation control section, 46 display control section, 47 UI control section, 901 bus, 902 CPU, 903 ROM, 904 RAM, 905 hard disk, 906 output section, 907 input section, 908 communication section, 909 drive, 910 input / output interface, 911 removable recording medium

Claims

1. a file control unit that generates the HEIF file including specific information before being assigned to the external data, the specific information specifying external data outside the HEIF file to be associated with an image stored in the HEIF file, and storing relationship information relating to the association between the image and the specific information; The file control unit generates the HEIF file in which specific information of the external data is stored in an mdat box and association information that associates the image with the specific information stored in the mdat box is stored in a meta box. File processing device.

2. The file control unit generates the HEIF file in which the specific information is stored in the mdat box, and the association information that associates an item ID that identifies the image with an item ID that identifies the specific information stored in the mdat box is stored in a meta box. The file processing device according to claim 1 .

3. The file control unit generates the identification information for each image stored in the HEIF file, and generates the HEIF file in which related information including the identification information is stored. The file processing device according to claim 1 .

4. generating a HEIF file including specific information before being assigned to the external data, the specific information specifying external data outside the HEIF file to be associated with an image stored in the HEIF file, and storing relationship information relating to the association between the image and the specific information; generating the HEIF file in which specific information of the external data is stored in an mdat box and association information associating the image with the specific information stored in the mdat box is stored in a meta box; A file processing method comprising:

5. a file control unit that generates the HEIF file including specific information before being assigned to the external data, the specific information specifying external data outside the HEIF file to be associated with an image stored in the HEIF file, and storing relationship information relating to the association between the image and the specific information; The file control unit generates the HEIF file in which specific information of the external data is stored in an mdat box and association information that associates the image with the specific information stored in the mdat box is stored in a meta box. A program that enables a computer to function as a file processing device.

6. a file control unit that writes the specific information stored in the HEIF file, which includes specific information before being assigned to the external data, specifying external data outside the HEIF file to be associated with an image stored in the HEIF file, the specific information being stored in the HEIF file and which stores relationship information relating to the association between the image and the specific information, to a file in which the external data is stored; The file control unit writes the specific information stored in the HEIF file, in which specific information of the external data is stored in an mdat box and association information associating the image with the specific information stored in the mdat box is stored in a meta box, to a file in which the external data is stored. File processing device.

7. The file control unit writes the specific information stored in the HEIF file, in which the specific information is stored in the mdat box and the association information, which associates an item ID that identifies the image with an item ID that identifies the specific information stored in the mdat box, is stored in a meta box, to a file in which the external data is stored. The file processing device according to claim 6.

8. The file control unit writes the specific information stored in the HEIF file in which the relationship information including the specific information generated for each image stored in the HEIF file is stored to a file in which the external data is stored. The file processing device according to claim 6.

9. writing the specific information stored in the HEIF file, which includes specific information before being assigned to the external data, that identifies external data outside the HEIF file to be associated with an image stored in the HEIF file, and in which relationship information relating to the association between the image and the specific information is stored, to a file in which the external data is stored; writing the specific information stored in the HEIF file, in which specific information of the external data is stored in an mdat box and association information associating the image with the specific information stored in the mdat box is stored in a meta box, to a file in which the external data is stored; A file processing method comprising:

10. a file control unit that writes the specific information stored in the HEIF file, which includes specific information before being assigned to the external data, specifying external data outside the HEIF file to be associated with an image stored in the HEIF file, the specific information being stored in the HEIF file and which stores relationship information relating to the association between the image and the specific information, to a file in which the external data is stored; The file control unit writes the specific information stored in the HEIF file, in which specific information of the external data is stored in an mdat box and association information associating the image with the specific information stored in the mdat box is stored in a meta box, to a file in which the external data is stored. A program that enables a computer to function as a file processing device.

Citation Information

Patent Citations

  • IEC23008-12

  • Information processing system, information processor, and information generating device

    JP2011041230A

  • Adjustment for irregular sensor movement during spectral imaging

    US9784615B1

  • Media encapsulating and decapsulating

    WO2017140939A1