Image processing device, control method and program

Abstract

To generate an image file which stores an image group shot synchronously and which can be used appropriately.SOLUTION: An image processing apparatus is configured to generate an image file having a structure including a first storage region and a second storage region, the first storage region storing a plurality of instances of image data and the second storage region storing metadata pertaining to the plurality of instances of image data. The metadata includes: synchronous group information that groups image data shot synchronously with respect to time, the synchronous group information indicating image data, among the plurality of instances of image data, belonging to a group created by the grouping; and indicator information that indicates an indicator of grouping for the group indicated by the synchronous group information. The image processing apparatus includes obtainment means configured to obtain the plurality of instances of image data and shooting information pertaining to shooting of the image data; configuration means configured to configure the synchronous group information and the indicator information based on the shooting information obtained by the obtainment means; and generation means configured to generate the image file storing the plurality of instances of image data obtained by the obtainment means, and the synchronous group information and the indicator information configured by the configuration means.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an image processing device, a control method, and a program, and more particularly to a technique for generating an image file capable of storing data of a plurality of still images and moving images. [Background technology]

[0002] By using captured images acquired at the same time, it is possible to present expanded information that is not limited to information contained in a single captured image.Patent Document 1 discloses a photography system that takes images using two cameras with synchronized built-in clocks, and generates a 3D video using an image captured by one camera and an image captured by the other camera at the time closest to the capture time of the first image.

[0003] Recently, file formats have emerged that encode multiple still images or video images and store them as a single image file, making it possible to easily manage groups of highly related images. For example, a file format called High Efficiency Image File Format (HEIF), internationally standardized by ISO / IEC 23008-12, allows still images encoded with H.265 (HEVC) to be stored as a single image file. For such file formats, a normative structure including metadata is defined, specifying a method for associating the metadata with stored images and the structure of specific metadata formats. Furthermore, by describing the metadata in the metadata field, a single image representation consisting of multiple still images, known as a derived image, can be recorded as an image file.

[0004] As in Patent Document 1, images captured by synchronized shooting are highly correlated, and storing them as a synchronized group within a single image file is expected to improve the convenience of displaying and processing them. Regarding HEIF, standardization is underway in ISO / IEC 23008-12:2017 FDAM2 to add a 'tsyn' group to the metadata, which indicates that multiple still images or videos were captured in a synchronized manner. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2018-007031 Summary of the Invention [Problem to be solved by the invention]

[0006] However, no proposals have been made so far regarding the structure of metadata that takes into account the specific use of image files that group and store images obtained through synchronized shooting. The HEIF 'tsyn' format has also only been considered as a way to describe information that identifies a group of images that were shot simultaneously over the same period.

[0007] The present invention has been made in consideration of the above-mentioned problems, and aims to provide an image processing device, control method, and program that generate a suitably usable image file that stores a group of images captured synchronously. [Means for solving the problem]

[0008] In order to achieve the above object, an image processing device according to one aspect of the present invention is an image processing device that generates an image file having a structure including a first storage area for storing a plurality of image data and a second storage area for storing metadata relating to the plurality of image data. , complex an acquisition means for acquiring a number of image data and photographing information relating to photographing of the image data; and (1) Information for grouping image data captured in a time-synchronized manner and information indicating image data included in the group among the plurality of image data Sync Group Information 、 and (2) For groups related to synchronous group information, indicate the grouping index. A configuration means for configuring index information, a plurality of image data acquired by an acquisition means, and synchronization group information and index information configured by the configuration means Metadata containing and a generating means for generating an image file that stores the image data. [Effects of the Invention]

[0009] With this configuration, according to the present invention, it is possible to generate an image file that stores a group of images captured synchronously and that can be used appropriately. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 illustrates an example of a configuration of a synchronized imaging system according to an embodiment and a modification of the present invention. [Figure 2] FIG. 1 is a block diagram illustrating the functional configuration of a synchronous imaging controller 100 according to an embodiment and a modification of the present invention. [Figure 3] FIG. 1 is a block diagram illustrating the functional configuration of a camera device 200 according to an embodiment and a modification of the present invention. [Figure 4] Diagram showing the file structure of a HEIF file [Figure 5] FIG. 10 is a diagram showing an example of a definition of the data structure of a 'tsyn' box 411 according to the first embodiment of the present invention. [Figure 6] FIG. 10 is a diagram showing an example of a definition of the data structure of an 'iinf' box 406 according to the first embodiment of the present invention. [Figure 7] FIG. 10 is a diagram showing an example of a definition of the data structure of 'infe' according to the first embodiment of the present invention. [Figure 8] FIG. 10 is a diagram illustrating a file generation procedure in a synchronous imaging system according to an embodiment and a modification of the present invention. [Figure 9] FIG. 10 is a diagram showing an example of a definition of the data structure of a 'tsyn' box 411 according to the second embodiment of the present invention. [Figure 10] FIG. 10 is a diagram showing an example of a definition of the data structure of attribute information 'catt' according to the second embodiment of the present invention. [Figure 11] FIG. 10 is a diagram showing an example of a definition of the data structure of attribute information 'sytm' according to the second embodiment of the present invention. [Figure 12] FIG. 10 is a diagram showing an example of a definition of the data structure of attribute information 'syct' according to the second embodiment of the present invention. [Figure 13] FIG. 10 is a diagram showing an example of a definition of the data structure of attribute information 'sytd' according to the second embodiment of the present invention. [Figure 14] FIG. 10 is a diagram showing an example of a definition of the data structure of attribute information 'sytc' according to the second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] [Embodiment 1] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] In the embodiment described below, the present invention is applied to a synchronized shooting controller, which is an example of an image processing device, capable of acquiring image data captured by a plurality of camera devices connected via a network and generating an image file. However, the present invention can be applied to any device that can acquire image data and the shooting time associated with a predetermined master clock at which the image data was captured and generate an image file.

[0013] In this specification, "image data" refers to digital data representing one or more images obtained by photography, and may include either or both of still images and moving images. Furthermore, "image data" may be data whose format is determined according to the file format of the image file generated by storing the image data, and is not limited to encoded data.

[0014] Furthermore, "synchronization accuracy (of photography)" is a concept that quantifies the simultaneity of photography for a group of image data that have been photographed in a timely synchronized manner. For example, in one aspect, "synchronization accuracy" may be derived based on the difference in photography timestamps between image data. In another aspect, "synchronization accuracy" may be derived based on the time difference between the built-in clocks of different devices when different devices photograph at designated photography times. Alternatively, in yet another aspect, "synchronization accuracy" may be derived by combining these pieces of information.

[0015] <Configuration of synchronized imaging system> Fig. 1 is a schematic diagram illustrating the configuration of a synchronized photography system according to this embodiment. The synchronized photography system is configured to allow a plurality of camera devices 200 to cooperate with one another and to enable a synchronized photography controller 100 to record image data obtained by each of the camera devices 200 performing temporally synchronized photography as a single image file. In the example of Fig. 1, the synchronized photography controller 100 and four camera devices 200a-d are connected via a network 300, enabling transmission and reception of information. In this embodiment, the synchronized photography controller 100 sends a photography instruction, which causes each camera device 200 to execute photography processing, and the image data (encoded data) obtained by photography is returned to the synchronized photography controller 100.

[0016] In this embodiment, the system will be described assuming that four camera devices 200 are included, but it will be readily understood that the number of camera devices 200 is not limited to this in the practice of the present invention. In addition, in this embodiment, the system will be described assuming that the photographing units that photograph the subject at each of the four locations are provided in different camera devices 200, but the practice of the present invention is not limited to this. For example, it goes without saying that multiple photographing units may be provided in one camera device 200. In addition, in the following description, to facilitate understanding of the invention, the four camera devices 200 will be described assuming that they all have the same functional configuration, but it will be readily understood that the practice of the present invention is not limited to this.

[0017] Furthermore, the network 300 may be, for example, a LAN or a public line such as the Internet. The connections between the network 300 and each device may be realized by any method, whether wired or wireless.

[0018] <Functional configuration of the synchronized imaging controller> 2 is a block diagram illustrating the functional configuration of the synchronous imaging controller 100 according to this embodiment. Each functional configuration is configured to be able to transmit information via a system bus 110.

[0019] The synchronized shooting system of this embodiment performs synchronized shooting using the multiple camera devices 200 and generates an image file containing multiple images obtained by the synchronized shooting. The CPU 101 executes a predetermined application program. The program and a system program related to the operation of the synchronized shooting controller 100 are stored in a nonvolatile storage device, ROM 102, and the CPU 101 executes the program by reading and loading them into RAM 103. The RAM 103 is a temporary storage device and is used as a program loading area and a work area for the operation of each block. More specifically, the RAM 103 is also used as an input / output buffer related to encoding and decoding by a codec 105 (described later) and as a data buffer for image file generation processing by a file generation unit 104. The ROM 102 also stores user setting data for the synchronized shooting controller 100.

[0020] The file generation unit 104 executes processes related to the generation and editing of image files. Processable image files may include files in formats such as HEIF, JPEG, JPEG2000, JPEG XR, WebP, MP4, MOV, AVI, and WebM. The file generation unit 104 also reads encoded data (image data) from image files stored on a recording medium 109, and transmits it to a codec 105 for decoding.

[0021] The codec 105 is a video codec for moving images and still images, such as H.265 (HEVC), H.264 (AVC), AV1, JPEG, etc. The codec 105 encodes and decodes data of still images and moving images handled by the synchronous shooting controller 100.

[0022] The display unit 106 is a display device such as a liquid crystal display (LCD) that is configured integrally with the synchronous shooting controller 100 or that is detachably provided on the synchronous shooting controller 100. In this embodiment, the display unit 106 displays a screen of an application related to synchronous shooting.

[0023] The operation input unit 107 is various user interfaces provided in the synchronized shooting controller 100, such as operation buttons, switches, a mouse, and a keyboard. In addition, in an aspect in which the display unit 106 is a touch panel, the operation input unit 107 may include a touch panel sensor. When the operation input unit 107 detects that an operation input has been made to the user interface, it outputs a control signal indicating that fact to the CPU 101.

[0024] The communication control unit 108 is a network interface that connects to the network 300 and transmits and receives transmission frames. The communication control unit 108 may be, for example, a PHY and MAC (transmission media control processing) that can connect to a wired LAN using Ethernet (registered trademark). Alternatively, if the synchronous imaging controller 100 can connect to a wireless LAN, the communication control unit 108 may include a controller that executes wireless LAN control such as IEEE802.11a / b / g / n / ac / ax, an RF circuit, and an antenna.

[0025] The communication control unit 108 also has a built-in hardware clock for PTP. As will be described in detail later, the synchronized shooting system of this embodiment performs time-synchronized communication using the Precision Time Protocol (PTP) defined in IEEE1588-2008 (also known as IEEE1588 version 2). Synchronized shooting is achieved by having multiple camera devices 200 work together under the control of the synchronized shooting controller 100, so the synchronized shooting controller 100 has a PTP grandmaster clock (GMC) function. The PTP grandmaster clock function is implemented as PTP stack software executed by the CPU 101, and uses the time of a hardware clock built into the communication control unit 108 as its reference. The communication control unit 108 is configured to be able to output timestamps for sending and receiving transmission frames based on the time of the hardware clock.

[0026] The recording medium 109 is a non-volatile recording device with a large storage capacity, such as a hard disk drive, an SD card, a CompactFlash (registered trademark) card, etc. In this embodiment, the recording medium 109 is used as a storage destination for image files generated based on synchronous shooting.

[0027] <Functional configuration of the camera device> Next, an example of the functional configuration of each camera device 200 of this embodiment will be described using the block diagram of Fig. 3. As shown in the figure, each functional configuration of the camera device 200 is configured to be able to transmit information via a camera system bus 210. Note that among the functional configurations of the camera device 200, those that play the same role as the functional configuration of the synchronous shooting controller 100 are prefixed with the letter "camera" to clearly distinguish them.

[0028] The processing related to time synchronization and photography for synchronous photography is realized by the camera CPU 201 executing a predetermined application program. This program and system programs related to the operation of the camera device 200 are stored in the camera ROM 202, which is a non-volatile storage device, and the camera CPU 201 executes them by reading them and expanding them in the camera RAM 203. The camera RAM 203 is a temporary storage device and is used as a program expansion area and a work area for the operation of each block. More specifically, the camera RAM 203 is also used as an input / output buffer related to encoding and decoding by the camera codec 205, which will be described later. The camera ROM 202 also stores user setting data for the camera device 200, etc.

[0029] The photographing unit 204 is a unit with an imaging function that includes a lens group, a CCD (photoelectric conversion element), a CCD control unit, and an image processing unit. The lens group is composed of multiple lenses that optically form an image of a subject on the imaging surface of the CCD. The CCD control unit includes a timing generator that supplies a transfer clock signal and a shutter signal to the CCD, a circuit that performs noise reduction and gain processing on the CCD output signal, and an A / D conversion circuit that converts analog signals into digital signals. The image processing unit performs image processing such as gamma conversion, color space conversion, white balance, and exposure correction on the digital signal output from the CCD control unit, and outputs digital image signal data (image data) that can be encoded by the camera codec 205 to the RAM 203.

[0030] The camera codec 205 is a video codec for moving images and still images, such as H.265 (HEVC), H.264 (AVC), AV1, JPEG, etc. The camera codec 205 executes encoding and decoding processes for still image and moving image data handled by the camera device 200.

[0031] The camera display unit 206 is a display device such as a liquid crystal display (LCD) that is configured integrally with the camera device 200 or that is detachably provided on the camera device 200. In this embodiment, the camera display unit 206 displays a screen of an application related to synchronous shooting.

[0032] The camera operation input unit 207 is a variety of user interfaces provided in the camera device 200, such as operation buttons and switches. In addition, in an aspect in which the camera display unit 206 is a touch panel, the camera operation input unit 207 may include a touch panel sensor. When the camera operation input unit 207 detects that an operation input has been made to the user interface, it outputs a control signal indicating that fact to the camera CPU 201.

[0033] The camera communication control unit 208 is a network interface that connects to the network 300 and transmits and receives transmission frames. The camera communication control unit 208 may be, for example, a PHY and MAC (transmission media control processing) that can connect to a wired LAN using Ethernet (registered trademark). Alternatively, if the camera device 200 can connect to a wireless LAN, the camera communication control unit 208 may include a controller that executes wireless LAN control such as IEEE802.11a / b / g / n / ac / ax, an RF circuit, and an antenna.

[0034] The camera communication control unit 208 also has a built-in hardware clock for PTP. In the synchronized shooting system of this embodiment, the camera device 200 has a PTP slave function. The PTP slave function is implemented as PTP stack software executed by the camera CPU 201, and synchronizes the hardware clock built into the camera device 200 based on time information supplied from the synchronized shooting controller 100. Thereafter, time synchronization is performed by synchronizing the time of the system timer of the operation system of the camera device 200 executed by the camera CPU 201 with the hardware clock of the camera device 200. The camera communication control unit 208 is configured to be able to output timestamps for sending and receiving transmission frames based on the time of the built-in hardware clock.

[0035] The camera recording medium 209 is a non-volatile recording device with a large storage capacity, such as a hard disk drive, an SD card, a CompactFlash (registered trademark) card, etc. In this embodiment, the camera recording medium 209 may be used to store image data such as still images and moving images obtained by shooting.

[0036] "Synchronized shooting" The following describes the synchronous imaging performed in the synchronous imaging system of this embodiment and the generation of one image file that stores multiple image data acquired by the synchronous imaging.

[0037] As described above, in the synchronized shooting system of this embodiment, the synchronized shooting controller 100 serves as a grandmaster clock, and the system timers of the devices in the synchronized shooting system are synchronized by time-synchronized communication. Then, the synchronized shooting controller 100 provides a shooting time, and each camera device 200 captures an image when its internal clock reaches the shooting time. Image data obtained by the capture is sent to the synchronized shooting controller 100, which receives multiple image data captured at designated shooting times and stores and records them in a single image file. At this time, the multiple image data can be considered to be image data captured in a time-synchronized manner, and are grouped in the image file as synchronously captured images. Grouping may be achieved, for example, by including an identifier for the grouped image data in the metadata of the image file.

[0038] Here, the file format of the image file may be any format that stores multiple image data and has metadata that describes groupings of image data that were captured synchronously, but the following explanation will use the HEIF format as an example.

[0039] HEIF file structure The file structure of a HEIF file will be explained using FIG. 4. As shown in FIG. 4, a HEIF file 400 is generally composed of the following three boxes (storage areas). The first is FileTypeBox('ftyp') 401, which stores the brand name used by the reader of the HEIF file 400 to identify the file specifications. In the HEIF file 400 generated by the synchronized capture controller 100 of this embodiment, the 'ftyp' box 401 is set to 'mif1' as the brand name and 'heic' as the compatible brand name. The second is MetaBox('meta') 402, which stores metadata describing various information about the image data stored in the HEIF file 400. As shown in the figure, the 'meta' box 402 stores multiple types of information related to the image data, each stored in a separate box, as described in detail below. The third is MediaDataBox('mdat') 412, which stores multiple pieces of encoded data (image data) as image items. In the following description, the coded data stored in the 'mdat' box 412 will be referred to as either an "image item" or "image data", depending on the context.

[0040] HandlerReferenceBox('hdlr') 403 stores a declaration of the handler type for analyzing the structure of the 'meta' box 402. In the HEIF file 400 generated by the synchronized shooting controller 100 of this embodiment, all of the encoded data stored is assumed to be still images, and the handler type name 'pict' is set in the 'hdlr' box 403.

[0041] PrimaryItemBox('pitm') 404 specifies the identifier (item ID) of the encoded data that is to be the representative item among the image items stored in the HEIF file 400.

[0042] ItemLocationBox('iloc') 405 stores information indicating the storage location of the data of each image item in the HEIF file 400. The 'iloc' box 405 typically describes the storage location of the image item as a byte offset from the beginning of the HEIF file 400 and the data length from that beginning. In other words, the information in the 'iloc' box 405 makes it possible to identify the location of each piece of encoded data stored in the 'mdat' box 412.

[0043] ItemInfoBox('iinf') 406 defines basic information such as item ID, item type indicating the item type, and item name for all image items included in the HEIF file 400.

[0044] The ItemReferenceBox('iref') 407 stores information describing the association between image items included in the HEIF file 400. In an embodiment where the image item is a photographed image, the 'iref' box 407 is used to describe the association between the image item and an item of its photographic information (such as Exif data). In an embodiment where multiple image items are in a derived image relationship, the 'iref' box 407 is used to describe the association between the image items.

[0045] ItemPropertiesBox ('iprp') 408 stores various attribute information about image items included in the HEIF file 400. More specifically, the 'iprp' box 408 includes an ItemPropertyContainerBox ('ipco') 409 that describes the attribute information, and an array of ItemPropertyAssociation ('ipma') boxes 410 that indicate the association between the attribute information and each image item. The 'ipco' box 409 may list and store, for example, the creation date and time of the image item, information indicating the width and height of the image item in pixels, and HEVC parameter set data required for decoding the HEVC image item. Meanwhile, the 'ipma' box 410 indicates the association by storing, for example, the item ID of the corresponding image item for each piece of attribute information stored in the 'ipco' box 409. Here, if multiple image items correspond to the same attribute information, this may be stated in a single 'ipma' box 410.

[0046] TimeSychronizedEntityToGroupBox('tsyn') 411 stores information identifying a group (synchronization group) defined by grouping image items captured in a time-synchronized manner. The information stored in the 'tsyn' box 411 may include at least information that identifies the image items included in one synchronization group, for example, in the form of a list of item IDs. Note that multiple synchronization groups may be defined for one HEIF file 400, and the same number of 'tsyn' boxes 411 may be stored in the 'meta' box 402 as there are synchronization groups.

[0047] Note that the file structure of the HEIF file 400 shown in FIG. 4 is one example, and the order in which the boxes are stored, the manner in which the storage areas are separated, and the type of stored information are not limited to this.

[0048] As mentioned above, conventional HEIF files only specify that image data captured at the same time over the same period be grouped and described in the 'tsyn' box 411, and are not structured with this use in mind.

[0049] For example, when adding new image data to a HEIF file, it was not possible to determine whether the new image data could be added to the synchronization group defined in the 'tsyn' box 411, and the extensibility of the synchronization group was not ensured. In other words, when adding image data at a time other than when the HEIF file was created, it was difficult to determine whether the image data could be "image data taken at the same time over the same period" as image data already stored in the synchronization group.

[0050] Furthermore, particularly in an embodiment where image data captured by multiple different camera devices are stored in an HEIF file, such as in this embodiment, it is difficult to completely synchronize the system timers between the devices even when time-synchronized communication is performed. Therefore, in an embodiment where multiple image data requiring simultaneous capture is used for processing, such as when generating 3D video as in Patent Document 1, it is not possible to determine whether the image data included in one synchronization group meets the required capture synchronization accuracy. In other words, when using multiple image data included in a synchronization group in an HEIF file, it is difficult to determine whether the multiple image data ensures the capture synchronization accuracy appropriate for the intended use. In other words, it is difficult to determine the degree to which the multiple image data included in the synchronization group are temporally synchronized.

[0051] Taking such file usage into consideration, the synchronized capture controller 100 of this embodiment describes a new configuration method for the 'meta' box 402 that makes it possible to easily evaluate the synchronization accuracy of image data included in a synchronization group. More specifically, in this embodiment, within the framework of the file structure of the HEIF file 400 shown in Figure 4, the data structures of the 'tsyn' box 411 and the 'iinf' box 406 are defined so that information for evaluating the synchronization accuracy related to the synchronization group can be included.

[0052] 5 shows a definition 501 of the data structure of the 'tsyn' box 411 in a HEIF file according to this embodiment. As shown in definition 501, the 'tsyn' box 411 in this embodiment, which relates to one synchronization group, includes synchronization group information 502 that identifies the grouped image items, and index information 503 that indicates the synchronization accuracy of the capture of the image data relating to that group.

[0053] As shown in the figure, the synchronization group information 502 describes, for example, a group ID (group_id) that is an identifier for the synchronization group, the number of entries included in the group (num_entities_in_group), and a list of entry IDs (entity_id). Here, the entry ID specifies the item ID of the image item stored in the HEIF file. In other words, the synchronization group information 502 is information that identifies image data that has been grouped as having been captured in a time-synchronized manner among the image data stored in the 'mdat' box 412, as previously considered.

[0054] On the other hand, considering future uses not limited to synchronized shooting systems for HEIF files, it is preferable that index information 503 indicating the synchronization accuracy of shooting be configured to enable a variety of determinations, not just at the time of HEIF file generation. Therefore, in this embodiment, index information 503 is configured to include synchronization method information 504, reference time information 505, synchronization error information 506, and allowable error information 507.

[0055] The synchronization method information 504 describes information that identifies the time synchronization method used by the devices that captured the image data included in the synchronization group. Specifically, the synchronization method information 504 may be information that specifies a time synchronization standard or protocol. In addition, the synchronization method information 504 may include information specific to the time synchronization standard or protocol. For example, if the synchronization group is composed of image data captured using the synchronized capture system of this embodiment, the information may be information indicating that the time synchronization standard is IEEE1588-2008 and that the grandmaster clock is the synchronized capture controller 100.

[0056] The reference time information 505 describes information specifying the time (reference time) that is the reference for synchronous shooting for the image data included in the synchronous group. For example, when the synchronous group is made up of image data captured by the synchronous shooting system of this embodiment, the reference time information 505 may be information indicating the designated shooting time (shooting timestamp or time) supplied from the synchronous shooting controller 100 to each camera device 200.

[0057] The synchronization error information 506 describes information indicating the state of synchronization error between devices in synchronized shooting related to the synchronization group. In other words, the synchronization error information 506 describes information indicating the error in simultaneity related to image data included in the synchronization group. For example, when PTP or NTP is used for time synchronization, the synchronization error information 506 may be the maximum time difference between the system clock of the camera device 200 that captured the image data and the master clock of the synchronized shooting controller 100 at the time the image data was captured. In other words, it may be the maximum deviation of the image data capture time from the designated capture time on a time axis based on the grand master clock. Alternatively, the synchronization error information 506 may be the maximum time difference between the system clocks of the camera devices 200 that captured the image data included in the synchronization group. In other words, it may be the time difference between the earliest and latest capture times of multiple image data on a time axis based on the grand master clock.

[0058] The allowable error information 507 describes information indicating the error range of the synchronization time allowed for inclusion in a synchronization group. In other words, the allowable error information 507 describes information defined for a synchronization group indicating the minimum level of synchronicity that must be maintained for image data in the synchronization group. The allowable error information 507 may indicate the conditions under which image data can be included in a synchronization group in absolute terms, in the form of the time difference between the system clock of the camera device 200 that captured the image data and the master clock of the synchronized capture controller 100. Alternatively, the allowable error information 507 may indicate the conditions in relative terms, in the form of the time difference between the system clocks of the camera devices 200 that captured the image data. As will be described in detail later, the information described in the allowable error information 507 is determined based on the intended use of the HEIF file and the synchronization accuracy options to be guaranteed, which are specified, for example, when the synchronized capture controller 100 performs processing related to HEIF file generation. In other words, the information in the tolerance information 507 is not determined based on the image data acquired in connection with file generation or the state of the camera device 200 that captured the image data, but is determined in advance taking into account the extensibility of the HEIF file.

[0059] That is, while the synchronization error information 506 is information indicating the synchronization accuracy of photography determined based on the image data included in the synchronization group, the allowable error information 507 is information indicating the synchronization accuracy of photography absolutely determined for the synchronization group. Therefore, for multiple image data included in the synchronization group, the minimum synchronization accuracy of photography specified by the allowable error information 507 is guaranteed, and more specific synchronization accuracy is indicated by the synchronization error information 506.

[0060] By defining the data structure of the 'tsyn' box 411 in this way, when using an HEIF file, it is possible to understand the synchronization accuracy of the image capture of the corresponding synchronization group by referencing the index information 503. More specifically, it is possible to understand that image data captured at a specified capture time on a device time-synchronized by specific time synchronization communication is grouped using the synchronization method information 504 and the reference time information 505. Furthermore, by further referencing the synchronization error information 506, the maximum time difference from the specified capture time of the image data included in the synchronization group or the maximum time difference between the capture times of the image data within the group can be obtained, thereby understanding the synchronization accuracy of the image capture. Alternatively, by further referencing the allowable error information 507, the allowable time difference from the specified capture time of the image data that can be included in the synchronization group or the allowable time difference between the capture times of the image data can be obtained, thereby understanding the synchronization accuracy of the synchronization group. In other words, the index information 503 not only indicates the synchronization accuracy of the image data captured in the synchronization group, but also indicates an index of the grouping of the image data for the synchronization group.

[0061] In a synchronized shooting system like this embodiment, which generates an HEIF file using image data obtained by providing a designated shooting time from the synchronized shooting controller 100, the shooting timestamps of the resulting image data are consistent. That is, after time-synchronized communication, the system clock of each camera device 200 instructs the camera to shoot at the designated shooting time, and the shooting timestamp attached to the image data captured by each camera device 200 indicates the designated shooting time. Therefore, in the synchronized shooting system of this embodiment, when image data is provided from each camera device 200, information on the synchronization error (device error information) of the camera device's 200 system clock relative to the grandmaster clock is associated with and transmitted. The synchronized shooting controller 100 then constructs the 'tsyn' box 411 (synchronization group information 502 and index information 503) of the HEIF file based on the shooting timestamp and device error information associated with the image data. While this embodiment describes the device error information as being derived by each camera device 200, the implementation of the present invention is not limited to this. The device error information may be derived by at least one of the synchronized shooting controller 100 and the camera device 200.

[0062] On the other hand, considering the scalability of synchronized groups, it is preferable that HEIF files be configured to allow for the addition of image data captured without a designated capture time by camera devices time-synchronized using the same method as the devices in the synchronized shooting system. In other words, it is preferable that HEIF files be configured to allow for the definition of synchronized groups without being limited to image data captured synchronously by specifying a capture time under the control of the synchronized shooting controller 100. The simultaneity of shooting in a synchronized group can be measured if it is possible to compare the shooting times on a common time axis for the multiple camera devices 200 that capture image data. Therefore, even image data obtained without controlling the shooting time can be grouped. For example, image data obtained by each camera device 200 capturing images upon receiving a shooting instruction simultaneously transmitted from the synchronized shooting controller 100 may be grouped in an HEIF file. Alternatively, for example, in a configuration in which multiple camera devices (shooting units) are operated independently without forming a synchronized shooting system, image data captured at the same time may be grouped in an HEIF file.

[0063] Here, time synchronization between these devices does not need to be achieved as in the synchronized shooting system of this embodiment. In other words, in the synchronized shooting system of this embodiment, the synchronized shooting controller 100 has a grandmaster clock function and time-synchronizes the camera devices 200 belonging to the system, but the time reference for synchronization may be provided by an external device. In other words, the device that acquires image data and generates the HEIF file does not need to be the device that provides the time reference for synchronization; at least the time synchronization of the multiple camera devices that perform shooting may be achieved through time-synchronous communication with any device. In other words, the generation of the HEIF file according to the present invention can be achieved using image data captured by the camera devices 200 belonging to the synchronized shooting system, as well as image data captured by any device that is time-synchronized using the same method.

[0064] Therefore, the 'meta' box 402 may be configured to store information (photography timestamp) indicating the time of shooting of each image data stored in the 'mdat' box 412. The photography timestamp may be obtained from photography information (including device error information) that is provided in association with the image data and stores various information related to the shooting of the image data. Information related to the image data is stored by defining the data structure of the 'iinf' box 406 (601) as shown in Figure 6.

[0065] As shown in the figure, definition 601 includes number of entries 602 (16 bits or 32 bits) indicating the number of image items stored in 'mdat' box 412, and an array 603 of ItemInfoEntry ('infe') that stores various information related to each image item. 'infe' array 603 is an array in which data equal to the number of entries 602 is arranged, and stores various information about each image item.

[0066] The data structure of 'infe' for one image item may be defined (701) as shown in FIG. 7. As shown in the figure, definition 701 defines basic information 702, such as the item ID of the image item, the item type indicating the type of image item, and the item name of the image item, as well as timestamp information 703 describing the capture timestamp. In the example shown in the figure, the timestamp information 703 stores the capture timestamp as a 64-bit integer value representing the number of elapsed seconds in Coordinated Universal Time (UTC) in item_ts_sec, and as a 32-bit integer value representing nanoseconds less than an integer second in item_ts_nsec. Note that the description format of timestamp information 703 is not limited to this; for example, the capture timestamp may be described in NTP timestamp format using a 32-bit second value and a 32-bit nanosecond value. Alternatively, the capture timestamp may be described in PTP timestamp format using International Atomic Time Interchange (TAI) using a 48-bit second value and a 32-bit nanosecond value.

[0067] By allowing the shooting timestamp of each image data item to be stored in the 'meta' box 402 in this way, it is also possible to perform grouping using the shooting timestamp of image data already stored in the 'mdat' box 412. In this embodiment, for example, instead of storing a specified shooting time in the reference time information 505 of the 'tsyn' box 411, the item ID of the image data may be written to specify the shooting timestamp of the image data as the reference time for the synchronization group. In this case, the synchronization error information 506 may store the maximum time difference between the shooting time of the image data specified in the reference time information 505 and other image data in the synchronization group. Furthermore, the allowable error information 507 may describe the conditions under which image data can be included in a synchronization group in the form of the time difference from the shooting time of the image data specified in the reference time information 505.

[0068] <File generation procedure> The file generation procedure for generating an HEIF file containing multiple pieces of image data synchronously captured in the synchronized capture system of this embodiment will be described in detail below with reference to Fig. 8. As shown in the figure, the file generation procedure is realized by the synchronized capture controller 100 and camera device 200 performing the relevant processes, i.e., by the CPU 101 and camera CPU 201 executing the corresponding processes. Note that the camera device 200 procedure shown in Fig. 8 is executed by each of the camera devices 200a-d communicating with the synchronized capture controller 100.

[0069] When generating an HEIF file, the intended use of the multiple image data captured synchronously is selected. The information on the selected intended use is used to determine the upper limit to be stored in the allowable error information 507 in the 'tsyn' box 411. The intended use may be selected by specifying the application that will use the HEIF file generated in connection with the synchronized shooting, or the synchronization accuracy required when combining the image data included in the synchronization group.

[0070] In step 801, time synchronous communication is performed between the synchronized shooting controller 100 and the camera device 200, and the times on the system timers of each device are synchronized. As described above, in the synchronized shooting system of this embodiment, the synchronized shooting controller 100 has a PTP grandmaster clock function, and therefore the time on the system timer of the camera device 200 is synchronized to the grandmaster clock through time synchronous communication. Note that in the description of the file generation procedure in FIG. 8 , time synchronous communication is described as being performed only in step 801 to facilitate understanding of the invention. However, the present invention is not limited to this, and time synchronous communication may be performed at a predetermined frequency. When time synchronous communication is performed, the camera device 200 stores the PTP offset value used to adjust the system timer or hardware clock. Here, the system timers of the synchronized shooting controller 100 and the camera device 200 are temporarily synchronized through time synchronous communication, but strictly speaking, there may be slight deviations even after time synchronization. Furthermore, the PTP offset value may fluctuate each time time synchronous communication is performed. Therefore, it is assumed that the camera device 200 holds PTP offset values ​​for multiple time-synchronized communications.

[0071] When the time-synchronized communication is completed, in step 802, the synchronized shooting controller 100 transmits information about the designated shooting time and a shooting instruction for synchronized shooting to the camera device 200. Note that in this embodiment, the description will be given assuming that the information about the designated shooting time is supplied and the camera device 200 performs shooting, but as mentioned above, the implementation of the present invention is not limited to this.

[0072] In step 803, a photographing process is executed at the designated photographing time in the camera device 200. The photographed image obtained by the photographing process is coded by the camera codec 205 and stored as image data.

[0073] In step 804, the image data generated in step 803 is transmitted together with the photographing information from the camera device 200 to the synchronous photographing controller 100. At this time, the device error information included in the photographing information may be the PTP offset value or the moving median of the PTP offset values ​​for the most recent predetermined number of times.

[0074] In step 805, the synchronized shooting controller 100 generates an HEIF file based on the multiple image data received in connection with synchronized shooting. For simplicity's sake, in this embodiment, it is assumed that the multiple image data captured by each camera device 200 in response to a synchronized shooting command are stored in the 'mdat' box 412 of a single HEIF file. Furthermore, based on information regarding the synchronization accuracy of the synchronized shooting that is to be guaranteed and that is preset for synchronized shooting, the image data among the multiple image data that meets the conditions for inclusion in a synchronization group is included in the synchronization group. Furthermore, it is assumed that only one synchronization group is defined in a single generated HEIF file.

[0075] At this time, under the control of the CPU 101, the file generation unit 104 configures various information to be stored in the 'meta' box 402 related to the HEIF file to be generated based on the shooting information received for the multiple pieces of image data. For example, in a mode in which the allowable upper limit of the PTP offset value of the camera device 200 is set as a condition for inclusion in a synchronization group for synchronized shooting performed when a specified shooting time is provided, as in this embodiment, the 'meta' box 402 is configured as follows:

[0076] The file generation unit 104 generates synchronization method information 504 and reference time information 505 for the 'tsyn' box 411 based on information on synchronous time communication and specified shooting time, and stores an upper limit tolerance value based on information on the selected usage together with allowable error information 507. The file generation unit 104 also determines whether to include each image data in a synchronization group based on information in the allowable error information 507, and updates the synchronization error information 506 as appropriate while generating synchronization group information 502. Specifically, if the image data meets the conditions for inclusion in a synchronization group, the file generation unit 104 adds information about the image data as an entry in the synchronization group list and updates the synchronization group information 502. If the image data meets the conditions for inclusion in a synchronization group, the file generation unit 104 also updates the PTP offset value (absolute value) of the camera device 200 that captured the image data if it is greater than the current value of the synchronization error information 506. Here, the synchronization error information 506 has an initial value of 0, and is updated to that value if the absolute value of the PTP offset value for image data to be included in the group is greater. In this way, the file generation unit 104 can configure synchronization group information 502 and index information 503 for the synchronization group defined in the HEIF file to be generated.

[0077] Furthermore, the file generation unit 104 configures the number of entries 602 and the 'infe' array 603 based on the received image data for the 'iinf' box 406. When configuring 'infe' for each image data, a shooting timestamp is stored based on the shooting information received in association with the image data.

[0078] In this way, it is possible to generate an HEIF file for a defined synchronization group that includes information indicating the synchronization accuracy of shooting related to multiple image data included in the synchronization group.

[0079] Note that, in this embodiment, a mode in which a designated shooting time is supplied and an HEIF file containing multiple pieces of image data captured at that time is generated has been described, but the implementation of the present invention is not limited to this. For example, in a mode in which the camera device 200 captures images in response to receiving a shooting instruction from the synchronous shooting controller 100, the 'tsyn' box 411 may be configured differently as described above to generate an HEIF file. That is, when an HEIF file containing multiple pieces of image data captured without receiving a designated shooting time from the synchronous shooting controller 100 is generated, the 'meta' box 402 may be configured differently as follows:

[0080] In this case, the shooting timestamps of the image data may differ, and the shooting timestamps stored in 'infe' for each image data in the 'iinf' box 406 will not be uniform. Therefore, for the reference time information 505 in the 'tsyn' box 411, the file generation unit 104 stores, for example, the item ID of the image data indicating the earliest shooting timestamp among the image data to be grouped, and specifies that shooting timestamp as the reference time. The file generation unit 104 then identifies image data whose shooting timestamps fall within a time range determined based on the reference time and the upper tolerance limit, and constructs the synchronization group information 502. Furthermore, the file generation unit 104 constructs the synchronization error information 506 based on the shooting timestamp farthest from the reference time among the shooting timestamps of the image data included in the synchronization group. This also makes it possible to generate a HEIF file that includes a definition of the synchronization group and information indicating the synchronization accuracy of the shooting of the multiple image data included in the synchronization group.

[0081] <<Using HEIF files>> Next, we will explain how to use the HEIF file created in this way. It should be easy to understand that the HEIF file can be used in any device, not just the synchronized shooting controller 100 that created the file. When using the file, a processor such as the device's CPU reads the 'meta' box 402 of the HEIF file to be processed, and plays or changes the image data stored in the 'mdat' box 412, and edits the 'meta' box 402 as necessary.

[0082] One aspect of using HEIF files involves presenting the shooting synchronization accuracy for multiple image data included in a synchronization group defined in the 'tsyn' box 411 of the HEIF file based on the information in the synchronization group. The presentation of the shooting synchronization accuracy may be, for example, a mode in which at least one piece of information in the index information 503 for the HEIF file is displayed, or a mode in which the HEIF file is displayed if the requested synchronization accuracy is met.

[0083] In the former case, for example, it may be displayed that the image data of the group was taken within the range of the time difference indicated by the synchronization error information 506 with respect to the shooting time indicated by the reference time information 505. Alternatively, for example, it may be displayed that at least the shooting synchronization accuracy indicated by the allowable error information 507 is guaranteed for the image data of the group.

[0084] In the latter case, for example, when a file viewing application sets the shooting synchronization accuracy as an extraction condition, HEIF files that satisfy the extraction condition may be displayed based on the synchronization error information 506 or the allowable error information 507. Alternatively, for example, in an application that performs a compositing process using image data captured synchronously, HEIF files that satisfy the synchronization accuracy required for the compositing process may be displayed based on the synchronization error information 506 or the allowable error information 507.

[0085] When presenting the synchronization accuracy for these synchronization groups, some or all of the multiple image data classified into the synchronization groups defined by the synchronization group information 502 may also be presented.

[0086] Another aspect of HEIF file usage involves determining whether new image data should be added to a synchronization group when the new image data is added to the HEIF file (stored in the 'mdat' box 412). For example, when an instruction is given to add new image data to an HEIF file, the 'tsyn' box 411 is referenced to determine whether a defined synchronization group exists. If a defined synchronization group exists, the shooting information associated with the new image data is compared with the index information 503 to determine whether the new image data can be added to the synchronization group (grouped).

[0087] For example, if the synchronization method information 504 of the device that captured the new image data is different, it may be determined that the device cannot be added to the synchronization group. Furthermore, if the synchronization method information 504 is the same, it may be determined that the device can be added to the synchronization group if, for example, the capture timestamp of the new image data is within a time range defined by the reference time information 505 and the allowable error information 507. Alternatively, this determination may be made by taking into account the PTP offset value of the device that captured the new image data in addition to the capture timestamp. If it is determined that the device can be added to the synchronization group, information related to the new image data is added to the synchronization group information 502, thereby forming the image data into a group. Furthermore, if the synchronization accuracy of the image data within the group changes due to the addition of the new image data to the synchronization group, the synchronization error information 506 is updated.

[0088] Note that adding new image data to a synchronization group does not necessarily have to be performed on the condition that the synchronization method information 504 is common. For example, even if the device that captured the new image data was not time-synchronized at the time of capture using the same method as the synchronization method information 504 related to the synchronization group, the time of capture may be estimated based on information on time synchronization performed at any time after capture, and the addition determination may be performed.

[0089] As described above, by including index information 503 for synchronization groups defined in HEIF files according to this embodiment, the usability of image data included in the groups can be improved. More specifically, by including index information 503 for synchronization groups, it is possible to quantitatively indicate the degree to which the image data included in the groups were captured in chronological order, making it easier to determine whether the image data is suitable for a particular application. Furthermore, because index information 503 indicates the grouping conditions, it is easy to determine whether a predefined synchronization group should be updated when image data is added to an HEIF file. Alternatively, for example, an operation to directly add new image data to a predefined synchronization group can be accepted, and if the addition changes the synchronization accuracy of the group, the information in index information 503 can be updated. Alternatively, for example, an operation to delete any image data from a predefined synchronization group can be accepted, and if the deletion changes the synchronization accuracy of the group, the information in index information 503 can be updated.

[0090] [Embodiment 2] In the embodiment described above, synchronization group information and index information are included in the 'tsyn' box 411 of the HEIF file, and the shooting timestamp of each image data is included in the 'iinf' box 406, but the present invention is not limited to this. In the synchronized shooting system of this embodiment described below, the shooting timestamp of each image data and index information related to the synchronization group are stored in the 'iprp' box 408 instead of the 'tsyn' box 411 to generate an HEIF file.

[0091] 《HEIF file structure》 The following describes the structure of the HEIF file generated for synchronized shooting by the synchronized shooting controller 100 of this embodiment. More specifically, the structure of the generated HEIF file is configured similarly to that described in the first embodiment using Fig. 4, but the data structure definition for some boxes is different, and index information related to the synchronization group is included.

[0092] As described above, the 'iprp' box 408 stores various attribute information ('ipco' box 409) about image items included in the 'mdat' box 412, as well as information ('ipma' box 410) that identifies the image item corresponding to each attribute information. In other words, the 'ipco' box 409 and the 'ipma' box 410 are information indicating the association between attribute information that can be held by image items included in a HEIF file and the image item to which that attribute information belongs. From this perspective, the 'iprp' box 408 stores attribute information about any entry in a HEIF file in the 'ipco' box 409, and defines the association between that attribute information and the entry in the 'ipma' box 410. Therefore, in this embodiment, index information related to each synchronization group defined in the 'tsyn' box 411 is stored as attribute information in the 'ipco' box 409, and information indicating the association between the index information and the synchronization group is stored in the 'ipma' box 410.

[0093] To achieve this information storage, the data structure of the 'tsyn' box 411 is defined as shown in Figure 9, which differs from that of embodiment 1. As shown in the figure, in the HEIF file of this embodiment, the definition 901 of the data structure of the 'tsyn' box 411 for one synchronization group includes only synchronization group information 502 that identifies the grouped image items, and does not include index information.

[0094] Meanwhile, for index information, attribute information relating to each of synchronization method information, reference time information, synchronization error information, and allowable error information is newly defined so that it can be used when generating and using HEIF files, and can be stored in the 'ipco' box 409. Note that, as described in the first embodiment, to cover the case where synchronization error information and allowable error information are stored as differential values ​​of shooting timestamps, information on the shooting timestamp for each image item is also defined so that it can be stored in the 'ipco' box 409. That is, in the HEIF file of this embodiment, information on the shooting timestamp of an image item is stored in the 'iprp' box 408, not the 'iinf' box 406, as in the first embodiment.

[0095] First, the storage of the capture timestamp is realized by newly defining the data structure of CaptureTimeProperty('catt') shown in FIG. 10 as attribute information. As shown in the figure, the definition 1001 of 'catt' has a structure that can store the capture timestamp in a 64-bit timestamp format (1002) or a timestamp format (1003) that is a combination of a 64-bit second value and a 32-bit nanosecond value. In the former format (1002), the capture timestamp can be stored in a 64-bit NTP timestamp format with the upper 32 bits as a second value and the lower 32 bits as a fractional second value. In the latter format (1003), the capture timestamp can be stored as a PTP timestamp. Definition 1001 makes it possible to store attribute information of the capture timestamp of an image item as 'catt' in the 'ipco' box 409. Note that the HEIF specification already defines attribute information for CreationTimeProperty ('crtt'), which stores the creation date and time of an image item, so the shooting timestamp can be stored by extending the data structure of that attribute information.

[0096] Next, the synchronization method information in the index information is stored by newly defining, as attribute information, the data structure of SynchronizedTimeMethodProperty('sytm') shown in Fig. 11. As shown in the figure, a definition 1101 of 'sytm' has a structure that can store character string information 1102 indicating the time synchronization standard or protocol for time synchronization performed in synchronized shooting, and a 64-bit PTP grandmaster clock identifier 1103. The information 1102 may further include unique information according to the standard or protocol.

[0097] Furthermore, the storage of the reference time information from the index information is realized by newly defining as attribute information the data structure of SynchronizedCaptureTimeProperty('syct') shown in Fig. 12. As shown in the figure, the definition 1201 of 'syct' has a structure that can store information specifying the reference time of synchronized shooting in the form of a shooting timestamp (1202) or the item ID of an image item associated with the shooting timestamp of the reference time.

[0098] Furthermore, the synchronization error information included in the index information can be stored by, for example, newly defining the data structure of SynchronizedTimeDifferenceProperty('sytd') shown in Fig. 13 as attribute information. As shown in the figure, the definition 1301 of 'sytd' has a structure that can store information indicating the synchronicity error related to image items included in a synchronization group as the maximum value of the time difference between the shooting timestamps (1302). Alternatively, it has a structure that can store the same information as the maximum value (offset value) (1303) of the time difference between the system clock at the time of shooting in the camera device 200 that performed the shooting and the master clock of the synchronized shooting controller 100.

[0099] Furthermore, storage of the allowable error information from the index information is realized by, for example, newly defining the data structure of SynchronizedTimeCriteria('sytc') shown in Fig. 14 as attribute information. As shown in the figure, the definition 1401 of 'sytc' has a structure that can store the time difference format (1402) between the system clock at the time of shooting in the camera device 200 that performed the shooting and the master clock of the synchronized shooting controller 100. Alternatively, it has a structure that can store information indicating the allowable error range of simultaneity for image items included in a synchronization group in the format of the time difference of a 64-bit shooting timestamp (1403).

[0100] By defining five new types of attribute information in this way, index information related to synchronization groups can be stored in the HEIF file's 'iprp' box 408. That is, the defined attribute information for 'sytm', 'syct', 'sytd', and 'sytc' is stored in the 'ipco' box 409, and information relating these to the synchronization group's group ID is stored in the 'ipma' box 410, allowing the index information to be included.

[0101] Therefore, in the file generation procedure in the synchronized photography system of this embodiment, the file generation unit 104 stores attribute information for 'catt' in the 'ipco' box 409 based on the photography information received for multiple image data. Furthermore, in the grouping process, the file generation unit 104 stores attribute information for 'sytm' and 'syct' in the 'ipco' box 409 based on information about synchronous time communication and the specified photography time. Furthermore, the file generation unit 104 stores attribute information for 'sytc' in the 'ipco' box 409 based on information about the selected usage. Furthermore, the file generation unit 104 determines whether each image data item should be included in a synchronization group based on the 'sytc' information, and stores / updates 'sytd' as appropriate while configuring the synchronization group information 502. Specifically, if the image data meets the conditions for inclusion in a synchronization group, the file generation unit 104 adds information about the image data item to the synchronization group as an entry and updates the synchronization group information 502. Furthermore, when image data meets the conditions for inclusion in a synchronization group, the file generation unit 104 updates the PTP offset value (absolute value) of the camera device 200 that captured the image data if it is greater than the current 'sytd' value. Here, 'sytd' has an initial value of 0, and is updated to that value if, for example, the absolute value of the PTP offset value for image data to be included in the group is greater. After the file generation unit 104 has determined whether or not to include a received image data group in a synchronization group, it stores information in the 'ipma' box 410 that associates the group ID of the synchronization group with four types of attribute information related to the configured index information. In this way, the file generation unit 104 can configure synchronization group information 502 and index information for the synchronization group defined in the HEIF file it generates.

[0102] The use of the HEIF file generated in this manner may be the same as in the first embodiment except for the method of reading the index information, and therefore the description will be omitted.

[0103] [Variation 1] In the above-described first and second embodiments, an HEIF file including synchronization group information indicating that multiple pieces of image data captured in a time-synchronized manner have been grouped, and index information indicating an index for the synchronization accuracy of the capture of the multiple pieces of image data or the grouping, has been described as an example. However, as described above, the implementation of the present invention is not limited to files in the HEIF format, and can be applied to any file including synchronization group information and index information.

[0104] For example, the present invention can be applied to a synchronized shooting system in which multiple camera devices 200 each capture video and store the resulting video data in a single file. That is, according to the present invention, a single video file can be generated that stores multiple video data sets and includes, as metadata, synchronization group information that defines groups of video data captured in a time-synchronized manner among the stored video data sets and index information that serves as the basis for the grouping. In this case, the generated video file may be in a file format based on the ISO Base Media File Format (ISOBMFF), similar to HEIF. For example, in a case in which the camera devices 200 generate video data encoded using H.264, the synchronized shooting controller 100 may generate an MP4 file that stores the video data captured by each camera device 200 as separate video and audio tracks.

[0105] In an MP4 file, metadata is written in a MovieBox ('moov'), so by including a 'tsyn' box in 'moov', it is possible to describe track grouping. Note that the list of entry IDs in the 'tsyn' box may specify track IDs. In addition, in the case of video data, the shooting timestamp is the start time of video shooting in each camera device 200, and can be stored in a TrackHeaderBox ('tkhd') box in 'moov'. Index information may be stored in the 'tsyn' box, for example, as in the first embodiment.

[0106] [Variation 2] In the above-described embodiment, the allowable upper limit value stored in the allowable error information is determined based on the selected use of the image data captured synchronously when generating an HEIF file. However, the present invention is not limited to this. For example, the allowable upper limit value may be fixed for a synchronous shooting system, or may be dynamically determined depending on the synchronous shooting method (e.g., whether or not a specified shooting time is provided) or the status of time synchronization between devices.

[0107] [Variation 3] In the above-described embodiment and modified examples, the index information has been described as including synchronization method information, reference time information, synchronization error information, and allowable error information, but the implementation of the present invention is not limited to this and may include only some of the information. In other words, from the perspective of using multiple image data included in a synchronization group, the index information only needs to indicate the simultaneity of capture of this image data, and does not need to be configured to include all of the information.

[0108] Furthermore, from the perspective of indicating the synchronization accuracy of the capture of multiple image data included in a synchronization group, it is sufficient to include either synchronization error information or tolerance information. Furthermore, from the perspective of indicating the time range that can be included in a synchronization group, it is sufficient to include reference time information and tolerance information, or reference time information and synchronization error information. In this case, if it is ensured that the time synchronization method of the capture device is the same for image data stored / added to / in a HEIF file, the index information does not need to include synchronization method information.

[0109] [Other embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0110] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0111] 100: Synchronous shooting controller, 101: CPU, 104: File generation unit

Claims

1. 1. An image processing device that generates an image file having a structure including a first storage area for storing a plurality of image data, and a second storage area for storing metadata relating to the plurality of image data, an acquisition means for acquiring the plurality of image data and photographing information relating to photographing of the image data; A configuration means for configuring, based on the photographing information acquired by the acquisition means, (1) synchronized group information including information for grouping image data photographed in a time-synchronized manner and information indicating image data included in the group among the plurality of image data, and (2) index information indicating a grouping index for the group related to the synchronized group information; a generating means for generating the image file that stores the plurality of image data acquired by the acquiring means and metadata including the synchronization group information and the index information configured by the configuring means; 1. An image processing device comprising:

2. The image processing device according to claim 1 , wherein the index information includes synchronization error information indicating the simultaneity of capturing image data included in the group related to the synchronization group information.

3. the index information further includes reference time information indicating a reference time of shooting for image data included in the group related to the synchronization group information; 3. The image processing device according to claim 2, wherein the synchronization error information indicates a time difference between a shooting time of image data included in a group related to the synchronization group information and the reference time.

4. The plurality of image data acquired by the acquisition means are image data captured by a plurality of devices with designated capture times, 3. The image processing apparatus according to claim 2, wherein the synchronization error information indicates a time difference between the internal clocks of the plurality of apparatuses.

5. 5. The image processing device according to claim 1, wherein the index information includes tolerance information indicating a range of shooting times of image data that can be included in a group related to the synchronization group information.

6. the index information further includes reference time information indicating a reference time of shooting for image data included in the group related to the synchronization group information; 6. The image processing apparatus according to claim 5, wherein the allowable error information indicates an upper limit of a time difference from the reference time indicated by the reference time information.

7. the photographing information relating to one image data includes at least information indicating a photographing time of the image data; 7. The image processing apparatus according to claim 6, wherein said constructing means constructs said synchronization group information by grouping based on said reference time information, said allowable error information and said photographing information.

8. the plurality of image data acquired by the acquisition means are image data captured by a plurality of devices that are time-synchronized using a common synchronization method; 8. The image processing device according to claim 1, wherein the index information further includes synchronization method information indicating the synchronization method.

9. 1. An image processing device that processes an image file having a structure including a first storage area for storing a plurality of image data, and a second storage area for storing metadata relating to the plurality of image data, The metadata includes: (1) Synchronous group information including information for grouping image data captured in a time-synchronized manner and information indicating image data included in the group among the plurality of image data; (2) For a group related to the synchronization group information, index information indicating a grouping index, the index information including synchronization error information indicating the simultaneity of shooting of image data included in the group or allowable error information indicating the range of shooting times of image data that can be included in the group; Including, The image processing device includes: an acquisition means for acquiring the image file to be processed; a presentation means for presenting information relating to synchronization accuracy of photography for image data included in a group related to the synchronization group information based on the index information stored in the image file to be processed; 1. An image processing device comprising:

10. 1. An image processing device that processes an image file having a structure including a first storage area for storing a plurality of image data, and a second storage area for storing metadata relating to the plurality of image data, The metadata includes: (1) Synchronous group information including information for grouping image data captured in a time-synchronized manner and information indicating image data included in the group among the plurality of image data; (2) Index information indicating a grouping index for a group related to the synchronization group information, the index information including allowable error information indicating a range of shooting times of image data that can be included in the group; Including, The image processing device includes: a first acquisition means for acquiring the image file to be processed; a second acquiring means for acquiring image data to be added to the first storage area of ​​the image file to be processed and photographing information relating to the photographing of the image data to be added; a determination means for determining whether or not to add the image data to the group related to the synchronization group information based on the index information of the image file to be processed and the shooting information related to the image data to be added; a configuration means for configuring the synchronization group information based on the determination result of the determination means; a generating means for generating a new image file by storing the image data to be added in the first storage area of ​​the image file to be processed and storing the synchronization group information configured by the configuring means in the second storage area; 1. An image processing device comprising:

11. 11. The image processing device according to claim 1, wherein the image data stored in the first storage area is at least one of a still image and a moving image.

12. 12. The image processing device according to claim 1, wherein the file format of the image file is High Efficiency Image File Format (HEIF) or ISO Base Media File Format (ISOBMFF).

13. 1. A control method for an image processing device that generates an image file having a structure including a first storage area for storing a plurality of image data and a second storage area for storing metadata relating to the plurality of image data, the method comprising: an acquisition step of acquiring the plurality of image data and photographing information relating to photographing of the image data; A configuration step of configuring, based on the photographing information acquired in the acquisition step, (1) synchronous group information including information for grouping image data photographed in a time-synchronized manner and information indicating image data included in the group among the plurality of image data, and (2) index information indicating a grouping index for the group related to the synchronous group information; a generating step of generating the image file that stores the plurality of image data acquired in the acquiring step and metadata including the synchronization group information and the index information configured in the configuring step; A control method comprising:

14. 1. A control method for an image processing device that processes an image file having a structure including a first storage area for storing a plurality of image data and a second storage area for storing metadata relating to the plurality of image data, comprising: The metadata includes: (1) Synchronous group information including information for grouping image data captured in a time-synchronized manner and information indicating image data included in the group among the plurality of image data; (2) For a group related to the synchronization group information, index information indicating a grouping index, the index information including synchronization error information indicating the simultaneity of shooting of image data included in the group or allowable error information indicating the range of shooting times of image data that can be included in the group; Including, The control method includes: an acquisition step of acquiring the image file to be processed; a presentation step of presenting information regarding synchronization accuracy of image capture for image data included in a group related to the synchronization group information based on the index information stored in the image file to be processed; A control method comprising:

15. 1. A control method for an image processing device that processes an image file having a structure including a first storage area for storing a plurality of image data and a second storage area for storing metadata relating to the plurality of image data, comprising: The metadata includes: (1) Synchronous group information including information for grouping image data captured in a time-synchronized manner and information indicating image data included in the group among the plurality of image data; (2) Index information indicating a grouping index for a group related to the synchronization group information, the index information including allowable error information indicating a range of shooting times of image data that can be included in the group; Including, The control method includes: a first acquisition step of acquiring the image file to be processed; a second acquisition step of acquiring image data to be added to the first storage area of ​​the image file to be processed and photographing information related to photographing of the image data to be added; a determination step of determining whether or not to add the image data to be added to a group related to the synchronization group information based on the index information of the image file to be processed and the shooting information related to the image data to be added; a configuration step of configuring the synchronization group information based on the determination result in the determination step; a generating step of generating a new image file by storing the image data to be added in the first storage area of ​​the image file to be processed and storing the synchronization group information configured in the configuring step in the second storage area; A control method comprising:

16. A program that causes a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 12.

Images