Image processing device, control method and program

Abstract

To provides an image processing apparatus, a control method, and a program that generate an image file which stores an image group shot synchronously and which can be used appropriately.SOLUTION: An image processing apparatus is configured to generate an image file having a structure including a first storage region and a second storage region, the first storage region storing a plurality of instances of image data and the second storage region storing metadata pertaining to the plurality of instances of image data. A processing method performed by the image processing apparatus includes: first configuration means configured to configure synchronous group information 502 obtained by grouping the image data determined to have been shot synchronously with respect to time; second configuration means configured to configure indicator information 503 including at least information on a determination reference of whether to have been shot synchronously with respect to time; and generation means configured to generate the image file storing the plurality of instances of image data obtained by obtainment means, and the synchronous group information configured by the first configuration means and the indicator information configured by the second configuration means.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an image processing device, a control method, and a program, and more particularly to a technique for generating an image file capable of storing data of a plurality of still images and moving images. [Background technology]

[0002] By using captured images acquired at the same time, it is possible to present expanded information that is not limited to information contained in a single captured image.Patent Document 1 discloses a photography system that takes images using two cameras with synchronized built-in clocks, and generates a 3D video using an image captured by one camera and an image captured by the other camera at the time closest to the capture time of the first image.

[0003] Recently, file formats have emerged that encode multiple still images or video images and store them as a single image file, making it possible to easily manage groups of highly related images. For example, a file format called High Efficiency Image File Format (HEIF), internationally standardized by ISO / IEC 23008-12, allows still images encoded with H.265 (HEVC) to be stored as a single image file. For such file formats, a normative structure including metadata is defined, specifying a method for associating the metadata with stored images and the structure of specific metadata formats. Furthermore, by describing the metadata in the metadata field, a single image representation consisting of multiple still images, known as a derived image, can be recorded as an image file.

[0004] As in Patent Document 1, images captured by synchronized shooting are highly correlated, and storing them as a synchronized group within a single image file is expected to improve the convenience of displaying and processing them. Regarding HEIF, standardization is underway in ISO / IEC 23008-12:2017 FDAM2 to add a 'tsyn' group to the metadata, which indicates that multiple still images or videos were captured in a synchronized manner. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2018-007031 Summary of the Invention [Problem to be solved by the invention]

[0006] However, no metadata structure has been proposed to date that takes into account the specific use of image files that store a group of images captured by synchronized shooting. The HEIF 'tsyn' specification has also been considered as merely describing information identifying a group of images captured simultaneously over the same period. Therefore, for such conventional image files, it has not been anticipated that the simultaneity of the shooting of the group of images in the image file can be evaluated. For example, in a case where multiple images that require simultaneous shooting are used for processing, such as when generating 3D video as described in Patent Document 1, it has not been possible to determine the degree to which the group of images were shot synchronously.

[0007] The present invention has been made in consideration of the above-mentioned problems, and aims to provide an image processing device, control method, and program that generate a suitably usable image file that stores a group of images captured synchronously. [Means for solving the problem]

[0008] In order to achieve the above object, an image processing device of the present invention is an image processing device that generates an image file having a structure including a first storage area for storing a plurality of image data and a second storage area for storing metadata relating to the plurality of image data. , complex an acquisition means for acquiring image data of a number of images and photographing information relating to the photographing of the image data; Multiple The photographing information related to the image data and the setting means , whether they were taken in sync or not Based on the criteria, For each of the plurality of image data acquired by the acquisition means, A determination means for determining whether or not images have been taken in a time-synchronized manner, and grouping image data determined by the determination means to have been taken in a time-synchronized manner. and information indicating the image data included in the group among the plurality of image data. a first configuration means for configuring synchronization group information; Index information indicating a grouping index for a group related to the synchronization group information, A second configuration means for configuring index information including at least information on a determination criterion, a plurality of image data acquired by an acquisition means, synchronization group information configured by the first configuration means, and index information configured by the second configuration means. Metadata containing and a generating means for generating an image file that stores the image data. [Effects of the Invention]

[0009] With this configuration, according to the present invention, it is possible to generate an image file that stores a group of images captured synchronously and that can be used appropriately. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 illustrates an example of a configuration of a synchronized imaging system according to an embodiment and a modification of the present invention. [Figure 2] FIG. 1 is a block diagram illustrating the functional configuration of a synchronous imaging controller 100 according to an embodiment and a modification of the present invention. [Figure 3] FIG. 1 is a block diagram illustrating the functional configuration of a camera device 200 according to an embodiment and a modification of the present invention. [Figure 4] Diagram showing the file structure of a HEIF file [Figure 5] FIG. 10 is a diagram showing an example of a definition of the data structure of a 'tsyn' box 411 according to an embodiment of the present invention. [Figure 6] FIG. 10 is a diagram showing an example of a definition of the data structure of an 'iinf' box 406 according to an embodiment of the present invention. [Figure 7] FIG. 10 is a diagram showing an example of a definition of the data structure of 'infe' according to an embodiment of the present invention. [Figure 8] FIG. 10 is a diagram illustrating a file generation procedure in a synchronous imaging system according to an embodiment and a modification of the present invention. [Figure 9] FIG. 10 is a diagram illustrating a procedure for time-synchronized communication in a synchronized photography system according to an embodiment and a modified example of the present invention. [Figure 10] 1 is a flowchart illustrating a synchronous shooting process executed by the synchronous shooting controller 100 according to an embodiment of the present invention; [Figure 11] A flowchart illustrating a process for generating an HEIF file executed by the synchronous shooting controller 100 according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] [Embodiment] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] In the embodiment described below, the present invention is applied to a synchronized shooting controller, which is an example of an image processing device, capable of acquiring image data captured by a plurality of camera devices connected via a network and generating an image file. However, the present invention can be applied to any device that can acquire image data and the shooting time associated with a predetermined master clock at which the image data was captured and generate an image file.

[0013] In this specification, "image data" refers to digital data representing one or more images obtained by photography, and may include either or both of still images and moving images. Furthermore, "image data" may be data whose format is determined according to the file format of the image file generated by storing the image data, and is not limited to encoded data.

[0014] Furthermore, "synchronization accuracy (of photography)" is a concept that quantifies the simultaneity of photography for a group of image data that have been photographed in a timely synchronized manner. For example, in one aspect, "synchronization accuracy" may be derived based on the difference in photography timestamps between image data. In another aspect, "synchronization accuracy" may be derived based on the time difference between the built-in clocks of different devices when different devices photograph at designated photography times. Alternatively, in yet another aspect, "synchronization accuracy" may be derived by combining these pieces of information.

[0015] "Photographing time" / "photography timestamp" are parameters associated with captured image data that are determined based on the time information indicated by the built-in clock (system timer) of the photographing device when the photograph was taken. On the other hand, "photography timing" is a concept that indicates the absolute time when the photograph was taken, independent of the built-in clock of a single photographing device, and is referred to as necessary, especially when photographs are taken with multiple photographing devices and these are to be compared.

[0016] <Configuration of synchronized imaging system> Fig. 1 is a schematic diagram illustrating the configuration of a synchronized photography system according to this embodiment. The synchronized photography system is configured to allow a plurality of camera devices 200 to cooperate with each other and to enable a synchronized photography controller 100 to record image data obtained by each camera device through temporally synchronized photography as a single image file. In the example of Fig. 1, the synchronized photography controller 100 and four camera devices 200a-d are connected via a network 300, enabling transmission and reception of information. In this embodiment, the synchronized photography controller 100 sends a photography instruction, which causes each camera device 200 to execute photography processing, and the image data (encoded data) obtained through photography is returned to the synchronized photography controller 100.

[0017] In this embodiment, the system will be described assuming that four camera devices 200 are included, but it will be readily understood that the number of camera devices 200 is not limited to this in the practice of the present invention. In addition, in this embodiment, the system will be described assuming that the photographing units that photograph the subject at each of the four locations are provided in different camera devices 200, but the practice of the present invention is not limited to this. For example, it goes without saying that multiple photographing units may be provided in one camera device 200. In addition, in the following description, to facilitate understanding of the invention, the four camera devices 200 will be described assuming that they all have the same functional configuration, but it will be readily understood that the practice of the present invention is not limited to this.

[0018] Furthermore, the network 300 may be, for example, a LAN or a public line such as the Internet. The connections between the network 300 and each device may be realized by any method, whether wired or wireless.

[0019] <Functional configuration of the synchronized imaging controller> 2 is a block diagram illustrating the functional configuration of the synchronous imaging controller 100 according to this embodiment. Each functional configuration is configured to be able to transmit information via a system bus 110.

[0020] The synchronized shooting system of this embodiment performs synchronized shooting using the multiple camera devices 200 and generates an image file containing multiple images obtained by the synchronized shooting. The CPU 101 executes a predetermined application program. The program and a system program related to the operation of the synchronized shooting controller 100 are stored in a nonvolatile storage device, ROM 102, and the CPU 101 executes the program by reading and loading them into RAM 103. The RAM 103 is a temporary storage device and is used as a program loading area and a work area for the operation of each block. More specifically, the RAM 103 is also used as an input / output buffer related to encoding and decoding by a codec 105 (described later) and as a data buffer for image file generation processing by a file generation unit 104. The ROM 102 also stores user setting data for the synchronized shooting controller 100.

[0021] The file generation unit 104 executes processes related to the generation and editing of image files. Processable image files may include files in formats such as HEIF, JPEG, JPEG2000, JPEG XR, WebP, MP4, MOV, AVI, and WebM. The file generation unit 104 also reads encoded data (image data) from image files stored on a recording medium 109, and transmits it to a codec 105 for decoding.

[0022] The codec 105 is a video codec for moving images and still images, such as H.265 (HEVC), H.264 (AVC), AV1, JPEG, etc. The codec 105 encodes and decodes data of still images and moving images handled by the synchronous shooting controller 100.

[0023] The display unit 106 is a display device such as a liquid crystal display (LCD) that is configured integrally with the synchronous shooting controller 100 or that is detachably provided on the synchronous shooting controller 100. In this embodiment, the display unit 106 displays a screen of an application related to synchronous shooting.

[0024] The operation input unit 107 is various user interfaces provided in the synchronized shooting controller 100, such as operation buttons, switches, a mouse, and a keyboard. In addition, in an aspect in which the display unit 106 is a touch panel, the operation input unit 107 may include a touch panel sensor. When the operation input unit 107 detects that an operation input has been made to the user interface, it outputs a control signal indicating that fact to the CPU 101.

[0025] The communication control unit 108 is a network interface that connects to the network 300 and transmits and receives transmission frames. The communication control unit 108 may be, for example, a PHY and MAC (transmission media control processing) that can connect to a wired LAN using Ethernet (registered trademark). Alternatively, if the synchronous imaging controller 100 can connect to a wireless LAN, the communication control unit 108 may include a controller that executes wireless LAN control such as IEEE802.11a / b / g / n / ac / ax, an RF circuit, and an antenna.

[0026] The communication control unit 108 also has a built-in hardware clock for PTP. As will be described in detail later, the synchronized shooting system of this embodiment performs time-synchronized communication using the Precision Time Protocol (PTP) defined in IEEE1588-2008 (also known as IEEE1588 version 2). Synchronized shooting is achieved by having multiple camera devices 200 work together under the control of the synchronized shooting controller 100, so the synchronized shooting controller 100 has a PTP grandmaster clock (GMC) function. The PTP grandmaster clock function is implemented as PTP stack software executed by the CPU 101, and uses the time of a hardware clock built into the communication control unit 108 as its reference. The communication control unit 108 is configured to be able to output timestamps for sending and receiving transmission frames based on the time of the hardware clock.

[0027] The recording medium 109 is a non-volatile recording device with a large storage capacity, such as a hard disk drive, an SD card, a CompactFlash (registered trademark) card, etc. In this embodiment, the recording medium 109 is used as a storage destination for image files generated based on synchronous shooting.

[0028] <Functional configuration of the camera device> Next, an example of the functional configuration of each camera device 200 of this embodiment will be described using the block diagram of Fig. 3. As shown in the figure, each functional configuration of the camera device 200 is configured to be able to transmit information via a camera system bus 210. Note that among the functional configurations of the camera device 200, those that play the same role as the functional configuration of the synchronous shooting controller 100 are prefixed with the letter "camera" to clearly distinguish them.

[0029] The processing related to time synchronization and photography for synchronous photography is realized by the camera CPU 201 executing a predetermined application program. This program and system programs related to the operation of the camera device 200 are stored in the camera ROM 202, which is a non-volatile storage device, and the camera CPU 201 executes them by reading them and expanding them in the camera RAM 203. The camera RAM 203 is a temporary storage device and is used as a program expansion area and a work area for the operation of each block. More specifically, the camera RAM 203 is also used as an input / output buffer related to encoding and decoding by the camera codec 205, which will be described later. The camera ROM 202 also stores user setting data for the camera device 200, etc.

[0030] The photographing unit 204 is a unit with an imaging function that includes a lens group, a CCD (photoelectric conversion element), a CCD control unit, and an image processing unit. The lens group is composed of multiple lenses that optically form an image of a subject on the imaging surface of the CCD. The CCD control unit includes a timing generator that supplies a transfer clock signal and a shutter signal to the CCD, a circuit that performs noise reduction and gain processing on the CCD output signal, and an A / D conversion circuit that converts analog signals into digital signals. The image processing unit performs image processing such as gamma conversion, color space conversion, white balance, and exposure correction on the digital signal output from the CCD control unit, and outputs digital image signal data (image data) that can be encoded by the camera codec 205 to the RAM 203.

[0031] The camera codec 205 is a video codec for moving images and still images, such as H.265 (HEVC), H.264 (AVC), AV1, JPEG, etc. The camera codec 205 executes encoding and decoding processes for still image and moving image data handled by the camera device 200.

[0032] The camera display unit 206 is a display device such as a liquid crystal display (LCD) that is configured integrally with the camera device 200 or that is detachably provided on the camera device 200. In this embodiment, the camera display unit 206 displays a screen of an application related to synchronous shooting.

[0033] The camera operation input unit 207 is a variety of user interfaces provided in the camera device 200, such as operation buttons and switches. In addition, in an aspect in which the camera display unit 206 is a touch panel, the camera operation input unit 207 may include a touch panel sensor. When the camera operation input unit 207 detects that an operation input has been made to the user interface, it outputs a control signal indicating that fact to the camera CPU 201.

[0034] The camera communication control unit 208 is a network interface that connects to the network 300 and transmits and receives transmission frames. The camera communication control unit 208 may be, for example, a PHY and MAC (transmission media control processing) that can connect to a wired LAN using Ethernet (registered trademark). Alternatively, if the camera device 200 can connect to a wireless LAN, the camera communication control unit 208 may include a controller that executes wireless LAN control such as IEEE802.11a / b / g / n / ac / ax, an RF circuit, and an antenna.

[0035] The camera communication control unit 208 also has a built-in hardware clock for PTP. In the synchronized shooting system of this embodiment, the camera device 200 has a PTP slave function. The PTP slave function is implemented as PTP stack software executed by the camera CPU 201, and synchronizes the hardware clock built into the camera device 200 based on time information supplied from the synchronized shooting controller 100. Thereafter, time synchronization is performed by synchronizing the time of the system timer of the operation system of the camera device 200 executed by the camera CPU 201 with the hardware clock of the camera device 200. The camera communication control unit 208 is configured to be able to output timestamps for sending and receiving transmission frames based on the time of the built-in hardware clock.

[0036] The camera recording medium 209 is a non-volatile recording device with a large storage capacity, such as a hard disk drive, an SD card, a CompactFlash (registered trademark) card, etc. In this embodiment, the camera recording medium 209 may be used to store image data such as still images and moving images obtained by shooting.

[0037] "Synchronized shooting" The following describes the synchronous imaging performed in the synchronous imaging system of this embodiment and the generation of one image file that stores multiple image data acquired by the synchronous imaging.

[0038] As described above, in the synchronized shooting system of this embodiment, time synchronization communication is performed using the synchronized shooting controller 100 as a grandmaster clock, thereby synchronizing the times on the system timers (internal clocks) of the devices in the synchronized shooting system. Then, the synchronized shooting controller 100 supplies a shooting time, and each camera device 200 captures an image when its system timer reaches the shooting time. Image data obtained by the capture is sent to the synchronized shooting controller 100, which receives multiple image data captured at specified shooting times and stores and records them in a single image file. At this time, the multiple image data can be considered to be image data captured in a time-synchronized manner, and are grouped in the image file as synchronously captured images. Grouping may be achieved, for example, by including an identifier for the grouped image data in the metadata of the image file.

[0039] Here, the file format of the image file may be any format that stores multiple image data and has metadata that describes groupings of image data that were captured synchronously, but the following explanation will use the HEIF format as an example.

[0040] HEIF file structure The file structure of a HEIF file will be explained using FIG. 4. As shown in FIG. 4, a HEIF file 400 is generally composed of the following three boxes (storage areas). The first is FileTypeBox('ftyp') 401, which stores the brand name used by the reader of the HEIF file 400 to identify the file specifications. In the HEIF file 400 generated by the synchronized capture controller 100 of this embodiment, the 'ftyp' box 401 is set to 'mif1' as the brand name and 'heic' as the compatible brand name. The second is MetaBox('meta') 402, which stores metadata describing various information about the image data stored in the HEIF file 400. As shown in the figure, the 'meta' box 402 stores multiple types of information related to the image data, each stored in a separate box, as described in detail below. The third is MediaDataBox('mdat') 412, which stores multiple pieces of encoded data (image data) as image items. In the following description, the coded data stored in the 'mdat' box 412 will be referred to as either an "image item" or "image data", depending on the context.

[0041] HandlerReferenceBox('hdlr') 403 stores a declaration of the handler type for analyzing the structure of the 'meta' box 402. In the HEIF file 400 generated by the synchronized shooting controller 100 of this embodiment, all of the encoded data stored is assumed to be still images, and the handler type name 'pict' is set in the 'hdlr' box 403.

[0042] PrimaryItemBox('pitm') 404 specifies the identifier (item ID) of the encoded data that is to be the representative item among the image items stored in the HEIF file 400.

[0043] ItemLocationBox('iloc') 405 stores information indicating the storage location of the data of each image item in the HEIF file 400. The 'iloc' box 405 typically describes the storage location of the image item as a byte offset from the beginning of the HEIF file 400 and the data length from that beginning. In other words, the information in the 'iloc' box 405 makes it possible to identify the location of each piece of encoded data stored in the 'mdat' box 412.

[0044] ItemInfoBox('iinf') 406 defines basic information such as item ID, item type indicating the item type, and item name for all image items included in the HEIF file 400.

[0045] The ItemReferenceBox('iref') 407 stores information describing the association between image items included in the HEIF file 400. In an embodiment where the image item is a photographed image, the 'iref' box 407 is used to describe the association between the image item and an item of its photographic information (such as Exif data). In an embodiment where multiple image items are in a derived image relationship, the 'iref' box 407 is used to describe the association between the image items.

[0046] ItemPropertiesBox ('iprp') 408 stores various attribute information about image items included in the HEIF file 400. More specifically, the 'iprp' box 408 includes an ItemPropertyContainerBox ('ipco') 409 that describes the attribute information, and an array of ItemPropertyAssociation ('ipma') boxes 410 that indicate the association between the attribute information and each image item. The 'ipco' box 409 may list and store, for example, the creation date and time of the image item, information indicating the width and height of the image item in pixels, and HEVC parameter set data required for decoding the HEVC image item. Meanwhile, the 'ipma' box 410 indicates the association by storing, for example, the item ID of the corresponding image item for each piece of attribute information stored in the 'ipco' box 409. Here, if multiple image items correspond to the same attribute information, this may be stated in a single 'ipma' box 410.

[0047] TimeSychronizedEntityToGroupBox('tsyn') 411 stores information identifying a group (synchronization group) defined by grouping image items captured in a time-synchronized manner. The information stored in the 'tsyn' box 411 may include at least information that identifies the image items included in one synchronization group, for example, in the form of a list of item IDs. Note that multiple synchronization groups may be defined for one HEIF file 400, and the same number of 'tsyn' boxes 411 may be stored in the 'meta' box 402 as there are synchronization groups.

[0048] Note that the file structure of the HEIF file 400 shown in FIG. 4 is one example, and the order in which the boxes are stored, the manner in which the storage areas are separated, and the type of stored information are not limited to this.

[0049] As mentioned above, conventional HEIF files only specify that image data captured at the same time over the same period be grouped and described in the 'tsyn' box 411, and are not structured with this use in mind.

[0050] For example, when adding new image data to a HEIF file, it was not possible to determine whether the new image data could be added to the synchronization group defined in the 'tsyn' box 411, and the extensibility of the synchronization group was not ensured. In other words, when adding image data at a time other than when the HEIF file was created, it was difficult to determine whether the image data could be "image data taken at the same time over the same period" as image data already stored in the synchronization group.

[0051] Furthermore, particularly in an embodiment where image data captured by multiple different camera devices are stored in a HEIF file, as in the present embodiment, it is difficult to perfectly synchronize the system timers of the devices even when time-synchronized communication is performed. This is due, for example, to the constant fluctuation of the difference between the time (reference time) of the system timer of the synchronized capture controller 100, which serves as the grandmaster clock, and the time of the system timer of the synchronized camera device 200. In other words, even if a capture time is specified after time synchronization and each camera device 200 captures at that time, there may be a difference in the timing of capture between the camera devices. Therefore, depending on the application of multiple image data included in a synchronized group of an HEIF file, the difference in capture timing between the image data may make it unsuitable for use. For example, applications such as generating 3D images, as described in Patent Document 1, require higher synchronization accuracy than applications simply viewing image data captured at the same time. Thus, depending on the application of the synchronized image data, the required synchronization accuracy varies, for example, from an error of several milliseconds or less to an error of several microseconds or less. Therefore, simply grouping and describing image data captured simultaneously over the same period of time has not allowed for determining whether the image data included in one synchronization group meets the required shooting synchronization accuracy. In other words, when using multiple image data included in a synchronization group of an HEIF file, it has been difficult to determine whether the multiple image data ensures the shooting synchronization accuracy appropriate for the intended use. In other words, it has been difficult to determine the degree to which the multiple image data included in the synchronization group were captured in sync.

[0052] Taking such file usage into consideration, the synchronized capture controller 100 of this embodiment describes a new configuration method for the 'meta' box 402 that makes it possible to easily evaluate the synchronization accuracy of image data included in a synchronization group. More specifically, in this embodiment, within the framework of the file structure of the HEIF file 400 shown in Figure 4, the data structures of the 'tsyn' box 411 and the 'iinf' box 406 are defined so that information for evaluating the synchronization accuracy related to the synchronization group can be included.

[0053] 5 shows a definition 501 of the data structure of the 'tsyn' box 411 in a HEIF file according to this embodiment. As shown in definition 501, the 'tsyn' box 411 in this embodiment, which relates to one synchronization group, includes synchronization group information 502 that identifies the grouped image items, and index information 503 that indicates the synchronization accuracy of the capture of the image data relating to that group.

[0054] As shown in the figure, the synchronization group information 502 describes, for example, a group ID (group_id) that is an identifier for the synchronization group, the number of entries included in the group (num_entities_in_group), and a list of entry IDs (entity_id). Here, the entry ID specifies the item ID of the image item stored in the HEIF file. In other words, the synchronization group information 502 is information that identifies image data that has been grouped as having been captured in a time-synchronized manner among the image data stored in the 'mdat' box 412, as previously considered.

[0055] On the other hand, considering future uses not limited to synchronized shooting systems for HEIF files, it is preferable that index information 503 indicating the synchronization accuracy of shooting be configured to enable a variety of determinations, not just at the time of HEIF file generation. Therefore, in this embodiment, index information 503 is configured to include synchronization method information 504, reference time information 505, synchronization error information 506, and allowable error information 507.

[0056] The synchronization method information 504 describes information that identifies the method of time synchronization used by the devices that captured the image data included in the synchronization group. Specifically, the synchronization method information 504 may be information that specifies the time synchronization standard or protocol to be followed. In addition, the synchronization method information 504 may include information specific to the time synchronization standard or protocol. For example, if the synchronization group is composed of image data captured using the synchronized capture system of this embodiment, the information may be information indicating that the time synchronization standard is IEEE1588-2008 and that the grandmaster clock is the synchronized capture controller 100.

[0057] The reference time information 505 describes information specifying the time (reference time) that is the reference for synchronous shooting for the image data included in the synchronous group. For example, when the synchronous group is made up of image data captured by the synchronous shooting system of this embodiment, the reference time information 505 may be information indicating the designated shooting time (shooting timestamp or time) supplied from the synchronous shooting controller 100 to each camera device 200.

[0058] The synchronization error information 506 describes information indicating the state of synchronization error between the devices that captured the image data included in the synchronization group. In other words, the synchronization error information 506 describes information indicating the simultaneity of capture of the image data included in the synchronization group. The synchronization error is, for example, when PTP or NTP is used for time synchronization, the time difference between the system timer of the camera device 200 and the grand master clock. In this embodiment, synchronous capture is performed by providing a designated capture time, so the synchronization error of the camera device 200 indicates the time difference between the capture timing of the image data from the designated capture time. Therefore, in this embodiment, to objectively indicate the simultaneity of capture of the image data included in the synchronization group, the synchronization error information 506 indicates the maximum value of the synchronization errors of the camera devices 200 that captured the image data within the group. However, the implementation of the present invention is not limited to this, and the synchronization error information 506 may also be the maximum value of the time difference between the system timers of the camera devices 200 that captured the image data included in the synchronization group. In other words, the synchronization error information 506 may be the time difference between the earliest and latest times indicating the capture timing of the image data included in the synchronization group on a time axis based on the grand master clock.

[0059] The allowable error information 507 describes information indicating the allowable range of synchronization error of the device that captured image data that can be included in a synchronization group, more specifically, information indicating the time range within which the image data are considered synchronized when grouping. In other words, the allowable error information 507 describes information indicating the minimum guaranteed simultaneity of capture for the image data included in the synchronization group. In other words, the allowable error information 507 describes information that serves as a criterion for determining whether multiple image data were captured in a timely synchronized manner when grouping, i.e., whether image data is allowed to be included in a synchronization group. In this embodiment, the allowable error information 507 indicates this criterion in the form of the allowable upper limit of the synchronization error of the camera device 200, i.e., the upper limit of the time difference between the capture timing of the image data and the designated capture time. In other words, the allowable error information 507, in combination with the reference time information 505, determines the time range within which the image data are considered synchronized with the designated capture time. However, the implementation of the present invention is not limited to this, and the allowable error information 507 may be the upper limit of the allowable time lag of the system timer between the camera devices 200 that captured image data that can be included in a synchronization group. As will be described in detail later, the information described in the allowable error information 507 is determined based on the intended use of the file and the synchronization accuracy options to be guaranteed, which are specified when processing related to HEIF file generation is performed in the synchronized capture controller 100, for example.

[0060] That is, while the synchronization error information 506 is information indicating the synchronization accuracy of photography determined based on the image data included in the synchronization group, the allowable error information 507 is information indicating the synchronization accuracy of photography fixedly determined for the synchronization group. Therefore, for multiple image data included in the synchronization group, the minimum synchronization accuracy of photography specified by the allowable error information 507 is guaranteed, and more specific synchronization accuracy is indicated by the synchronization error information 506.

[0061] By defining the data structure of the 'tsyn' box 411 in this way, when using a HEIF file, it is possible to understand the synchronization accuracy of the image capture of the corresponding synchronization group by referencing the index information 503. More specifically, it is possible to understand that image data captured at a specified capture time on a device time-synchronized by specific time synchronization communication is grouped using the synchronization method information 504 and the reference time information 505. Furthermore, by referencing the synchronization error information 506, the maximum time difference from the specified capture time of the image data included in the synchronization group can be obtained, thereby understanding the synchronization accuracy of the image capture. Alternatively, by referencing the allowable error information 507, the upper limit of the time difference from the specified capture time of the image data that can be included in the synchronization group can be obtained, thereby understanding the synchronization accuracy of the synchronization group. In other words, the index information 503 not only indicates the synchronization accuracy of the image data captured in the synchronization group, but also indicates an index for grouping the image data for the synchronization group.

[0062] In a synchronized shooting system like this embodiment, which generates an HEIF file using image data obtained by supplying a designated shooting time from the synchronized shooting controller 100, the shooting timestamps of the resulting image data are consistent. That is, after time-synchronized communication, the system timer of each camera device 200 instructs the camera to capture an image at the designated shooting time, so the shooting timestamp attached to the image data captured by each camera device 200 indicates the designated shooting time. However, even if the designated shooting time is the same, the shooting timing between the camera devices 200 may differ depending on the type of synchronization error in the system timer. Therefore, in the synchronized shooting system of this embodiment, when image data is provided from each camera device 200, information on the synchronization error of the camera device's 200 system timer (device error information) is associated and sent. The synchronized shooting controller 100 then constructs the 'tsyn' box 411 (synchronization group information 502 and index information 503) of the HEIF file based on the shooting timestamp and device error information associated with the image data. Here, in this embodiment, the device error information is described as being derived in each camera device 200, but the implementation of the present invention is not limited to this, and the device error information may be derived in at least one of the synchronous shooting controller 100 and the camera device 200.

[0063] On the other hand, considering the scalability of synchronized groups, it is preferable that HEIF files be configured so that image data captured without a designated capture time can also be added by camera devices time-synchronized using the same method as the devices in the synchronized shooting system. In other words, it is preferable that HEIF files be configured so that synchronized groups can be defined without being limited to image data captured synchronously by specifying a capture time. The simultaneity of capture in a synchronized group can be measured if it is possible to compare the capture timing on a common time axis for the multiple camera devices 200 that capture image data. Therefore, even image data obtained without controlling the capture time can be grouped. For example, image data obtained by each camera device 200 capturing images upon receiving a capture instruction simultaneously transmitted from the synchronized shooting controller 100 may be grouped in an HEIF file. Alternatively, for example, in a configuration in which multiple camera devices (shooting units) are operated independently without forming a synchronized shooting system, image data captured at the same time may be grouped in an HEIF file.

[0064] Here, time synchronization between these devices does not need to be achieved as in the synchronized shooting system of this embodiment. In other words, in the synchronized shooting system of this embodiment, the synchronized shooting controller 100 has a grandmaster clock function and time-synchronizes the camera devices 200 belonging to the system, but the time reference for synchronization may be provided by an external device. In other words, the device that acquires image data and generates the HEIF file does not need to be the device that provides the time reference for synchronization; at least the time synchronization of the multiple camera devices that perform shooting may be achieved through time-synchronous communication with any device. In other words, the generation of the HEIF file according to the present invention can be achieved using image data captured by the camera devices 200 belonging to the synchronized shooting system, as well as image data captured by any device that is time-synchronized using the same method.

[0065] Therefore, the 'meta' box 402 may be configured to store information (photography timestamp) indicating the time of shooting of each image data stored in the 'mdat' box 412. The photography timestamp may be obtained from photography information (including device error information) that is provided in association with the image data and stores various information related to the shooting of the image data. Information related to the image data is stored by defining the data structure of the 'iinf' box 406 (601) as shown in Figure 6.

[0066] As shown in the figure, definition 601 includes number of entries 602 (16 bits or 32 bits) indicating the number of image items stored in 'mdat' box 412, and an array 603 of ItemInfoEntry ('infe') that stores various information related to each image item. 'infe' array 603 is an array in which data equal to the number of entries 602 is arranged, and stores various information about each image item.

[0067] The data structure of 'infe' for one image item may be defined (701) as shown in FIG. 7. As shown in the figure, definition 701 defines basic information 702, such as the item ID of the image item, the item type indicating the type of image item, and the item name of the image item, as well as timestamp information 703 describing the capture timestamp. In the example shown in the figure, the timestamp information 703 stores the capture timestamp as a 64-bit integer value representing the number of elapsed seconds in Coordinated Universal Time (UTC) in item_ts_sec, and as a 32-bit integer value representing nanoseconds less than an integer second in item_ts_nsec. Note that the description format of timestamp information 703 is not limited to this; for example, the capture timestamp may be described in NTP timestamp format using a 32-bit second value and a 32-bit nanosecond value. Alternatively, the capture timestamp may be described in PTP timestamp format using International Atomic Time Interchange (TAI) using a 48-bit second value and a 32-bit nanosecond value.

[0068] By allowing the shooting timestamp of each image data item to be stored in the 'meta' box 402 in this way, it is also possible to perform grouping using the shooting timestamp of image data already stored in the 'mdat' box 412. In this embodiment, for example, instead of storing a specified shooting time in the reference time information 505 of the 'tsyn' box 411, the item ID of the image data may be written to specify the shooting timestamp of the image data as the reference time for the synchronization group. In this case, the synchronization error information 506 may store the maximum value of the time difference between the shooting time (shooting timestamp) of the image data specified in the reference time information 505 and other image data in the synchronization group. Furthermore, the allowable error information 507 may describe the conditions under which image data can be included in the synchronization group in the form of an upper limit of the time difference from the shooting time of the image data specified in the reference time information 505.

[0069] <File generation procedure> The file generation procedure for generating an HEIF file containing multiple pieces of image data synchronously captured in the synchronized capture system of this embodiment will be described in detail below with reference to Fig. 8. As shown in the figure, the file generation procedure is realized by the synchronized capture controller 100 and camera device 200 performing the relevant processes, i.e., by the CPU 101 and camera CPU 201 executing the corresponding processes. Note that the camera device 200 procedure shown in Fig. 8 is executed by each of the camera devices 200a-d communicating with the synchronized capture controller 100.

[0070] In step 800, the synchronized capture controller 100 sets the capture synchronization accuracy for image data to be included in one synchronization group defined in the HEIF file to be generated. In the synchronized capture system of this embodiment, setting the capture synchronization accuracy refers to determining the upper limit of the allowable synchronization error of the camera device 200 as a criterion for grouping. This upper limit is also stored as allowable error information 507 in the 'tsyn' box 411 of the generated HEIF file. Here, the capture synchronization accuracy may be set based on, for example, a user's operational input selecting the intended use of the generated HEIF file or an operational input specifying an application that will use the HEIF file. Alternatively, a numerical input may be received for the upper limit of the allowable synchronization error required when combining image data included in the synchronization group, and the setting may be based on the input numerical value. Alternatively, the capture synchronization accuracy may be fixed for the synchronized capture system, for example.

[0071] In step 801, time synchronous communication is performed between the synchronized capture controller 100 and the camera device 200, and the times on the system timers of each device are synchronized. As described above, in the synchronized capture system of this embodiment, the synchronized capture controller 100 has a PTP grandmaster clock function, and therefore the time on the system timer of the camera device 200 is synchronized with the grandmaster clock through time synchronous communication. Note that in the description of the file generation procedure in FIG. 8, to facilitate understanding of the invention, time synchronous communication is described as being performed only in step 801, but the implementation of the present invention is not limited to this, and time synchronous communication may be performed at a predetermined frequency. When time synchronous communication is performed in the camera device 200, the PTP offset value used to adjust the system timer or hardware clock is stored.

[0072] Here, the time synchronous communication procedure performed in this procedure will be described in detail with reference to Fig. 9. As shown in the figure, the time synchronous communication procedure is realized by the synchronous shooting controller 100 and the camera device 200 performing the relevant processes, that is, by the CPU 101 and the camera CPU 201 executing the corresponding processes. Note that the procedure of the camera device 200 shown in Fig. 9 is executed by each of the camera devices 200a to 200d communicating with the synchronous shooting controller 100, similar to the file generation procedure (Fig. 8).

[0073] In step 901, a Sync message 911 is multicast from the synchronous shooting controller 100 to other devices (camera devices 200) included in the synchronous shooting system. At this time, a transmission timestamp (transmission time t1) is stored in, for example, the RAM 103 based on the output of a hardware clock built in the communication control unit 108 of the synchronous shooting controller 100. Furthermore, when the transmitted Sync message 911 is received by the camera device 200, a reception timestamp (reception time T2) is stored in, for example, the camera RAM 203 based on the output of a hardware clock built in the camera communication control unit 208.

[0074] Next, in step 902, a Follow-up message 912 is multicast from the synchronized photography controller 100 to other devices (camera devices 200) included in the synchronized photography system. The Follow-up message 912 includes a transmission time t1. When the transmitted Follow-up message 912 is received by the camera device 200, the transmission time t1 included in the message is stored in the camera RAM 203. That is, the transmission time t1 of the Sync message 911 in the grandmaster clock is transmitted to the camera device 200 by the Follow-up message 912.

[0075] In step 903, a DelayRequest message 913 is transmitted from the camera device 200 to the synchronous shooting controller 100. At this time, a transmission time stamp (transmission time T3) is stored in the camera RAM 203 based on the output of a hardware clock built in the camera communication control unit 208 of the camera device 200. When the transmitted DelayRequest message 913 is received by the synchronous shooting controller 100, a reception time stamp (reception time t4) is stored in the RAM 103 based on the output of a hardware clock built in the communication control unit 108.

[0076] Then, in step 904, the synchronous shooting controller 100 transmits a DelayResponse message 914 to the camera device 200 that transmitted the DelayRequest message 913. The DelayResponse message 914 includes a reception time t4. When the transmitted DelayResponse message 914 is received by the camera device 200, the reception time t4 included in the message is stored in the camera RAM 203. That is, the reception time t4 of the DelayRequest message 913 on the grandmaster clock is transmitted to the camera device 200 by the DelayResponse message 914.

[0077] In step 905, the camera device 200 derives the time difference (PTP offset value) between the hardware clocks of the camera device 200 and the synchronized shooting controller 100, and stores it in the camera RAM 203. Based on the derived PTP offset value, the time of the hardware clock of the camera communication control unit 208 is corrected, and based on the corrected hardware clock, the system timer of the camera CPU 201 is corrected, thereby completing time synchronization. Here, the PTP offset value is (PTP offset value) = ((T2-t1)-(t4-T3)) / 2 In this way, time synchronization between the synchronous shooting controller 100 and the camera device 200 is achieved.

[0078] Although the system timers of the synchronous shooting controller 100 and the camera device 200 are temporarily synchronized by time synchronous communication, strictly speaking, they may deviate even after time synchronization. Furthermore, the PTP offset value may change each time time synchronous communication is executed. Therefore, the camera device 200 stores and holds PTP offset values ​​for multiple time synchronous communications in the camera RAM 203.

[0079] When the time-synchronized communication is completed, in step 802, the synchronized shooting controller 100 transmits information about the designated shooting time and a shooting instruction for synchronized shooting to the camera device 200. Note that in this embodiment, the description will be given assuming that the information about the designated shooting time is supplied and the camera device 200 performs shooting, but as mentioned above, the implementation of the present invention is not limited to this.

[0080] In step 803, a photographing process is executed at the designated photographing time in the camera device 200. The photographed image obtained by the photographing process is coded by the camera codec 205 and stored as image data.

[0081] In step 804, the image data generated in step 803 is transmitted together with the photographing information from the camera device 200 to the synchronous photographing controller 100. In this embodiment, the photographing information transmitted together with the image data includes device synchronization method information that identifies the method of time synchronization used in the camera device 200, a photographing timestamp of the image data, and device error information.

[0082] The device synchronization method information is information that identifies the method of time synchronization used for the device that captured the image data. The device synchronization method information includes, for example, a UUID for identifying the synchronized imaging system and the synchronized imaging that was performed, the protocol type, and information on the clock master used as the reference for time synchronization. Here, in the synchronized imaging system of this embodiment, time synchronization is performed using PTP communication, so the protocol type indicates IEEE1588-2008. Furthermore, the clock master information indicates the value of the ClockIdentity field included in the PTP Sync message 911 received by time synchronization communication. The field value indicates the identifier of the PTP grandmaster clock.

[0083] The shooting timestamp indicates the timing of shooting (shooting time) when the shooting process of step 803 was executed for the image data. Here, the shooting timestamp indicates the time indicated by the system timer of the camera device 200 that captured the image data when the image data was captured. In the synchronized shooting system of this embodiment, the shooting process is performed based on the designated shooting time supplied from the synchronized shooting controller 100, so the shooting timestamps related to the image data sent from the camera devices 200 in the system will match.

[0084] The device error information also indicates the PTP offset value derived for the camera device 200 that captured the image data. As described above, the PTP offset value may vary each time PTP time synchronous communication is performed, so the device error information may store, for example, a value (moving median) that is the middle ranking when PTP offset values ​​derived multiple times in the past are sorted. That is, in an embodiment targeting PTP offset values ​​obtained for the past nine PTP time synchronous communications, the fifth-ranked value among these is stored in the device error information. However, the PTP offset value stored in the device error information is not limited to the moving median, and a value obtained based on one or more derivation results may be stored.

[0085] In step 805, the synchronized shooting controller 100 generates an HEIF file based on the multiple image data received during synchronized shooting. At this time, grouping processing is performed based on the information on the synchronization accuracy of shooting set in step 800 and the shooting information related to the multiple image data received, and synchronization group information 502 and index information 503 are generated. This information is stored in the 'tsyn' box 411 of the generated HEIF file.

[0086] <Synchronized shooting processing> 10, an example of synchronized shooting processing executed by the synchronized shooting controller 100 in relation to such a file generation procedure will be described in detail below. The processing corresponding to the flowchart can be realized by the CPU 101 reading out a corresponding processing program stored in, for example, the ROM 102, and loading and executing the program in the RAM 103. This synchronized shooting processing will be described as being started when, for example, the synchronized shooting controller 100 detects an operation input related to the execution of synchronized shooting.

[0087] For simplicity's sake, this embodiment will be described assuming that one HEIF file is generated by one synchronized shooting process. Furthermore, it is assumed that multiple pieces of image data captured by each camera device 200 in the synchronized shooting system in response to a shooting instruction related to synchronized shooting are stored in the 'mdat' box 412 of one HEIF file. Furthermore, it is assumed that only one synchronization group is defined in one generated HEIF file. However, the implementation of the present invention is not limited to this. That is, multiple HEIF files may be generated by one synchronized shooting process, or not all of the multiple pieces of image data captured in response to a shooting instruction may be stored in a HEIF file, or multiple synchronization groups may be defined in a single HEIF file.

[0088] In S1001, CPU 101 sets the synchronization accuracy of image capture for one synchronization group defined in the HEIF file to be generated. More specifically, CPU 101 sets the allowable upper limit of the PTP offset value that serves as the criterion for grouping. As described above, the allowable upper limit may be set by CPU 101 based on, for example, a user's operation input selecting the intended use of the HEIF file to be generated or an operation input specifying an application that will use the HEIF file. Alternatively, CPU 101 may accept a numerical input for the allowable upper limit of the synchronization error required when combining image data included in the synchronization group, and set the value based on that numerical value.

[0089] In S1002, the CPU 101 performs a time synchronization process to synchronize the times of the camera devices 200a to 200d through PTP communication. This time synchronization process may be executed, for example, until the times of all the camera devices 200 in the synchronized shooting system are synchronized with the time of the hardware clock built into the synchronized shooting controller 100. Here, the time synchronization between the synchronized shooting controller 100 and the camera devices 200 may be determined based on the shooting synchronization accuracy set in S1001. For example, the CPU 101 may communicate with each camera device 200 to inquire whether the derived PTP offset value satisfies the shooting synchronization accuracy, and determine the time synchronization as a result.

[0090] In S1003, the CPU 101 transmits a shooting instruction for synchronous shooting to each of the camera devices 200 in the synchronous shooting system. Here, the shooting instruction for synchronous shooting includes a 128-bit UUID for identifying the synchronous shooting system and the synchronous shooting, and information on the specified shooting time.

[0091] In S1004, under the control of the CPU 101, the file generation unit 104 generates a new HEIF file for the current synchronized shooting. At this step, the HEIF file does not yet contain image data or some information, but it can contain and configure the 'tsyn' box 411 and some information. For example, in this step, some of the index information 503 in the 'tsyn' box 411 is stored. Specifically, the file generation unit 104 stores, in the synchronization method information 504, information specifying the time synchronization standard IEEE1588-2008, the UUID transmitted in S1003, and the identifier of the PTP grandmaster clock broadcast in the time synchronization communication. The file generation unit 104 also stores, in the reference time information 505, the information on the specified shooting time transmitted in S1003. The file generation unit 104 also stores, in the allowable error information 507, the upper limit of the allowable PTP offset value set in S1001.

[0092] Thereafter, each camera device 200 takes a photograph at the designated photographing time, and the image data obtained by the photographing is transmitted as photographing information to the synchronous photographing controller 100. Therefore, in S1005, the CPU 101 stores the image data and photographing information received from each camera device 200 via the communication control unit 108 in the RAM 103.

[0093] In S1006, under the control of the CPU 101, the file generation unit 104 stores the received image data in a HEIF file, configures each piece of information in the 'tsyn' box 411 to define a synchronization group, and completes the HEIF file.

[0094] Here, the processing performed in this step (storing image data and configuring the 'tsyn' box 411) will be described in detail using the flowchart in Fig. 11. The processing shown in this flowchart is performed for each set of image data and shooting information received from the camera device 200, and is therefore performed as many times as the number of camera devices 200 included in the synchronized shooting system.

[0095] In S1101, the file generation unit 104 stores the image data to be processed as an image item (target image item) in the 'mdat' box 412 of the HEIF file. At this time, the file generation unit 104 determines the item ID of the target image item. The file generation unit 104 then constructs 'infe' for the image item, updates the number of entries 602 to the number of image items stored in the 'mdat' box 412, and updates the 'iinf' box 406. When constructing 'infe' for the target image item, a shooting timestamp based on the shooting information received in association with the image data is stored.

[0096] In S1102, the file generation unit 104 references the photography information (target photography information) received along with the target image item and determines whether the time synchronization method and photography timestamp used to photograph the target image item match the information in the 'tsyn' box 411. Specifically, the file generation unit 104 determines whether the time synchronization method information and photography timestamp in the target photography information match the synchronization method information 504 and reference time information 505 included in the 'tsyn' box 411 of the HEIF file being generated. Here, in the synchronized photography system of this embodiment, a designated photography time is supplied and photography processing is performed in each camera device 200, so the photography timestamp matches the designated photography time in the reference time information 505. If the file generation unit 104 determines that this information matches the information in the 'tsyn' box 411, it proceeds to S1103; if it determines that they do not match, it completes this processing flow without updating the information in the 'tsyn' box 411.

[0097] In S1103, the file generation unit 104 determines whether the target image item satisfies the shooting synchronization accuracy (criterion). More specifically, the file generation unit 104 makes this determination based on the PTP offset value (of the camera device 200 that captured the target image item) included in the target shooting information and the allowable error information 507 in the 'tsyn' box 411 of the HEIF file being generated. That is, the file generation unit 104 determines whether the shooting synchronization accuracy is met based on whether the PTP offset value is within the allowable upper limit indicated by the allowable error information 507 (e.g., whether the absolute value of the PTP offset value does not exceed the allowable upper limit). If the file generation unit 104 determines that the target image item satisfies the shooting synchronization accuracy, it proceeds to S1104; if it determines that the target image item does not satisfy the shooting synchronization accuracy, it terminates this processing flow. As described above, in this synchronized shooting system, in S1102, the time synchronization method and shooting timestamp must be the same, and then in S1103, it is determined whether to group the images based on the time synchronization error.

[0098] In S1104, the file generation unit 104 performs processing to include the target image item in a synchronization group related to synchronized shooting. Specifically, the file generation unit 104 updates the synchronization group information 502 in the 'tsyn' box 411 by adding the item ID of the target image item as an entry. The file generation unit 104 also updates the PTP offset value (absolute value) of the camera device 200 that captured the target image item if it is greater than the current value of the synchronization error information 506. Here, the synchronization error information 506 has an initial value of 0, and is updated to the value if the absolute value of the PTP offset value related to the target image item is greater. In this way, the file generation unit 104 configures the synchronization group information 502 and index information 503 for the synchronization group defined in the HEIF file to be generated, thereby completing the HEIF file.

[0099] In this way, by executing the process of S1006 of the synchronized shooting process, a HEIF file is generated that groups together image data that meets the shooting synchronization accuracy from among the received image data. In other words, the generated HEIF file defines a synchronization group that ensures the set shooting synchronization accuracy, and is configured to include information that allows for evaluating the simultaneity of shooting for the multiple image data included in that synchronization group.

[0100] <<Using HEIF files>> Next, we will explain how to use the HEIF file created in this way. It should be easy to understand that the HEIF file can be used in any device, not just the synchronized shooting controller 100 that created the file. When using the file, a processor such as the device's CPU reads the 'meta' box 402 of the HEIF file to be processed, and plays or changes the image data stored in the 'mdat' box 412, and edits the 'meta' box 402 as necessary.

[0101] One aspect of using HEIF files involves presenting the shooting synchronization accuracy for multiple image data included in a synchronization group defined in the 'tsyn' box 411 of the HEIF file based on the information in the synchronization group. The presentation of the shooting synchronization accuracy may be, for example, a mode in which at least one piece of information in the index information 503 for the HEIF file is displayed, or a mode in which the HEIF file is displayed if the requested synchronization accuracy is met.

[0102] In the former case, for example, it may be displayed that at least the synchronization accuracy of the image data of the group indicated by the allowable error information 507 is ensured with respect to the shooting time indicated by the reference time information 505. Alternatively, it may be displayed that the image data of the group was shot within the range of the time difference indicated by the synchronization error information 506.

[0103] In the latter case, for example, when a file viewing application sets the shooting synchronization accuracy as an extraction condition, HEIF files that satisfy the extraction condition may be displayed based on the synchronization error information 506 or the allowable error information 507. Alternatively, for example, in an application that performs a compositing process using image data captured synchronously, HEIF files that satisfy the synchronization accuracy required for the compositing process may be displayed based on the synchronization error information 506 or the allowable error information 507.

[0104] When presenting the synchronization accuracy for these synchronization groups, some or all of the multiple image data classified into the synchronization groups defined by the synchronization group information 502 may also be presented.

[0105] Another aspect of using HEIF files involves determining whether new image data should be added to a synchronization group when the new image data is added to the HEIF file (stored in the 'mdat' box 412). For example, when an instruction is given to add new image data to an HEIF file, the 'tsyn' box 411 is referenced to determine whether a defined synchronization group exists. If a defined synchronization group exists, the shooting information associated with the new image data is compared with the index information 503 to determine whether the new image data can be added to the synchronization group (grouped).

[0106] This determination may be performed, for example, in the same manner as in the processing flow shown in FIG. 11. That is, a determination as to whether to add the new image data to the synchronization group is made on the assumption that the information on the time synchronization method used by the device that captured the new image data and the capture timestamp of the new image data are the same as the synchronization method information 504 and reference time information 505 of the HEIF file. If the prerequisite conditions are met and the PTP offset value at the time the new image data was captured falls within the upper limit of the allowable error indicated by the allowable error information 507, it is determined that the new image data can be added to the synchronization group. At this time, grouping is performed by adding information (item ID) related to the new image data to the synchronization group information 502. Furthermore, if the addition of the new image data to the synchronization group changes the synchronization accuracy of the image data in the group, the information in the synchronization error information 506 is updated. On the other hand, if the determination result indicates that the prerequisite conditions are not met, or that even if the prerequisite conditions are met, the PTP offset value does not fall within the upper limit of the allowable error indicated by the allowable error information 507, it is determined that the new image data cannot be added to the synchronization group.

[0107] Note that adding new image data to a synchronization group does not necessarily require that the time synchronization method and shooting timestamp of the device that captured the image be the same as those in the synchronization method information 504 and the reference time information 505. For example, if the time synchronization method and synchronization method information 504 are the same, the shooting time on a predetermined time axis (a time axis based on a master clock) may be identified based on the shooting timestamp and PTP offset value of the new image data, and then a determination of addition may be made. In this case, the shooting time on the common time axis of the synchronization group defined in the HEIF file is the designated shooting time indicated in the reference time information 505. Therefore, if the time difference between the shooting time of the new image data and the designated shooting time is within the allowable range of synchronization error indicated in the allowable error information 507, the image data may be determined to be addable to the synchronization group. Furthermore, even if the device that captured the new image data was not time-synchronized using the same method as the synchronization method information 504 for the synchronization group at the time of capture, the time of capture may be estimated based on information on time synchronization performed at any time after capture, for example, and then a determination of addition may be made.

[0108] As described above, by including tolerance information 507 indicating the accuracy of synchronization of images captured in a synchronization group defined in a HEIF file according to this embodiment, the usability of the image data included in the group can be improved. More specifically, by including tolerance information 507 for a synchronization group, it is possible to quantitatively indicate the degree to which the image data included in the group were captured in chronological order, making it easier to determine whether the image data is suitable for a particular purpose. Furthermore, because the index information 503 including tolerance information 507 indicates the grouping conditions, it is easy to determine whether a predefined synchronization group should be updated when image data is added to a HEIF file.

[0109] In this embodiment, when generating an HEIF file, it has been described that all received image data is stored in the 'mdat' box 412 even if it is not added to a synchronization group, but it is also possible not to store it if it is not added to a synchronization group.

[0110] [Variations] In the above-described embodiment, an HEIF file containing synchronization group information indicating that multiple pieces of image data captured in a time-synchronized manner have been grouped, and tolerance information indicating an index for the synchronization accuracy or grouping of the captured pieces of image data, was described as an example. However, as described above, the implementation of the present invention is not limited to files in the HEIF format, and can be applied to any file containing synchronization group information and index information including tolerance information.

[0111] For example, the present invention can be applied to a synchronized shooting system in which multiple camera devices 200 each capture video and store the resulting video data in a single file. That is, according to the present invention, a single video file can be generated that stores multiple video data sets and includes, as metadata, synchronization group information that defines groups of video data captured in a time-synchronized manner among the stored video data sets and index information that serves as the basis for the grouping. In this case, the generated video file may be in a file format based on the ISO Base Media File Format (ISOBMFF), similar to HEIF. For example, in a case in which the camera devices 200 generate video data encoded using H.264, the synchronized shooting controller 100 may generate an MP4 file that stores the video data captured by each camera device 200 as separate video and audio tracks.

[0112] In an MP4 file, metadata is written in a MovieBox ('moov'), so by including a 'tsyn' box in 'moov', it is possible to describe track grouping. Note that the list of entry IDs in the 'tsyn' box may specify track IDs. In addition, in the case of video data, the shooting timestamp is the start time of video shooting in each camera device 200, and can be stored in a TrackHeaderBox ('tkhd') box in 'moov'. Index information may be stored in the 'tsyn' box, for example, as in the above-mentioned embodiment.

[0113] [Other embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0114] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0115] 100: Synchronous shooting controller, 101: CPU, 104: File generation unit

Claims

1. 1. An image processing device that generates an image file having a structure including a first storage area for storing a plurality of image data, and a second storage area for storing metadata relating to the plurality of image data, an acquisition means for acquiring the plurality of image data and photographing information relating to photographing of the image data; a determination means for determining whether or not each of the plurality of image data acquired by the acquisition means has been photographed in a time-synchronized manner, based on the photographing information relating to the plurality of image data and a determination criterion for determining whether or not the images have been photographed in a time-synchronized manner; a first configuration means for configuring synchronization group information including information for grouping image data determined by the determination means to have been captured in a time-synchronized manner and information indicating image data included in the group among the plurality of image data; a second configuration means for configuring index information indicating a grouping index for a group related to the synchronization group information, the index information including at least information on the judgment criterion; a generating means for generating the image file that stores the plurality of image data acquired by the acquiring means, and metadata including the synchronization group information configured by the first configuring means and the index information configured by the second configuring means; 1. An image processing device comprising:

2. The apparatus further includes a designation unit for designating a reference time of photography for a group related to the synchronization group information, 2. The image processing apparatus according to claim 1, wherein the determination criterion defines a time range that is considered to be synchronized with the reference time.

3. the plurality of image data acquired by the acquisition means are image data captured by a plurality of devices whose built-in clocks are time-synchronized by a common synchronization method, the plurality of image data are image data captured in each of the plurality of devices when a built-in clock of the device is set to the reference time, the photographing information relating to one image data includes information on a synchronization error of a built-in clock of a device that photographed the image data with respect to a master clock that is a reference for the time synchronization; 3. The image processing apparatus according to claim 2, wherein the determination criterion defines the time range in the form of an upper limit of an allowable synchronization error with respect to the master clock.

4. the plurality of image data acquired by the acquisition means are image data captured by a plurality of devices whose built-in clocks are time-synchronized by a common synchronization method, The photographing information relating to one image data includes information on the photographing time of a built-in clock of a device that photographed the image data, the designation means designates a photographing time of any one of the plurality of image data as the reference time, 3. The image processing apparatus according to claim 2, wherein the determination criterion defines the time range in the form of an allowable upper limit value of a time difference from the reference time.

5. a master clock that is a reference for the time synchronization; a time synchronization unit that synchronizes the times of the built-in clocks of the plurality of devices based on the master clock; 5. The image processing device according to claim 3, further comprising:

6. 6. The image processing device according to claim 3, wherein the synchronization method complies with the time synchronization standard IEEE1588-2008.

7. 7. The image processing device according to claim 3, wherein the index information further includes synchronization method information indicating the synchronization method.

8. 8. The image processing apparatus according to claim 2, wherein the index information further includes reference time information indicating the reference time.

9. 1. An image processing device that processes an image file having a structure including a first storage area for storing a plurality of image data, and a second storage area for storing metadata relating to the plurality of image data, The metadata includes: (1) Synchronous group information including information for grouping image data captured in a time-synchronized manner and information indicating image data included in the group among the plurality of image data; (2) index information indicating a grouping index for a group related to the synchronization group information, the index information including information on a determination criterion used in the grouping to determine whether or not the images were captured in a time-synchronized manner; Including, The image processing device includes: an acquisition means for acquiring the image file to be processed; a presentation means for presenting information regarding the synchronization accuracy of image data included in a group related to the synchronization group information based on the information regarding the determination criteria stored in the image file to be processed; 1. An image processing device comprising:

10. 1. An image processing device that processes an image file having a structure including a first storage area for storing a plurality of image data, and a second storage area for storing metadata relating to the plurality of image data, The metadata includes: (1) Synchronous group information including information for grouping image data captured in a time-synchronized manner and information indicating image data included in the group among the plurality of image data; (2) Index information indicating a grouping index for a group related to the synchronized group information, the index information including information on a determination criterion used in the grouping as to whether or not images were captured in a time-synchronized manner, and reference time information indicating a reference time for capturing image data included in the group; Including, The image processing device includes: a first acquisition means for acquiring the image file to be processed; a second acquiring means for acquiring image data to be added to the first storage area of ​​the image file to be processed and photographing information relating to the photographing of the image data to be added; a determination means for determining whether or not to add the image data to the group related to the synchronization group information based on the determination criterion information and the reference time information of the image file to be processed and the shooting information related to the image data to be added; a configuration means for configuring the synchronization group information based on the determination result by the determination means; a generating means for generating a new image file by storing the image data to be added in the first storage area of ​​the image file to be processed and storing the synchronization group information configured by the configuring means in the second storage area; 1. An image processing device comprising:

11. 11. The image processing device according to claim 1, wherein the image data stored in the first storage area is at least one of a still image and a moving image.

12. 12. The image processing device according to claim 1, wherein the file format of the image file is High Efficiency Image File Format (HEIF) or ISO Base Media File Format (ISOBMFF).

13. 1. A control method for an image processing device that generates an image file having a structure including a first storage area for storing a plurality of image data and a second storage area for storing metadata relating to the plurality of image data, the method comprising: an acquisition step of acquiring the plurality of image data and photographing information relating to photographing of the image data; a determination step of determining whether or not each of the plurality of image data acquired in the acquisition step has been photographed in a time-synchronized manner based on the photographing information related to the plurality of image data and a determination criterion for determining whether or not the images have been photographed in a time-synchronized manner; a first configuration step of configuring synchronization group information including information for grouping image data determined to have been captured in a time-synchronized manner in the determination step and information indicating image data included in the group among the plurality of image data; a second configuration step of configuring index information indicating a grouping index for a group related to the synchronization group information, the index information including at least information on the determination criterion; a generating step of generating the image file storing the plurality of image data acquired in the acquiring step, and metadata including the synchronization group information configured in the first configuring step and the index information configured in the second configuring step; A control method comprising:

14. 1. A control method for an image processing device that processes an image file having a structure including a first storage area for storing a plurality of image data and a second storage area for storing metadata relating to the plurality of image data, comprising: The metadata includes: (1) Synchronous group information including information for grouping image data captured in a time-synchronized manner and information indicating image data included in the group among the plurality of image data; (2) index information indicating a grouping index for a group related to the synchronization group information, the index information including information on a determination criterion used in the grouping to determine whether or not the images were captured in a time-synchronized manner; Including, The control method includes: an acquisition step of acquiring the image file to be processed; a presentation step of presenting information regarding the synchronization accuracy of image data included in a group related to the synchronization group information based on information about the determination criteria stored in the image file to be processed; A control method comprising:

15. 1. A control method for an image processing device that processes an image file having a structure including a first storage area for storing a plurality of image data and a second storage area for storing metadata relating to the plurality of image data, comprising: The metadata includes: (1) Synchronous group information including information for grouping image data captured in a time-synchronized manner and information indicating image data included in the group among the plurality of image data; (2) Index information indicating a grouping index for a group related to the synchronized group information, the index information including information on a determination criterion used in the grouping as to whether or not images were captured in a time-synchronized manner, and reference time information indicating a reference time for capturing image data included in the group; Including, The control method includes: a first acquisition step of acquiring the image file to be processed; a second acquisition step of acquiring image data to be added to the first storage area of ​​the image file to be processed and photographing information related to photographing of the image data to be added; a determination step of determining whether or not to add the image data to the group related to the synchronization group information based on the determination criterion information and the reference time information of the image file to be processed and the shooting information related to the image data to be added; a configuration step of configuring the synchronization group information based on the determination result in the determination step; a generating step of generating a new image file by storing the image data to be added in the first storage area of ​​the image file to be processed and storing the synchronization group information configured in the configuring step in the second storage area; A control method comprising:

16. A program that causes a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 12.

Images