Synchronization device

Abstract

High-precision wireless synchronization can be achieved between multiple devices. [Solution] The synchronization device comprises a control unit, a clock, and a synchronization unit, wherein the clock is equipped with a self-propelled oscillator that acts as a clock source, and the synchronization unit controls its own oscillator to synchronize with an oscillator provided by another synchronization device. Furthermore, the synchronization unit controls the rise time of its own oscillator to synchronize its frequency and phase with an oscillator provided by another synchronization device.

Description

[Technical Field]

[0001] This invention relates to a synchronization device. [Background technology]

[0002] Conventionally, there are technologies for synchronizing the operation of multiple devices. Generally, the synchronization of multiple devices is achieved by wired connections for accuracy reasons. For example, synchronized shooting of a subject using multiple cameras is achieved by a technology that connects the cameras via wires (see, for example, Patent Document 1). Specifically, technologies have been disclosed that synchronize multiple cameras using broadcast synchronization signals including black burst signals (BBS) and tri-level signals (collectively referred to as Genlock signals) via coaxial cables, or network synchronization such as PTP (Precision Time Protocol), to provide a frame-synchronized shooting environment. Such camera synchronization technologies can reduce synchronization between microphones and cameras, and reduce disruptions in distribution caused by camera switching (i.e., viewpoint switching) in applications of video distribution.

[0003] Furthermore, in recent years, synchronization technology has also been used for video synthesis using synchronized captured data (such as free-viewpoint synthesis, Structure from Motion, or 3D model construction) (see, for example, Patent Document 2). In such cases, by capturing moving subjects at precisely the same timing, more accurate synthesis results can be obtained. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Patent No. 7340957 [Patent Document 2] Japanese Patent Publication No. 2024-112399 [Overview of the project] [Problems that the invention aims to solve]

[0005] However, the technology described in Patent Document 1 requires multiple devices (e.g., photographic equipment) to be connected to each other by cables, making it difficult to move the devices easily. Also, because it requires cable connections, it may be suitable for applications such as permanent installation in stadiums, but it may be difficult to apply to applications such as one-off sporting events or live events where cable laying work would be burdensome and costly. Furthermore, depending on the type and length of the cable, a timing difference of several hundred nanoseconds to several microseconds may occur, making synchronized operation difficult.

[0006] Furthermore, while the technology described in Patent Document 2 guarantees that the frames will not be off by more than one because the frame alignment of each captured frame is performed after shooting, it is not possible to strictly synchronize the timing of frame captures because each piece of equipment shoots independently. When using the technology described in Patent Document 2 to shoot fast-moving subjects, the difference in how the subject is captured due to mismatched shooting timings can become large, making it difficult to properly composite the images. In addition, in images with very few feature points (such as images of vast spaces or subjects with rotational symmetry such as spheres), post-processing of frames using the captured images is difficult.

[0007] Therefore, the present invention has been made in view of the above points, and aims to present a technology that enables high-precision wireless synchronization between multiple devices. [Means for solving the problem]

[0008] One aspect of the present invention is a synchronization device characterized in that the clock has a self-propelled oscillator as its clock source, and the synchronization unit controls its own oscillator so as to synchronize with an oscillator provided by another synchronization device.

[0009] Furthermore, in one embodiment of the present invention, the synchronization unit is characterized by controlling the rise time of its own oscillator to synchronize its frequency and phase with that of an oscillator provided by another synchronization device.

[0010] Furthermore, in one embodiment of the present invention, the synchronization unit controls its own oscillator so that it becomes a clock source with the same frequency and phase as other synchronization devices.

[0011] Furthermore, in one embodiment of the present invention, the control unit is (1a) Instead of adjusting the machine's own clock based on clock correction information regarding time discrepancies with other synchronization devices, indirect adjustment is performed, which is control that takes into account time discrepancies with other synchronization devices. (1b) Or the synchronization unit directly adjusts the clock of the machine, It is characterized by the ability to switch between any of the following synchronization methods.

[0012] Furthermore, in one embodiment of the present invention, the synchronization device is characterized by using its own time system.

[0013] Furthermore, in one embodiment of the present invention, the synchronization unit is controlled to perform phase correction after frequency correction.

[0014] Furthermore, in one embodiment of the present invention, the synchronization unit is (2a) Perform frequency correction on the oscillator of the aircraft, (2b) Perform phase correction on the oscillator of the own machine so that the rising edge timing matches that of the oscillators of other synchronization devices. (2c) The system is characterized by controlling the phase of its own oscillator so as to maintain only the phase-matched state after synchronization with other synchronization devices is complete.

[0015] In addition, one aspect of the present invention further includes a wireless communication unit that transmits and receives communication information by radio waves and measures the transmission and reception timing of the communication information. The information regarding the transmission and reception timing includes time information indicating the time when the radio waves are transmitted and received, and phase information φ indicating the phase difference of the transmitted and received radio waves. The synchronization unit (3a) Based on the phase information with high resolution, identify "φ" among "Φ = φ + 2nπ". (3b) Based on the time information with low resolution, identify "2nπ" among "Φ = φ + 2nπ". (3c) Control the oscillator of the own device so as to adjust the time difference from another synchronization device based on the identified phase information Φ.

Effect of the Invention

[0016] According to the present invention, high-precision wireless synchronization can be realized among a plurality of devices.

Brief Description of the Drawings

[0017] [Figure 1] It is a block diagram showing a configuration example of a synchronization system including a plurality of synchronization devices. [Figure 2] It is a flowchart showing the flow of processing performed by a plurality of synchronization devices. [Figure 3] It is a diagram for explaining an example of calculation performed by the synchronization unit. [Figure 4] It is a diagram for explaining a clock adjustment method by the synchronization device according to the embodiment. [Figure 5] It is a diagram for explaining an example of a clock synchronization method. [Figure 6] It is a flowchart for explaining an example of the flow of clock time adjustment performed by the synchronization device. [Figure 7] It is a schematic diagram for explaining an aspect of a synchronous shooting system which is an example of a synchronization system. [Figure 8] It is a block diagram for explaining a configuration example of a photographing device. [Figure 9]This diagram compares a time synchronization method that only synchronizes the time of a real-time clock with a time synchronization method that synchronizes the oscillator or clock by frequency and phase. [Figure 10] This diagram illustrates the relationship between the type of trigger signal and the timing difference in the image capture. [Figure 11] This diagram illustrates an example of a synchronization method that involves staggering the timing of operations between different imaging devices. [Figure 12] This figure shows an example of communication information, including synchronization information and information related to synchronized shooting. [Figure 13] This is a diagram illustrating an example of how to assign a timestamp. [Figure 14] This is a diagram illustrating an example of a method for determining location. [Figure 15] This figure shows a schematic example of the hardware configuration of the information processing device applied to this embodiment. [Modes for carrying out the invention]

[0018] [Embodiment] The synchronization device according to this embodiment will be described in detail below with reference to the attached drawings, with reference to preferred embodiments. In the drawings, identical or similar parts are denoted by the same or similar reference numerals. This embodiment is not limited to these embodiments and includes various modifications or improvements. In other words, the components described below include those that can be easily imagined by a person skilled in the art, and those that are substantially the same, and the components described below can be combined as appropriate. Furthermore, this embodiment may include various omissions, substitutions, or modifications of components without departing from the spirit of the present invention.

[0019] In all the figures used to illustrate the embodiments, components with the same function are given the same reference numerals, and repeated explanations are omitted. Furthermore, in this application, "based on XX" means "based on at least XX," and includes cases where it is based on other elements in addition to XX. Also, "based on XX" is not limited to cases where XX is used directly, but also includes cases where it is based on XX after calculations or processing have been performed on it. "XX" is any element (for example, any information). Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0020] First, an example of a synchronization device according to the embodiment will be described with reference to Figure 1.

[0021] [Example of a synchronization system configuration] Figure 1 is a block diagram showing an example configuration of a synchronization system comprising multiple synchronization devices. Synchronization system 1 comprises multiple synchronization devices 10 and a control terminal 20. The first synchronization device 101 and the second synchronization device 102 illustrate multiple synchronization devices 10. When the first synchronization device 101 and the second synchronization device 102 are not distinguished, they may simply be referred to as synchronization device 10. Synchronization system 1 may comprise multiple synchronization devices 10 and multiple control terminals 20, or it may comprise multiple synchronization devices 10 and one control terminal 20. In the following description, one example is that a control terminal 20 is provided for each synchronization device 10.

[0022] The synchronization system 1 may, if necessary, communicate with an external system 2 via a network. The network may use wireless communication or wired communication. The network may be configured using, for example, the Internet or a local area network (LAN). The network may be configured by combining multiple networks.

[0023] External system 2 is a system that aggregates and processes information acquired from the devices (synchronization device 10 and control terminal 20) of synchronization system 1 and provides it.

[0024] Synchronization device 10 is a device that operates in synchronization with other synchronization devices 10. Synchronization includes synchronizing specific operations or processes, time synchronization, spatial synchronization, and spatiotemporal synchronization. Time synchronization means that the times of multiple synchronization devices 10 are synchronized. Spatial synchronization means that multiple synchronization devices 10 are aware of each other's positions (relative positions). Spatiotemporal synchronization is the combination of time synchronization and spatial synchronization. In the following description, the case where each synchronization device 10 is independent will be described as an example, but this embodiment is not limited to this example, and multiple synchronization devices 10 may be provided within a single device.

[0025] The control terminal 20 assists in the synchronization of the synchronization devices 10 by, as needed, controlling the synchronization devices 10 and executing some of the processes performed by the synchronization devices 10. In addition, the control terminal 20 acts as a bridge for information communication between the synchronization devices 10 and the external system 2 as needed.

[0026] [Example of a synchronization device configuration] The synchronization device 10 comprises a sensor 11, a wireless communication unit 12, a synchronization unit 13, a clock 14, and a control unit 15 as its functional components. The synchronization device 10 is configured using an information processing device such as a smartphone, tablet, personal computer, or dedicated equipment.

[0027] All or part of the functions of the synchronization device 10 may be implemented using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The above program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and semiconductor storage devices (e.g., SSDs: Solid State Drives), as well as storage devices such as hard disks and semiconductor storage devices built into computer systems. The above program may be transmitted via a telecommunications line.

[0028] Sensor 11 operates based on a trigger signal and acquires information. The synchronization device 10 may have one or more sensors 11. Furthermore, the sensors 11 in each synchronization device 10 may be of the same type, different types of sensors, or some sensors may be of the same type and some sensors may be of different types. Sensor 11 is a sensor selected according to the application of the synchronization device 10, such as an image sensor, temperature sensor, light sensor, laser sensor, sound sensor (microphone), vibration sensor, pressure sensor, humidity sensor, ultrasonic sensor, acceleration sensor, current sensor, or frequency sensor. Note that sensor 11 does not have to be provided in the synchronization device 10. Sensor 11 may be a sensor provided outside the synchronization device 10.

[0029] Sensor 11 may acquire information at a predetermined point in time based on a trigger signal, or it may acquire information for a specific period based on a trigger signal. For example, sensor 11 may acquire information at at least one of the following points in time (hereinafter sometimes referred to as the start time): the time when the trigger signal becomes active, a predetermined time before the time when the trigger signal becomes active, or a predetermined time after the time when the trigger signal becomes active. Sensor 11 may also acquire information for a predetermined period from the start time. Furthermore, sensor 11 may acquire information from the start time up to the time when there is an instruction to end (for example, another trigger signal). In the following description, the information acquired by sensor 11 may be referred to as acquired information.

[0030] The wireless communication unit 12 transmits and receives (hereinafter sometimes simply referred to as transmitting and receiving) communication information with other synchronization devices 10. The wireless communication unit 12 is, for example, a wireless device that transmits and receives communication information using radio waves. Communication information is, for example, a code or a packet. The communication information includes synchronization information used to synchronize the clock 14. The wireless communication unit 12 receives communication information transmitted from the wireless communication unit 12 of other synchronization devices 10 and obtains synchronization information by decoding the received communication information.

[0031] The wireless communication unit 12 may be provided as one unit or as multiple units in the synchronization device 10. Alternatively, one wireless communication unit 12 may communicate on one frequency or on multiple frequencies.

[0032] [Synchronization Information] Figure 2 is a flowchart showing the processing flow performed by multiple synchronization devices 10. An example of the information included in the synchronization information will be explained with reference to Figure 2. Figure 2 shows an example in which the first synchronization device 101 and the second synchronization device 102 send and receive communication information to each other. In the following explanation, the wireless communication unit 12 and synchronization unit 13 of the first synchronization device 101 will be referred to as wireless communication unit 12A and synchronization unit 13A, and the wireless communication unit 12 and synchronization unit 13 of the second synchronization device 102 will be referred to as wireless communication unit 12B and synchronization unit 13B to distinguish them from each other.

[0033] (Steps S101, S201) The first synchronization device 101 and the second synchronization device 102 start transmitting radio waves.

[0034] (Step S102) The wireless communication unit 12A transmits communication information to the second synchronization device 102. The wireless communication unit 12A also measures the transmission timing at which the communication information was transmitted.

[0035] The transmission timing may be determined by at least one of the following: the phase of the transmitted radio waves and the time at which the communication information was transmitted. Specifically, the transmission timing may be, for example, the transmission phase of the radio waves at a specific symbol or timing, and the time at which that phase occurred.

[0036] Phase refers to the phase difference between the electric or magnetic field of a radio wave and a reference source. Phase may be, for example, the carrier phase of a radio wave, or it may be the phase difference measured relative to the clock of a clock. Alternatively, phase may refer to the phase of an I / Q signal generated from a received radio wave signal, with the carrier wave as the reference. Phase may be measured using the carrier phase of a radio wave, or it may be the phase in an encoded state, or the phase after down-conversion; any signal can be used as long as the phases of the transmitting and receiving sides can be compared. For phase detection, for example, I / Q conversion, a time interval counter, or DMTD (Dual Mixer Time Difference) may be used, and up / down conversion may be performed using an appropriate mixer beforehand. Time may refer to the time indicated by the clock 14 or counter information.

[0037] A specific symbol may be, for example, a symbol included in the preamble contained within the packet, or a specific symbol included in a packet or symbol synchronization symbol such as a sync word (Start of Frame Delimiter: SFD). A specific timing may be a timing based on predetermined information, such as immediately after detection of the preamble or 0.4 ms after detection of the preamble, or other preamble-based timing.

[0038] Hereinafter, the phase information indicating the measured phase will be referred to as "φ," and the time information indicating the measured time will be referred to as "t." In addition, multiple phase information and multiple time information may be referred to as "φ1" and "t1" respectively to distinguish them. For example, in step S102, the wireless communication unit 12 of the first synchronization device 101 measures the phase information φ1 and the time information t1.

[0039] (Step S202) The wireless communication unit 12B receives communication information from the first synchronization device 101. The wireless communication unit 12B also measures phase information φ2 and time information t2 as the reception timing when it receives the communication information.

[0040] The reception timing may be determined by at least one of the following: the phase of the received radio wave and the time the communication information was received. In the following explanation, the transmission timing and reception timing may be collectively referred to as the transmission / reception timing.

[0041] (Step S203) The wireless communication unit 12B transmits communication information including synchronization information to the first synchronization device 101. Synchronization information is information including information related to transmission and reception timing, such as phase information φ and time information t. For example, the wireless communication unit 12B transmits communication information including phase information φ2 and time information t2. That is, the wireless communication unit 12 stores information related to the reception timing of receiving the communication information in the communication information and transmits it to the synchronization device 10 that is the target of communication. The wireless communication unit 12B measures phase information φ3 and time information t3 as the transmission timing when it transmitted the communication information including phase information φ2 and time information t2.

[0042] (Step S103) The wireless communication unit 12A receives communication information including phase information φ2 and time information t2 from the second synchronization device 102. The wireless communication unit 12A also measures phase information φ4 and time information t4 as the reception timing when it has acquired the communication information including phase information φ2 and time information t2.

[0043] (Step S104) The wireless communication unit 12A transmits communication information including phase information φ1, phase information φ4, time information t1, and time information t4 to the second synchronization device 102. Since the phase information at the time of transmission (phase information φ1) is obtained as a result of transmission, it is not possible to include the phase information at the time of transmission in the communication information (e.g., packet) at the same time as transmission. For this reason, the wireless communication unit 12 stores the information related to the transmission timing of the transmitted communication information in the communication information for subsequent transmissions and transmits it to the synchronization device 10 that is the target of communication. In other words, the information related to the transmission timing is transmitted with a delay. The wireless communication unit 12A measures the phase information φ5 and time information t5 as the transmission timing of the communication information including phase information φ1, phase information φ4, time information t1, and time information t4.

[0044] (Step S204) The wireless communication unit 12B receives communication information from the first synchronization device 101, including phase information φ1, phase information φ4, time information t1, and time information t4. The wireless communication unit 12B also measures phase information φ6 and time information t6 as the reception timing of the communication information including phase information φ1, phase information φ4, time information t1, and time information t4.

[0045] (Step S205) The wireless communication unit 12B transmits communication information including phase information φ3, phase information φ6, time information t3, and time information t6 to the second synchronization device 102. The wireless communication unit 12B also measures phase information φ7 and time information t7 as the transmission timing for the communication information including phase information φ3, phase information φ6, time information t3, and time information t6.

[0046] (Step S105) The wireless communication unit 12A receives communication information from the second synchronization device 102, including phase information φ3, phase information φ6, time information t3, and time information t6. The wireless communication unit 12A also measures phase information φ8 and time information t8 as the reception timing of the communication information including phase information φ3, phase information φ6, time information t3, and time information t6.

[0047] (Step S106) The synchronization unit 13A acquires phase information φ1, phase information φ2, phase information φ3, phase information φ4, time information t1, time information t2, time information t3, and time information t4. Based on this information, the synchronization unit 13A calculates the time difference and propagation delay time. That is, the synchronization unit 13A acquires information relating to the timing at which communication information transmitted from the unit is transmitted and the timing at which it is received by the communication destination, and the timing at which communication information transmitted from the communication destination is transmitted and the timing at which it is received by the unit. Based on the information relating to the four types of transmission and reception timings, the synchronization unit 13A calculates information indicating the propagation delay time and time difference. Details of the calculations performed by the synchronization unit 13 will be described later.

[0048] (Step S107) The wireless communication unit 12A transmits communication information including phase information φ5, phase information φ8, time information t5, and time information t8 to the second synchronization device 102. The wireless communication unit 12A also measures phase information φ9 and time information t9 as the transmission timing for the communication information including phase information φ5, phase information φ8, time information t5, and time information t8.

[0049] (Step S206) The wireless communication unit 12B receives communication information from the first synchronization device 101, including phase information φ5, phase information φ8, time information t5, and time information t8. The wireless communication unit 12B also measures phase information φ10 and time information t9 as the reception timing of the communication information including phase information φ5, phase information φ8, time information t5, and time information t8.

[0050] (Step S207) The synchronization unit 13B acquires phase information φ3, phase information φ4, phase information φ5, phase information φ6, time information t3, time information t4, time information t5, and time information t6. That is, the synchronization unit 13B calculates information indicating propagation delay time and time shift based on the same four types of transmission and reception timing information as the synchronization unit 13A.

[0051] Multiple synchronization devices 10 continuously calculate information indicating propagation delay time and time difference by repeatedly performing steps S205, S105, S106, S107, S206, and S207 with respect to each other. The time difference and propagation delay time are used in the process of synchronizing the multiple synchronization devices 10 with each other.

[0052] The wireless communication unit 12 periodically transmits communication information as radio waves. However, for synchronization between synchronization devices 10, the wireless communication unit 12 only needs to transmit communication information multiple times; it is not necessarily required to transmit periodically. The wireless communication unit 12 may transmit communication information irregularly. Furthermore, to achieve higher-precision synchronization with other synchronization devices 10, it is preferable to transmit communication information at a high frequency; for example, the radio wave transmission frequency is 20 Hz.

[0053] Wireless communication standards include, for example, LPWA (Low Power Wide Area), Bluetooth (registered trademark), BLE (Bluetooth Low Energy), and Wi-Fi. Radio frequencies are ISM (Industrial, Scientific, and Medical radio band) bands such as the 920 MHz band and the 2.4 GHz band. This allows for synchronization over long distances (km) to short distances (m). While technologies exist that use Bluetooth and other technologies to simultaneously drive remote devices, these remote control technologies assume that multiple devices receiving a wireless signal transmitted by one device are driven simultaneously. Therefore, they do not take into account the calculation delay required for radio wave transmission or reception, or the radio wave propagation delay time, and thus do not strictly perform synchronization. Furthermore, conventional wireless synchronization technologies have low synchronization accuracy, making it difficult to achieve high-precision synchronization like that of the synchronization device 10.

[0054] [Calculation of clock discrepancy] The synchronization unit 13 adjusts the time of the clock 14 with other synchronization devices 10 based on synchronization information. The synchronization unit 13 calculates the information used to adjust the clock 14. The synchronization unit 13 may calculate the information used to adjust the clock 14 according to, for example, Cristian's algorithm. Figure 3 is a diagram illustrating an example of the calculation performed by the synchronization unit 13. The synchronization unit 13 adjusts the time of the clock 14 using the calculated information. Note that the phase and time values ​​shown in Figure 3 are for illustrative purposes only, and the synchronization device 10 according to this embodiment does not need to operate as shown in Figure 3.

[0055] FIG. 3(A) shows an example of the case where the phase information φ of the transmitted and received radio waves is used. The synchronization unit 13 calculates, for example, the clock phase difference between the two synchronization devices 10 and the propagation delay phase of the radio wave based on the transmission phase on the transmission side and the reception phase on the reception side obtained by the wireless communication unit 12. Specifically, radio wave transmission and reception are performed bidirectionally, the transmission phase at the time of transmission of the first synchronization device 101 is the phase information Φ1, the reception phase at the time of reception of the second synchronization device 102 is the phase information Φ2, the transmission phase at the time of transmission of the second synchronization device 102 in the return path is the phase information Φ3, and the reception phase at the time of reception of the first synchronization device 101 is the phase information Φ4. In this case, the synchronization unit 13 calculates the clock phase difference δ of the internal clock (clock 14) and the propagation delay phase RTT (round trip time) / 2 of the radio wave based on the information related to the above four types of transmission and reception timings according to the following formula (1). The propagation delay phase RTT of the radio wave is the phase shift required for the round trip of the radio wave.

[0056]

Equation

[0057] Specifically, the transmission phase “φ S ” of the first synchronization device 101, and the phase of the second synchronization device 102 at this timing is “φ<00000​​​​​​​​​​​​​​​​​​According to equation (1), the clock phase difference δ between the clock 14 of the first synchronous device 101 and the clock 14 of the second synchronous device 102 is "φ S -φ M Therefore, the radio wave propagation delay phase RTT / 2 can be calculated as "5π / 2".

[0058] Figure 3(B) shows an example of using time information t, which indicates the time when communication information was sent and received. The synchronization unit 13 calculates the time difference (time shift) between the two synchronization devices 10 and the radio wave propagation delay time, similar to the case when using phase information φ, based on the transmission time of the transmitting side and the reception time of the receiving side obtained by the wireless communication unit 12. Specifically, radio wave transmission and reception are performed bidirectionally, with the transmission time of the first synchronization device 101 as time information T1, the reception time of the second synchronization device 102 as time information T2, the transmission time of the second synchronization device 102 on the return path as time information T3, and the reception time of the first synchronization device 101 as time information T4. In this case, the synchronization unit 13 calculates the time difference of the internal clock and the radio wave propagation delay time RTT / 2 based on the following equation (2) from the four types of transmission and reception timing information described above.

[0059]

number

[0060] Specifically, the transmission time of the first synchronization device 101 is set to "10:00:00", and the time of the second synchronization device 102 at this time is set to "10:00:03". Also, the reception time of the second synchronization device 102 is set to "10:00:04", and the time of the first synchronization device 101 at this time is set to "10:00:01". Furthermore, the transmission time of the second synchronization device 102 on the return journey is set to "10:00:07", and the time of the first synchronization device 101 at this time is set to "10:00:04". Also, the reception time of the first synchronization device 101 on the return journey is set to "10:00:08", and the time of the second synchronization device 102 at this time is set to "10:00:05". In this case, time information T1 is "10:00:00", time information T2 is "10:00:04", time information T3 is "10:00:07", and time information T4 is "10:00:05". According to equation (2), the time difference δ between the clock 14 of the first synchronization device 101 and the clock 14 of the second synchronization device 102 is "3 [sec]", and the radio wave propagation delay time RTT / 2 can be calculated as "1 [sec]".

[0061] The synchronization unit 13 can calculate the time difference of the clock 14 by calculating the time difference of the clock 14 based on equations (1) and (2), thereby canceling out the propagation delay phase and propagation delay time.

[0062] The synchronization unit 13 may calculate the clock phase difference δ based on the phase information φ, or it may calculate the time difference δ based on the time information t. The synchronization unit 13 may also calculate the phase information φ and the time difference δ based on both the phase information φ and the time information t. This allows the phase ambiguity arising from the phase-based calculation to be determined by detecting the time difference δ using time. Phase ambiguity refers to the fact that of Φ = φ + 2nπ (where n is an integer), only φ is physically observable, while n is not; that is, it is impossible to distinguish between phase 0[°] and 360[°]. By determining φ using a phase difference δ with high resolution and capable of precisely detecting time deviations, and by combining this with a method using a time difference δ with lower resolution, the remaining n due to the ambiguity of phase measurement can be determined. By combining these two methods, the time of the clock 14 of the synchronization device 10 can be synchronized with high precision. By using such a synchronization device 10 in, for example, a shooting device, it is possible to synchronize shooting frames without discrepancies between multiple shooting devices, and to increase the accuracy of the timestamps assigned to the frames, thereby improving the quality of photogrammetry synthesis and enhancing the viewing experience for viewers.

[0063] The clock 14 is not limited to a real-time clock that handles time, but may also be a counter, oscillator, oscillator circuit, or oscillator that performs timing operations. If the clock 14 is a time or counter, the synchronization unit 13 will operate to match its value. If the clock 14 is an oscillator or the like, the synchronization unit 13 will operate to match the timing of the rising or falling edges of the clock. In the following description, the clock 14 will be assumed to include an oscillator 141 and a real-time clock 142.

[0064] The synchronization system 1 may designate one of the multiple synchronization devices 10 as a master and the other synchronization devices 10 as slaves, synchronizing the clocks 14 of the other synchronization devices 10 to match the time on the master synchronization device 10. In the following description, adjusting the time between multiple synchronization devices 10 may be referred to as synchronizing with other synchronization devices 10. In this case, the synchronization device 10 that takes on the role of master does not need to be a fixed single unit; control may be performed to switch to another synchronization device 10 as the master depending on the situation. Alternatively, without designating a master role, a predetermined time such as the average time of all devices or the time of any one device may be considered as a common time, and control may be performed to synchronize all devices so that they show the same time. The control may be performed by the synchronization device 10 or by the control terminal 20. Synchronization may mean adjusting each device to be strictly synchronized, or adjusting them to be synchronized toward a predetermined synchronization standard.

[0065] In this embodiment, the time, time information t, or time indicating a specific point in time such as timing may be in the format of UTC (Universal Time Coordinated), JST (Japan Standard Time), or year, month, day, hour, minute, second. Furthermore, this time may be a time uniquely set among the synchronization devices 10, starting from the master machine's startup time, the startup time of the first synchronization device 10 to be started, or any other arbitrarily set operation. In other words, the time may be the elapsed time from the starting point. Also, the elapsed time does not need to follow a seconds basis; it may be a counter value based on the oscillation period of the oscillator 141 provided in the synchronization device 10. Using a synchronization device 10 with its own unique time system enables high-precision synchronization among the group of synchronization devices 10, even in indoor use where GPS (Global Positioning System) / GNSS (Global Navigation Satellite System) cannot be provided, or when it is difficult to obtain UTC time due to the availability of a PTP (Precision Time Protocol) master clock, etc.

[0066] As described above, the synchronization device 10 does not need to synchronize to a global reference time, and local high-precision time synchronization is possible within the group (synchronization system 1) formed by the target synchronization device 10, etc. This allows the synchronization system 1 to switch between master and slave (i.e., source and destination) as needed, without being bound by the concept of a master clock. If the master designated by the user becomes dysfunctional due to failure or other reasons, or if wireless communication is interrupted, or if other situations arise that make it impossible to guarantee synchronization accuracy, the synchronization system 1 may autonomously switch to another synchronization device 10 as the synchronization source.

[0067] In the synchronization system 1, if there are no synchronization sources other than the synchronization device 10, i.e., no external synchronization sources, the synchronization performance will not degrade regardless of which synchronization device 10 becomes the synchronization source. Therefore, the synchronization system 1 does not necessarily require the user to specify a synchronization device 10 that will act as the master. The synchronization device 10 may autonomously select the synchronization device 10 to be used as the synchronization source. The synchronization device 10 may autonomously select the synchronization device 10 to be used as the synchronization source, for example, based on its suitability as a synchronization source, such as the synchronization status and synchronization accuracy with other synchronization devices 10.

[0068] Furthermore, for similar reasons, the connection configuration is not limited to a single synchronization source where all synchronization devices 10 are connected, and synchronization may be performed using a mesh topology. Therefore, there may be multiple synchronization devices 10 that act as synchronization sources within a group. Also, a synchronization device 10 that synchronizes its own clock 14 based on the clock 14 of a synchronization device 10 that acts as a synchronization source may further become a synchronization source for other synchronization devices 10 (chain topology). With such a connection configuration, it becomes possible to substantially place and synchronize the synchronization devices 10 over a wider range than the range that the radio waves of the synchronization devices 10 can normally reach. In addition, since it is generally known that time synchronization errors follow a Gaussian distribution, there is no accumulation of synchronization errors associated with the chain in the synchronization means according to this embodiment. Thus, with the synchronization system 1, which does not have a master-slave distinction, operation and configuration can be performed with any connection configuration.

[0069] [Adjusting the time discrepancy] The synchronization unit 13 adjusts the time of the aforementioned clock 14 with other synchronization devices 10 based on the calculated clock deviation (time difference δ). Figure 4 is a diagram illustrating the adjustment method of the clock 14 by the synchronization device 10 according to this embodiment. An example of the adjustment method will be specifically explained with reference to Figure 4.

[0070] Figure 4(A) shows an example of how the synchronization unit 13 indirectly adjusts the clock 14 by providing clock correction information to the control unit 15. In this case, the clock 14 itself is not adjusted. The synchronization unit 13 outputs clock correction information to the control unit 15 regarding the time difference between the machine's clock 14 and the clock 14 of another synchronization device 10, such as being +1 microsecond ahead or -5 microseconds behind. The control unit 15 corrects the outputted clock correction information and then controls the operation of the sensor 11 and assigns time information to the acquired information obtained by the sensor 11. In other words, the control unit 15 does not adjust the clock 14 itself, but performs control that takes into account the time difference of the clock 14.

[0071] Figure 4(B) shows an example of how the synchronization unit 13 directly adjusts the clock 14. The synchronization unit 13 adjusts (corrects) the clock's delay or advance as appropriate based on the detected time deviation. The synchronization unit 13 may correct the clock's deviation by, for example, increasing or decreasing a counter from an external source. The synchronization unit 13 may also perform feedback control so that at least one of the frequency and phase of an adjustable oscillator such as a VCXO (Voltage-Controlled Crystal Oscillator), OCXO (Oven-Controlled Crystal Oscillator), or DCXO (Digital-Controlled Crystal Oscillator) synchronizes with other synchronization devices 10. Note that correcting the clock 14's time deviation by a large amount at once can cause frame skipping, so it is desirable for the synchronization unit 13 to correct the time deviation gradually to approach the target time based on the detected time deviation. Furthermore, when performing feedback control to the oscillator, it is desirable for the synchronization unit 13 to correct in stages, such as first correcting the frequency and then correcting the phase. Furthermore, when the synchronization method according to this embodiment is adopted in the 920MHz band wireless, the synchronization accuracy is approximately several tens of nanoseconds, which is extremely high precision.

[0072] Figure 5 illustrates an example of a synchronization method for clock 14. Figure 5(A) shows an example of synchronizing clock 14 by only adjusting its time with the real-time clock 142. This is the same method used for time synchronization with time servers such as NTP (Network Time Protocol) and PTP. In this case, the clock source, such as oscillator 141, operates autonomously, and the resulting time discrepancy is adjusted every few seconds or minutes.

[0073] Figure 5(B) shows an example of synchronizing the clock 14 by performing frequency-phase synchronization. The synchronization unit 13 may synchronize the clock source such as the oscillator 141 or the clock 14 with other synchronization devices 10 in frequency-phase synchronization and keep time, or it may perform feedback control so that the rising edge of the clock is always synchronized. When the clock 14 is controlled in this way, the time difference does not increase as time passes from the time of periodic time synchronization, as in NTP, PTP, etc., and when used in a camera, no frame shift occurs over time. For this reason, it is desirable to perform the frequency-phase synchronization shown in Figure 5(B). Furthermore, even if wireless communication is temporarily interrupted, long-term stability can be ensured because both the frequency and phase of the clock source are matched at that point. Therefore, even when used in an environment with unstable communication, synchronization of multiple synchronization devices 10 can be achieved for a long period of time.

[0074] The synchronization unit 13 may synchronize the clock 14 based on signals from an external time source other than the synchronization device 10, i.e., signals from an external device. Specifically, the synchronization unit 13 may synchronize based on time information output from a system such as NTP or PTP. This allows for time synchronization with systems such as PTP used by broadcasting stations.

[0075] [Synchronization process flow] Figure 6 is a flowchart illustrating an example of the time adjustment process performed by the synchronization device 10 on the clock 14.

[0076] (Step S301) The synchronization device 10 starts synchronization operation with other synchronization devices 10. Starting synchronization operation may mean, for example, that each synchronization device 10 starts transmitting radio waves (Steps S101 and S201).

[0077] (Step S302) The wireless communication unit 12 of the synchronization device 10 sends and receives communication information including synchronization information with other synchronization devices 10 to measure the transmission and reception timing (phase information φ and time information t).

[0078] (Step S303) The synchronization unit 13 of the synchronization device 10 calculates the clock phase difference δ of the clock deviation 14 based on the phase information φ. Because the detection of the clock deviation 14 based on the phase information φ has high resolution, it is possible to accurately identify "φ" in "Φ = φ + 2nπ". Calculating the clock deviation 14 may also be referred to as determining the clock deviation.

[0079] (Step S304) The synchronization unit 13 of the synchronization device 10 calculates the time difference δ of the clock 14 based on the time information t. Since the detection of the clock 14's deviation based on the time information t does not detect the phase, it is possible to identify "2nπ" from "Φ = φ + 2nπ".

[0080] The synchronization device 10 continuously synchronizes the time of the clock 14 with other synchronization devices 10 by repeatedly executing the processes from step S302 to step S305. The order in which the processes of step S303 and step S304 are executed does not matter during synchronization. Furthermore, during the repeated processing, the processes to be executed may be switched as needed, such as executing only step S303, only step S304, or both steps S303 and S304.

[0081] (Step S305) The synchronization unit 13 of the synchronization device 10 performs feedback control to adjust the deviation of the clock 14 by adjusting the clock 14 itself and providing clock correction information to the control unit 15. As a result, the synchronization device 10 can achieve high-precision time synchronization with other synchronization devices 10. Furthermore, with high-precision time synchronization, the synchronization device 10 can achieve high-precision spatial synchronization.

[0082] When the synchronization device 10 and the control terminal 20 that controls the synchronization device 10 transmit acquired information to the external system 2, if they send packets at the same time, network constraints may result in bandwidth limitations, preventing smooth transmission. The synchronization system 1, which synchronizes each synchronization device 10, can perform time-division multiplexing by staggering the timing of video transmission. Alternatively, the synchronization device 10 may be equipped with a wired communication unit instead of, or in addition to, the wireless communication unit 12, to perform the same communication as wireless communication via a coaxial cable or the like. Furthermore, by using a battery or the like as a power source, the synchronization device 10 does not require a power cable, and multiple shooting devices can be synchronized without any connections by cables or the like.

[0083] The control unit 15 is composed of a processor such as a CPU (Central Processing Unit) and memory (main memory). The control unit 15 functions when the processor executes a program. The control unit 15 controls the sensor 11. For example, the synchronization device 10 outputs a trigger signal to operate the sensor 11 and acquires information from the sensor 11. The control unit 15 may also synchronize the operation of its own sensor 11 with the sensor 11 of another synchronization device 10. Furthermore, the control unit 15 may synchronize the time with other synchronization devices 10 and add time information t indicating the time of acquisition to the acquired information obtained by the sensor 11 (time assignment).

[0084] The synchronization device 10 may also include a storage unit (not shown). The storage unit stores data used by the control unit and the synchronization unit. The storage unit also stores data required when the control and synchronization units perform processing. Each functional unit of the synchronization device 10 may output information directly to other functional units, or it may store the information in a storage unit (not shown), and then the receiving functional unit may refer to the information stored in the storage unit.

[0085] [Summary of Synchronization] According to the embodiment described above, the synchronization device 10 comprises a control unit 15, a clock 14, a synchronization unit 13, and a wireless communication unit 12. The wireless communication unit 12 transmits and receives communication information, including synchronization information, via radio waves. The synchronization unit 13 adjusts the time of the clock 14 with other synchronization devices 10 based on the synchronization information. The control unit 15 controls a sensor 11 that operates based on a trigger signal and acquires information from the sensor 11. The control unit 15 operates the sensor 11 in synchronization with a sensor 11 controlled by another synchronization device 10, or synchronizes with another synchronization device 10 to add time information t indicating the acquisition time to the acquired information acquired by the sensor 11. With the synchronization device 10 described above, the clock 14 is synchronized by exchanging synchronization information wirelessly between the synchronization devices 10, enabling synchronized operations such as synchronized shooting with synchronized shutter timing control between multiple synchronization devices 10, and the addition of time codes (timestamps) on a common time axis. By repeatedly correcting the deviation of the clock 14 periodically, the synchronization accuracy of the clock 14 can be maintained with high precision. Furthermore, it can capture images in frame-synchronized mode while mounted on a mobile device, making post-capture video editing, processing, and analysis easier. Additionally, since no wiring is required, installation and operating costs are reduced, as is the workload associated with installation.

[0086] Furthermore, according to the embodiment described above, the wireless communication unit 12 measures the timing of transmission and reception of communication information, and the synchronization information includes information related to the transmission and reception timing. The synchronization device 10 obtains information used to synchronize its own clock 14 with other clocks 14 by measuring the timing of transmission and reception of radio waves, thereby preventing the effects of processing delays in the control unit 15 and enabling accurate synchronization of the clocks 14.

[0087] Furthermore, according to the embodiment described above, the transmission / reception timing is the time when radio waves are transmitted or received, and the information regarding the transmission / reception timing includes time information t indicating the time when radio waves were transmitted or received. By measuring the time of transmission and reception, the time difference and propagation delay difference can be calculated according to the accuracy and resolution of the counter used for time measurement, enabling highly accurate synchronization that takes into account the propagation delay of radio waves.

[0088] Furthermore, according to the embodiment described above, the transmission and reception timing is the phase difference between the transmitted and received radio waves, which is the phase difference with respect to the radio waves of the reference source, and the information regarding the transmission and reception timing includes phase information φ indicating the phase difference between the transmitted and received radio waves. By measuring the phase of the radio waves, the clock 14 can be synchronized with higher precision compared to time synchronization using transmission and reception timing by a counter. In addition, by combining this with time measurement, the uncertainty of integer multiples of 2π caused by phase ambiguity can be suppressed, the clock 14 can be synchronized with higher precision, and precise exposure control (operation control), etc., becomes possible.

[0089] Furthermore, according to the embodiment described above, the synchronization unit 13 detects the discrepancy in the clock 14 by canceling out the propagation delay time of radio waves transmitted and received with other synchronization devices 10 based on the synchronization information and transmission / reception timing. By canceling out the time taken for radio wave propagation, the processing delay of the control unit 15 and the propagation delay can be canceled out, enabling highly accurate synchronization of the clock 14 and precise exposure control (operation control), etc. This makes it possible to provide a shooting environment (acquisition environment) without frame drift even at high frame rates such as 100 fps. In addition, it is also effective in applications where information acquisition is frequent, such as with high-speed cameras.

[0090] Furthermore, according to the embodiment described above, the clock 14 is equipped with an oscillator 141, and the synchronization unit 13 synchronizes it with the oscillator 141 of the other synchronization device 10 by controlling the rising timing of the oscillator 141. By controlling the oscillator 141 to become a clock source with the same frequency and phase as the other synchronization device 10, even in situations where wireless communication is temporarily unstable, drift begins from a state where the frequencies match, and as a result, stable exposure control (operation control) and time code assignment over the long term become possible.

[0091] Furthermore, according to the embodiment described above, the clock 14 uses its own time system in relation to other synchronization devices 10. Since the synchronization device 10 described above does not require a reference time, it can be used by the synchronization device 10 alone or in combination with multiple synchronization devices 10, without the need to prepare a synchronization source such as GPS, GNSS, NTP, or PTP.

[0092] Furthermore, according to the embodiment described above, the synchronization unit 13 autonomously selects the synchronization source 10 based on at least one of the synchronization status with other synchronization devices 10 or the synchronization accuracy with other synchronization devices 10. While it is common for the synchronization source and synchronization destination to be pre-set, by eliminating this restriction, it is possible to switch to another synchronization device 10 as the synchronization source in the event of a failure, thereby ensuring robustness. In addition, if a unique time system is used and it is not synchronized with a reference time, it is not necessary to consider deviations from the reference time during switching, thus simplifying the configuration.

[0093] Furthermore, according to the embodiment described above, the synchronization unit 13 synchronizes its own clock 14 using the clock 14 of another synchronization device 10 as the synchronization source, and its own clock 14 is used to synchronize the clock 14 of a synchronization device 10 different from the synchronization device 10 used as the synchronization source. With the synchronization device 10, which can synchronize clocks at the nanosecond level by frequency phase synchronization using phase, the accumulation of errors when connected in a chain is extremely small. Also, since errors due to phase synchronization occur in a Gaussian distribution, they do not increase monotonically and converge to zero probabilistically. Therefore, synchronization can be established over a wider range than the range of radio waves, and the synchronization state can be maintained among multiple synchronization devices 10, thus improving redundancy.

[0094] Furthermore, according to the embodiment described above, the synchronization unit 13 synchronizes the clock 14 based on a signal from an external time source. This allows the device's own clock 14 to be synchronized with devices of other standards or principles that are not wireless synchronization.

[0095] Furthermore, according to the embodiment described above, the external time source is a synchronization signal output from an external device. This allows the synchronization device 10 to be synchronized with a broadcasting station's synchronization system such as PTP.

[0096] Furthermore, according to the embodiment described above, the control unit 15 performs time-division multiplexing communication by shifting the transmission timing of acquired information with other synchronization devices 10.

[0097] Furthermore, according to the embodiment described above, the synchronization system 1 includes multiple synchronization devices 10. This makes it possible to construct a system that synchronizes the operation of multiple imaging devices 10A.

[0098] [Example of a synchronized imaging system configuration] Next, we will specifically explain the processing performed by the control unit 15, referring to the imaging device 10A, which is an example of the synchronization device 10.

[0099] Figure 7 is a schematic diagram illustrating an example of the synchronous shooting system 1A, which is an example of the synchronous system 1. The synchronous shooting system 1A is used, for example, to photograph, record, or transmit the movements of players in sports scenes, collisions, destruction, or explosions between objects, or other dynamic phenomena. The synchronous shooting system 1A comprises a plurality of shooting devices 10A, one or more control terminals 20, an image acquisition device 30, and a light-emitting device 40. The wireless image acquisition device 31 and the wired image acquisition device 32 are examples of the image acquisition device 30, and when not distinguished, they may simply be referred to as the image acquisition device 30.

[0100] The video acquisition device 30 collects image data and video data captured by each of the shooting devices 10A. The video collected by the video acquisition device 30 is either stored or distributed to viewers via the external system 2. The video acquisition device 30 may acquire image data and video data from the shooting devices 10A, or it may acquire image data and video data via the control terminal 20. The video acquisition device 30 may collect image data and video data wirelessly, as with the wireless video acquisition device 31, or it may collect image data and video data via a wire, as with the wired video acquisition device 32.

[0101] The light-emitting device 40 is a light-emitting device used when the imaging device 10A takes a picture. The light-emitting device 40 is a light-emitting device used to illuminate the subject when sufficient light cannot be obtained from ambient light alone, and for example, it emits a strobe flash. The synchronized imaging system 1A may be equipped with one light-emitting device 40 or multiple devices.

[0102] The distribution system 2A aggregates image data and video data captured by each of the shooting devices 10A, performs processing such as video synthesis, and then provides it to the viewer. The external system 2 may acquire image data and video data directly from the shooting devices 10A, or it may acquire image data and video data via a control terminal 20, a video acquisition device 30, etc.

[0103] [Example of a synchronized imaging system configuration] Figure 8 is a block diagram illustrating an example configuration of the imaging device 10A. The imaging device 10A comprises an image sensor 11A, a wireless communication unit 12, a synchronization unit 13, a clock 14, and a control unit 15 as its functional units. In other words, the imaging device 10A differs from the synchronization device 10 in that the sensor 11 is an image sensor 11A. The imaging device 10A performs imaging by incorporating the image sensor 11A.

[0104] The image sensor 11A captures a subject based on a trigger signal. By capturing, the image sensor 11A acquires image data or video data. The following description will focus on the case where the image sensor 11A acquires video data, but the same control is applied when acquiring image data as when acquiring video data. Video data and image data refer to one or more frame images captured by the image sensor 11A.

[0105] [Exposure control] The control unit 15 controls the preparation for shooting, such as setting the shooting conditions and external trigger of the image sensor 11A, and starts shooting. The image sensor 11A controls the shooting timing based on the trigger signal provided by the control unit 15. Specifically, the image sensor 11A may start shooting (exposure) when the trigger signal becomes active, or a predetermined time before or after the trigger signal becomes active, or a predetermined time after the trigger signal becomes active. The trigger signal is input to, for example, an external trigger terminal or a vertical sync signal terminal provided on the image sensor 11A. The control unit 15 acquires the frame image captured by the image sensor 11A.

[0106] The synchronization unit 13 described above is adjusted, that is, information for clock correction is provided to the control unit 15 and the clock 14 itself is adjusted, thereby synchronizing the timing between the multiple imaging devices 10A. As a result, the control unit 15 can perform exposure control at the same timing among the multiple imaging devices 10A that are to be synchronized. In other words, the control unit 15 can synchronize the exposure timing of the multiple image sensors 11A in time and perform shooting. By controlling the exposure at the same timing, all video frames obtained from each image sensor 11A will have the same shooting timing, and the subject can be captured in synchronous video from multiple angles and positions. Note that if the frames are synchronized but the exposure timing is not synchronized, and a subject moving at high speed is captured, the position of the subject in the frame may change due to a small difference in exposure timing, and it may not be easy to perform video synthesis. With the synchronized shooting system 1A that synchronizes the exposure timing and shoots at the same timing, even a subject moving at high speed can be captured without discrepancies by multiple imaging devices 10A. In the following description, unless otherwise specified, it is assumed that the multiple imaging devices 10A are shooting in synchronous order.

[0107] Figure 9 is a diagram for comparing a time synchronization method that only synchronizes the time of the real-time clock 142 with a time synchronization method that synchronizes the oscillator 141 and the clock 14 by frequency phase. Figure 9 shows the time shift, frequency shift, and the timing of each captured frame in relation to the change in time.

[0108] Figure 9(A) shows the time difference, frequency difference, and shooting timing of each shooting frame in a time synchronization method that only synchronizes the time of the real-time clock 142. In a time synchronization method that only synchronizes the time of the real-time clock 142, the time difference and frequency difference increase as time passes from the time of synchronization, and the shooting timing also differs between the first shooting device 101A and the second shooting device 102A.

[0109] Figure 9(B) shows the time shift, frequency shift, and shooting timing of each shooting frame in a time synchronization method that uses frequency phase synchronization. In a time synchronization method that phase-synchronizes oscillator 141 and clock 14, both the frequency and phase of the clock source are matched, so the time shift and frequency shift do not tend to increase significantly over time, and the shooting timing remains stable and synchronized over the long term. As a result, shooting frames other than the one immediately after time synchronization can also be captured in sync, making it easy to perform image synthesis.

[0110] [Relationship between trigger signal and shooting timing] Figure 10 illustrates the relationship between the type of trigger signal and the timing difference of the exposure. Exposures A through F shown in Figure 10 are examples of exposure control using different references. Note that "exposure timing" refers to the exposure timing.

[0111] Figure 10(A) shows an example where the trigger signal to the image sensor 11A is the trigger for each frame capture, i.e., the trigger signal with a period of the desired frame rate (exposure A to exposure D). When the trigger signal for each frame capture is provided to the image sensor 11A, all captured frames can be synchronized without any discrepancies in the start timing of the capture. Exposure A is an example where the start timing of the capture is synchronized with the rising edge of the trigger signal. Exposure B is an example where the start timing of the capture is synchronized with the falling edge of the trigger signal. Exposure C is an example where the start timing of the capture is synchronized to a predetermined time after the falling edge of the trigger signal. Exposure D is an example where the end timing of the capture is synchronized with the rising edge of the trigger signal. Note that the image sensor 11A may also synchronize the start timing of the capture to a predetermined time after the rising edge of the trigger signal. Alternatively, the image sensor 11A may synchronize the end timing of the capture to the falling edge of the trigger signal.

[0112] Figure 10(B) shows an example where the trigger signal to the image sensor 11A is a single trigger signal indicating the start of shooting (exposure G and exposure H). When a single trigger signal indicating the start of shooting is given to the image sensor 11A, it is difficult to precisely control the start timing of all captured frames, and a gradual frame shift may occur as time passes from the start of shooting. However, this frame shift is not a problem in the case of short-duration video shooting. Exposure G is an example of aligning the start timing of the first frame with the rising edge of the trigger signal. Exposure H is an example of aligning the start timing of the first frame with the falling edge of the trigger signal. The image sensor 11A may also align the start timing of the first frame with a predetermined time after the falling edge or rising edge of the trigger signal.

[0113] Figure 10(C) shows an example where the trigger signal for the image sensor 11A is a trigger signal indicating the exposure start timing and exposure end timing, i.e., the exposure time (exposure E and exposure F). By controlling the exposure start timing and exposure end timing in synchronization with other imaging devices 10A, the image sensor 11A can perform multi-view imaging that captures only a specific point in time of the subject more precisely than exposure control from exposure A to exposure H. If the exposure times differ for each imaging device 10A, it is desirable to control the exposure start time (shooting start) and exposure end time (shooting end) so that they are the same for each imaging device 10A. That is, the control unit 15 synchronizes the exposure timing of the image sensor with the other imaging devices 10A based on at least one of the following points: the exposure start time within the frame, the exposure end time within the frame, or the point in the middle between the exposure start time and the exposure end time within the frame.

[0114] In the above description, the synchronization system 1 was explained in an example of performing synchronized shooting by temporally matching the shooting timing between each shooting device 10A. However, this embodiment is not limited to this example, and other methods of synchronized shooting may be used. Figure 11 is a diagram illustrating an example of a synchronization method in which the operating timing is shifted between other shooting devices 10A. The synchronization method shown in Figure 11 can substantially increase the frame rate by synchronizing multiple shooting devices 10A that shoot from the same direction. In Figure 11, the shooting devices 10A to be synchronized are the first shooting device 101A, the second shooting device 102A, and the third shooting device 103A.

[0115] Figure 11(A) shows an example of a synchronization method in which the shooting timings of each imaging device 10A are shifted by a predetermined amount of time. Each imaging device 10A shifts its shooting timing by a predetermined amount of time. Specifically, the second imaging device 102A starts shooting one-third of a frame later than the first imaging device 101A. Similarly, the third imaging device 103A starts shooting one-third of a frame later than the second imaging device 102A. By shifting the shooting timing of each imaging device 10A by a time equal to the number of imaging devices 10A dividing one frame equally, the frame rate can be effectively increased without reducing the exposure time. The predetermined time for shifting the shooting timing may also be called a predetermined time difference. It is desirable that the reciprocal of the predetermined time difference is greater than the shooting frame rate.

[0116] Figure 11(B) shows an example of a synchronization method in which the shooting timing is shifted by a predetermined amount of time between each imaging device 10A when the image sensor 11A uses a line exposure sequential readout method (rolling shutter method). With the line exposure sequential readout method image sensor 11A, the scanning of pixels or lines (one or more lines that constitute the readout unit) can be intentionally shifted by applying a smaller trigger delay (a predetermined time difference). Specifically, when the sensor 11 has 9 lines, the first imaging device 101A starts line reading a time required to read out 3 lines later than the start time of line reading of the second imaging device 102A. Similarly, the second imaging device 102A starts line reading a time required to read out 3 lines later than the start time of line reading of the third imaging device 103A. By each imaging device 10A shifting the shooting timing by a time equal to the number of imaging devices 10A dividing the image sensor 11A, the frame rate can be substantially increased without reducing the exposure time. Furthermore, it is desirable that the specified time difference be shorter than the total exposure time of one frame.

[0117] Both Figures 11(A) and 11(B) above can be realized with similar control. Specifically, the control unit 15 shifts the line if the trigger is shifted for less than the total exposure time of the frame (Figure 11(B)), and shifts the frame if it is greater than or equal to the total exposure time (Figure 11(A)). According to the synchronization method shown in Figure 11, the movement of a subject moving at extremely high speed, or a subject changing at extremely high speed, over a slight time difference can be inferred from the movement of pixels and lines, and rolling shutter distortion can be corrected based on images captured from multiple viewpoints. By setting an appropriate amount of shift, it is possible to observe changes in the same subject at different times (even at the expense of the number of viewpoints of the shooting device 10A), and motion compensation is applied to estimate the movement of the subject from a different viewpoint and correct any rolling shutter distortion that may occur.

[0118] The shooting device 10A may dynamically switch between matching the trigger signal (matching the shooting timing) or intentionally shifting the shooting timing. Control commands can be incorporated into the synchronization information in order to dynamically switch the synchronization method or to share and control the amount of the shift among multiple synchronization devices 10. This makes it possible to switch between matching the shooting timing or intentionally shifting the shooting timing depending on the situation of the subject. For example, when the shooting device 10A detects that the subject is moving at high speed, it can perform fine-grained video shooting control, such as increasing the effective frame rate. In this specification, "operating in sync" may mean that the operation timings are temporally synchronized, or that the operation timings are intentionally shifted. Furthermore, "multiple synchronization devices 10 and shooting devices 10A operating in sync" may mean that they are operating in sync, or that they are synchronized in time, space, or spatiotemporal.

[0119] The trigger signal may be either positive logic or negative logic. Furthermore, the control unit 15 is not limited to outputting a pulse signal to the image sensor 11A, but may also provide some kind of control signal for exposure control. When the control unit 15 provides a control signal to the image sensor 11A, the control signal includes information indicating the image capture (exposure) timing of the image sensor 11A.

[0120] [Synchronization of shooting time] In the above, we described a method to ensure that the shooting timing does not differ among multiple imaging devices 10A by synchronizing the clocks 14 so that no discrepancies occur. However, imaging devices 10A cannot synchronize their shooting start times simply by synchronizing the clocks 14; therefore, it is necessary to share the timing (time) related to shooting among the imaging devices 10A. For example, if shooting is to start from "2020 / 1 / 1 10:00:00.000000", this will be shared among each imaging device 10A performing the shooting, allowing all devices to start shooting at the same time based on the synchronized clocks 14.

[0121] The imaging devices 10A acquire a common time axis for each imaging device 10A performing imaging through the synchronous operation described above. The control unit 15 shares information regarding synchronous imaging (synchronous operation) with other imaging devices 10A by transmitting the imaging start time to the other imaging devices 10A via the wireless communication unit 12. The control unit 15 may control the start or stop of imaging by the image sensor 11A based on the time indicated by its own clock 14 and the imaging period included in the information regarding synchronous imaging. Specifically, the control unit 15 may refer to the time on the clock 14 and start imaging by the image sensor 11A if it falls within the imaging period. Alternatively, the control unit 15 may refer to the time on the clock 14 and stop imaging by the image sensor 11A if it falls outside the imaging period.

[0122] Information regarding synchronized shooting may include, for example, information regarding shooting timing (exposure timing), such as the start time of video recording, the end time of video recording, the duration of video recording, the start time of frame recording, the end time of frame recording, and the duration of frame recording. Information regarding synchronized shooting may also include, for example, information regarding shooting parameters that are changed according to the color and brightness of the subject being photographed, such as gain, exposure time, and contrast. Furthermore, information regarding synchronized shooting may include, for example, information that is set according to the speed of movement of the subject being photographed, such as frame rate and frame shift amount (exposure shift amount). This allows for adjustment of the frame rate when the movement of a moving object becomes rapid, changes in the degree of frame shift, or adjustment of the brightness of the image for integrated analysis processing of multi-view images. Information regarding synchronized shooting may also be referred to as time-sharing information.

[0123] The wireless communication unit 12 may transmit communication information that includes synchronization information, as well as communication information that includes information related to synchronized shooting, such as the shooting start time. Alternatively, the wireless communication unit 12 may add information related to synchronized shooting to any surplus communication information that includes synchronization information that is periodically transmitted and received, and transmit it together with the synchronization information. Figure 12 shows an example of communication information that includes synchronization information and information related to synchronized shooting. By transmitting and receiving information related to synchronized shooting simultaneously with synchronization information, unnecessary radio wave emission can be avoided, reducing the waste of radio wave resources and lowering power consumption. Furthermore, the configuration of the control unit 15 and the wireless communication unit 12 can be simplified.

[0124] Unless there is a specific reason otherwise, it is desirable that the measurement start time included in the information regarding synchronized imaging be a future time a certain period after the current time, taking into account the settings of the imaging device 10A and the delay time of time sharing via wireless communication. A future time is, for example, several hundred milliseconds to several seconds from the current time.

[0125] Furthermore, the shared time is not limited to the start time of shooting, but may also be the end time of shooting, the time during shooting, or any time that indicates any frame. In the case of sharing the time that indicates any frame during shooting, a frame number or the like that identifies the frame at that time may also be shared in conjunction with the time. This allows each shooting device 10A to synchronize the number of frames to be shot by coordinating the timing of the end of shooting, or to skip frames during shooting, etc.

[0126] Furthermore, the clock 14 of the imaging device 10A uses its own time system, as described above. Unlike the synchronization of broadcast equipment using PTP, etc., the imaging device 10A, which uses its own time system, does not need to acquire time dependent on UTC (Coordinated Universal Time) or JST (Japan Standard Time). Therefore, the synchronized imaging system 1A can easily perform synchronized imaging even in indoor use where GPS or GNSS is not available, or in situations where it is difficult to prepare a PTP master clock, etc. In addition to sharing the imaging start time via the wireless communication unit 12, the synchronized imaging system 1A may also instruct the imaging start time, etc., using devices other than the imaging device 10A, such as the control terminal 20 or the distribution system 2A.

[0127] [Summary of shooting timing control] According to the embodiment described above, the sensor 11 is an image sensor 11A, and the control unit 15 controls the exposure of the image sensor 11A, thereby synchronizing the exposure timing of the image sensor 11A in time. With the synchronized shooting system 1A described above, multiple shooting devices 10A can shoot subjects at the same time with high precision in synchronization. In addition, since the video output is also synchronized, it becomes possible to switch viewpoints without frame shift using an external switcher or the like.

[0128] Furthermore, according to the embodiment described above, the control unit 15 synchronizes the exposure timing of the image sensor 11A with other synchronization devices 10 based on one of the following points: the exposure start time within the frame, the exposure end time within the frame, or the point midway between the exposure start time and the exposure end time within the frame. With the synchronized shooting system 1A described above, at the exposure start time or the exposure end time, the subject at the same point in time can be captured with high precision across multiple shooting devices. In addition, if the exposure times differ for each shooting device 10A, synchronized shooting without blur can be achieved by simultaneously shooting at the timing of the center of the exposure time.

[0129] Furthermore, according to the embodiment described above, the control unit 15 shares the exposure timing with other imaging devices 10A via the wireless communication unit 12 and controls the exposure of the image sensor 11A according to the exposure timing shared with the other imaging devices 10A. By sharing the exposure timing via wireless communication, multiple imaging devices 10A can start shooting at the same timing. In other words, by utilizing the synchronization of the clocks 14 and specifying the same exposure timing, they can all start operating simultaneously.

[0130] Furthermore, according to the embodiment described above, the communication information includes information about the shared exposure timing. By including schedule information in the time-synchronizing packets, additional radio wave transmission becomes unnecessary. This simplifies the configuration and reduces power consumption.

[0131] Furthermore, according to the embodiment described above, the control unit 15 controls the start or stop of shooting based on the time indicated by the clock 14 and the shooting period. With the shooting device 10A described above, it is possible to shoot video with the same number of frames between different shooting devices 10A. As a result, the frames of the video data match from the beginning, which simplifies subsequent video processing and analysis. Also, when distributing, since the starting frames match, there is no packet congestion of unnecessary video data before the start, and distribution can be performed with minimal network delay from the start of shooting.

[0132] Furthermore, according to the embodiment described above, the sensor 11 is an image sensor 11A, and the control unit 15 controls the exposure of the image sensor 11A, thereby creating a predetermined time difference in the exposure timing of the image sensor 11A compared to other synchronization devices 10. By slightly shifting the shooting time, the effective frame rate for shooting the same subject can be increased.

[0133] Furthermore, according to the embodiment described above, the reciprocal of the predetermined time difference is greater than the shooting frame rate. By having multiple shooting devices 10A take turns shooting with a certain delay within the frame rate, the effective frame rate for shooting the same subject can be increased.

[0134] Furthermore, according to the embodiment described above, the sensor 11 is an image sensor 11A using a line exposure sequential readout method, and the control unit 15 controls the exposure of the image sensor 11A, thereby creating a predetermined time difference in the exposure timing of the readout lines between it and other imaging devices 10A. By intentionally shifting the readout lines, if rolling shutter distortion occurs, the movement of the object can be estimated by analyzing multi-view images and distortion correction can be applied. In addition, the effective frame rate for capturing the same subject can be increased.

[0135] Furthermore, according to the embodiment described above, the synchronized shooting system 1A further comprises an image acquisition device 30, the sensor 11 is an image sensor 11A, and the multiple synchronized devices 10 (shooting devices 10A) synchronously shoot one or more subjects, the captured images are aggregated in the image acquisition device 30, and the images aggregated in the image acquisition device 30 are saved or distributed to viewers. With this, multiple shooting devices 10A can be synchronized to acquire frame-synchronized images, which can then be distributed.

[0136] [Timestamp addition] Next, we will specifically explain the process related to timestamp assignment performed by the control unit 15. Timestamp assignment means recording time information t associated with the acquired information.

[0137] The control unit 15 reads the captured frame image based on the trigger signal and buffers it as needed. The control unit 15 assigns a timestamp to the frame image obtained through the camera interface, indicating the time the frame image was received, the time it was captured (i.e., the time the trigger signal was issued), or the time the vertical synchronization signal became active. That is, the time information t assigned as a timestamp is the time on the clock 14 at one of the following points in time: when the transmission of the frame image to the control unit 15 begins, when the transmission of the frame image to the control unit 15 ends, or when the control unit 15 issues the trigger signal. The reference time for the timestamp is determined to be the same as that of the other imaging devices 10A, and is therefore set in common with the other imaging devices 10A.

[0138] The timestamp is recorded based on clock 14 and is preferably implemented in hardware, such as by a logic circuit, CPLD (Complex Programmable Logic Device), or FPGA, when a signal occurs at any of the above points in time. The timestamp may be stored or transmitted as individual data, or it may be stored in an appropriate timestamp field for the encoded video format, such as a video compression standard. For example, in the case of the H.264 format, it may be contained in the SEI Message (pic_timing) timecode format. Alternatively, when using a distribution protocol such as HLS (HTTP Live Streaming), a timestamp may be added to each segment of video data (segment file) divided into fixed time segments before distribution.

[0139] Figure 13 is a diagram illustrating an example of timestamp assignment. Figure 13(A) shows an example of assigning timestamps to all captured frames included in the video data. Assigning timestamps to all captured frames provides more reliable and accurate information about the frame capture timing. However, considering that the capture timing of the image sensor 11A is strictly controlled by a trigger, it is also possible to assign timestamps to only one or more specific frames, within the range where the capture time of each frame can be calculated backward from the frame rate.

[0140] Figure 13(B) shows an example where a timestamp is applied only to the first frame. Figure 13(C) shows an example where a timestamp is applied to frames at regular intervals. Figure 13(D) shows an example where a timestamp is applied to any frame selected from a segment file according to predetermined criteria. The timestamping methods shown in Figures 13(B) to 13(D) allow for the calculation of the shooting time by applying timestamps according to predetermined criteria.

[0141] In the timestamping method shown in Figures 13(B) to (D), a full-size timestamp may be applied only to specific frames, while other frames may only be given the timestamp difference from the frame with the full-size timestamp (for example, 33.333 ms for 30 fps). According to the timestamping method shown in Figures 13(B) to 13(D), the amount of data required for timestamping can be reduced. Note that if all frames are associated with a frame number, the time can be determined from the frame number, so it is not necessary to apply the timestamp difference.

[0142] In addition to the timestamp (time information t), the frame may also be accompanied by information indicating the reliability of the timestamp, such as information showing the confidence interval and confidence level of the timestamp, or information showing the accuracy of the synchronization between the clock 14 of another imaging device 10A and the clock 14 of the own device. Specifically, this information may be information indicating the upper and lower limits of the deviation range, such as ±0.250 [ms] or ±0.2 [%], or it may be information indicating the route variance involved in synchronization, the detected time difference, its temporal variation, or other aspects of the stability of time synchronization. This provides an indicator of how reliable the shooting timing is when analyzing multi-view images obtained by the synchronized imaging system 1A. This reliability allows for more appropriate weighting of multiple estimated parameters obtained from image analysis, thereby improving the reliability and accuracy of the final analysis results. For example, if the reliability of the timestamp is low, processing such as giving higher weight to the camera position and time estimated from the image feature points than to the timestamp attached to the frame and incorporating them into the estimation may be performed.

[0143] The control unit 15 combines and encodes the frame images as needed and transmits them to the distribution system. The video data transmission protocol can be any protocol, such as MIPI (Mobile Industry Processor Interface), CSI (Camera Serial Interface)-2, LVDS (Low Voltage Differential Signaling), SLVS-EC (Scalable Low Voltage Signaling - Embedded Clock), or CoaXPress (CXP). Video transmission to the distribution system 2A can be done by any method, whether wired or wireless.

[0144] Some of the operations performed within the control unit 15 may be performed by an external control terminal 20, or by the distribution system 2A if possible. Furthermore, in cases where video retrieval after shooting is sufficient and real-time distribution is unnecessary, the shooting device 10A or the control terminal 20 may be equipped with internal storage to save the video data. In this case as well, a timestamp may be added to the video data, or the timestamp and frame number may be linked and saved as a separate file.

[0145] The control unit 15 described above provides both the effect of synchronizing the frame shooting timing and the effect of adding a timestamp to the captured image. When shooting fast-moving or rapidly changing subjects, synchronizing the frame shooting timing is effective when precise timing matching is required. On the other hand, if frame matching can be done retrospectively, adding a timestamp is sufficient. The shooting device 10A is functional even with only synchronized shooting or timestamping. When traceability of the shooting time is important, combining both is more effective in guaranteeing the shooting time.

[0146] By keeping the time shift within the frame rate period, frame shifts of one or more frames can be avoided (for example, within approximately 8.3 ms for 120 fps). However, in cases where the subject undergoes rapid changes over time, such as high-speed movement, even a shift within a frame may result in different states of the subject being captured by each shooting device 10A. While this may not be noticeable during simple viewing, it can become a problem when analyzing video captured from multiple viewpoints for purposes such as viewpoint interpolation using Structure from Motion (SfM) or construction of three-dimensional models using feature point extraction, as each viewpoint captures the subject at different times. Therefore, a shooting device 10A that synchronizes exposure timing rather than frames is highly effective.

[0147] Furthermore, when using a line exposure sequential readout camera (rolling shutter method), the state of the subject detected by each scan line and each pixel will not match among multiple imaging devices 10A unless the scanning timing is synchronized. Therefore, when photographing extremely fast-moving subjects, it is desirable to synchronize the distortion (rolling shutter phenomenon) among multiple imaging devices 10A. Since imaging devices 10A have synchronization performance of less than a microsecond, they are very effective for photographing such subjects.

[0148] Furthermore, generally, acquiring high-resolution and high-frame-rate video requires shortening the exposure time for each pixel, thus demanding high time synchronization accuracy and high temporal resolution. The aforementioned imaging device 10A has extremely high time synchronization accuracy and high temporal resolution by intentionally shifting the shooting timing, making it possible to acquire high-resolution and high-frame-rate video relatively easily. In addition to being able to acquire high-resolution and high-frame-rate video, the imaging device 10A can precisely synchronize the frame shooting (exposure) timing due to its high time synchronization accuracy, so it can capture multi-view images captured at specific timings even for fast-moving subjects. This improves the accuracy of image processing and video processing for fast-moving subjects.

[0149] [Summary of Timestamp Addition] According to the embodiment described above, the control unit 15 records time information t associated with the acquired information obtained by the sensor 11. By attaching time information t to the acquired information, traceability of the acquisition time can be ensured. Furthermore, even if the frames delivered are shifted due to network delays or the like, the simultaneously captured video frames can be matched, and processing such as constructing a three-dimensional model using multi-view images of subjects in the same state can be performed with high accuracy.

[0150] Furthermore, according to the embodiment described above, the time information t is the time on the clock 14 at a point commonly determined with other synchronization devices 10, whichever is correct: the time when the transmission of acquired information to the control unit 15 begins, the time when the transmission of acquired information to the control unit 15 ends, or the time when the control unit 15 issues a trigger signal. This prevents the reference time of the time information t from being out of sync with other synchronization devices 10. In particular, in the case of the time when the trigger signal is issued, the time it was scheduled to be issued is known, so measurement is not required for timestamp assignment, and reliable time code assignment is possible with a simple configuration.

[0151] Furthermore, according to the embodiment described above, the sensor 11 is an image sensor 11A, and the control unit 15 adds time information t to all captured frames included in the video data captured using the image sensor 11A. By adding time information to all frames, it is possible to estimate synthesis errors caused by discrepancies in the time of capture, and image processing can be performed with high accuracy by removing images with large discrepancies.

[0152] Furthermore, according to the embodiment described above, the control unit 15 adds time information t, indicating the time the information was acquired, to some of the acquired information among the multiple acquired information acquired by the sensor 11, according to a predetermined assignment criterion. By adding time information t to some of the acquired information according to the assignment criterion, the data size required to add time information t can be reduced. In particular, when an external trigger is used, a precise frame rate can be actively obtained, thus reducing the amount of information in the timestamp.

[0153] Furthermore, according to the embodiment described above, the control unit 15 adds information indicating at least one of the accuracy or reliability of the synchronization of the clock 14, along with the time information t. By adding reliability information, traceability can be ensured in the event of fluctuations in synchronization accuracy, and it can also serve as a basis for controlling the weighting of parameters in the processing of multi-view images.

[0154] [Adding location information] Next, we will specifically explain the process related to assigning location information performed by the control unit 15. Assigning location information means recording location information linked to the acquired information (hereinafter sometimes referred to as acquired location information).

[0155] Figure 14 is a diagram illustrating an example of a position determination method. Figure 14(A) is a diagram illustrating a position determination method during the time synchronization process. As described above, the imaging device 10A can measure the radio wave propagation time and RTT during the time synchronization process, and this information can be used to determine the positional relationship (relative distance) of each imaging device 10A. In other words, similar to positioning means such as GPS, if the distance between three or more imaging devices 10A is determined, two-dimensional position determination becomes possible, and if the distance between four or more imaging devices is determined, three-dimensional position determination becomes possible. Furthermore, if the number of imaging devices 10A is insufficient, an imaging device 10A without imaging functionality, i.e., a synchronization device 10, can be prepared to provide a reference point and can be applied to position determination. The control unit 15 acquires information obtained during the time synchronization process from the synchronization unit 13. Based on the information obtained during the time synchronization process, the control unit 15 determines the relative position of its own device and adds position information indicating the determined relative position to the frame. This position information indicates the position captured by the imaging device 10A. The accuracy of distance measurement using the above phase measurement method is approximately a few millimeters, which is sufficient for use in image processing.

[0156] Figure 14(B) is a diagram illustrating a position identification means that determines location by having a positioning and distance measuring means separate from time synchronization. The imaging device 10A includes, for example, a positioning and distance measuring means 16. The positioning and distance measuring means 16 performs at least one of positioning or distance measuring. That is, the positioning and distance measuring means 16 may use positioning or distance measuring, or it may use positioning and distance measuring in combination. In Figure 14(B), the positioning and distance measuring means 16 includes a positioning means 161 and a distance measuring means 162. The positioning means 161 measures the position of the device itself and may be, for example, GPS, GNSS, or RTK (Real Time Kinematic). The distance measuring means 162 measures the distance from the device to an arbitrary position and may be, for example, UWB (Ultra-Wide Band), RSSI (Received Signal Strength Indicator) distance measuring, or laser distance measuring. The control unit 15 acquires output information (positioning information, distance measurement information, etc.) output from the positioning and distance measurement means 16. The control unit 15 adds a timestamp to the output information and records it. For example, when using GPS, a timestamp is added to the PPS (Pulse Per Second) signal obtained from the GPS module and managed in association with the positioning information contained in the NMEA (National Marine Electronics Association) data transmitted in tandem. In other words, the control unit 15 adds a timestamp that takes into account the delay of serial communication.

[0157] The timestamped output information obtained by the control unit 15 may be stored and managed separately. The control unit 15 may also compare the timestamp attached to the frame with the timestamp attached to the output information to link it with the positioning and distance measurement information of the nearest or most recent timestamp, thereby identifying (calculating) the output information at the time the frame was captured, i.e., the acquired position information. The control unit 15 adds the identified acquired position information (relative positional relationship) as metadata to each frame. This allows for the understanding of the time and location at which each frame of video obtained from multiple viewpoints using multiple shooting devices 10A was captured, improving the quality of photogrammetry and the accuracy of 3D modeling. Furthermore, adding acquired position information can achieve the same effect as adding timestamps.

[0158] The control unit 15 may share the output information from the positioning and ranging means 16 with other imaging devices 10A via the wireless communication unit 12. In this case, the control unit 15 may calculate relative position information indicating the relative positional relationship based on the output information shared from the wireless communication unit 12 of the other imaging devices 10A and the output information output from its own positioning and ranging means 16, and record this relative position information linked to the frame. In most cases, the range in which each imaging device 10A is installed is within a radius of several kilometers, so by recording relative position information instead of, or in conjunction with, the output information from the positioning and ranging means 16, the amount of positional information can be reduced. Furthermore, by recording relative position information, the relative position of each imaging device 10A required for image processing can be directly recorded in the frame.

[0159] Furthermore, the control unit 15 may also be assigned information indicating at least one of the accuracy, confidence interval, or confidence level of the positioning and ranging accuracy, similar to the case of time stamp assignment. For example, in GPS positioning information, horizontal and altitude accuracy can be obtained in the positioning result, so this may be recorded.

[0160] The relative positional relationship may be the positions of the other imaging devices 10A relative to one of the imaging devices 10A, or it may be the positions of each imaging device 10A relative to the synchronization device 10 which is used as a reference point. The relative positional information only needs to be sufficient to calculate the relative positional relationship, and may include radio wave propagation time, round-trip time (RTT), or other indicators. Based on this primary information, the relative positional relationship can also be calculated and processed.

[0161] In the above, we described an example of adding positioning information and distance measurement information using the positioning and distance measurement means 16. Generally, for multi-viewpoint video synthesis, relative positional relationships are sufficient, and a method of calculating acquired position information during the time synchronization process (Figure 14(A)) is effective. It should be noted that the above method involves adding relative position information and positioning information in a way that is linked to each frame, which differs from the generally practiced method of recording a single shooting location as metadata for the video.

[0162] Although the above description uses the imaging device 10A, this embodiment is not limited to this example, and the control of the synchronization device 10 can also be described by appropriately reinterpreting the wording. For example, the wording may be reinterpreted to be independent of imaging, such as using the frame image as acquisition information, the frame rate as the acquisition interval for acquisition information, and synchronized imaging as synchronized operation. Furthermore, the synchronization device 10 is not limited to the example of controlling the sensor 11. The synchronization device 10 may be, for example, a simple synchronization signal output device that outputs a synchronization signal to the device equipped in the synchronization device 10 or to an external device (for example, the sensor 11) of the synchronization device 10. In other words, the synchronization device 10 may be used to synchronize something other than the sensor 11, or it may be used as a reference point for synchronizing other synchronization devices 10.

[0163] The acquired position information described above may be the phase information φ itself, i.e., the propagation delay phase difference. Alternatively, the acquired position information may be information obtained by processing the phase information φ, for example, relative position information.

[0164] The information exchanged between each functional unit may be exchanged directly between the functional units, or it may be stored in a memory unit (not shown) and then referenced by the functional unit that uses that information.

[0165] [Summary of location information assignment] According to the embodiment described above, the control unit 15 records acquisition location information indicating the location where the acquisition information was obtained by the sensor 11, linked to the acquisition information obtained by the sensor 11. By adding location information indicating the location where the information was obtained to the acquisition information, it becomes possible to construct a three-dimensional model using multi-view images with high accuracy. In particular, location information is essential for constructing the generated three-dimensional model at actual size and can be used as georeferencing information.

[0166] Furthermore, according to the embodiment described above, the control unit 15 records acquisition position information indicating the location where the acquisition information was obtained by the sensor 11, linked to the acquisition information obtained by the sensor 11. The acquisition position information is phase information φ or information obtained by processing phase information φ. That is, the acquisition position information may be primary information such as propagation delay difference obtained secondarily by time synchronization, or it may be processed information such as the calculated propagation distance, i.e., the distance between the synchronization devices 10. Based on this information, the approximate positional relationship between the synchronization devices 10 can be grasped.

[0167] Furthermore, according to the embodiment described above, the control unit 15 records acquisition position information indicating the location where the acquisition information was obtained by the sensor 11, linked to the acquisition information obtained by the sensor 11. The acquisition position information is information indicating the relative position with other synchronization devices 10 calculated based on the phase information φ. If the propagation distance between three or more synchronization devices 10 is obtained, the two-dimensional and three-dimensional positional relationships can be determined, and these can be calculated and used as relative position information. In addition, since the information required for image processing is relative position rather than absolute position, it can be recorded in a state where it has been calculated in advance, simplifying image processing and analysis.

[0168] Furthermore, according to the embodiment described above, the synchronization device 10 further includes a positioning and ranging means 16 that performs positioning or distance measurement by means different from the method for identifying acquired position information. The control unit 15 adds time information t to the output information (positioning information and distance measurement information) from the positioning and ranging means 16, calculates acquired position information by comparing the time information t added to the output information with the time information t added to the acquired information, and adds the acquired position information to the acquired information. By using other positioning and ranging means, position information can be added in combination with other positioning means when the positioning means using phase is temporarily unavailable (e.g., due to radio interference). In addition, highly accurate position information can be added by sensor fusion. Furthermore, redundancy can be ensured by other positioning and ranging means, and position information can be obtained at the precise timing when the image was taken, allowing for accurate recording of positional relationships. In addition, the position of the synchronization device 10 on a map can be determined by the acquired position information based on the relative position information between the synchronization devices 10 and positioning information such as GPS. Furthermore, if there is at least one synchronization device 10 equipped with positioning and distance measuring means 16 within the synchronization system 1, the map location of each synchronization device 10 can be determined.

[0169] Furthermore, according to the embodiment described above, the control unit 15 shares output information with other synchronization devices 10 via the wireless communication unit 12, calculates relative position information based on the output information from its own positioning and ranging means 16 and the output information from the positioning and ranging means 16 of the other synchronization device 10, and records the relative position information. A synchronization device 10 that adds relative position information to acquired information can record it after replacing it with relative position coordinates used in image processing, etc., thereby reducing the amount of processing required in subsequent stages. In addition, if absolute position information is included in the packets used for time synchronization and relative position information is calculated based on that, unnecessary radio wave emission can be suppressed and power consumption can be reduced.

[0170] Furthermore, according to the embodiment described above, the control unit 15 also records information indicating at least one of the accuracy or reliability of the acquired position information. By also adding information indicating the reliability of the acquired position information, traceability can be ensured in the event of fluctuations in position detection accuracy, and it can also serve as a basis for controlling the weighting of parameters in the processing of multi-view images.

[0171] Figure 15 is a schematic diagram of an example hardware configuration of an information processing device 90 applied to this embodiment. The information processing device 90 comprises a processor 91, main memory 92, communication interface 93, auxiliary storage device 94, input / output interface 95, and internal bus 96. The processor 91, main memory 92, communication interface 93, auxiliary storage device 94, and input / output interface 95 are connected to each other via the internal bus 96 so as to be able to communicate with each other. The information processing device 90 may be applied to, for example, a synchronization device 10 and an imaging device 10A. In this case, for example, the wireless communication unit 12 may be configured using the communication interface 93. For example, the storage unit (not shown) may be configured using the auxiliary storage device 94. The control unit 15 may also be configured using the processor 91 and the main memory 92. In this embodiment, the synchronization unit 13 and the control unit 15 are configured as separate devices, but they may be configured as an integrated device.

[0172] The synchronization device 10 and the imaging device 10A may be implemented using multiple information processing devices. For example, in the synchronization device 10 and the imaging device 10A, the synchronization unit 13 and the control unit 15 may be implemented in different information processing devices. For example, the storage units of the synchronization device 10 and the imaging device 10A may be distributed and implemented in multiple information processing devices.

[0173] Furthermore, the entirety or a part thereof of the functions of each part of the synchronization device 10 and imaging device 10A in the above-described embodiment may be realized by recording a program for realizing these functions on a computer-readable recording medium, having a computer system read the program recorded on this recording medium, and executing it. The term "computer system" here includes hardware such as an operating system and peripheral devices.

[0174] Although one embodiment of this invention has been described in detail above with reference to the drawings, the specific configuration is not limited to that described above, and various design changes can be made without departing from the spirit of this invention. Furthermore, the configurations described in each embodiment and example above may be combined. [Explanation of symbols]

[0175] 1...Synchronization system, 2...External system, 10...Synchronization device, 11...Sensor, 12...Wireless communication unit, 13...Synchronization unit, 14...Clock, 141...Oscillator, 142...Real-time clock, 15...Control unit, 16...Positioning and distance measurement means, 161...Positioning means, 162...Distance measurement means, 20...Control terminal, 21...Motion generation unit, 30...Image acquisition device, 40...Light emission device, 1A...Synchronized shooting system, 10A...Shooting device, 11A...Image sensor, 2A...Distribution system

Claims

1. A synchronization device comprising a control unit, a clock, and a synchronization unit, The aforementioned clock is equipped with a self-propelled oscillator as its clock source. The synchronization unit controls its own oscillator so as to synchronize with the oscillators of other synchronization devices. A synchronization device characterized by the following features.

2. The synchronization unit controls the rising edge timing of its own oscillator to synchronize its frequency and phase with that of the oscillators of other synchronization devices. The synchronization device according to feature 1.

3. The synchronization unit controls its own oscillator so that it becomes a clock source with the same frequency and phase as other synchronization devices. The synchronization device according to feature 2.

4. The control unit, (1a) Instead of adjusting the machine's own clock based on clock correction information regarding time discrepancies with other synchronization devices, perform indirect adjustments that take into account time discrepancies with other synchronization devices, (1b) Or the synchronization unit directly adjusts the clock of the machine, It is possible to switch between any of the following synchronization methods. The synchronization device according to any one of claims 1 to 3.

5. Uses its own time system, The synchronization device according to any one of claims 1 to 3.

6. The synchronization unit is controlled to perform phase correction after frequency correction. The synchronization device according to feature 2.

7. The aforementioned synchronization unit, (2a) Perform frequency correction on the oscillator of the own machine, (2b) Perform phase correction on the oscillator of the own machine so that the rising edge timing matches that of the oscillators of other synchronization devices. (2c) After synchronization with other synchronization devices is complete, control the phase of the transmitter of the own unit so as to maintain only the phase-matched state. The synchronization device according to feature 6.

8. The unit further comprises a wireless communication unit that transmits and receives communication information by radio waves and measures the timing of the transmission and reception of the communication information. The information regarding transmission and reception timing includes time information indicating the time when the radio waves were transmitted and received, and phase information φ indicating the phase difference between the transmitted and received radio waves. The aforementioned synchronization unit, (3a) Based on the phase information with high resolution, identify "φ" from "Φ = φ + 2nπ", (3b) Based on the time information with low resolution, identify "2nπ" from "Φ = φ + 2nπ", (3c) Based on the identified phase information Φ, the oscillator of the unit is controlled to adjust for the time difference with other synchronization devices. The synchronization device according to feature 1.

Images