Imaging system, method for controlling imaging system, and program

Abstract

To provide an imaging system, a method for controlling an imaging system, and a program that can reduce the processing load in image processing.SOLUTION: An imaging system 1000 comprises a plurality of imaging apparatuses 101 that can communicate with each other. The imaging system 1000 comprises: time synchronization means (synchronization control unit 214) that synchronizes time between the imaging apparatuses 101; control means (CPU 207) that controls imaging timing in each imaging apparatus 101 on the basis of the time synchronized by the time synchronization means; image processing means (GPU 204) that performs image processing, by using a learned model, on an image picked up by each imaging apparatus 101 at the imaging timing controlled by the control means; and determination means (CPU 202) that determines the learned model used by the image processing means. The determination means determines the learned model on the basis of accuracy information on the accuracy of synchronization of time performed by the time synchronization means.SELECTED DRAWING: Figure 4A

Description

[Technical field]

[0001] The present invention relates to an imaging system, a control method for an imaging system, and a program. [Background technology]

[0002] There is known an imaging system that includes a plurality of imaging devices capable of imaging, and the imaging devices are capable of communicating with each other. In this imaging system, imaging can be performed simultaneously at a predetermined time by each imaging device while synchronizing the time of the imaging devices. The imaging system can generate a three-dimensional image by performing a synthesis process or the like for synthesizing images captured by each imaging device. However, in the imaging system, an error occurs in the time synchronized between the imaging devices due to, for example, variations in processing time in each imaging device or variations in communication time between the imaging devices. Depending on the degree of this error, a deviation occurs in the imaging timing of each imaging device. In this case, the image quality of the three-dimensional image obtained by synthesizing the images captured by each imaging device is degraded. For example, Patent Document 1 discloses an imaging device that measures a delay time in communication between the imaging devices in advance and adjusts the imaging timing based on the measured delay time. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2009-296323 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the imaging device described in Patent Document 1, for example, when the processing time is not constant due to individual differences between imaging devices, or when the delay time is not constant, such as in communication via a network, the time between the imaging devices cannot be accurately synchronized. As a result, the imaging timing cannot be synchronized, and the image quality of the three-dimensional image obtained by synthesizing the images from each imaging device may be degraded. It is preferable to improve this degradation in image quality, but there is a problem that the processing load in image processing during image synthesis increases in order to improve the image quality.

[0005] The present invention has been made in view of the above problems, and an object of the present invention is to provide an imaging system, an imaging system control method, and a program that can reduce the processing load in image processing. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, the imaging system of the present invention is an imaging system comprising a plurality of imaging devices capable of imaging, the imaging devices being capable of communicating with each other, and comprising: a time synchronization means for synchronizing the time between the imaging devices; a control means for controlling the imaging timing of each imaging device based on the time synchronized by the time synchronization means; an image processing means for performing image processing using a trained model on images captured by each imaging device at the imaging timing controlled by the control means; and a determination means for determining the trained model to be used by the image processing means, wherein the determination means determines the trained model based on accuracy information regarding the synchronization accuracy of the time by the time synchronization means. Effect of the Invention

[0007] According to the present invention, it is possible to reduce the processing load in image processing. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of the overall configuration of an imaging system according to a first embodiment. [Diagram 2] 2 is a block diagram showing an example of a hardware configuration of the imaging system shown in FIG. 1. [Diagram 3] 10 is a flowchart showing a series of processes until the imaging devices capture an image in synchronization with each other. [Figure 4A] 10 is a flowchart showing a series of processes until the server combines images from the image capture devices. [Figure 4B] FIG. 1 illustrates an example of a neural network. [Diagram 5] 4 is a flowchart showing the process (time synchronization process) in step S304 (subroutine) of the flowchart shown in FIG. [Figure 6] 6 is a sequence diagram showing a time synchronization packet transmission / reception process executed in step S502 of the flowchart shown in FIG. 5. [Figure 7] 4 is a flowchart showing the process (synchronization accuracy calculation process) in step S305 (subroutine) of the flowchart shown in FIG. [Figure 8] 13 is a flowchart showing a series of processes until a server of an imaging system according to a second embodiment combines images from each imaging device. [Figure 9] 11 is a diagram for explaining how a neural network is selected depending on the number of image capture devices having a relatively large error in synchronization accuracy. FIG. [Figure 10] 13 is a flowchart showing a series of processes until a server of an imaging system according to a third embodiment combines images from the imaging devices. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Each embodiment of the present invention will be described in detail below with reference to the drawings. However, the configurations described in each of the following embodiments are merely examples, and the scope of the present invention is not limited by the configurations described in each embodiment. For example, each part constituting the present invention can be replaced with any configuration that can exert a similar function. In addition, any configuration may be added. In addition, any two or more configurations (features) of each embodiment can be combined.

[0010] First Embodiment Hereinafter, the first embodiment will be described with reference to FIG. 1 to FIG. 7. FIG. 1 is a diagram showing an example of the overall configuration of an imaging system according to the first embodiment. As shown in FIG. 1, an imaging system 1000 has four imaging devices 101 capable of imaging, and a server 103. The imaging devices 101 are connected to each other via a communication network 102 such as a LAN or the Internet so as to be able to communicate with each other, and each imaging device 101 and the server 103 are also connected to each other via the communication network 102 so as to be able to communicate with each other. Images captured by each imaging device 101 are transmitted to the server 103. This allows the server 103 to receive images from each imaging device 101. Then, the server 103 can synthesize each image to create a three-dimensional image. Thus, in this embodiment, the server 103 has a function as an image processing means that performs processing to create a three-dimensional image. In the following, the four imaging devices 101 may be referred to as "imaging device 101a", "imaging device 101b", "imaging device 101c", and "imaging device 101d", respectively. 1, the imaging device 101 is a digital camera, but is not limited thereto and may be, for example, a tablet terminal or a smartphone having an imaging function. Also, the number of imaging devices 101 arranged in the imaging system 1000 is four in the configuration shown in FIG. 1, but is not limited thereto and may be, for example, two, three, five or more.

[0011] FIG. 2 is a block diagram showing an example of a hardware configuration of the imaging system shown in FIG. 1. As shown in FIG. 2, the imaging device 101 has a CPU (control means) 207, a ROM 208, a memory 209, a network processing unit 210, an input unit 211, a display unit 212, an imaging unit 213, and a synchronization control unit (time synchronization means) 214. These are also connected to each other via a system bus 206 so as to be able to communicate with each other. The CPU 207 controls the operation of the ROM 208, the memory 209, the network processing unit 210, the input unit 211, the display unit 212, the imaging unit 213, and the synchronization control unit 214. The ROM 208 stores various programs and the like. The programs are not particularly limited, and include, for example, programs for causing the CPU 207 (computer) to execute each unit and each unit (a control method of the imaging system) of the imaging system 1000. The various programs are not limited to being stored in the ROM 208, and may be stored, for example, in a hard disk or the like. The memory 209 is, for example, configured with a RAM. When executing a program stored in the ROM 208, the CPU 207 uses the memory 209 as a work memory. The input unit 211 can receive an input operation from a user who uses the imaging device 101. The input unit 211 is not particularly limited, and can be configured, for example, with a mechanical operation button or a touch panel. The input unit 211 can also generate a control signal according to the input operation and transmit it to the CPU 207. This allows the CPU 207 to control the network processing unit 210, the display unit 212, the imaging unit 213, and the synchronization control unit 214 according to a program started by a control signal from the input unit 211.

[0012] The display unit 212 is composed of, for example, a display, and outputs a display signal for displaying various images such as an image captured by the imaging unit 213. When a touch panel is used as the input unit 211, the input unit 211 and the display (display unit 212) can be configured integrally. In this case, for example, a touch panel having a light transmittance that allows the image displayed by the display to be sufficiently visible is attached to the surface of the display. Then, the input coordinates for the touch panel and the display coordinates on the display are associated with each other, so that the input unit 211 and the display can be configured integrally. The imaging unit 213 captures an image, and includes, for example, a lens, a shutter having an aperture function, an imaging element that converts an optical image into an electrical signal, and an image processing unit that performs various image processes such as exposure control and distance measurement control by the electrical signal from the imaging element. The imaging element is not particularly limited, and examples thereof include a CCD and a CMOS element. An imaging instruction to the imaging unit 213 can be given by the input unit 211. The network processor 210 communicates with the server 103 via the communication network 102 .

[0013] In the imaging system 1000, one of the imaging devices 101a to 101d is a primary device, i.e., a main imaging device, and the remaining imaging devices 101 are secondary devices, i.e., slave imaging devices. The "primary device" is a device that manages the time that serves as a reference when synchronizing the times of the imaging devices 101a to 101d. The "secondary device" is a device that synchronizes the time with the time of the primary device. By synchronizing the time of the secondary device with the time of the primary device, the times of the imaging devices 101a to 101d can be synchronized. For example, in this embodiment, the imaging device 101a is a primary device, and the imaging devices 101b to 101d are secondary devices. Note that in the imaging system 1000, an imaging device serving as a primary device may be arranged in addition to the imaging devices 101a to 101d, and the imaging devices 101a to 101d may be secondary devices. The synchronization control unit 214 synchronizes the times of the imaging devices 101b to 101d, which are secondary devices, with the time of the imaging device 101a, which is the primary device, thereby synchronizing the times of the imaging devices 101 (time synchronization step).

[0014] The CPU 207 can control the imaging timing of each imaging device 101 based on the time synchronized by the synchronization control unit 214 (control process). The synchronization control unit 214 is composed of a clock circuit 214a and a synchronization signal output circuit 214b. The clock circuit 214a is a circuit for synchronizing the time with other imaging devices 101 via the network processing unit 210. The synchronization signal output circuit 214b is a circuit for outputting a synchronization pulse signal (synchronization signal) from the clock circuit 214a to the imaging unit 213 to adjust the imaging timing. The clock circuit 214a is a hardware counter that counts up at a predetermined timing to manage the time. The clock circuit 214a also has a function of correcting the counter value in order to synchronize the time with other imaging devices 101 via the network processing unit 210. The synchronization signal output circuit 214b outputs the synchronization pulse signal from the clock circuit 214a to the imaging unit 213. The imaging timing of the imaging unit 213 is controlled by using this synchronization signal as a trigger.

[0015] As shown in FIG. 2, the server 103 has a CPU 202, a memory 203, a GPU (Graphics Processing Unit) 204, a server communication unit 205, and a ROM (storage means) 215. These are also connected to each other via a system bus 201 so as to be able to communicate with each other. The CPU 202 controls the operation of the memory 203, the GPU 204, the server communication unit 205, and the ROM 215. The ROM 215 stores various programs and the like. The programs are not particularly limited, and examples thereof include programs for causing the CPU 202 (computer) to execute each unit and each unit (a control method of the imaging system) of the imaging system 1000. The various programs are not limited to being stored in the ROM 215, and may be stored, for example, in a hard disk or the like.

[0016] The server 103 receives the images captured by each imaging device 101 at the above-mentioned imaging timing. The server 103 then performs image processing on the images from each imaging device 101 (image processing step). The image processing in this embodiment is a process of performing a synthesis process for synthesizing each image to generate a three-dimensional image, but is not limited to this. The memory 203 is, for example, configured with a RAM, and is used as a work memory when the CPU 202 and the GPU 204 operate. The server communication unit 205 communicates with the imaging devices 101a to 101d via the communication network 102. In this embodiment, the CPU 202 accepts a communication request from the imaging devices 101a to 101d, generates a control signal corresponding to the communication request, and operates the GPU 204. The GPU 204 is a computing device capable of performing processing specialized for computer graphic calculations. The GPU 204 can generally perform calculations required for the neural network 410 (see FIG. 4B), such as matrix calculations, in a shorter time than the CPU 202. The GPU 204 can also handle the arithmetic processing together with the CPU 202. The server 103 may also have, for example, a TPU (Tensor Processing Unit) instead of the GPU 204. The server 103 may also have an NPU (Neural network Processing Unit), and the NPU may handle the arithmetic processing together with the CPU 202.

[0017] FIG. 3 is a flowchart showing a series of processes until the imaging devices capture images in synchronization with each other. In this embodiment, the program based on the flowchart shown in FIG. 3 is started by the user operating one of the imaging devices 101a to 101d, but is not limited to this. For example, the program start may be instructed by remote control. As shown in FIG. 3, in step S300, the CPU 207 accepts the setting of the primary device and the secondary device. As described above, in this embodiment, of the imaging devices 101a to 101d that are communicably connected to each other, the imaging device 101a is set as the primary device, and the remaining imaging devices 101, that is, the imaging devices 101b to 101d are set as the secondary devices. This setting is possible, for example, by operating a setting screen (setting means) on a touch panel in which the input unit 211 and the display unit 212 of the imaging device 101a are integrally configured. The CPU 207 can then accept setting information on the setting screen, that is, information that sets the imaging device 101a as the primary device and the imaging devices 101b to 101d as the secondary devices. This information is then shared among the imaging devices 101a to 101d.

[0018] In step S301, when the CPU 207 of the imaging device 101a, which is the primary device, determines that the input unit 211 has received an instruction to start synchronous imaging among the imaging devices 101a to 101d, the CPU 207 constructs a network among the imaging devices 101a to 101d. The "network construction" refers to communication processing performed among the imaging devices 101a to 101d to perform synchronous imaging. Specifically, the CPU 207 of the imaging device 101a controls the network processing unit 210 to communicate with the imaging devices 101b to 101d connected to the communication network 102. At this time, the CPU 207 of the imaging device 101a transmits a packet for inquiring about the presence of the imaging device 101 capable of synchronous imaging to the communication network 102, and waits for a reply packet. Then, the CPU 207 of the imaging device 101a identifies the imaging device 101 that will participate in the synchronous imaging based on the reply packet. In this embodiment, the identified imaging device 101 becomes the imaging device 101b to imaging device 101d, which are secondary devices.

[0019] In step S302, the CPU 207 of the imaging device 101a performs live view imaging using the imaging unit 213, and also displays an image obtained by the live view imaging on the display unit 212. It is also preferable that the imaging devices 101b to 101d also perform live view display following the imaging device 101a.

[0020] In step S303, the CPU 207 of the imaging device 101a identifies a main subject in the image displayed in live view. In this embodiment, the subject is a moving subject. The subject is identified, for example, by the user operating the input unit 211 to specify a subject to be specified as a main subject in the image displayed in live view, and the CPU 207 accepts the specification. The CPU 207 of the imaging device 101a then notifies the imaging devices 101b to 101d of subject information related to the specified subject. This allows the subject information to be shared among the imaging devices 101a to 101d.

[0021] In step S304, the CPU 207 of the image capture device 101a executes a time synchronization process for synchronizing the times among the image capture devices 101a to 101d. Details of the time synchronization process in step S304 will be described later with reference to FIG.

[0022] In step S305, the CPU 207 of the imaging device 101a executes a synchronization accuracy calculation process to calculate the synchronization accuracy (amount of time discrepancy) of the times synchronized in step S304. In this manner, in this embodiment, the CPU 207 has a function as an accuracy detection means for calculating, that is, detecting, the synchronization accuracy. The CPU 207 of the imaging device 101a also controls the network processing unit 210 to transmit the result of the synchronization accuracy calculation process, that is, synchronization accuracy information related to the synchronization accuracy, to the server 103. Details of the synchronization accuracy calculation process in step S305 will be described later with reference to FIG. 7.

[0023] In step S306, when the CPU 207 of the imaging device 101a determines that the input unit 211 has received an imaging instruction from the user, the CPU 207 controls the synchronization control unit 214 to prepare for imaging with the imaging start time set. In addition to this imaging preparation, the CPU 207 of the imaging device 101a controls the network processing unit 210 to notify the imaging devices 101b to 101d of information on imaging timing. Then, when the imaging start time arrives, the CPU 207 of the imaging device 101a outputs an imaging synchronization signal to the imaging unit 213 to start imaging in the imaging unit 213. The imaging devices 101b to 101d also start imaging following the imaging device 101a. After the start of imaging, when the CPU 207 of the imaging device 101a determines that the input unit 211 has received an imaging end instruction from the user, the CPU 207 controls the imaging unit 213 and the synchronization control unit 214 to end imaging. Furthermore, the imaging devices 101b to 101d also finish imaging following the imaging device 101a.

[0024] In step S307, the CPU 207 of the imaging device 101a calculates the moving speed of the subject identified in step S303 in the image captured in step S306. Specifically, the CPU 207 of the imaging device 101a determines that the object on which the imaging unit 213 is focused in the image captured in step S306 is the subject. Then, the CPU 207 calculates the moving speed of the subject based on difference information between frames in the captured image. Thus, in this embodiment, the CPU 207 functions as a speed calculation means for calculating the moving speed of the subject, that is, a speed detection means for detecting the moving speed of the subject.

[0025] In step S308, the CPU 207 of the imaging device 101a adds moving speed information regarding the moving speed of the subject calculated in step S307 to the captured image stored in the memory 209. Then, the CPU 207 of the imaging device 101a controls the network processing unit 210 to transmit the captured image with the moving speed information added to the server 103. The imaging devices 101b to 101d also transmit their captured images to the server 103 following the imaging device 101a. Note that steps S301 to S308 are preferably repeated.

[0026] The server 103 can receive captured images (synchronous captured images) from the imaging devices 101a to 101d. The server 103 performs image processing on these captured images to generate three-dimensional images. Note that, although the image processing on the captured images is executed by the server 103 in this embodiment, the present invention is not limited to this, and may be executed by, for example, one of the imaging devices 101a to 101d. For example, when the image processing is executed by the imaging device 101a of the imaging devices 101a to 101d, the captured images of the imaging devices 101b to 101d are transmitted to the imaging device 101a.

[0027] Fig. 4A is a flow chart showing a series of processes until the server synthesizes images from each imaging device. Fig. 4B is a diagram showing an example of a neural network. As shown in Fig. 4A, in step S401, the CPU 202 of the server 103 judges whether or not the synchronization accuracy information transmitted in step S305 has been received by the server communication unit 205. If the CPU 202 judges that the synchronization accuracy information has been received as a result of the judgment in step S401, the process proceeds to step S402. On the other hand, if the CPU 202 judges that the synchronization accuracy information has not been received as a result of the judgment in step S401, the process remains in step S401 and waits.

[0028] In step S402, the CPU 202 determines a trained model to be used in step S404 (image processing) based on the synchronization accuracy information received in step S401 (determination step). Thus, in this embodiment, the CPU 202 has a function as a determination means for determining a trained model. This makes it possible to omit providing a part having the function as a determination means separately from the CPU 202, and therefore the configuration of the server 103 can be simplified. Note that in the server 103, a part having the function as a determination means may be provided separately from the CPU 202. The trained model includes, for example, a neural network 410 shown in FIG. 4B. The neural network 410 has an input layer 411, an intermediate layer 412, and an output layer 413.

[0029] By the way, a state where the synchronization accuracy is relatively low, that is, the amount of time difference is relatively large, is a state where the image capturing timing is shifted among the image capturing devices 101a to 101d. In this state where the image capturing timing is shifted, if an object moving at high speed is captured by each of the image capturing devices 101a to 101d, the object will be captured in a state where the position is shifted more than desired. If all captured images in a state where the object's position is shifted are synthesized to generate a three-dimensional image, the three-dimensional image may be blurred. Therefore, in order to prevent or suppress blurring in this three-dimensional image, image correction using the neural network 410, that is, blur correction, is performed when performing image processing in step S404.

[0030] A plurality of neural networks 410 are stored in the ROM 215 of the server 103. These neural networks 410 are different from each other in at least one of the number of layers of the layer composed of the input layer 411, the intermediate layer 412, and the output layer 413 and the total number of the input layer 411, the intermediate layer 412, and the output layer 413. Then, the CPU 202 selects one neural network 410 from the plurality of neural networks 410 according to the amount of blur correction. Specifically, the CPU 202 selects a neural network 410 having a larger number of layers and a larger total number of layers (quantity) and a longer processing time as the amount of blur correction is larger. Also, the CPU 202 selects a neural network 410 having a smaller number of layers and a smaller total number of layers and a shorter processing time as the amount of blur correction is smaller. Also, the CPU 202 can select the neural network 410 used in step S404 based on the synchronization accuracy information and the moving speed information of the subject.

[0031] In step S403, the CPU 202 determines whether or not the captured images transmitted in step S308 from the imaging devices 101a to 101d have been received by the server communication unit 205. If the CPU 202 determines in step S403 that all the captured images have been received, the process proceeds to step S404. On the other hand, if the CPU 202 determines in step S403 that all the captured images have not been received, the process remains in step S403 and waits.

[0032] In step S404, the GPU (image processing means) 204 performs image processing on all the captured images received in step S403 using the neural network 410 determined in step S402, that is, synthesizes all the captured images to generate a three-dimensional image. As a result, the three-dimensional image becomes an image with blur corrected. Specifically, all the captured images received in step S403 are input to the input layer 411 of the neural network 410. Then, in the neural network 410, image processing such as synthesis processing and blur correction is performed on all the captured images in the intermediate layer 412, and a three-dimensional image with blur prevention is output from the output layer 413. Note that steps S401 to S404 are preferably repeated.

[0033] As described above, in the imaging system 1000, one neural network 410 suitable for the correction can be selected according to the amount of blur correction. This can reduce the processing load in the image processing in the neural network 410. In addition, the three-dimensional image (synthetic image) generated from the synchronous imaging has reduced image quality degradation due to blur. The learning in the trained model is supervised learning. As teacher data in this supervised learning, an input image when training the trained model and an output image obtained by processing the input image and in which blurring has been prevented are prepared. In the trained model, when various input images are input, a large number of learnings are performed to output a target output image. In addition, as a machine learning algorithm, deep learning is preferable in this embodiment, but is not limited thereto, and may be, for example, a support vector machine, a logistic regression, a decision tree, or the like.

[0034] FIG. 5 is a flowchart showing the process (time synchronization process) in step S304 (subroutine) of the flowchart shown in FIG. 3. As shown in FIG. 5, in step S501, the CPU 207 of the imaging device 101 judges whether the imaging device 101 itself is the primary device. If the CPU 207 judges in step S501 that the imaging device 101 itself is the primary device, the process proceeds to step S502. On the other hand, if the CPU 207 judges in step S501 that the imaging device 101 itself is not the primary device, the process proceeds to step S506. As described above, in this embodiment, the imaging device 101a is the primary device, and the imaging devices 101b to 101d are the secondary devices. In this case, the CPU 207 of the imaging device 101a as the primary device judges in step S501 that the imaging device 101 itself is the primary device, and the process proceeds to step S502. Furthermore, the CPU 207 of each of the image capture devices 101b to 101d acting as a secondary device determines in step S501 that the image capture device 101 itself is not the primary device, and the process proceeds to step S506.

[0035] In step S502, the CPU 207 of the imaging device 101a transmits and receives a time synchronization packet as information related to time synchronization to and from the imaging devices 101b to 101d via the network processing unit 210. Details of the transmission and reception of the time synchronization packet will be described later with reference to FIG.

[0036] In step S503, the CPU 207 of the imaging device 101a determines whether or not a notification of time synchronization completion has been received from the imaging devices 101b to 101d via the network processing unit 210. If the CPU 207 of the imaging device 101a determines in step S503 that a notification of time synchronization completion has been received, the process proceeds to step S504. On the other hand, if the CPU 207 of the imaging device 101a determines in step S503 that a notification of time synchronization completion has not been received, the process remains in step S503 and waits.

[0037] In step S504, the CPU 207 of the imaging device 101a transmits a notification to the imaging devices 101b to 101d via the network processing unit 210 that time synchronization has been completed in the imaging system 1000, that is, in the imaging devices 101a to 101d.

[0038] In step S505, the CPU 207 of the image capturing apparatus 101a continues to perform processing for periodically continuing communication for time synchronization with the image capturing apparatuses 101b to 101d.

[0039] In step S506, the CPUs 207 of the imaging devices 101b to 101d acting as secondary devices each determine whether or not a time synchronization packet has been received from the imaging device 101a via the network processing unit 210. If the result of the determination in step S506 indicates that the CPUs 207 of the imaging devices 101b to 101d have each received a time synchronization packet, the process proceeds to step S507. On the other hand, if the result of the determination in step S506 indicates that the CPUs 207 of the imaging devices 101b to 101d have each not received a time synchronization packet, the process remains in step S506 and waits.

[0040] In step S507, the CPU 207 of each of the image pickup devices 101b to 101d controls the synchronization control unit 214 based on the time synchronization packet received in step S506 to correct the time of the clock circuit 214a.

[0041] In step S508, the CPU 207 of the imaging devices 101b to 101d continues to correct the time of the clock circuit 214a based on the communication for time synchronization periodically transmitted from the imaging device 101a. Then, the CPU 207 of the imaging devices 101b to 101d judges whether or not synchronization is complete. This judgment is made based on whether or not the width of the correction value at the predetermined time falls within a certain range. Then, if the width of the correction value at the predetermined time falls within the certain range, the CPU 207 of the imaging devices 101b to 101d judges that synchronization is complete, and the process proceeds to step S509. On the other hand, if the width of the correction value at the predetermined time does not fall within the certain range, the CPU 207 does not judge that synchronization is complete, and the process remains in step S508 and waits.

[0042] In step S509, the CPUs 207 of the image capturing devices 101b to 101d transmit a notification of synchronization completion to the image capturing device 101a via the network processor 210. Note that steps S501 to S509 are preferably repeated.

[0043] FIG. 6 is a sequence diagram showing the time synchronization packet transmission / reception process executed in step S502 of the flowchart shown in FIG. 5. In the imaging system 1000, the time synchronization packet transmission / reception process estimates the delay of the network path in the imaging system 1000, and based on the delay amount, the times of the imaging devices 101b to 101d can be synchronized with the time of the imaging device 101a. The method used in the time synchronization packet transmission / reception process is the TWO-STEP method in this embodiment, but is not limited thereto, and may be, for example, the ONE-STEP method. As shown in FIG. 6, in step S601, the CPU 207 of the imaging device 101a, which is the primary device, transmits a SYNC packet to the imaging devices 101b to 101d, which are secondary devices, via the network processing unit 210. The SYNC packet includes information that the synchronization method is the TWO-STEP method. The CPU 207 of each of the imaging devices 101b to 101d receives the SYNC packet from the imaging device 101a via the network processor 210. Then, the CPU 207 of each of the imaging devices 101b to 101d stores the reception time of the SYNC packet.

[0044] In step S602, the CPU 207 of the imaging device 101a transmits a FOLLOW-UP packet to the imaging devices 101b to 101d via the network processing unit 210. The FOLLOW-UP packet includes the transmission time of the SYNC packet transmitted in step S601. The CPU 207 of the imaging devices 101b to 101d receives the FOLLOW-UP packet from the imaging device 101a via the network processing unit 210. The CPU 207 of the imaging devices 101b to 101d calculates the delay time on the communication path from the primary device to the secondary device based on the difference between the SYNC packet transmission time included in the FOLLOW-UP packet and the reception time of the SYNC packet.

[0045] In step S603, the CPUs 207 of the imaging devices 101b to 101d transmit Delay_req packets to the imaging device 101a via the network processing unit 210. At this time, the CPUs 207 of the imaging devices 101b to 101d store the transmission times of the Delay_req packets. The CPU 207 of the imaging device 101a receives the Delay_req packets from the imaging devices 101b to 101d via the network processing unit 210. Then, the CPU 207 of the imaging device 101a stores the reception times at which the Delay_req packets are received.

[0046] In step S604, the CPU 207 of the imaging device 101a transmits a Delay_resp packet to the imaging devices 101b to 101d via the network processing unit 210. The Delay_resp packet contains information about the reception time when the imaging device 101a received the Delay_req packet. The CPU 207 of the imaging devices 101b to 101d receives the Delay_resp packet from the imaging device 101a via the network processing unit 210. Then, the CPU 207 of the imaging devices 101b to 101d calculates the delay time on the communication path from the secondary device to the primary device based on the difference between the transmission time of the Delay_req packet and the reception time of the Delay_req packet.

[0047] By periodically transmitting and receiving packets as described above, the imaging devices 101b to 101d, which are secondary devices, can periodically perform correction to match the time of the imaging device 101a, which is the primary device, thereby enabling periodic time synchronization.

[0048] FIG. 7 is a flowchart showing the process (synchronization accuracy calculation process) in step S305 (subroutine) of the flowchart shown in FIG. 3. As shown in FIG. 7, in step S701, the CPU 207 of each of the imaging devices 101b to 101d, which are secondary devices, judges whether or not a time synchronization packet has been received from the imaging device 101a, which is the primary device, via the network processing unit 210. The "time synchronization packet" refers to two packets, a Sync packet (see step S601 in FIG. 6) and a Follow-Up packet (see step S602 in FIG. 6). If the CPU 207 of each of the imaging devices 101b to 101d judges in step S701 that the time synchronization packet has been received, the process proceeds to step S702. On the other hand, if the CPU 207 of each of the imaging devices 101b to 101d judges in step S701 that the time synchronization packet has not been received, the process remains in step S701 and waits.

[0049] In step S702, the CPU 207 of each of the imaging devices 101b to 101d calculates a delay value of the reception path based on the Sync packet and the Follow-Up packet received in step S701. This delay value is calculated by comparing the reception time of the Sync packet with the transmission time included in the Follow-Up packet, and is the difference from the time of the imaging device 101a.

[0050] In step S703, the CPU 207 of each of the imaging devices 101b to 101d transmits a Delay_req packet (see step S603 in FIG. 6) to the imaging device 101a via the network processing unit 210. The Delay_req packet includes time information of each of the imaging devices 101b to 101d at the time of transmitting the packet.

[0051] In step S704, the CPUs 207 of the imaging devices 101b to 101d determine whether or not a Delay_resp packet (see step S604 in FIG. 6) has been received from the imaging device 101a via the network processing unit 210. If the result of the determination in step S704 indicates that the CPUs 207 of the imaging devices 101b to 101d have received a Delay_resp packet, the process proceeds to step S705. On the other hand, if the result of the determination in step S704 indicates that the CPUs 207 of the imaging devices 101b to 101d have not received a Delay_resp packet, the process remains in step S704 and waits.

[0052] In step S705, the CPU 207 of each of the image pickup devices 101b to 101d calculates a delay value of the transmission path based on the transmission time of the Delay_req packet in step S703 and the Delay_resp packet received in step S704.

[0053] Next, the CPU 207 of the imaging devices 101b to 101d calculates the path delay of the transmission and reception paths based on the delay value of the transmission path and the delay value of the reception path in step S702. Then, the CPU 207 of the imaging devices 101b to 101d corrects the time of the imaging devices 101b to 101d to match the time of the imaging device 101a based on the calculation result value, and records it. In addition, the CPU 207 of the imaging devices 101b to 101d measures the number of hops of the network path of the time synchronization packet.

[0054] In step S706, the CPUs 207 of the imaging devices 101b to 101d determine whether the mode in which the time error is calculated from the time of the imaging device 101a is the first mode or the second mode. The first mode is a mode in which the time error is calculated based on the amount of correction when the time is corrected. The second mode is a mode in which the time error is calculated based on the number of hops in the network path when transmitting and receiving a time synchronization packet. If the CPUs 207 of the imaging devices 101b to 101d determine that the first mode is selected in step S706, the process proceeds to step S707. On the other hand, if the CPUs 207 of the imaging devices 101b to 101d determine that the second mode is selected in step S706, the process proceeds to step S708.

[0055] In step S707, the CPU 207 of each of the imaging devices 101b to 101d calculates the time error range between the imaging device 101a and the imaging device 101b based on the history of the time correction information acquired by receiving the time synchronization packet. Specifically, the CPU 207 of each of the imaging devices 101b to 101d calculates the time error range using the maximum correction amount of time adjustment between the imaging device 101a and the imaging device 101b to 101d. This makes it possible to calculate a quantitative value indicating the degree of time error between the imaging device 101a and the imaging device 101b. Then, based on this calculated value, the server 103 can determine the neural network 410 to be used for image correction. After step S707 is executed, the process proceeds to step S709.

[0056] In step S708, the CPU 207 of each of the imaging devices 101b to 101d calculates the time error range between the imaging device 101b and the imaging device 101a based on the number of hops of the network path acquired by receiving the time synchronization packet. This makes it possible to calculate the delay error of the network path. Then, based on this calculated value, the neural network 410 can be determined. After executing step S708, the process proceeds to step S709.

[0057] In step S709, the CPU 207 of the imaging device 101a calculates the synchronization accuracy based on the time error range calculated in step S707 or step S708. Next, the CPU 207 of the imaging device 101a transmits synchronization accuracy information related to the synchronization accuracy to the server 103 via the network processing unit 210. This allows the server 103 to determine the neural network 410 based on the synchronization accuracy information, as described above. Note that steps S701 to S709 are preferably repeated. As described above, in this embodiment, the CPU 207 can calculate the synchronization accuracy based on the communication speed when the imaging devices 101 communicate with each other.

[0058] <Second embodiment> Hereinafter, the second embodiment will be described with reference to FIG. 8 and FIG. 9, but the differences from the above-mentioned embodiment will be mainly described, and the description of the similar matters will be omitted. In this embodiment, a configuration for performing 3D synthesis according to the number of allowable errors of the imaging device 101 will be described. FIG. 8 is a flowchart showing a series of processes until the server of the imaging system according to the second embodiment synthesizes images from each imaging device. As shown in FIG. 8, in step S801, similar to step S401, the CPU 202 of the server 103 judges whether or not the server communication unit 205 has received the synchronization accuracy information transmitted in step S305. If the CPU 202 judges that the synchronization accuracy information has been received as a result of the judgment in step S801, the process proceeds to step S802. On the other hand, if the CPU 202 judges that the synchronization accuracy information has not been received as a result of the judgment in step S801, the process waits in step S801.

[0059] In step S802, the CPU 202 judges whether the synchronization accuracy included in each piece of synchronization accuracy information received in step S801 is within an allowable range. Thus, in this embodiment, the CPU 202 has a function as a judgment means for judging whether the synchronization accuracy is within an allowable range. Here, "the synchronization accuracy is within an allowable range" means that the neural network 410 determined based on the synchronization accuracy can be determined to be a neural network 410 having a small number of layers or total number of layers and a short processing time. In contrast, "the synchronization accuracy is not within an allowable range" means that the neural network 410 determined based on the synchronization accuracy is preferably determined to be a neural network 410 having a large number of layers or total number of layers and a long processing time. Hereinafter, the neural network 410 having a small number of layers or total number of layers is referred to as "neural network A," and the neural network 410 having a large number of layers or total number of layers is referred to as "neural network B." Then, if the CPU 202 judges that the synchronization accuracy is within an allowable range as a result of the judgment in step S802, the process proceeds to step S803. On the other hand, if the CPU 202 determines in step S802 that the synchronization accuracy is not within the permissible range, the process proceeds to step S804.

[0060] In step S803, CPU 202 determines neural network A as the neural network 410 to be used in step S810.

[0061] In step S804, the CPU 202 acquires the number of image capture devices 101 whose synchronization accuracy has been determined to be outside the allowable range in step S802, and the process proceeds to step S805.

[0062] In step S805, the CPU 202 judges whether the number of the imaging devices 101 acquired in step S804 is equal to or less than the threshold value N. A method for determining the threshold value N will be described. As described above, the neural network 410 includes the neural network A and the neural network B. The difference in the amount of calculation between the neural network A and the neural network B is large. For example, when the neural network A is used as the neural network 410 for correcting the blur of a three-dimensional image after performing a time correction process according to the amount of time shift for one image based on, for example, vector information, the calculation time may be short. Here, the calculation time in the neural network A is X1, and the calculation time in the neural network B is X2. Also, the time required for the time correction process according to the amount of time shift for one image is X3. The calculation time X1, the calculation time X2, the required time X3, and the threshold value N satisfy the following formula (1). X1+X3×N <X2···(1) The threshold value N can be derived from equation (1).

[0063] If the CPU 202 determines in step S805 that the number of image capture devices 101 is equal to or less than the threshold value N, the process proceeds to step S806. On the other hand, if the CPU 202 determines in step S805 that the number of image capture devices 101 is not equal to or less than the threshold value N, the process proceeds to step S807.

[0064] In step S806, the CPU 202 determines to perform time correction processing according to the amount of time difference for the images acquired from the image capture device 101 for which the synchronization accuracy was determined not to be within the allowable range in step S802, and the process proceeds to step S803. The time correction processing can be performed by generating a pseudo image corresponding to the frame interval based on the vector information. Note that a separate neural network may also be used for this processing.

[0065] In step S807, CPU 202 determines neural network B as neural network 410, and the process proceeds to step S808.

[0066] In step S808, the CPU 202 determines whether or not the captured images transmitted in step S308 from the imaging devices 101a to 101d have been received by the server communication unit 205. If the CPU 202 determines in step S808 that all the captured images have been received, the process proceeds to step S809. On the other hand, if the CPU 202 determines in step S808 that all the captured images have not been received, the process remains in step S808 and waits.

[0067] In step S809, the CPU 202 executes a time correction process, and the process proceeds to step S810. Note that the process in step S809 is executed according to the amount of time deviation for the image of the imaging device 101 for which the synchronization accuracy determined in step S806 is not within the allowable range.

[0068] In step S810, the GPU 204 performs image processing on all the captured images received in step S808 using the neural network 410 determined in step S803 or step S807. That is, all the captured images are synthesized to generate a three-dimensional image. As a result, a three-dimensional image is generated by synthesizing all the captured images. The three-dimensional image is an image in which blur has been corrected. Note that steps S801 to S810 are preferably repeated.

[0069] FIG. 9 is a diagram for explaining how a neural network is selected depending on the number of imaging devices having a relatively large error in synchronization accuracy. Here, as an example, the threshold value for the number of imaging devices 101 whose synchronization accuracy is determined not to be within the allowable range is "3". FIG. 9(a) is a diagram showing a case where the actual number of imaging devices 101 whose synchronization accuracy is determined not to be within the allowable range is "2". In FIG. 9(a), the actual number of imaging devices 101 is smaller than the threshold value for the number of imaging devices 101. FIG. 9(b) is a diagram showing a case where the actual number of imaging devices 101 whose synchronization accuracy is determined not to be within the allowable range is "3". In FIG. 9(b), the actual number of imaging devices 101 is the same as the threshold value for the number of imaging devices 101 (equal to or greater than the threshold value for the number of imaging devices 101).

[0070] In FIG. 9(a), it is assumed that the imaging devices 101a and 101c have relatively large errors in synchronization accuracy. In this case, the imaging device 101a is subjected to a time correction process to create a pseudo image A, and the imaging device 101c is subjected to a time correction process to create a pseudo image B. Then, the neural network A is used as the neural network 410 for the pseudo image A of the imaging device 101a, the image of the imaging device 101b, the pseudo image B of the imaging device 101c, and the image of the imaging device 101d. As a result, a three-dimensional image Y in which image quality degradation due to blurring is reduced is obtained. In FIG. 9(b), the neural network B is used as the neural network 410 for the images of the imaging devices 101a to 101d. As a result, a three-dimensional image Y in which image quality degradation due to blurring is reduced is obtained. As described above, a neural network can be appropriately selected according to the number of imaging devices with relatively large errors in synchronization accuracy. As a result, the processing load in image processing can be reduced. Furthermore, the three-dimensional image is an image in which degradation of image quality due to blurring is reduced.

[0071] <Third embodiment> Hereinafter, the third embodiment will be described with reference to FIG. 10. The differences from the above-mentioned embodiments will be mainly described, and the description of the similar matters will be omitted. For example, when there is no limit to the processing time of the three-dimensional image formation in the server 103, it is preferable to use the processing described in the first and second embodiments. On the other hand, when it is desired to view a three-dimensional image in real time, it is preferable to set a limit to the processing time of the three-dimensional image formation and perform the three-dimensional image formation within the processing time. Here, as the number of the imaging devices 101 increases, the neural network B tends to be selected as the neural network 410. However, depending on the number of the imaging devices 101, it may be difficult to use the neural network B. Therefore, in this embodiment, a configuration in which the neural network 410 is selected according to the available processing time of the CPU 202 of the server 103 will be described.

[0072] Fig. 10 is a flowchart showing a series of processes until the server of the imaging system according to the third embodiment combines images from each imaging device. First, the process in the server 103 will be described. As shown in Fig. 10, in step S1001, the CPU 202 of the server 103 issues an instruction to the imaging devices 101a to 101d to execute a synchronization process 1021, and the process proceeds to step S1002. Note that the synchronization process 1021 has been described with reference to Fig. 6, and will not be described here.

[0073] In step S1002, CPU 202 determines whether or not the synchronization accuracy information transmitted in step S305 has been received by server communication unit 205. If CPU 202 determines in step S1002 that the synchronization accuracy information has been received, the process proceeds to step S1004. On the other hand, if CPU 202 determines in step S1002 that the synchronization accuracy information has not been received, the process remains in step S1002 and waits.

[0074] In step S1004, the CPU 202 acquires the processing time allowed to be consumed during three-dimensional image formation (image synthesis), and the process proceeds to step S1005. This allowed processing time may be input by the user, for example, or may be stored in advance in the ROM 215 and acquired from the ROM 215.

[0075] In step S1005, the CPU 202 determines the neural network 410 (trained model) to be used in step S1008 based on the total number of image capture devices 101 in the image capture system 1000 and the allowable processing time acquired in step S1004. A plurality of types of neural networks 410 are stored in advance in the ROM 215, and one neural network 410 suitable for processing within the allowable processing time is selected from these. After executing step S1005, the process proceeds to step S1006.

[0076] In step S1006, the CPU 202 notifies the image capturing devices 101a to 101d of the permissible time 1023 for the time synchronization error according to the neural network 410 determined in step S1005, and the process proceeds to step S1007.

[0077] In step S1007, the CPU 202 determines whether or not the captured images from the imaging devices 101a to 101d have been received by the server communication unit 205. If the CPU 202 determines in step S1007 that all the captured images have been received, the process proceeds to step S1008. On the other hand, if the CPU 202 determines in step S1007 that all the captured images have not been received, the process remains in step S1007 and waits.

[0078] In step S1008, the GPU 204 generates a 3D image by synthesizing all the captured images received in step S1007 using the neural network 410 determined in step S1005. As a result, the 3D image becomes an image in which blur has been corrected.

[0079] Next, the processing in the imaging devices 101a to 101d will be described. As shown in Fig. 10, in step S1011, the CPU 207 of the imaging device 101 (imaging devices 101a to 101d) determines whether or not the synchronization processing 1021 has been received in step S1001. If the CPU 207 determines in step S1011 that the synchronization processing 1021 has been received, the processing proceeds to step S1012. On the other hand, if the CPU 207 determines in step S1011 that the synchronization processing 1021 has not been received, the processing remains in step S1011 and waits.

[0080] In step S1012, the CPU 207 executes a synchronization process, and the process proceeds to step S1016.

[0081] In step S1016, CPU 207 determines whether or not it has received allowable time 1023 in step S1006. If CPU 207 determines in step S1016 that it has received allowable time 1023, the process proceeds to step S1017. On the other hand, if CPU 207 determines in step S1016 that it has not received allowable time 1023, the process remains in step S1016 and waits.

[0082] In step S1017, the CPU 207 determines whether the error of the synchronized captured image is within the allowable time 1023 received in step S1016. If the CPU 207 determines in step S1017 that the error is within the allowable time 1023, the process proceeds to step S1019. On the other hand, if the CPU 207 determines in step S1017 that the error is not within the allowable time 1023, the process proceeds to step S1018.

[0083] In step S1018, the CPU 207 determines to execute time correction processing for the captured image according to the amount of time deviation when the imaging device 101 having the CPU 207 is an imaging device 101 whose synchronization accuracy information is outside the allowable range. After executing step S1018, the process proceeds to step S1019.

[0084] In step S 1019 , the CPU 207 transmits the captured image 1024 from the network processing unit 210 to the server 103 .

[0085] The time correction process may be executed by the imaging devices 101a to 101d, or may not be executed by some of the imaging devices 101. The CPU 207 also functions as a switching means for switching an imaging device 101 set as a primary device to a secondary device, and switching an imaging device 101 set as a secondary device to a primary device. In this case, the primary device can be selected from the imaging devices 101 for which the time correction process is not executed, and switched. This makes it possible to omit the time correction process in the primary device.

[0086] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the above-mentioned embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. The present invention can also be realized by supplying a program that realizes one or more functions of the above-mentioned embodiments to a system or device via a network or storage medium, and having one or more processors of a computer in the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0087] The disclosure of each embodiment includes the following configurations, methods, and programs. (Configuration 1) An imaging system including a plurality of imaging devices capable of imaging, the imaging devices being capable of communicating with each other, a time synchronization unit for synchronizing the times of the imaging devices; a control unit that controls the image capture timing of each of the image capture devices based on the time synchronized by the time synchronization unit; An image processing means for performing image processing on the images captured by each of the imaging devices at the imaging timing controlled by the control means, using a trained model; A determination means for determining the trained model to be used in the image processing means, An imaging system characterized in that the determination means determines the trained model based on accuracy information regarding the time synchronization accuracy of the time synchronization means. (Configuration 2) The imaging system according to configuration 1, wherein the image processing means performs, as the image processing, a synthesis process for synthesizing the images to generate a three-dimensional image. (Configuration 3) The imaging system according to configuration 2, wherein the image processing means corrects blurring when blurring occurs in the three-dimensional image by using the trained model. (Configuration 4) The trained model includes a neural network having an input layer, an intermediate layer, and an output layer, the imaging system includes a storage means for storing a plurality of the neural networks, each of which differs from the others in at least one of the number of layers of a hierarchy constituted by the input layer, the intermediate layer, and the output layer, and the total number of the input layer, the intermediate layer, and the output layer; 4. The imaging system according to configuration 3, wherein the determining means selects one neural network from the plurality of neural networks in accordance with the amount of blur correction. (Configuration 5) The imaging system described in Configuration 4, characterized in that the determination means selects a neural network with a larger quantity the greater the amount of blur correction, and selects a neural network with a smaller quantity the smaller the amount of blur correction. (Configuration 6) A determination unit is provided for determining whether the synchronization accuracy is within an allowable range, The imaging system according to configuration 4 or 5, wherein the determining means selects the neural network with the smaller quantity when the judging means judges that the synchronization accuracy is within an allowable range. (Configuration 7) In the imaging system described in Configuration 6, when the judgment means determines that the synchronization accuracy is not within the acceptable range, it determines whether the number of the imaging devices whose synchronization accuracy is judged to be outside the acceptable range is equal to or less than a threshold value. (Configuration 8) An imaging system as described in Configuration 6 or 7, wherein the determination means selects the neural network with the larger quantity when the judgment means determines that the number of imaging devices is equal to or less than a threshold, and selects the neural network with the smaller quantity when the judgment means determines that the number of imaging devices is not equal to or less than the threshold. (Configuration 9) The imaging system according to any one of configurations 1 to 8, wherein the determining means determines the total number of the imaging devices and the time consumed during the image processing. (Configuration 10) The image includes a moving subject, a speed detection means for detecting a moving speed of the subject, The imaging system according to any one of configurations 1 to 8, wherein the determination means determines the trained model based on a detection result by the speed detection means. (Configuration 11) The imaging system according to any one of configurations 1 to 10, further comprising an accuracy detection unit that detects the synchronization accuracy. (Configuration 12) The imaging system described in Configuration 11, characterized in that the accuracy detection means detects the synchronization accuracy based on the amount of time adjustment made when the time synchronization means synchronizes the time between the imaging devices. (Configuration 13) The imaging system according to configuration 11, wherein the accuracy detection means detects the synchronization accuracy based on a communication speed when the communication is performed between the imaging devices. (Configuration 14) The imaging system described in any one of configurations 1 to 13, characterized in that the time synchronization means synchronizes the time between the imaging devices by designating one of the multiple imaging devices as a primary device and the remaining imaging devices as secondary devices, and adjusting the time of the secondary device to the time of the primary device. (Configuration 15) The imaging system according to configuration 14, further comprising a setting unit which sets one of the plurality of imaging devices as the primary device and sets the remaining imaging devices as the secondary devices. (Configuration 16) The imaging system according to configuration 15, further comprising a switching means for switching the imaging device set as the primary device to the secondary device, and switching the imaging device set as the secondary device to the primary device. (Method 1) A method for controlling an imaging system including a plurality of imaging devices capable of imaging, the imaging devices being capable of communicating with each other, comprising the steps of: a time synchronization step of synchronizing the times of the imaging devices; a control step of controlling an image capturing timing of each of the image capturing devices based on the time synchronized in the time synchronization step; an image processing step of performing image processing on the images captured by each of the imaging devices at the imaging timing controlled by the control step, using a trained model; A determination step of determining the trained model to be used in the image processing step, A control method for an imaging system, characterized in that the determination step determines the trained model based on accuracy information regarding the time synchronization accuracy in the time synchronization step. (Program 1) A program for causing a computer to execute each of the means of the imaging system according to any one of configurations 1 to 16. [Explanation of symbols]

[0088] 101, 101a, 101b, 101c, 101d Imaging device 103 Server 204 GPU(Graphics Processing Unit) 207 CPU (control means) 214 Synchronization control unit (time synchronization means) 410 Neural Networks 1000 Imaging System

Claims

1. An imaging system including a plurality of imaging devices capable of imaging, the imaging devices being capable of communicating with each other, a time synchronization unit for synchronizing the times of the imaging devices; a control unit that controls the image capture timing of each of the image capture devices based on the time synchronized by the time synchronization unit; An image processing means for performing image processing on the images captured by each of the imaging devices at the imaging timing controlled by the control means, using a trained model; A determination means for determining the trained model to be used in the image processing means, An imaging system characterized in that the determination means determines the trained model based on accuracy information regarding the time synchronization accuracy of the time synchronization means.

2. 2. The imaging system according to claim 1, wherein the image processing means performs a synthesis process for synthesizing the images to generate a three-dimensional image.

3. The imaging system according to claim 2 , wherein when blurring occurs in the three-dimensional image, the image processing means corrects the blurring by using the trained model.

4. The trained model includes a neural network having an input layer, an intermediate layer, and an output layer, the imaging system includes a storage means for storing a plurality of the neural networks, each of which differs from the others in at least one of the number of layers of a hierarchy constituted by the input layer, the intermediate layer, and the output layer, and the total number of the input layer, the intermediate layer, and the output layer; 4. The imaging system according to claim 3, wherein the determining means selects one neural network from among the plurality of neural networks in accordance with the amount of blur correction.

5. 5. The imaging system according to claim 4, wherein the determining means selects a neural network having a larger quantity as the amount of blur correction is larger, and selects a neural network having a smaller quantity as the amount of blur correction is smaller.

6. a determination unit for determining whether the synchronization accuracy is within an allowable range; 5. The imaging system according to claim 4, wherein the determining means selects the neural network with the smallest number of neural networks when the judging means judges that the synchronization accuracy is within an allowable range.

7. The imaging system according to claim 6 , wherein, when the determination means determines that the synchronization accuracy is not within an acceptable range, the determination means determines whether the number of the imaging devices for which the synchronization accuracy is determined to be not within an acceptable range is equal to or less than a threshold value.

8. The imaging system of claim 6, wherein the determination means selects the neural network with the larger quantity when the judgment means determines that the number of imaging devices is equal to or less than a threshold, and selects the neural network with the smaller quantity when the judgment means determines that the number of imaging devices is not equal to or less than the threshold.

9. The imaging system according to claim 1 , wherein the determination means determines the trained model based on a total number of the imaging devices and a processing time allowed to be consumed during the image processing.

10. The image includes a moving object, a speed detection means for detecting a moving speed of the subject, The imaging system according to claim 1 , wherein the determining means determines the trained model based on a detection result by the speed detecting means.

11. The imaging system according to claim 1 , further comprising an accuracy detection unit for detecting the accuracy of the synchronization.

12. 12. The imaging system according to claim 11, wherein the accuracy detection unit detects the synchronization accuracy based on an error range when the time synchronization unit synchronizes the times of the imaging devices.

13. 12. The imaging system according to claim 11, wherein the accuracy detection unit detects the synchronization accuracy based on a communication speed when the communication is performed between the imaging devices.

14. The imaging system according to claim 1, characterized in that the time synchronization means synchronizes the times of the imaging devices by designating one of the plurality of imaging devices as a primary device and the remaining imaging devices as secondary devices, and adjusting the time of the secondary device to the time of the primary device.

15. 15. The imaging system according to claim 14, further comprising a setting unit which sets one of the plurality of imaging devices as the primary device and sets the remaining imaging devices as the secondary devices.

16. 16. The imaging system according to claim 15, further comprising a switching unit that switches the imaging device set as the primary device to the secondary device, and switches the imaging device set as the secondary device to the primary device.

17. A method for controlling an imaging system including a plurality of imaging devices capable of imaging, the imaging devices being capable of communicating with each other, comprising: a time synchronization step of synchronizing the times of the imaging devices; a control step of controlling an image capturing timing of each of the image capturing devices based on the time synchronized in the time synchronization step; an image processing step of performing image processing on the images captured by each of the imaging devices at the imaging timing controlled by the control step, using a trained model; A determination step of determining the trained model to be used in the image processing step, A control method for an imaging system, characterized in that the determination step determines the trained model based on accuracy information regarding the time synchronization accuracy in the time synchronization step.

18. A program for causing a computer to execute each of the means of the imaging system according to claim 1.

Images