Image Processing System

The image processing system synchronizes multiple imaging devices to acquire images at the same time, addressing the challenge of asynchronous image capture and enhancing virtual viewpoint image quality.

JP7577617B2Active Publication Date: 2024-11-05CANON KK
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
JP2021104058
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-06-23
Publication Date
2024-11-05
Estimated Expiration
2041-06-23

AI Technical Summary

Technical Problem

Existing technologies fail to synchronously acquire multiple captured images from different imaging devices, which is necessary for generating high-precision virtual viewpoint images and synchronized image displays.

Method used

An image processing system that includes multiple image processing devices synchronized by a synchronization signal, which acquires and outputs image information based on time information associated with the imaging time, ensuring all devices operate in sync to retrieve images at the same time.

Benefits of technology

Enables the synchronous acquisition of images from multiple imaging devices, allowing for the generation of high-quality virtual viewpoint images and synchronized image displays without quality deterioration.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007577617000001
    Figure 0007577617000001
  • Figure 0007577617000002
    Figure 0007577617000002
  • Figure 0007577617000003
    Figure 0007577617000003
Patent Text Reader

Abstract

To synchronously obtain information on a plurality of captured images obtained by being captured with a plurality of imaging devices from a storage device.SOLUTION: An image processing system 1 includes a plurality of image processing devices 100. Each of the plurality of image processing devices 100 obtains image information from a storage means configured to store image information indicating a captured image to be associated with time information indicating an imaging time of the captured image, on the basis of the time information. Specifically, the plurality of image processing devices 100 control a timing for obtaining the image information by outputting a synchronization signal synchronized with each other at a predetermined time interval. By controlling the timing of the synchronization signal output from each image processing device, the image processing system 1 obtains, from the storage device, the image information of the captured image captured at the same time by synchronizing the plurality of image processing devices 100 with each other.SELECTED DRAWING: Figure 3
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to a technique for synchronously acquiring a plurality of image data obtained by synchronously capturing images using a plurality of imaging devices. [Background technology]

[0002] There is a technology in which a plurality of imaging devices are installed at different positions, and images are captured synchronously from multiple viewpoints, and an image (hereinafter referred to as a "virtual viewpoint image") viewed from any viewpoint designated by a user is generated using a plurality of images (hereinafter referred to as "captured images") obtained by the imaging. The virtual viewpoint image allows a user to view highlight scenes of a soccer or basketball game, for example, from various angles, and therefore can provide a user with a higher sense of realism than a normal captured image. In order to generate a highly accurate virtual viewpoint image, it is necessary to generate three-dimensional shape data based on captured images captured at the same time by each imaging device, and therefore it is necessary to synchronously acquire the captured images captured at the same time by each imaging device. Also, in the surveillance field, in order to display captured images captured at the same time by a plurality of imaging devices side by side, it is necessary to synchronously acquire the captured images captured at the same time by each imaging device.

[0003] Patent Document 1 discloses a technique for synchronously playing back a plurality of moving images captured by a plurality of imaging devices with different frame rates. Specifically, the technique disclosed in Patent Document 1 predetermines frames (hereinafter referred to as "reference frames") to be synchronously played back for each of a plurality of moving images captured by each imaging device, and controls the reference frames to be played back at the same playback timing. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2013-225930 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology disclosed in Patent Document 1 is for synchronously playing back multiple moving images, but is not for synchronously acquiring multiple captured images captured at the same time by multiple imaging devices.

[0006] The present disclosure makes it possible to synchronously acquire information about a plurality of captured images obtained by capturing images using a plurality of imaging devices from a storage device. [Means for solving the problem]

[0007] The image processing system according to the present disclosure is an image processing system including a plurality of image processing devices, A synchronization signal that specifies the timing of imaging in an imaging device and is synchronized among a plurality of image processing devices. Image information indicating an image captured by the imaging device; 、 an image acquisition means for acquiring image information based on time information from a storage means storing the image information in association with time information indicating the image capturing time when the image was captured; an acquisition control means for controlling a timing at which the image acquisition means acquires the image information from the storage means; and a designation means for designating the image capturing time of the image information acquired by the image acquisition means from the storage means. an output means for selecting and outputting either the image information acquired by the image acquisition means or the image information not stored in the storage means; The acquisition control means includes: Get Synchronization with multiple image processing devices that define timing The sync signal is synchronized with the sync signal that specifies the timing of imaging. The specifying means specifies the same image capturing time in the plurality of image processing devices, and the image acquiring means outputs image information corresponding to the image capturing time specified by the specifying means from the acquisition control means. Specify the timing of acquisition Based on the synchronization signal, the plurality of image processing devices synchronize with each other and acquire the images from the storage means. Effect of the Invention

[0008] According to the present disclosure, information on a plurality of captured images obtained by capturing images using a plurality of imaging devices can be synchronously acquired from a storage device. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an example of a configuration of an image processing system according to a first embodiment. [Diagram 2] 1A is an explanatory diagram showing an example of the arrangement and connection of a sensor system according to the first embodiment, FIG. 1B is an explanatory diagram showing an example of a virtual camera and a position and orientation of the virtual camera based on a virtual camera path, and FIG. 1C is an explanatory diagram showing another example of the arrangement and connection of a sensor system according to the first embodiment. [Diagram 3] 1 is a block diagram showing an example of a configuration of functional blocks in an image processing device according to a first embodiment. [Figure 4] 1 is a block diagram showing an example of a hardware configuration of an image processing device according to a first embodiment. [Diagram 5] 4 is a diagram showing an example of scene information managed in the storage device according to the first embodiment. FIG. [Figure 6] 5 is a flowchart showing an example of a processing flow of the image processing device according to the first embodiment. [Figure 7] 5 is a flowchart showing an example of a processing flow of the image processing device according to the first embodiment. [Figure 8] FIG. 11 is a block diagram showing an example of the configuration of an image processing system according to a second embodiment. [Figure 9] 11 is a block diagram showing an example of the configuration of functional blocks in an image processing device 0 according to a second embodiment. FIG. [Figure 10] 10 is a flowchart showing an example of a processing flow of an image processing device according to a second embodiment. [Figure 11] 10 is a flowchart showing an example of a processing flow of an image processing device according to a second embodiment. [Figure 12] 13 is a flowchart showing an example of a processing flow of an image processing apparatus located at the frontmost stage among the image processing apparatuses according to the second embodiment. [Figure 13] 13 is a flowchart showing an example of a processing flow of an image processing device located at a stage subsequent to the image processing device located at the forefront stage among the image processing devices according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the attached drawings. Note that the configurations shown in the following embodiments are merely examples, and the scope of the present disclosure is not limited to only those configurations. In the embodiments of the present disclosure, as an example, an application to a system for generating a virtual viewpoint image is shown, but the scope of application of the present disclosure is not limited to that system. The present disclosure can also be applied to a system in the surveillance field, etc., which acquires multiple images stored in a storage device and displays the acquired multiple images side by side.

[0011] First Embodiment An image processing system 1 according to the first embodiment will be described with reference to Figs. 1 to 7. Fig. 1 is a block diagram showing an example of the configuration of the image processing system 1 according to the first embodiment. The image processing system 1 has a plurality of sensor systems 10 (10a-10x), an image computing server 13, a controller 16, a switching hub 18, an end user terminal 17, and a time server 12. The controller 16 has a control station 14 and a virtual camera operation UI 15. Each sensor system 10 (10a-10x) has an image processing device 100 (100a-100x) and an imaging device 11 (11a-11x). In the image processing system 1, each imaging device 11 is arranged at a different position in a facility such as a stadium or a concert hall, and captures an image of an imaging target.

[0012] In the first embodiment, each sensor system 10 is described as having one image processing device 100 and one imaging device 11, but the number of image processing devices 100 and imaging devices 11 in each sensor system 10 is not limited to this. For example, the number of image processing devices 100 in each sensor system 10 may be two or more, and the number of image processing devices 100 in each sensor system 10 may be two or more. The sensor system 10 may have other devices in addition to the imaging device 11 and the image processing device 100. The imaging device 11 and the image processing device 100 are described as separate devices exteriorly mounted in different housings, but the imaging device 11 and the image processing device 100 may be configured as an integrated device exteriorly mounted in the same housing. The sensor systems 10 are described as all having the same configuration, but each sensor system 10 may have a configuration in which some or all of the multiple sensor systems 10 are different from each other. Hereinafter, unless otherwise specified, the image processing devices 100a-100x and the imaging devices 11a-11x included in the sensor systems 10a-10x will be described without distinction as the imaging device 11 and the image processing device 100. In addition, in the first embodiment, the image processing system 1 will be described as having 24 sensor systems 10, but the number of sensor systems 10 may be less than 24 or more than 24 as long as it is two or more. Hereinafter, unless otherwise specified, the 24 sensor systems 10a-10x will be described without distinction, simply described as the sensor systems 10.

[0013] The imaging device 11 has a lens and an image sensor, not shown in FIG. 1, and outputs an image (hereinafter referred to as a "captured image") obtained by capturing an image with the image sensor as an image signal. The imaging device 11 may have a microphone, not shown in FIG. 1, in addition to the lens and image sensor, and may output sound collected by the microphone as an audio signal in addition to the image signal. The imaging device 11 may be fixed to a camera platform that can change the imaging direction of the imaging device 11 by panning, tilting, or rolling the imaging device 11. Hereinafter, unless otherwise specified, the term "image" will be described as including the concepts of moving images and still images. That is, the image processing system 1 according to the first embodiment is capable of processing both still images and moving images.

[0014] The image processing device 100 receives an image signal output by the imaging device 11, generates image information corresponding to the image signal, and outputs the generated image information to a storage device arranged inside or outside the image processing device 100, causing the storage device to store the image information. The image processing device 100 also acquires image information by reading the image information from the storage device. The image processing device 100 selects either the image information generated based on the image signal or the image information read from the storage device, and outputs the selected image information. Details of the image processing device 100 will be described later.

[0015] The block diagrams shown in FIG. 1(a) and FIG. 1(b) show an example of the connection form of each block of the image processing system 1. In the image processing system 1 shown in FIG. 1(a), a plurality of image processing devices 100 are connected in a daisy chain so as to be able to communicate with each other via networks 19a-19w. In addition, the image processing devices 100a, 100x, the image computing server 13, the time server 12, and the control station 14 are connected so as to be able to communicate with each other via a switching hub 18 and networks 18a, 18x, 12a, 13a, and 14a. The connection form of each block of the image processing system 1 shown in FIG. 1(a) is merely an example, and is not limited to the form shown in FIG. 1(a). For example, each block of the image processing system 1 may be connected in a form as shown in the block diagram of FIG. 1(b). Specifically, for example, as shown in FIG. 1(b), each block included in the image processing system 1 may be connected to each other so as to be able to communicate with each other via a switching hub 18 and networks 18a-18x, 12a, 13a, and 14a. Hereinafter, each block included in the image processing system 1 will be described as being connected in a connection form as shown in FIG. 1(a). Note that each block included in the image processing system 1 is configured, for example, by GbE (Gigabit Ethernet) or 10 GbE conforming to the IEEE standard, which is Ethernet (registered trademark). GbE or 10 GbE is only an example, and the network may be configured by an interconnect such as Infiniband, or an industrial Ethernet, or may be configured by a combination of these. In addition, the configuration is not limited to these, and may be configured by other types of networks.

[0016] The arrangement and connection of the sensor system 10 will be described with reference to FIG. 2(a) and FIG. 2(c). FIG. 2(a) and FIG. 2(c) are explanatory diagrams showing an example of the arrangement and connection of the sensor system 10. The imaging devices 11 of each sensor system 10 are arranged at different positions so as to surround a field such as a soccer field. Each imaging device 11 captures an image of an object to be imaged, such as a player playing on the field, from a plurality of different directions. The image processing devices 100 of the plurality of sensor systems 10 are connected to each other by a daisy chain. By connecting the image processing devices 100 to each other by a daisy chain, the number of connection cables can be reduced, or the labor required for wiring can be saved. FIG. 2(a) shows a form in which all the image processing devices 100a-100x are cascade-connected to form one daisy chain, but the connection form of the image processing system 1 is not limited to this.

[0017] For example, as shown in FIG. 2(c), the sensor systems 10 may be divided into two groups of 24 sensor systems 10, and the sensor systems 10, i.e., the image processing devices 100, may be connected to each other in a daisy chain for each group. The number of groups into which the sensor systems 10 are divided is not limited to two, and may be three or more. A connection form in which the sensor systems 10 are divided into several groups is particularly effective in a stadium or the like in which spectator seats are arranged in a cone shape around the field. For example, a case may be considered in which spectator seats are arranged across multiple floors, and a sensor system 10 is arranged on each floor on which the spectator seats are arranged. In this case, for example, the sensor systems 10 arranged on each floor are treated as one group, and the sensor systems 10 in the same group are connected to each other in a daisy chain. This type of connection can simplify wiring and make the system more flexible. Furthermore, this type of connection can divide data flowing on the network, and it is possible to increase the amount of image information generated by each image processing device 100 or acquired from a storage device.

[0018] The manner in which the sensor systems 10 are divided into several groups is not limited to the manner in which the sensor systems 10 arranged on each floor are grouped together. For example, the periphery of the field may be divided into two half-circles from a certain point, and the sensor systems 10 arranged at positions corresponding to each half-circle may be grouped together, and the sensor systems 10 in the same group may be connected together in a daisy chain. This type of connection can simplify the wiring and make the system more flexible. In the following, the multiple image processing devices 100 are described as being connected together in a daisy chain in the connection form shown in FIG. 2(a).

[0019] The control station 14 manages the operating state of each block constituting the image processing system 1, and sets or controls parameters. The virtual camera operation UI 15 provides a user interface for specifying the position and orientation of a virtual camera used when generating a virtual viewpoint image. A virtual camera is a virtual imaging device that enables acquisition of an image captured from a viewpoint different from any of the installed imaging devices. The operator of the controller 16 specifies the position and orientation of the virtual camera using an operation input device not shown in FIG. 1 such as a mouse or a joystick. For example, the position and orientation of the virtual camera may be specified by the operator of the controller 16 specifying a virtual camera path. The virtual camera path is a string of information indicating the position and orientation of the virtual camera for each frame. FIG. 2(b) is an explanatory diagram showing an example of a virtual camera and the position and orientation of the virtual camera based on the virtual camera path. The virtual camera operation UI 15 receives a signal indicating the position and orientation of the virtual camera specified by the operator from the operation input device, and outputs information corresponding to the signal as viewpoint information to the image computing server 13. The position and orientation of the virtual camera are not limited to being specified by the operator of the controller 16, and may be automatically specified based on the result of image analysis, etc. Hereinafter, an operation of transmitting 24 images from the sensor system 10x to the image computing server 13, with each image acquired by the 24 sensor systems 10a-10x being regarded as an image set, will be described.

[0020] The image computing server 13 is an information processing device configured by a computer or the like. The image computing server 13 receives image information output by each image processing device 100 as one image set, and acquires captured images (hereinafter referred to as "multiple viewpoint images") indicated by each image information in one image set. The image computing server 13 also acquires viewpoint information output by the virtual camera operation UI 15, and generates a virtual viewpoint image that looks like an image captured by a virtual camera based on the multiple viewpoint images and the viewpoint information. That is, the image computing server 13 can generate a virtual viewpoint image that looks as if it was captured by an imaging device located near the goal, for example. A method for generating a virtual viewpoint image based on multiple viewpoint images and viewpoint information is well known, so a detailed description will be omitted.

[0021] The image computing server 13 outputs the generated virtual viewpoint image to the end user terminal 17. The image computing server 13 may perform compression encoding or the like on the generated virtual viewpoint image by standard image encoding such as H.264 or HEVC (High Efficiency Video Coding), and then output the generated virtual viewpoint image to the end user terminal 17. For example, the image computing server 13 may output the compressed and encoded virtual viewpoint image by standard streaming distribution or the like using MPEG-DASH (Dynamic Adaptive Streaming over HTTP) protocol or the like. Note that the streaming distribution by MPEG-DASH is merely an example, and the distribution method of the virtual viewpoint image is not limited to MPEG-DASH. For example, a transmission method such as HLS (HTTP Live Streaming) may be used to distribute the virtual viewpoint image. The image computing server 13 may also output the generated virtual viewpoint image to the end user terminal 17 in an uncompressed state.

[0022] The image computing server 13 switches the control mode of image processing depending on whether the number of the image processing device 100 at the last stage that outputs image information to the image computing server 13 is one or two or more. That is, the image computing server 13 switches the control mode depending on whether the sensor system 10 is divided into a plurality of groups. As described above, the image processing devices 100 are connected to each other by a daisy chain. For example, in the case of the connection form shown in FIG. 2(a), the image information output from each image processing device 100 is transmitted to the image computing server 13 as follows. The image information acquired or generated by the image processing device 100a is transmitted to the image processing device 100b via the network 19a. The image processing device 100b transmits the image information generated by the image processing device 100b or acquired from the storage device and the image information transmitted from the image processing device 100a together to the image processing device 100c. Thereafter, the same transmission is repeated up to the image processing device 100x. As a result, information combining image information generated by each of the image processing devices 100a-100w or acquired from a storage device is transmitted from the image processing device 100w via the network 19w to the image processing device 100x. In addition, the image processing device 100x combines the information transmitted from the image processing device 100w with image information generated by the image processing device 100x or acquired from a storage device, and outputs the combined information to the image computing server 13. Specifically, the information output from the image processing device 100x, i.e., the combined information of image information generated by each of the image processing devices 100a-100x or acquired from a storage device, is transmitted to the image computing server 13 via the switching hub 18.

[0023] Therefore, in the case where there is one image processing device 100 that outputs image information to the image computing server 13, the image computing server 13 can acquire all image information generated or acquired by each of the multiple image processing devices 100 at the same timing. In other words, in the case where the sensor system 10 is not divided into groups, the timing of acquiring all image information corresponding to one image set by the image computing server 13 is synchronized. However, in the case where there are multiple image processing devices 100 that output image information to the image computing server 13 (when the multiple sensor systems 10 are divided into multiple groups), the following may occur. For example, in this case, the delay time until the image computing server 13 acquires the image information may differ for each lane (path) of the daisy chain. Therefore, in this case, the image computing server 13 may need to wait until all image information corresponding to one image set is available, and then perform subsequent image processing to generate a virtual viewpoint image. The image computing server 13 is configured by a computer or the like.

[0024] The end user terminal 17 is configured with a smartphone, a tablet terminal, a personal computer, or the like, and receives the virtual viewpoint image output by the image computing server 13 and displays the virtual viewpoint image. When the virtual viewpoint image output by the image computing server 13 is a compressed and encoded virtual viewpoint image, the end user terminal 17 receives the compressed and encoded virtual viewpoint image output by the image computing server 13 and decodes the virtual viewpoint image. The end user terminal 17 outputs the decoded virtual viewpoint image to a display not shown in FIG. 1, and displays the virtual viewpoint image on the display. When the virtual viewpoint image output by the image computing server 13 is an uncompressed virtual viewpoint image, the end user terminal 17 may be a monitor or the like that receives the uncompressed virtual viewpoint image as a display signal and displays the virtual viewpoint image based on the display signal.

[0025] The time server 12 outputs a reference time signal indicating a time serving as a reference for the system (hereinafter referred to as "reference time"), and a synchronization signal serving as a reference for the system (hereinafter referred to as "reference synchronization signal"). The reference time signal and the reference synchronization signal output by the time server 12 are distributed to the image processing devices 100a-100x of the sensor systems 10a-10x via a switching hub 18. For example, the time server 12 periodically outputs the reference synchronization signal at a predetermined time interval. Here, the predetermined time interval is a period corresponding to the frame interval of the imaging frame rate of the imaging device 11, such as 1 / 60 seconds.

[0026] The image processing device 100 will be described with reference to FIG. 3 to FIG. 7. The functional blocks of the image processing device 100 will be described with reference to FIG. 3. FIG. 3 is a block diagram showing an example of the configuration of the functional blocks in the image processing device 100 according to the first embodiment. The image processing device 100 has a reference signal acquisition unit 110, a reference signal output unit 112, a current time acquisition unit 114, an acquisition control unit 140, a designation unit 150, an image acquisition unit 170, and an image output unit 190. In addition to the above-mentioned configuration, the image processing device 100 may have an imaging control unit 120, an imaging output unit 130, a time point acquisition unit 152, or a start time acquisition unit 154. Hereinafter, the image processing device 100 will be described as having the imaging control unit 120, the imaging output unit 130, the time point acquisition unit 152, and the start time acquisition unit 154. Details of each unit shown in FIG. 3 will be described later.

[0027] The processing of each unit of the image processing device 100 is performed by hardware such as an ASIC (application specific integrated circuit) built into the image processing device 100. The processing may be performed by hardware such as an FPGA (field programmable gate array). Moreover, the processing may be performed by software using a CPU (Central Processor Unit) or a GPU (Graphic Processor Unit) and a memory.

[0028] With reference to Fig. 4, a hardware configuration of the image processing device 100 in the case where each unit of the image processing device 100 operates as software will be described. Fig. 4 is a block diagram showing an example of a hardware configuration in the image processing device 100 according to the first embodiment. The image processing device 100 is configured by a computer, and the computer has a CPU 411, a ROM 412, a RAM 413, an auxiliary storage device 414, a communication unit 417, and a bus 418 as shown in Fig. 4 as an example. In addition to the above-mentioned hardware, the computer may have a display unit 415 and an operation unit 416 as shown in Fig. 4.

[0029] The CPU 411 controls the computer using programs or data stored in the ROM 412 or the RAM 413, thereby causing the computer to function as each unit of the image processing device 100 shown in Fig. 1. The image processing device 100 may have one or more dedicated hardware components different from the CPU 411, and at least a part of the processing by the CPU 411 may be executed by the dedicated hardware components. Examples of the dedicated hardware components include an ASIC, an FPGA, and a DSP (digital signal processor).

[0030] The ROM 412 stores programs and the like that do not require modification. The RAM 413 temporarily stores programs or data supplied from the auxiliary storage device 414, or data and the like supplied from the outside via the communication unit 417. The auxiliary storage device 414 is constituted by, for example, a hard disk drive, and stores various data such as image data or audio data.

[0031] The display unit 415 is configured with, for example, a liquid crystal display or an LED, and displays a GUI (Graphical User Interface) or the like for the user to operate or view the image processing device 100. The operation unit 416 is configured with, for example, a keyboard, a mouse, a joystick, a touch panel, and the like, and receives operations by the user to input various instructions to the CPU 411. The CPU 411 also operates as a display control unit that controls the display unit 415 and an operation control unit that controls the operation unit 416.

[0032] The communication unit 417 is used for communication with an external device of the image processing device 100. For example, when the image processing device 100 is connected to an external device by wire, a communication cable is connected to the communication unit 417. When the image processing device 100 has a function of wirelessly communicating with an external device, the communication unit 417 is equipped with an antenna. The bus 418 connects each unit of the image processing device 100 to transmit information. In the first embodiment, the display unit 415 and the operation unit 416 are described as being present inside the image processing device 100, but at least one of the display unit 415 and the operation unit 416 may be present as a separate device outside the image processing device 100.

[0033] The reference signal acquisition unit 110 acquires a reference synchronization signal. The reference signal output unit 112 outputs the reference synchronization signal acquired by the reference signal acquisition unit 110. Specifically, when a plurality of image processing devices 100a-100x are connected by a daisy chain as shown in FIG. 1(a) or FIG. 2(a), the reference signal acquisition unit 110 of the image processing device 100a acquires a reference synchronization signal output by the time server 12. In this case, the reference signal acquisition unit 110 of each of the image processing devices 100b-100x acquires a reference synchronization signal output by a reference signal output unit 112 of the image processing device 100 preceding the image processing devices 100b-100x. More specifically, the reference signal acquisition unit 110 of the image processing device 100b acquires a reference synchronization signal output by a reference signal output unit 112 of the image processing device 100a. Furthermore, the reference signal acquisition unit 110 of the image processing device 100c acquires a reference synchronization signal output by the reference signal output unit 112 of the image processing device 100b. The same applies below, so a description will be omitted. When multiple image processing devices 100a-100x are connected in the connection form shown in FIG. 1(b), the reference signal acquisition unit 110 of each of the image processing devices 100a-100x acquires a reference synchronization signal output by the time server 12. In this case, the reference signal output unit 112 is not an essential component of the image processing device 100.

[0034] The current time acquisition unit 114 acquires information indicating the current time (hereinafter referred to as "current time information"). Specifically, for example, the current time acquisition unit 114 receives a reference time signal output by the time server 12 and acquires current time information assuming that the reference time indicated by the reference time signal is the current time. When a plurality of image processing devices 100a-100x are connected by a daisy chain as shown in FIG. 1(a) or FIG. 2(a), the current time acquisition unit 114 of the image processing device 100a acquires the reference time signal output by the time server 12. In this case, the current time acquisition unit 114 of each of the image processing devices 100b-100x acquires the reference time signal output by a current time output unit (not shown in FIG. 3) of the image processing device 100 preceding the image processing devices 100b-100x.

[0035] For example, the current time acquisition unit 114 may acquire current time information by acquiring the current time from an internal clock (not shown in FIG. 3) of the image processing device 100. In this case, the internal clock of the image processing device 100 is time-synchronized with the time server 12 periodically or irregularly using a timestamp of data transmitted to and received from the time server 12 in compliance with the IEEE 1588 standard or the like. Note that the time synchronization method with the time server 12 is not limited to a method using the IEEE 1588 standard, and may be a method using the EtherAVB standard or a predetermined unique protocol or the like. Also, for example, when all of the imaging devices 11 of each of the sensor systems 10a-10x are time-synchronized, the current time acquisition unit 114 may acquire current time information output by the corresponding imaging device 11.

[0036] The imaging control unit 120 is connected to the imaging device 11, and controls the imaging device 11, provides a synchronization signal, sets the time, and the like. The control of the imaging device 11 includes setting and referencing imaging parameters such as the number of pixels of the captured image, color depth, frame rate, or white balance. The control of the imaging device 11 may include obtaining the state of the imaging device 11 such as during imaging, during stop, during synchronization, or an error, controlling the start or stop of imaging, or adjusting focus or zoom. Note that lens adjustment such as focus or zoom may be performed via the imaging device 11, or may be performed directly by connecting the image processing device 100 to the lens when a removable lens is attached to the imaging device 11. The provision of the synchronization signal is performed by the imaging control unit 120 generating a synchronization signal based on a reference synchronization signal and the current time, and outputting the generated synchronization signal to the imaging device 11 in accordance with the imaging timing. The imaging device 11 receives a synchronization signal output from the image processing device 100 and performs GENLOCK (Generator Lock) on the synchronization signal, thereby capturing an image at the timing of receiving the synchronization signal from the image processing device 100. That is, the imaging timing of all of the imaging devices 11 is synchronized by the reference synchronization signal output from the imaging control unit 120 of each image processing device 100. The time is set by outputting information on the current time synchronized with the time server 12 to the imaging device 11 as, for example, a time code conforming to SMPTE 12M.

[0037] The imaging device 11 outputs the image signal including, as meta information, a time code corresponding to the time when the image was captured (hereinafter referred to as "capture time"), that is, a time code provided from the imaging control unit 120 to the imaging device 11, in addition to information indicating the captured image. The format of the time code is not limited to SMPTE12M, and may be other formats. The meta information may include information indicating a sequence number or data type of the captured image, or an identifier indicating an individual imaging device 11, in addition to the time code. In this way, the imaging control unit 120 of each image processing device 100 can obtain an image signal indicating the captured image captured by all the imaging devices 11 in high-precision synchronization. Up to this point, a form has been described in which the imaging device 11 includes, as meta information, information indicating a time code corresponding to the capture time of the captured image in the image signal using the time code provided from the imaging control unit 120, but this is not limiting. For example, the imaging control unit 120 may not provide a time code to the imaging device 11, but may add a time code corresponding to the time when the image signal was received as meta information to the image signal received from the imaging device 11 in the imaging control unit 120. The imaging control unit 120 generates image information indicating a captured image corresponding to the image signal and time information indicating the capture time of the captured image from the image signal output by the imaging device 11. The imaging control unit 120 outputs the generated image information and time information to the imaging output unit 130 and the image output unit 190. Note that when generating the image information, the imaging control unit 120 may compress or encode data of the captured image corresponding to the image signal, and generate information indicating the compressed or encoded captured image as the image information.

[0038] The imaging output unit 130 outputs the image information and time information generated by the imaging control unit 120 to a storage device arranged inside or outside the image processing device 100, and stores the image information and the time information in the storage device. Specifically, the imaging output unit 130 stores the image information and the time information in the storage device in association with the image information and the time information indicating the imaging time of the captured image indicated by the image information. With the above configuration, the storage device stores a plurality of image information obtained by imaging by each of the plurality of imaging devices 11, in which the imaging timing of all the imaging devices 11 is synchronized, in association with the time information corresponding to each image information. Note that the imaging output unit 130 may store meta information including information indicating the imaging time of the captured image and information indicating the sequence number or data type of the captured image, or an identifier indicating the individual imaging device 11, in association with the image information, in the storage device, instead of the time information.

[0039] In the first embodiment, as an example, the synchronization of the image capturing times in the multiple image capturing devices 11 is managed based on the reference synchronization signal output by the time server 12, but the method of synchronizing the image capturing times is not limited to this. In addition, in the first embodiment, as an example, the synchronization of the current times in the multiple image processing devices 100 and the image capturing device 11 is managed based on the reference time signal output by the time server 12, but the method of synchronizing the current times is not limited to this. For example, the process for synchronizing the current times may be independently performed by each image capturing device 11 or each image processing device 100. In addition, the process for synchronizing the image capturing times may be independently performed by each image capturing device 11 or each image processing device 100 based on the synchronized current time and a predetermined frame rate of the captured image.

[0040] The image acquiring unit 170 acquires image information based on the time information from a storage device that stores image information in association with time information. The image acquiring unit 170 may acquire time information corresponding to the image information in addition to the image information. The acquisition control unit 140 controls the timing at which the image acquiring unit 170 acquires the image information from the storage device. The designation unit 150 designates the imaging time of the image information that the image acquiring unit 170 acquires from the storage device.

[0041] Specifically, the acquisition control unit 140 of each image processing device 100 outputs a synchronization signal that specifies the timing at which the image acquisition unit 170 acquires image information from a storage device to the image acquisition unit 170. The synchronization signal output by the acquisition control unit 140 of each image processing device 100 is synchronized between the multiple image processing devices 100 and is output at a predetermined time interval, such as a time interval corresponding to the frame rate of the captured image indicated by the image information. More specifically, for example, the acquisition control unit 140 outputs a synchronization signal based on a reference synchronization signal acquired by the reference signal acquisition unit 110, thereby synchronizing the synchronization signals between the multiple image processing devices 100. In addition, the designation unit 150 of each image processing device 100 designates the same imaging time between the multiple image processing devices. The image acquisition unit 170 of each image processing device 100 reads image information corresponding to the imaging time designated by the designation unit 150 from the storage means based on the synchronization signal output from the acquisition control unit 140. With this configuration, the image acquisition unit 170 acquires image information corresponding to images captured at the same imaging time in synchronization with each other among the multiple image processing devices 100. The image acquisition unit 170 outputs the acquired image information to the image output unit 190. Note that the imaging time designated by the designation unit 150 is acquired from the control station 14, for example, assuming that the time input by the operator of the controller 16 or the like is the imaging time to be designated.

[0042] The image output unit 190 receives the image information generated by the imaging control unit 120 and the image information acquired by the image acquisition unit 170, selects one of the image information, and outputs the selected image information. Note that which image information is to be selected is determined by, for example, acquiring information indicating which image information is to be selected from the control station 14, which information is input by the operator of the controller 16 or the like. In addition, when the image output unit 190 receives the image information output from the image acquisition unit 170, it may select and output the image information acquired by the image acquisition unit 170, and when it does not receive the image information, it may select and output the image information generated by the imaging control unit 120. When a plurality of image processing devices 100 are connected to each other by a daisy chain as shown as an example in FIG. 1(a) or the like, the image processing device 100 acquires image information output by the image processing device 100 at the previous stage of the image processing device 100 by the image output unit 190 or the like. In this case, the image output unit 190 outputs information including the image information and the image information selected by the image output unit 190 to an image processing device 100 downstream of the image processing device 100 or the image computing server 13.

[0043] According to the above configuration, the image processing system 1 can acquire a plurality of captured images captured at the same imaging time at the same timing. As a result, the image processing system 1 can generate a virtual viewpoint image based on a plurality of captured images captured at the same imaging time thus acquired at the same timing. Therefore, according to the image processing system 1, it is possible to suppress deterioration in the quality of the virtual viewpoint image.

[0044] The image output unit 190 may perform foreground / background separation on the captured image indicated by the selected image information, and may output information indicating the foreground region or background region resulting from the foreground / background separation in addition to the image information. With this configuration, the image computing server 13 does not need to perform foreground / background separation on all captured images indicated by the image information acquired by the image computing server 13. As a result, the processing load in each component of the image processing system 1 as a whole can be leveled. Foreground / background separation on the captured image indicated by the image information does not necessarily need to be performed when the image output unit 190 outputs the image information, and may be performed when the image capture control unit 120 generates the image information or when the image acquisition unit 170 acquires the image information, for example. A method of performing foreground / background separation on a captured image and separating the captured image region into a foreground region and a background region is well known, so a description of the foreground / background separation method will be omitted.

[0045] The time acquisition unit 152 acquires time information indicating a first time point and a second time point after the first time point, which are time points within a period during which the captured image is captured by the imaging device 11. The acquisition control unit 140 of each image processing device 100 outputs a synchronization signal until the image acquisition unit 170 acquires image information corresponding to the captured image captured between the first time point and the second time point. For example, the designation unit 150 may determine and designate the imaging time based on the first time point, the second time point, and the time interval between the synchronization signal. Note that the time information acquired by the time acquisition unit 152 is, for example, the first time point and the second time point input by the operator of the controller 16 or the like, acquired from the control station 14 as time information. By configuring in this way, it is possible to cause the image acquisition unit 170 to acquire image information of the captured image captured during a desired period.

[0046] With reference to FIG. 5, a method for an operator of the controller 16 to set a desired first time point and a second time point will be described. FIG. 5 is a diagram showing an example of scene information managed by a storage device according to the first embodiment. The storage device stores scene information corresponding to each of a plurality of imaging devices 11. In the scene information, for example, as shown in FIG. 5 as an example, a start time code indicating the time when imaging of each scene started and an end time code indicating the time when imaging of each scene ended are managed in association with the scene number of each scene. The control station 14 acquires the scene information by reading it from the storage device. All the image processing devices 100 obtain captured images by causing the corresponding imaging devices 11 to capture images in synchronization based on a reference synchronization signal. Therefore, the scene information corresponding to each captured image is all the same, except in special cases such as when test imaging is performed using only one imaging device 11. In normal cases, the control station 14 only needs to acquire one of a plurality of scene information corresponding to a plurality of imaging devices 11. However, in consideration of the above-mentioned special cases, it is preferable that the control station 14 reads and acquires all the scene information corresponding to the multiple image capture devices 11 from the storage device.

[0047] In the following description, it is assumed that all scene information is the same as the scene information shown as an example in FIG. 5. For example, the control station 14 displays the acquired scene information on a display device not shown in FIG. 1. The operator of the controller 16 selects the scene number of the scene for which a virtual viewpoint image is to be generated from the scene information displayed on the display device. The control station 14 outputs information indicating a start time code and an end time code corresponding to the selected scene number to each image processing device 100. For example, the time acquisition unit 152 included in each image processing device 100 acquires the information output from the control station 14, and acquires a time corresponding to the start time code as a first time point and an end time code as a second time point. In this manner, the first time point and the second time point are set. The control station 14 may output information indicating the selected scene number to each image processing device 100. In this case, for example, the time acquisition unit 152 included in each image processing device 100 acquires the information output from the control station 14. Also, the time acquisition unit 152 included in each image processing device 100 acquires the scene information by reading it from a storage device. The time point acquisition unit 152 possessed by each image processing device 100 may acquire the time point of the start time code corresponding to the selected scene number as the first time point and the time point of the end time code corresponding to the scene number as the second time point based on information indicating the selected scene number and the scene information.

[0048] The start time acquisition unit 154 acquires information indicating an acquisition start time at which the image acquisition unit 170 starts acquiring image information (hereinafter referred to as "acquisition start time information"). Here, the acquisition start time is a time in the future from the current time, and the acquisition control unit 140 included in each image processing device 100 waits to output a synchronization signal until the current time reaches the acquisition start time, and starts outputting the synchronization signal when the current time reaches the acquisition start time. By configuring in this way, it is possible to cause the image acquisition unit 170 to start acquiring image information from the time when the current time becomes a desired time. In addition, the acquisition control unit 140 may start outputting a synchronization signal when the current time reaches the acquisition start time, and output the synchronization signal until the image information of the captured image captured between the first time point and the second time point is acquired by the image acquisition unit 170. Note that the acquisition start time information acquired by the start time acquisition unit 154 is, for example, the acquisition start time input by the operator of the controller 16 or the like, acquired from the control station 14 as the acquisition start time information. With this configuration, it is possible to cause the image acquisition section 170 to acquire image information of images captured during a desired period from the time when the current time reaches the desired time.

[0049] A method for an operator of the controller 16 to set a desired acquisition start time will be described. For example, the control station 14 receives a reference time signal output by the time server 12 and acquires the reference time indicated by the reference time signal as the current time. The control station 14 may acquire the current time by the following method. For example, the control station 14 acquires time information from all the image processing devices 100, which is image information output from the image processing devices 100 and corresponds to the image information output from the imaging device 11. The control station 14 acquires the latest time among the times indicated by the time information acquired from all the image processing devices 100 as the current time. For example, the control station 14 displays the acquired current time on a display device not shown in FIG. 1. For example, the operator of the controller 16 inputs a time later than the current time displayed on the display device to the control station 14 as the acquisition start time. In the following, the description will be made assuming that the current time acquired by the control station 14 is 9:31:13.250 a.m. (hereinafter, referred to as "09:31:13:25.0"). In this case, the operator of the controller 16 inputs a time of 09:35:00:00.0 to the control station 14, for example, taking into consideration the period during which all image processing devices 100 will acquire their acquisition start times, so that acquisition of image information will start at 9:35:00. The control station 14 outputs information indicating this time to each image processing device 100. For example, the start time acquisition unit 154 included in each image processing device 100 acquires the information output from the control station 14.

[0050] Here, the first time point and the second time point may be past time points or future time points compared to the current time acquired by the current time acquisition unit 114. For example, when the first time point is a time point earlier than the current time, image information of the captured image captured at the first time point is stored in the storage device, so the acquisition control unit 140 immediately starts outputting a synchronization signal to cause the image acquisition unit 170 to start acquiring image information. Also, for example, when the first time point is a time point later than the current time, the acquisition control unit 140 waits to output a synchronization signal until the current time reaches the first time point, and after the current time reaches the first time point, starts outputting a synchronization signal to cause the image acquisition unit 170 to start acquiring image information. In this case, the time indicated by the acquisition start time information acquired by the start time acquisition unit 154 is the time of the first time point or a time point later than the first time point.

[0051] The operation of the image processing device 100 will be described with reference to Fig. 6 and Fig. 7. Fig. 6 is a flowchart showing an example of a processing flow of the image processing device 100 according to the first embodiment. With reference to Fig. 6, the operation when the image processing device 100 generates image information based on an image signal received from the imaging device 11 and stores the generated image information in a storage device will be described.

[0052] First, in S601, the reference signal acquisition unit 110 acquires a reference synchronization signal. When a plurality of image processing devices 100 are connected by a daisy chain, in S602, the reference signal output unit 112 outputs the reference synchronization signal acquired by the reference signal acquisition unit 110 to the downstream image processing device 100. Next, in S603, the imaging control unit 120 outputs a synchronization signal based on the reference synchronization signal in accordance with the imaging timing, and acquires an image signal. Next, in S604, the imaging control unit 120 generates image information and time information based on the acquired image signal. The imaging control unit 120 outputs the generated image information and time information to the imaging output unit 130 and the image output unit 190. Next, in S605, the imaging output unit 130 outputs the image information and time information generated by the imaging control unit 120 to the storage device.

[0053] Next, in S606, the image output unit 190 judges whether or not the image information acquired from the storage device is set to be selected and output. If it is judged in S606 that the image information acquired from the storage device is set to be selected and output, the image processing device 100 ends the process of the flowchart shown in FIG. 6, returns to S601, and repeats the process of the flowchart shown in FIG. 6. If it is judged in S606 that the image information acquired from the storage device is not set to be selected and output, the image output unit 190 outputs the image information output from the imaging control unit 120 to the image computing server in S607. Note that in this case and when a plurality of image processing devices 100 are connected by a daisy chain, the image output unit 190 outputs the image information output from the imaging control unit 120 to the subsequent image processing device 100. After S607, the image processing device 100 ends the process of the flowchart shown in FIG. 6, returns to S601, and repeats the process of the flowchart shown in FIG. 6.

[0054] With reference to FIG. 7, an operation of the image processing device 100 when acquiring image information from a storage device and outputting the acquired image information to the outside will be described. FIG. 7 is a flowchart showing an example of a processing flow of the image processing device 100 according to the first embodiment. First, in S701, the image output unit 190 is set to select and output image information acquired from the storage device. Next, in S702, the time point acquisition unit 152 acquires a first time point and a second time point. Next, in S703, the start time acquisition unit 154 acquires an acquisition start time. Next, in S704, the current time acquisition unit 114 acquires a current time. Next, in S710, the acquisition control unit 140 determines whether the current time has reached the acquisition start time. If it is determined in S710 that the current time has not reached the acquisition start time, the image processing device 100 returns to S704 and repeatedly executes the processes of S704 and S710 until it is determined in S710 that the current time has reached the acquisition start time. If it is determined in S710 that the current time has reached the acquisition start time, the image processing apparatus 100 executes the processes in S711 and thereafter.

[0055] In S711, the reference signal acquisition unit 110 acquires a reference synchronization signal. The reference signal acquisition unit 110 may also acquire a reference synchronization signal at a predetermined time interval during the period in which the processes from S701 to S710 are being executed. When a plurality of image processing devices 100 are connected by a daisy chain, in S712, the reference signal output unit 112 outputs the reference synchronization signal acquired by the reference signal acquisition unit 110 to the subsequent image processing device. After S712, in S713, the acquisition control unit 140 outputs a synchronization signal based on the reference synchronization signal to the image acquisition unit 170. After S713, in S714, the image acquisition unit 170 acquires image information from the storage device in synchronization with the synchronization signal. After S714, in S715, the image output unit 190 outputs the image information acquired by the image acquisition unit 170 to the image computing server. When a plurality of image processing devices 100 are connected in a daisy chain, the image output unit 190 outputs the image information acquired by the image acquisition unit 170 to the downstream image processing device 100. After S715, in S716, the acquisition control unit 140 determines whether or not all image information from the first time point to the second time point has been acquired by the image acquisition unit 170.

[0056] If it is determined in S716 that all image information from the first time point to the second time point has not been acquired by the image acquisition unit 170, the image processing device 100 returns to S711. In this case, the image processing device 100 repeatedly executes the processes from S711 to S716 until it is determined in S716 that all image information from the first time point to the second time point has been acquired by the image acquisition unit 170. If it is determined in S716 that all image information from the first time point to the second time point has been acquired by the image acquisition unit 170, the image output unit 190 is set to select and output the image information output from the imaging control unit in S717. After S717, the image processing device 100 ends the process of the flowchart shown in FIG. 7.

[0057] Here, as described above, the timing when the imaging control unit 120 generates image information and outputs it to the image output unit 190 and the timing when the image acquisition unit 170 acquires image information and outputs it to the image output unit 190 are both synchronized by a synchronization signal. Therefore, the timing when the image output unit 190 outputs image information to the image computing server (or the image processing device 100 at the subsequent stage) depends on the synchronization signal and does not depend on the selection at the image output unit 190. Therefore, even if the image information selected at the image output unit 190 is switched, no disturbance or the like occurs in the captured image acquired by the image computing server. In addition, during the period in which the processes of S704 and S710 in the flowchart shown in FIG. 7 are repeatedly executed, that is, during the period in which the current time has not yet reached the acquisition start time, the image output unit 190 may output information of a predetermined image such as a black image, for example.

[0058] As described above, the image processing system 1 described above can synchronously acquire information on a plurality of captured images captured at the same time from a storage device for a plurality of captured images captured synchronously by a plurality of imaging devices 11. As a result, the image processing system 1 can generate a virtual viewpoint image using a plurality of captured images thus synchronously acquired and captured at the same capturing time. Therefore, the image processing system 1 can suppress deterioration in the quality of the virtual viewpoint image.

[0059] <Second embodiment> In the first embodiment, a configuration has been described in which the image processing device 100 performs synchronization processing based on a reference synchronization signal from a time server 12 in the image processing system 1 in which a plurality of sensor systems 10 having imaging devices 11 are installed in a facility such as a stadium. The image processing system 1 according to the first embodiment enables all imaging devices 11 to perform imaging in high-precision synchronization, and image information showing images captured by each imaging device 11 is stored in a storage device in association with time information. Furthermore, image information of images captured during a specified period among the image information stored in the storage device is read out in accordance with a synchronization signal output at the same timing as the imaging timing of the imaging device 11 when the current time reaches the specified acquisition start time. The image information read out from the storage device is output from the image processing device 100 in place of image information generated in real time based on the image signal output from the imaging device 11. As a result, the image processing device 100 according to the first embodiment can output image information with the same imaging time for all image information stored in the storage device, similar to image information generated in real time based on the image signal output from the imaging device 11. As a result, the image processing system 1 according to the first embodiment can generate a highly accurate virtual viewpoint image using the image information stored in the storage device.

[0060] In the second embodiment, with reference to Figs. 8 to 13, for example, an image processing system 8 according to the first embodiment is removed from a facility, and a virtual viewpoint image can be generated from image information stored in a storage device in another facility different from the facility. Fig. 8 is a block diagram showing an example of the configuration of the image processing system 8 according to the second embodiment. Hereinafter, the same components as those in the image processing system 1 according to the first embodiment shown in Fig. 1 are denoted by the same reference numerals, and description thereof is omitted. In the image processing system 8, imaging by the imaging device 11 as in the first embodiment is not required. Therefore, the image processing system 8 has one sensor system 80 and multiple image processing devices 800 instead of the multiple sensor systems 10 having the imaging devices 11 shown in Fig. 1. The sensor system 80 has an imaging device 81 and an image processing device 800. Hereinafter, the number of image processing devices 800, including the image processing devices 800 of the sensor system 80, is described as being the same as the number of the sensor systems 10 shown in Fig. 1. That is, the image processing system 8 has 24 image processing devices 800 (800a-800x) including the image processing device 800 of the sensor system 80. The number of image processing devices 800 of the image processing system 8 may be less than 24 or more than 24, so long as it is plural. Furthermore, the image processing system 8 does not need the time server 12 as shown in Fig. 1, and therefore the image processing system 8 does not have the time server 12 of the image processing system 1 according to the first embodiment.

[0061] In the first embodiment, the imaging device 11 receives a synchronization signal and current time information from the image processing device 100. In contrast, the imaging device 81 operates with a time code and a synchronization signal generated by free running inside the imaging device 81 without receiving a synchronization signal and current time information from the image processing device 800. The imaging device 81 provides a reference synchronization signal and time code to the image processing device 800, and a synchronization signal and a time code indicating the imaging time of the captured image corresponding to the image signal are superimposed on the image signal output by the imaging device 81. The imaging device 81 is connected to the image processing device 800a at the front stage of the multiple image processing devices 800 connected by a daisy chain. That is, in the second embodiment, time synchronization control using the time server 12 is not required, and therefore the time server 12 as shown in FIG. 1 is not required.

[0062] The image processing device 800a receives the image signal output by the imaging device 81, redrives the image signal, and outputs the image signal to the image processing device 800b in the subsequent stage. The image processing device 800b receives the image signal output by the image processing device 800a, redrives the image signal, and outputs the image signal to the image processing device 800c in the subsequent stage. Thereafter, the same image signal transmission is repeated up to the image processing device 800x. As a result, all the image processing devices 800 can acquire the synchronization signal and the time code superimposed on the image signal output from the imaging device 81, and all the image processing devices 800 operate in synchronization based on the acquired synchronization signal and time code. Note that, in the second embodiment, as an example, a form in which the imaging device 81 is used as a device that outputs a synchronization signal and a time code that are the reference for all the image processing devices 800 to operate in synchronization will be described, but the device is not limited to the imaging device 81. The device does not have to be the imaging device 81 as long as it is a device that can generate and output a synchronization signal and a time code corresponding to a reference time such as the current time by a free-run operation.

[0063] Each image processing device 800 acquires image information synchronously by reading image information from a storage device based on the acquired synchronization signal and time code. Each image processing device 800 outputs image information read from a storage device. Specifically, image information read from a storage device by the image processing device 800a is transmitted to a subsequent image processing device 800b connected by a daisy chain via the network 19a. The image processing device 800b acquires image information transmitted from the image processing device 800a. The image information read from a storage device by the image processing device 800b is transmitted to a subsequent image processing device 800c connected by a daisy chain via the network 19b together with image information transmitted from the image processing device 800a. Thereafter, the same transmission is repeated up to the image processing device 800x. As a result, information combining the image information acquired from the storage device by each of the image processing devices 800a-800w is transmitted from the image processing device 800w via the network 19w to the image processing device 800x. Moreover, the image processing device 800x outputs the information transmitted from the image processing device 800w together with the image information acquired by the image processing device 800x from the storage device to the image computing server 13. Specifically, the information output from the image processing device 800x, that is, the information combining the image information acquired by each of the image processing devices 800a-800x from the storage device, is transmitted to the image computing server 13 via the switching hub 18.

[0064] The image processing device 800 will be described with reference to FIG. 9 to FIG. 13. The functional blocks of the image processing device 800 will be described with reference to FIG. 9. FIG. 9 is a block diagram showing an example of the configuration of the functional blocks in the image processing device 800 according to the second embodiment. The image processing device 800 has an image signal acquisition unit 820, an image signal output unit 822, a reference signal acquisition unit 810, a current time acquisition unit 814, an acquisition control unit 140, a designation unit 150, an image acquisition unit 170, and an image output unit 890. In addition to the above-mentioned configuration, the image processing device 800 may have a time point acquisition unit 152, a start time acquisition unit 154, a start signal acquisition unit 850, or a start signal output unit 852. Hereinafter, the image processing device 800 will be described as having the time point acquisition unit 152, the start time acquisition unit 154, the start signal acquisition unit 850, and the start signal output unit 852. Hereinafter, the same reference numerals will be used to designate the same configuration as the image processing device 100 according to the first embodiment shown in FIG. 3, and the description thereof will be omitted. The processing of each unit of the image processing device 800 is performed by hardware such as an ASIC or an FPGA built into the image processing device 800. Moreover, the processing may be performed by software using the hardware shown as an example in FIG.

[0065] The image signal acquisition unit 820 acquires an image signal output by the imaging device 81 or the image processing device 800 at the previous stage connected by a daisy chain. As described above, a synchronization signal and a time code are superimposed on the acquired image signal. The image signal output unit 822 redrives the image signal acquired by the image signal acquisition unit 820 and outputs the image signal to the image processing device 800 at the subsequent stage connected by the daisy chain. The reference signal acquisition unit 810 extracts a synchronization signal from the image signal acquired by the image signal acquisition unit 820 and acquires the extracted synchronization signal as a reference synchronization signal. The current time acquisition unit 814 extracts a time code from the image signal acquired by the image signal acquisition unit 820 and acquires the extracted time code as current time information.

[0066] The acquisition control unit 140 controls the timing at which the image acquisition unit 170 acquires image information from the storage device. Specifically, the acquisition control unit 140 outputs a reference synchronization signal acquired by the reference signal acquisition unit 810 as a synchronization signal, thereby synchronizing the synchronization signals between the multiple image processing devices 800. The image output unit 890 outputs the image information acquired by the image acquisition unit 170. When the multiple image processing devices 800 are connected to each other by a daisy chain, the image processing device 800 acquires image information output by the image processing device 800 at the previous stage of the image processing device 800 by the image output unit 890 or the like. In this case, the image output unit 890 outputs information including the image information and the image information acquired by the image acquisition unit 170 as image information to the image processing device 800 at the subsequent stage of the image processing device 800 or the image computing server 13.

[0067] According to the above configuration, the image processing system 8 can acquire multiple captured images captured at the same imaging time at the same timing from a storage device in which image information and time information of captured images captured synchronously by multiple imaging devices 11 are stored in advance. As a result, the image processing system 8 can generate a virtual viewpoint image based on multiple captured images captured at the same imaging time thus acquired at the same timing. Therefore, the image processing system 8 can suppress deterioration in the quality of the virtual viewpoint image.

[0068] The start signal acquisition unit 850 and the start signal output unit 852 will be described later. Note that the designation unit 150, the time point acquisition unit 152, and the start time acquisition unit 154 are similar to those in the image processing device 100 according to the first embodiment, and therefore their explanations will be omitted. Also, the method of setting the first and second time points, and the method of setting the acquisition start time are similar to those in the first embodiment, and therefore their explanations will be omitted.

[0069] The control station 14 according to the first embodiment receives the reference time signal output by the time server 12 and acquires the reference time indicated by the reference time signal as the current time. In contrast, the control station 14 according to the second embodiment acquires the current time, for example, by the following method. First, the control station 14 acquires image signals output from all image processing devices 800 and extracts the time code superimposed on each image signal. The control station 14 selects the time code indicating the latest time from all the extracted time codes and acquires the selected time as the current time. In consideration of the delay in re-driving, a time code may be extracted using only the image signal output from the image processing device 800x located at the last stage among the multiple image processing devices 800 connected by a daisy chain, and the time of the selected time code may be acquired as the current time. In this case, the control station 14 may acquire only the image signal output from the image processing device 800x located at the last stage.

[0070] The operation of the image processing device 800 will be described with reference to FIG. 10. FIG. 10 is a flowchart showing an example of a processing flow of the image processing device 800 according to the second embodiment. Hereinafter, the same processes as those of the image processing device 100 according to the first embodiment shown in FIG. 7 are denoted by the same reference numerals, and the description thereof will be omitted. First, the image processing device 800 executes the processes from S701 to S703. Next, in S1001, the image signal acquisition unit 820 acquires an image signal. Next, in S1002, the image signal output unit 822 outputs the image signal acquired by the image signal acquisition unit 820 to the subsequent image processing device 800 connected by a daisy chain. Next, in S1003, the current time acquisition unit 814 extracts a time code from the image signal to acquire the current time. Next, the image processing device 800 executes the process of S710. If it is determined in S710 that the current time has not reached the acquisition start time, the image processing device 800 returns to S1001 and repeats the processes from S1001 to S1003 until it is determined in S710 that the current time has reached the acquisition start time. If it is determined in S710 that the current time has reached the acquisition start time, the image processing device 800 executes the processes from S1011 onwards.

[0071] In S1011, the image signal acquisition unit 820 acquires an image signal. Next, in S1012, the image signal output unit 822 outputs the image signal acquired by the image signal acquisition unit 820 to the downstream image processing device 800 connected by a daisy chain. Next, in S1013, the reference signal acquisition unit 810 extracts a synchronization signal from the image signal to acquire a reference synchronization signal. After S1013, the image processing device 800 executes the processes from S713 to S716. If it is determined in S716 that the image acquisition unit 170 has not acquired all the image information from the first time point to the second time point, the image processing device 800 returns to S1011. In this case, the image processing device 800 repeatedly executes the processes from S1011 to S1013 and from S713 to S716 until it is determined in S716 that the image acquisition unit 170 has acquired all the image information from the first time point to the second time point. If it is determined in S716 that all image information from the first time point to the second time point has been acquired by the image acquisition unit 170, the image processing device 800 executes the process of S717. After S717, the image processing device 800 ends the process of the flowchart shown in Fig. 10. Note that during the period in which the processes from S1001 to S1003 in Fig. 10 and the process of S710 are repeatedly executed, that is, during the period in which the current time has not yet reached the acquisition start time, the image output unit 890 may output information of a predetermined image, such as a black image.

[0072] According to the above configuration, the image processing system 8 can synchronously acquire information on a plurality of captured images captured at the same time from the storage device for a plurality of captured images captured synchronously by a plurality of imaging devices 11. As a result, the image processing system 8 can generate a virtual viewpoint image using a plurality of captured images thus synchronously acquired and captured at the same capturing time. Therefore, the image processing system 8 can suppress deterioration in the quality of the virtual viewpoint image.

[0073] Each image processing device 800 according to the second embodiment described above starts acquiring image information stored in a storage device when the current time based on the time code superimposed on the image signal output from the imaging device 81 reaches a preset acquisition start time. However, the timing to start acquiring image information stored in a storage device may be specified by a method other than specifying the acquisition start time.

[0074] Specifically, for example, each image processing device 800 may start acquiring image information stored in a storage device, triggered by receiving an acquisition start command issued from the control station 14 or the like via the network 14a or the like. The acquisition start command is issued from the control station 14, for example, when a predetermined input is made by the operator of the controller 16, or when the current time reaches a preset time. More specifically, for example, the start signal acquiring unit 850 acquires, as an acquisition start signal, an acquisition start command for instructing the image acquiring unit 170 to start acquiring image information. In addition to the acquisition start signal, the start signal acquiring unit 850 may also acquire, as an acquisition end signal, an acquisition end command for instructing the image acquiring unit 170 to end acquiring image information.

[0075] Furthermore, when a plurality of image processing devices 800 are connected by a daisy chain, the image processing device 800a at the front stage receives an acquisition start command issued from the control station 14 and outputs the acquisition start command to the image processing device 800b at the rear stage. Similarly, each image processing device 800 located at a stage rear of the image processing device 800a receives an acquisition start command output from the image processing device 800 at the front stage and outputs the acquisition start command to the image processing device 800 at the rear stage. For example, the start signal output unit 852 outputs an acquisition start signal acquired by the start signal acquisition unit 850, i.e., an acquisition start command, to the image processing device 800 at the rear stage connected by the daisy chain. When the start signal acquisition unit 850 acquires an acquisition end command as an acquisition end signal, the start signal output unit 852 outputs an acquisition end signal acquired by the start signal acquisition unit 850 to the image processing device 800 at the rear stage connected by the daisy chain. With this configuration, all image processing devices 800 can acquire image information from the storage device in a highly synchronized manner.

[0076] With reference to FIG. 11, the operation of the image processing device 800 when the image processing device 800 starts acquiring image information stored in a storage device using an acquisition start command as a trigger will be described. FIG. 11 is a flowchart showing an example of a processing flow of the image processing device 800 according to the second embodiment in this case. Hereinafter, the same processes as those of the image processing device 800 according to the second embodiment shown in FIG. 10 are denoted by the same reference numerals and will not be described. First, the image processing device 800 executes the processes of S701 and S702, as well as S1001 and S1002. Next, in S1110, the start signal acquisition unit 850 determines whether or not an acquisition start signal has been acquired. If it is determined in S1110 that the acquisition start signal has not been acquired, the image processing device 800 returns to S1001 and repeatedly executes the processes of S1001, S1002, and S1110 until it is determined in S1110 that the acquisition start signal has been acquired. If it is determined in S1110 that the acquisition start signal has been acquired, the image processing device 800 executes the processes from S1011 to S1013 and from S713 to S715.

[0077] After S715, in S1120, the start signal acquisition unit 850 judges whether or not an acquisition end signal has been acquired. If it is judged in S1120 that an acquisition end signal has not been acquired, the image processing device 800 executes the process of S716. If it is judged in S716 that all image information from the first time point to the second time point has not been acquired by the image acquisition unit 170, the image processing device 800 returns to S1011. In this case, the image processing device 800 repeatedly executes the processes from S1011 to S716 until it is judged in S716 that all image information from the first time point to the second time point has been acquired or it is judged in S1120 that an acquisition end signal has been acquired. If it is judged in S1120 that an acquisition end signal has been acquired, or if it is judged in S716 that all image information from the first time point to the second time point has been acquired, the image processing device 800 executes the process of S1121. Specifically, in this case, in S1121, the start signal output unit 852 outputs the acquisition end signal acquired by the start signal acquisition unit 850 to the downstream image processing device 800 connected by the daisy chain. After S1121, the image processing device 800 executes the process of S717 and ends the process of the flowchart shown in FIG.

[0078] According to the above configuration, the image processing system 8 can synchronously acquire information on a plurality of captured images captured at the same time from the storage device for a plurality of captured images captured synchronously by a plurality of imaging devices 11. As a result, the image processing system 8 can generate a virtual viewpoint image using a plurality of captured images thus synchronously acquired and captured at the same capturing time. Therefore, the image processing system 8 can suppress deterioration in the quality of the virtual viewpoint image.

[0079] The process of the flowchart shown in FIG. 11 is a form in which an acquisition start signal is output to the downstream image processing device 800 to cause the downstream image processing device 800 to start acquiring image information. The method of causing the downstream image processing device 800 to start acquiring image information is not limited to the method of outputting an acquisition start signal to the downstream image processing device 800. For example, the image processing device 800a outputs an image signal obtained by processing an image signal received from the imaging device 81 to the image processing device 800b from the time when the acquisition start signal is acquired until the acquisition end signal is acquired, or until all image information from the first time point to the second time point is acquired. Here, the processing of the image signal received from the imaging device 81 means, for example, setting a value of acquisition flag information indicating the start or stop of acquisition in a predetermined area of ​​1 bit or more of the image signal.

[0080] Specifically, for example, when the image processing device 800a acquires an acquisition start signal, it sets "1" indicating the start of acquisition in the corresponding area of ​​the image signal received from the imaging device 81 thereafter. Also, when the image processing device 800a acquires an acquisition end signal or completes acquisition of all image information from the first time point to the second time point, it sets "0" indicating the end of acquisition in the corresponding area of ​​the image signal received from the imaging device 81 thereafter. The image processing device 800a outputs the image signal after the setting to the image processing device 800b. The image processing devices 800b-800x located at the rear stage of the image processing device 800a read image information from the storage device in synchronization with the synchronization signal if the value of the acquisition flag information set in the corresponding area is "1" indicating the start of acquisition. Also, if the value of the acquisition flag information is "0" indicating the start of acquisition, it stops or ends the reading of image information from the storage device. With this configuration, all the image processing devices 800 can acquire image information from the storage device in synchronization with high accuracy.

[0081] With reference to Figs. 12 and 13, the operation of the image processing device 800 when the image processing device 800 starts acquiring image information stored in the storage device based on the value of the acquisition flag information set in the image signal will be described. Fig. 12 is a flowchart showing an example of a processing flow of the image processing device 800a located at the front stage among the image processing device 800 according to the second embodiment in this case. Also, Fig. 13 is a flowchart showing an example of a processing flow of the image processing devices 800b-800x located at a stage subsequent to the image processing device 800a located at the front stage among the image processing device 800 according to the second embodiment in this case. Hereinafter, the same processes as those of the image processing device 800 according to the second embodiment shown in Fig. 10 or Fig. 11 are denoted by the same reference numerals and will not be described. First, the operation of the image processing device 800a will be described with reference to Fig. 12.

[0082] First, the image processing device 800a executes the processes of S701 and S702, S1001 and S1002, and S1110. If it is determined in S1110 that the acquisition start signal has not been acquired, the image processing device 800a returns to S1001 and repeatedly executes the processes of S1001, S1002, and S1110 until it is determined in S1110 that the acquisition start signal has been acquired. If it is determined in S1110 that the acquisition start signal has been acquired, the image processing device 800a acquires an image signal from the imaging device 81 in S1011. After S1011, in S1212, the image output unit 890 sets the value of the acquisition flag information in the image signal acquired from the imaging device 81 to "1" and outputs the image signal after the setting to the subsequent image processing device 800b. After S1212, the image processing device 800a executes the processes of S1013, S713 to S715, and S1120.

[0083] When it is determined in S1120 that the acquisition end signal has not been acquired, the image processing device 800a executes the process of S716. When it is determined in S716 that all image information from the first time point to the second time point has not been acquired by the image acquisition unit 170, the image processing device 800a returns to S1011. In this case, the image processing device 800a repeatedly executes the processes from S1011 to S716 until it is determined in S716 that all image information from the first time point to the second time point has been acquired, or until it is determined in S1120 that the acquisition end signal has been acquired. When it is determined in S1120 that the acquisition end signal has been acquired, or when it is determined in S716 that all image information from the first time point to the second time point has been acquired, the image processing device 800a executes the process of S717 and ends the process of the flowchart shown in FIG. 11. Thereafter, until the image processing device 800a acquires the next acquisition start signal, when the image processing device 800a acquires an image signal output by the imaging device 81, the image processing device 800a sets the value of the acquisition flag information of the acquired image signal to "0" and outputs the image signal after the setting to the downstream image processing device 800b.

[0084] Next, the operation of each of the image processing devices 800b-800x located at a stage subsequent to the image processing device 800a will be described with reference to FIG. 13. In this description, each of the image processing devices 800b-800x will be simply referred to as the image processing device 800. First, the image processing device 800 executes the processes of S701, S702, and S1001. After S1001, in S1310, the image signal acquisition unit 820 determines whether or not the value of the acquisition flag information of the image signal is set to "1". If it is determined in S1310 that the value of the acquisition flag information of the image signal is not set to "1", the image processing device 800 executes the process of S1311. Specifically, in this case, in S1311, the image signal output unit 822 sets the value of the acquisition flag information of the image signal acquired by the image signal acquisition unit 820 to "0" and outputs the set image signal to the downstream image processing device 800.

[0085] After S1311, the image processing device 800 returns to S1001 and repeatedly executes the processes of S1001, S1310, and S1311 until it is determined in S1310 that the value of the acquisition flag information of the image signal is set to "1". If it is determined in S1310 that the value of the acquisition flag information of the image signal is set to "1", the image processing device 800 executes the processes of S1012 and S1012, S713 to S715, and S1120. After S715, the image processing device 800 executes the process of S716. If it is determined in S716 that all image information from the first time point to the second time point has not been acquired by the image acquisition unit 170, the image processing device 800 returns to S1011. In this case, the image processing device 800 repeatedly executes the processes of S1011 to S716 in FIG. 13 until it is determined in S716 that all image information from the first time point to the second time point has been acquired. If it is determined in S716 that all image information from the first time point to the second time point has been acquired, the image processing device 800 executes the process of S717 and ends the process of the flowchart shown in FIG.

[0086] According to the above configuration, the image processing system 8 can synchronously acquire information on a plurality of captured images captured at the same time from the storage device for a plurality of captured images captured synchronously by a plurality of imaging devices 11. As a result, the image processing system 8 can generate a virtual viewpoint image using a plurality of captured images thus synchronously acquired and captured at the same capturing time. Therefore, the image processing system 8 can suppress deterioration in the quality of the virtual viewpoint image.

[0087] <Other embodiments> The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) for implementing one or more functions.

[0088] It should be noted that, within the scope of the present disclosure, the embodiments may be freely combined, any component of each embodiment may be modified, or any component of each embodiment may be omitted. [Explanation of symbols]

[0089] 100 Image processing device 110 Reference signal acquisition section 112 Reference signal output section 114 Current time acquisition section 120 Imaging control section 130 Image output section 140 Acquisition control section 150 Specified section 154 Start time acquisition part 152 Time Acquisition Department 170 Image acquisition unit 190 Image output section

Claims

1. An image processing system including a plurality of image processing devices, The image processing device includes: an image acquiring means for acquiring, from a storage means storing image information indicating an image captured by the imaging device based on a synchronization signal that specifies a timing of imaging in the imaging device and is synchronized among the plurality of image processing devices, the image information being associated with time information indicating an imaging time at which the image was captured, based on the time information; an acquisition control means for controlling a timing at which the image acquisition means acquires the image information from the storage means; a designation means for designating the image capture time of the image information that the image acquisition means acquires from the storage means; an output means for selecting and outputting either the image information acquired by the image acquisition means or the image information not stored in the storage means; having the acquisition control means outputs a synchronization signal synchronized with a synchronization signal that specifies a timing of imaging as a synchronization signal synchronized among the plurality of image processing devices that specifies the timing of acquisition; The designation means designates the same imaging time in the plurality of image processing devices, The image acquisition means acquires the image information corresponding to the imaging time designated by the designation means from the storage means in a synchronized manner among the plurality of image processing devices based on a synchronization signal that specifies the acquisition timing output from the acquisition control means. An image processing system comprising:

2. the plurality of image processing devices are connected in a daisy chain; 2. The image processing system according to claim 1, wherein the image information is output to another of the image processing devices via a daisy chain.

3. The image processing device includes: A signal acquiring means for acquiring a reference synchronization signal; Further comprising: The acquisition control means outputs a synchronization signal that defines the acquisition timing based on the reference synchronization signal, thereby synchronizing the synchronization signal that defines the acquisition timing among the plurality of image processing devices.

3. The image processing system according to claim 1, wherein:

4. a signal output device that outputs the reference synchronization signal to the plurality of image processing devices; Further comprising: The signal acquiring means acquires the reference synchronization signal output by the signal output device.

4. The image processing system according to claim 3,

5. A plurality of the imaging devices outputting the reference synchronization signal; Further comprising: the imaging device outputs the reference synchronization signal to the corresponding image processing device; The signal acquisition means acquires the reference synchronization signal output from the corresponding imaging device.

4. The image processing system according to claim 3,

6. a signal output device for outputting the reference synchronization signal; Further comprising: the plurality of image processing devices are connected in a daisy chain; The image processing device includes: a signal output means for outputting the reference synchronization signal acquired by the signal acquisition means; Further comprising: a signal acquisition means included in the image processing device at the front stage among the plurality of image processing devices connected in the daisy chain acquires the reference synchronization signal outputted by the signal output device; a signal acquiring means included in an image processing device other than the image processing device at the forefront stage among the plurality of image processing devices connected by the daisy chain acquires the reference synchronization signal outputted by a signal output means included in the image processing device at the front stage connected by the daisy chain; 4. The image processing system according to claim 3,

7. The signal output device is a time server that outputs information for synchronizing the internal clock of the image processing device.

7. The image processing system according to claim 6,

8. The signal output device is an imaging device connected to the image processing device at the front stage.

7. The image processing system according to claim 6,

9. The image acquisition means acquires the image information and time information corresponding to the image information.

9. The image processing system according to claim 1, wherein:

10. The output means, when set to select and output the image information acquired from the storage means, selects and outputs the image information acquired by the image acquisition means.

10. The image processing system according to claim 1, wherein:

11. The output means, when the image acquisition means acquires the image information from the storage means, selects and outputs the image information acquired by the image acquisition means.

10. The image processing system according to claim 1, wherein:

12. A program for causing a computer to operate as each of the means of the image processing device included in the image processing system according to any one of claims 1 to 11.

13. An image processing method in an image processing system having a plurality of image processing devices, comprising: The image processing method in the image processing device includes: an image acquiring step of acquiring, from a storage means storing image information indicating an image captured by the imaging device based on a synchronization signal that specifies a timing of imaging in the imaging device and is synchronized among the plurality of image processing devices, the image information being associated with time information indicating an imaging time at which the image was captured, based on the time information; an acquisition control step of controlling a timing at which the image information is acquired from the storage means in the image acquisition step; a designation step of designating the image capturing time of the image information acquired from the storage means in the image acquisition step; an output step of selecting and outputting either the image information acquired in the image acquisition step or the image information not stored in the storage means; Including, In the acquisition control step, a synchronization signal that is synchronized with a synchronization signal that specifies a timing of imaging is output as a synchronization signal synchronized among the plurality of image processing devices that specifies the timing of acquisition, In the designation step, the same imaging time is designated in the plurality of image processing devices, In the image acquisition step, the image information corresponding to the imaging time designated in the designation step is acquired from the storage means in a synchronized manner by the plurality of image processing devices based on a synchronization signal that specifies the acquisition timing outputted in the acquisition control step. An image processing method comprising:

Citation Information

Patent Citations

  • Video conference system and video conference method

    JP2009239762A

  • Image processor and program

    JP2013225930A

  • Image processing system

    JP2015165628A

  • Image monitoring system

    JP2015185896A

  • Imaging apparatus

    JP2017163295A