Control device, imaging system, control method, and program

Abstract

The present invention provides a control device capable of acquiring high-quality visual effects regardless of the number of imaging devices. [Solution] A control device (104) for an imaging system (100) that captures background images displayed on a display device (103) using a plurality of imaging devices (101A to 101C), including a first imaging device and a second imaging device, comprising: a control means (301) that causes a first background image corresponding to the first position and orientation of the first imaging device and a second background image corresponding to the second position and orientation of the second imaging device to be displayed on the display device; and an acquisition means (313) that, while the first background image is being displayed on the display device, acquires a start instruction from the user to start displaying the second background image on the display device, wherein the control means causes the display device to start displaying the second background image in response to the start instruction.

Description

Technical Field

[0001] The present invention relates to a control device, an imaging system, a control method, and a program.

Background Art

[0002] Patent Document 1 discloses a method (in-camera VFX) for obtaining VFX (Visual Effects) video without synthesizing a background image and a live-action video by imaging a subject against an image corresponding to the position and orientation of a camera (imaging device).

[0003] In in-camera VFX, when shooting the same scene using multiple cameras, if the background image is changed according to the position and orientation of one camera, the position and orientation of the other cameras will not match the background anymore. Patent Document 2 discloses a method for shooting a video with a background video corresponding to multiple cameras by displaying multiple videos corresponding to the position and orientation of each camera on the background in a time-sharing manner and matching the display timing with the exposure timing of the camera corresponding to that video.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in order to increase the number of cameras in the method disclosed in Patent Document 2, it is necessary to shorten the exposure time of the cameras and the display time of the background image as the number of cameras increases. Also, if the exposure time of the cameras is shortened, it is difficult to smooth the captured video.

[0006] Therefore, the present invention aims to provide a control device that can acquire visually pleasing images regardless of the number of imaging devices. [Means for solving the problem]

[0007] One aspect of the present invention is a control device for an imaging system that captures background images displayed on a display device using a plurality of imaging devices, including a first imaging device and a second imaging device, and comprises control means for causing a first background image corresponding to a first position and orientation of the first imaging device and a second background image corresponding to a second position and orientation of the second imaging device to be displayed on the display device, and acquisition means for obtaining a start instruction from a user to start displaying the second background image on the display device while the first background image is displayed on the display device, wherein the control means causes the display device to start displaying the second background image in response to the start instruction.

[0008] Other objects and features of the present invention are described in the following examples. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a control device that can acquire visually pleasing images regardless of the number of imaging devices. [Brief explanation of the drawing]

[0010] [Figure 1] These are schematic diagrams of the imaging systems according to each embodiment. [Figure 2] This is a block diagram of the imaging device according to each embodiment. [Figure 3] This is a block diagram of the main camera selection device according to each embodiment. [Figure 4] This is a flowchart illustrating the operation of the main camera selection device according to Example 1. [Figure 5] This is a schematic diagram showing an example of the display of the display device according to Example 1. [Figure 6] This is a flowchart illustrating the operation of the main camera selection device according to Example 2. [Figure 7] This figure shows examples of display patterns and exposure patterns according to Example 2. [Figure 8] This figure shows examples of display patterns and exposure patterns according to Example 2. [Figure 9] This is a flowchart illustrating the operation of the main camera selection device according to Example 3. [Figure 10] This figure shows an example of information regarding the switching order of the main camera according to Example 3. [Figure 11] This figure shows examples of display patterns and exposure patterns according to Example 3. [Modes for carrying out the invention]

[0011] The embodiments of the present invention will be described in detail below with reference to the drawings. Note that each embodiment describes multiple features, but not all of them are essential to the present invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are given the same reference numeral, and redundant descriptions are omitted.

[0012] (Example of a virtual studio configuration) First, with reference to Figure 1, an example configuration of the imaging system (virtual studio) 100 in each embodiment will be described. Figure 1 is a schematic diagram of the imaging system 100. In the imaging system 100, multiple imaging devices (cameras 101A to 101C) capture in-camera VFX images by capturing real subjects with an image displayed on a large display device 103, also known as an LED wall, as the background. Any one of the cameras 101A to 101C in Figure 1 is the first imaging device, and any one of the other two cameras is the second imaging device. Note that the imaging system 100 is not limited to a configuration with three imaging devices, but may have two or four or more imaging devices.

[0013] The position and orientation detection devices 102A to 102C are respectively attached to the cameras 101A to 101C, and detect the position and orientation (the first position and orientation, the second position and orientation) of the corresponding camera. Further, the position and orientation detection devices 102A to 102C output information on the position and orientation of the cameras 101A to 101C to the main camera selection device 104. Note that since the basic configuration of the position and orientation detection devices 102A to 102C is known as a camera tracking system, the detailed description thereof is omitted. For example, the position and orientation detection devices 102A to 102C include cameras for position and orientation detection, image a marker whose absolute position is known but not shown in FIG. 1, and detect the position and orientation of the corresponding camera based on the position of the marker included in the image captured by the cameras for position and orientation detection. Note that the positions and orientations of the cameras 101A to 101C can be detected by any known method.

[0014] The main camera selection device 104 selects a main camera from the cameras 101A to 101C by a method described later. Further, the main camera selection device 104 outputs information on the position and orientation of the main camera to the background image generation device 105. Also, the main camera selection device 104 outputs the image captured by the main camera to the display device (image display means) 107. In addition, the main camera selection device 104 controls the start and end of the imaging of the in-camera VFX video (moving image). The main camera selection device 104 constitutes a control device of the imaging system 100 that images the respective background images displayed on the display device 103 using the cameras 101A to 101C, together with the background image generation device 105 and the display control device 106 described later.

[0015] The background image generation device 105 renders a three-dimensional model of a preset virtual space according to the position and orientation of the main camera selected by the main camera selection device 104, and generates a CG background image at a predetermined frame rate. Note that when the imaging direction of the main camera is not directly facing the display device 103, the background image generation device 105 applies coordinate conversion (deformation processing) necessary for displaying the background image on the display device 103. The background image generation device 105 outputs the generated background image to the display control device 106.

[0016] The display control device 106 causes the background image to be displayed on the display device 103 arranged in the real space in accordance with the imaging timing of the main camera. When the display device 103 is composed of a plurality of display panels, the display control device 106 divides the background image according to each display panel and then displays it.

[0017] The display device 107 is, for example, a monitor, and displays the image captured by the main camera by the operation of the main camera selection device 104 described later. The display device 107 also displays a UI for assisting various instruction inputs to the main camera selection device 104 by the user.

[0018] In addition, a synchronization signal is supplied from the main camera selection device 104 to the cameras 101A to 101C, the position and orientation detection devices 102A to 102C, the background image generation device 105, and the display control device 106. By controlling the operation timing of each device according to the reference clock, synchronization between the exposure timing of the cameras 101A to 101C and the display timing of the display device 103 is realized. Note that since the technology for synchronizing the operations between devices based on the reference clock is known as, for example, genlock (Generator Locking), the detailed description thereof is omitted.

[0019] (Configuration example of camera 101A) Next, a configuration example of the camera 101A will be described with reference to FIG. 2. Note that the cameras 101B and 101C may have the same configuration as the camera 101A. FIG. 2 is a block diagram showing a functional configuration example of the camera 101A.

[0020] The control unit 201 is, for example, a CPU (Central Processing Unit). The control unit 201 reads the control programs for each block of the camera 101A from the ROM (Read Only Memory) 202 (described later), loads them into the RAM (Random Access Memory) 203 (described later), and executes them. In this way, the control unit 201 controls the operation of each block of the camera 101A. The control unit 201 also controls each block of the camera 101A based on synchronization signals supplied via the communication unit 209. This enables synchronization of operation with external devices.

[0021] ROM202 is an electrically erasable and recordable non-volatile memory that stores the operation programs for each block of the camera 101A, as well as parameters necessary for the operation of each block. RAM203 is a rewritable volatile memory that is used for deploying programs executed by the control unit 201, etc., and for temporary storage of data generated by the operation of each block of the camera 101A.

[0022] The optical system 204 consists of a lens group including a zoom lens and a focus lens, and forms an image of the subject on the imaging surface of the imaging unit 205, which will be described later. The imaging unit 205 is an image sensor such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide-Semiconductor) sensor. The imaging unit 205 performs photoelectric conversion of the optical image formed on the imaging surface by the optical system 204 and outputs the resulting analog image signal to the A / D conversion unit 206.

[0023] The A / D conversion unit 206 converts the input analog image signal into digital image data. The digital image data output from the A / D conversion unit 206 is temporarily stored in the RAM 203.

[0024] The image processing unit 207 performs various image processing on the image data stored in the RAM 203. Specifically, it applies various image processing techniques to develop, display, and record digital image data, such as demosaicing, white balance correction, and gamma processing. It also applies various image processing techniques to improve the image quality of digital image data, such as spatial filtering and noise suppression by combining multiple image data.

[0025] The recording unit 208 records data, including image data, on its built-in recording medium. The communication unit 209 connects to external devices via wired or wireless connection and communicates image data and synchronization signals. The display unit 210 includes a display device such as an LCD (Liquid Crystal Display) and displays images stored in the RAM 203 and images recorded in the recording unit 208 on the display device. The display unit 210 also displays an operation user interface for receiving instructions from the user. The instruction input unit 211 is an input interface including various physical operating elements such as a touch panel or shutter button, and accepts instructions from the user.

[0026] (Example configuration of the main camera selection device 104) Next, the main camera selection device 104 will be described with reference to Figure 3. Figure 3 is a block diagram showing an example of the functional configuration of the main camera selection device 104. The main camera selection device 104 can be implemented, for example, using computer equipment.

[0027] The control unit 301 is, for example, a CPU, and it realizes the functions of the main camera selection device 104 by loading one or more application programs stored in the ROM 311 into the RAM 312 and executing them. The control unit 301 also controls the operating timing of the main camera selection device 104 according to the synchronization signal supplied from the clock generation unit 314. Furthermore, as will be described later, the control unit 301, together with the display control device 106, constitutes a control means for displaying a first background image corresponding to the first position and orientation of the first imaging device and a second background image corresponding to the second position and orientation of the second imaging device on the display device 103.

[0028] The first I / F302 to the ninth I / F310 are communication interfaces for connecting external devices. In each embodiment, camera 101A is connected to the first I / F302, camera 101B to the second I / F303, and camera 101C to the third I / F304. In each embodiment, position and attitude detection device 102A is connected to the fourth I / F305, position and attitude detection device 102B to the fifth I / F306, and position and attitude detection device 102C to the sixth I / F307. In each embodiment, background image generation device 105 is connected to the seventh I / F308, display control device 106 to the eighth I / F309, and display device 107 to the ninth I / F310. The first I / F302 to the ninth I / F310 shall comply with standards corresponding to the external devices to be connected and the types of signals to be communicated. For convenience, the main camera selection device 104 and the external device are shown connected through a single interface, but they may be connected using multiple interfaces.

[0029] The control unit 301 reads the control programs for each block of the main camera selection device 104 from the ROM 311 (described later), expands them into the RAM 312 (described later), and executes them. In this way, the control unit 301 controls the operation of each block of the main camera selection device 104. The ROM 311 stores some of the programs executed by the control unit 301 (BIOS, bootstrap loader, firmware), setting values ​​for the main camera selection device 104, etc. The RAM 312 is a rewritable volatile memory and is used for expanding programs executed by the control unit 301, etc., and for temporary storage of data generated by the operation of each block of the main camera selection device 104.

[0030] The instruction input unit (acquisition means) 313 has multiple user-operable input devices such as a keyboard, mouse, or touchpad. For example, when the first background image is displayed on the display device 103, the instruction input unit 313 acquires (receives) a start instruction from the user to start displaying the second background image on the display device 103. The clock generation unit 314 generates a synchronization signal (clock) to synchronize the operation of the main camera selection device 104 and the external devices, which are cameras 101A to 101C, position and orientation detection devices 102A to 102C, background image generation device 105, and display control device 106. The main camera selection device 104 selects the main camera from among the cameras 101A to 101C and outputs the image captured by the main camera and the position and orientation information of the main camera to the various external devices.

[0031] The following describes the switching operation of the main camera by the main camera selection device 104 in each embodiment. [Examples]

[0032] First, with reference to Figure 4, the switching operation of the main camera by the main camera selection device 104 in Embodiment 1 of the present invention will be described. Figure 4 is a flowchart of the operation of the main camera selection device 104 in this embodiment.

[0033] The operation shown in Figure 4 is performed from the start of in-camera VFX video (video) capture by cameras 101A to 101C. Here, the video may be for recording or for display. The video for display may be, for example, a video for live view display on the display unit 210 of camera 101A or the display device 107 when it is in imaging standby mode. The start of in-camera VFX video capture is performed, for example, by the user giving an instruction to the main camera selection device 104 to start capturing via the instruction input unit 313. Furthermore, in the following description, the operations performed by the main camera selection device 104 are actually realized by the control unit 301 executing an appropriate application program.

[0034] In the following description, the processing of the main camera selection device 104 in steps S403 to S407 is performed frame by frame based on the synchronization signal supplied from the clock generation unit 314. One frame is one frame in which cameras 101A to 101C capture images at a predetermined frame rate, or one frame in which the display device 103 displays a background image at a predetermined frame rate.

[0035] First, in step S401, the main camera selection device 104 sets one of the cameras 101A to 101C as the default main camera. For example, in this case, camera 101A is set as the default main camera. Note that the default main camera may be selectable by the user.

[0036] Next, in step S402, the main camera selection device 104 starts the imaging operation for each of the cameras 101A to 101C. The main camera selection device 104 also starts the position and attitude detection operation for cameras 101A to 101C for each of the position and attitude detection devices 102A to 102C. Furthermore, if recording of in-camera VFX video is to be performed, the main camera selection device 104 starts recording of captured images (first captured image, second captured image) for each of the cameras 101A to 101C. Note that the user may choose whether or not to perform recording.

[0037] Next, in step S403, the main camera selection device 104 acquires data input from external devices. Specifically, the main camera selection device 104 acquires the captured images (first captured image, second captured image) from each camera 101A to 101C. The main camera selection device 104 also acquires information on the position and orientation (first position and orientation, second position and orientation) of cameras 101A to 101C from the position and orientation detection devices 102A to 102C.

[0038] Next, in step S404, the main camera selection device 104 determines, via the instruction input unit 313, whether or not a main camera switching instruction has been received from the user. The main camera switching instruction includes information specifying which camera to switch to. If it is determined that a main camera switching instruction has been received, the process proceeds to step S405. On the other hand, if it is determined that there is no main camera switching instruction, the process proceeds to step S406.

[0039] In step S405, the main camera selection device 104 sets (selects) one of the cameras 101A to 101C as the main camera in response to the main camera switching instruction received in step S404. The main camera is set by the control unit 301 changing the value of a variable that indicates which camera is the main camera, which is stored in the RAM 312. For example, if camera 101A is set as the main camera, the variable value is set to 0; if camera 101B is set as the main camera, the variable value is set to 1; and if camera 101C is set as the main camera, the variable value is set to 2.

[0040] Next, in step S406, the main camera selection device 104 outputs various data related to the main camera to an external device. Specifically, the main camera selection device 104 outputs the position and orientation information of the main camera from the position and orientation information of cameras 101A to 101C input in step S403 to the background image generation device 105. The background image generation device 105 then renders a CG background image using the position and orientation information of the main camera input from the main camera selection device 104 and outputs it to the display control device 106. The display control device 106 then outputs the background image input from the background image generation device 105 to the display device 103 for display. The main camera selection device 104 also outputs the image captured by the main camera from the images captured by cameras 101A to 101C acquired in step S403 to the display device 107 for display.

[0041] Next, in step S407, the main camera selection device 104 determines, via the instruction input unit 313, whether or not there has been an instruction from the user to end imaging. If it is determined that there has been an instruction to end imaging, the imaging and recording operations of cameras 101A to 101C, as well as the position and orientation detection operations of position and orientation detection devices 102A to 102C, are terminated, and the flow shown in Figure 4 is terminated. On the other hand, if it is determined that there has been no instruction to end imaging, the process proceeds to step S403.

[0042] In this way, the main camera selection device 104 can switch the image displayed on the display device 103, which corresponds to the position and orientation of the camera, to an image corresponding to the position and orientation of a different camera, at a timing corresponding to a main camera switching instruction from the user. That is, the main camera selection device 104 can end the display of the background image corresponding to any of the cameras 101A to 101C and start the display of the background image corresponding to a different camera. As a result, the user can capture images in the desired shot (image from the desired camera during the desired imaging period) in which the background image matches the position and orientation of the camera.

[0043] Furthermore, according to the aforementioned flow, the main camera selection device 104 can display an image captured by the same predetermined camera on the display device 107 at the time when a background image corresponding to the predetermined camera position and orientation is displayed on the display device 103, regardless of whether the main camera is switched. This makes it easier for the user to confirm whether the desired image is being captured by the main camera while the in-camera VFX image is being captured, regardless of whether the main camera is switched.

[0044] In the above explanation, the background image is switched by using the camera position and orientation information input by the main camera selection device 104 to the background image generation device 105. However, the background image may be switched to correspond to the position and orientation of a different camera by another method.

[0045] For example, the background image generation device 105 may generate background images corresponding to the position and orientation of cameras 101A to 101C, and select the background image corresponding to the position and orientation of the main camera from among the multiple background images and output it to the display device 103. In this case, for example, the position and orientation detection devices 102A to 102C output the position and orientation information of each camera to the background image generation device 105 without going through the main camera selection device 104. The background image generation device 105 also generates an image corresponding to the position and orientation of each camera for each camera. The main camera selection device 104, based on the setting of which camera is the main camera, causes the background image generation device 105 to select the image corresponding to the main camera from among the generated background images and outputs it to the display control device 106.

[0046] This method also allows switching which camera's position and orientation corresponds to the background image displayed on the display device 103. However, in the method described with reference to Figure 4, the background image generation device 105 only needs to generate a background image corresponding to the position and orientation of the main camera, thus reducing the computational cost of rendering compared to the method of generating multiple background images and then selecting one.

[0047] In the above explanation, the main camera selection device 104 displays only the image captured by the main camera on the display device 107. However, it may also be controlled to display the image captured by the main camera and the images captured by other cameras simultaneously, and to highlight the image captured by the main camera. Highlighting means, for example, displaying the area of ​​the image captured by the main camera larger than the area of ​​the images captured by other cameras, or fixing a predetermined area on the screen as the display area for the image captured by the main camera, regardless of the passage of time.

[0048] Figure 5 is a schematic diagram showing an example of an image with the main camera's captured image highlighted. Regions 501A to 501C each indicate areas where the captured images from cameras 101A to 101C are displayed in a reduced size. Region 502 indicates an area where the captured image from the main camera among cameras 101A to 101C is displayed in a larger area than each of regions 501A to 501C. In this case, the main camera selection device 104 executes an appropriate application to process the captured images from cameras 101A to 101C to the state shown in Figure 5 based on the main camera's information, and outputs the processed image to the display device 107. This method also makes it easier for the user to confirm whether the desired image is being captured by the main camera while the in-camera VFX image is being captured, regardless of the main camera switching.

[0049] In addition, as described above, the background image generation device 105 may, during rendering, determine the actual imaging range on the display surface of the display device 103 based on the position and orientation information of the main camera and the angle of view (focal length of the imaging lens) information acquired from the main camera. The background image generation device 105 can, for example, determine the first imaging range of the first imaging device and the second imaging range of the second imaging device. Furthermore, the background image generation device 105 can, for example, change the first imaging range to the second imaging range in response to a start command from the user to start displaying the second background image on the display device 103.

[0050] In addition, the background image generation device 105 may reduce the rendering processing load by rendering the CG with lower image quality outside the imaging range than within the imaging range, or by rendering the CG only within the imaging range. In this case, in step S403, the main camera selection device 104 further acquires field of view information from cameras 101A to 101C. Also, in step S406, the main camera selection device 104 outputs the field of view information of the main camera, in addition to the position and orientation information of the main camera, to the background image generation device 105. The background image generation device 105 identifies the imaging range according to the position and orientation information and field of view information of the main camera, and renders the CG based on the position and orientation information only within the imaging range. By this method, the main camera selection device 104 can match the rendering range of the background image regardless of the main camera switching.

[0051] In addition, as described above, camera 101A may record information indicating whether or not the background image corresponds to the position and orientation of camera 101A along with the captured image. For example, the main camera selection device 104 can determine whether or not the second captured image was captured at the time the second background image was displayed. In this case, in step S406, the main camera selection device 104 outputs a flag value (e.g., 1) indicating that the background image corresponds to the main camera. It also outputs a flag value (e.g., 0) indicating that the background image does not correspond to any camera other than the main camera. Furthermore, camera 101A records the flag value input from the main camera selection device 104 along with the captured image in the recording unit 208.

[0052] This makes it easier to determine which period of the captured video corresponds to the display period of the background image (i.e., is consistent with the camera's position and orientation) by referring to the flag value of each frame of the recorded captured video. Note that the form of the supplementary information described above is just one example, and the information may be attached in any form that makes it clear which period corresponds to the display period of the background image. For example, camera 101A may attach a flag indicating the start of the period in which the background corresponds to begins, and a flag indicating the end of the period in which the background corresponds to ends. Similarly, cameras 101B and 101C may record captured video with information indicating whether or not the background image corresponds to their own position and orientation. In this way, the main camera selection device 104 can, for example, attach information to the second captured image indicating whether or not the second captured image was captured at the display time of the second background image.

[0053] Alternatively, information indicating whether or not the background attached to the captured images of cameras 101A to 101C corresponds to the captured images may be used to extract images from the captured images for the period in which the background corresponds. That is, the main camera selection device 104 can, for example, use information indicating whether or not the second captured image was captured at the time the second background image was displayed to extract images from the second captured images that were captured at the time the second background image was displayed.

[0054] In this case, the image extraction process is performed, for example, after the completion of the flow shown in Figure 4 (after the capture of in-camera VFX footage is complete) by a video editing device (not shown in Figure 1). The video editing device is, for example, a computer device, and by running an appropriate application, it extracts the captured video footage for the period in which the background corresponds, based on the accompanying information of the captured video. For example, if frames in which the background image corresponds are assigned a flag value of 1 and frames in which the background image does not correspond are assigned a flag value of 0, the video editing device will extract the frames with a flag value of 1. This makes it possible to extract the captured video footage for the valid period in which the background corresponds.

[0055] In addition, as described above, the background image generation device 105 may temporarily suspend the recording operation of captured images for any camera among cameras 101A to 101C that is not set as the main camera. In this case, in step S405, the main camera selection device 104 starts the recording operation of captured images for the newly set main camera and ends the recording operation of captured images for the original main camera. Furthermore, in step S402, the main camera selection device 104 starts the recording operation of captured images only for the main camera that was initially set in step S401. As a result, cameras 101A to 101C can each record captured images only during the effective period in which a background image corresponding to their own position and orientation is displayed. [Examples]

[0056] Next, with reference to Figure 6, the switching operation of the main camera in Embodiment 2 of the present invention will be described. Figure 6 is a flowchart of the operation of the main camera selection device 104 in this embodiment.

[0057] Referring to Figure 4, in the main camera switching operation described in Example 1, the display of the background image corresponding to the newly set position and orientation of the main camera is started by the main camera switching instruction, and the display of the background image corresponding to the originally set position and orientation of the main camera is stopped. As a result, after the completion of imaging, a series of scenes without any gaps in time can be constructed by stitching together the periods (cuts) in which the background images are aligned in the images captured by each camera in chronological order.

[0058] However, if the cut transition time is changed to either before or after the main camera switching instruction time, a period of inconsistent background image will occur in either the cut before or after the transition. Therefore, in the method of Example 1, it is difficult for the user to adjust the cut transition time from the main camera switching instruction time after the image capture is complete.

[0059] On the other hand, in this embodiment, time-division display is performed on the display device to display multiple types of background images almost simultaneously. Furthermore, after the main camera switching instruction, the display of the background image corresponding to the originally set position and orientation of the main camera is not terminated for at least a predetermined period (it is continued for only the predetermined period), making it possible to adjust the time of the cut switch to a later date after the completion of imaging.

[0060] In Figure 6, the steps that perform the operations described in Figure 4 are denoted by the same reference numerals as in Figure 4, and their explanations are omitted. Furthermore, in the following description, the processing of the main camera selection device 104 in steps S403 to S407 is performed every frame based on the synchronization signal supplied from the clock generation unit 314, similar to Embodiment 1.

[0061] First, in step S601, the main camera selection device 104 sets one of the cameras 101A to 101C as the initial first main camera. For example, in this case, camera 101A is set as the initial main camera. Note that the initial main camera may be selectable by the user.

[0062] In step S404, the main camera selection device 104 determines whether or not a main camera switching instruction has been received. If it is determined that a main camera switching instruction has been received, the process proceeds to step S602. In step S602, the main camera selection device 104 sets one of the cameras 101A to 101C as either the first main camera or the second main camera, in accordance with the main camera switching instruction received in step S404.

[0063] Specifically, the first main camera and the second main camera are set to the main camera that was not set in the previous step S602. For example, if camera 101A was set as the first main camera in the previous step S602, then in the current step S602, camera 101B or camera 101C will be set as the second main camera. Also, if camera 101A was set as the second main camera in the previous step S602, then in the current step S602, camera 101B or camera 101C will be set as the first main camera. The setting of the first and second main cameras is performed by the control unit 301 changing the value of a variable that indicates which main camera is stored in the RAM 312.

[0064] Furthermore, the main camera selection device 104 updates a flag in the final step S602 to indicate whether the camera in question was set as the first main camera or the second main camera, for subsequent processing. For example, if the camera in question was set as the first main camera in the current step S602, the flag is set to 0. If the camera in question was set as the second main camera, the flag is set to 1. By referring to the flag, the main camera selection device 104 can determine whether the first or second main camera was set in the previous step S602. Note that in the initial state after the start of imaging, the flag is set to 0, corresponding to the setting of the first main camera in step S601. In other words, in the first main camera switching instruction after the start of imaging, the camera specified by the main camera switching instruction is set as the second main camera.

[0065] Furthermore, as described later, the first main camera displays the corresponding background image (according to the camera's position and orientation) in the first display pattern, and captures the image in the first exposure pattern. In addition, the second main camera displays the corresponding background image in the second display pattern, and captures the image in the second exposure pattern.

[0066] The first and second display patterns will be described with reference to Figure 7. Figure 7 shows examples of display patterns and exposure patterns. In this embodiment, the display control device 106 causes the display device 103 to display two types of background images in a time-division format with non-overlapping display periods (first display period and second display period).

[0067] Specifically, the background image corresponding to the first main camera is displayed during the first display period, and the background image corresponding to the second main camera is displayed during the second display period, which does not overlap with the first display period. In the example shown in Figure 7, the first and second display periods are designed to fit within one frame period according to the frame rate, with the first half of one frame allocated to the first display period and the second half to the second display period. The first and second display periods are repeatedly allocated at a cycle according to the frame rate (hereinafter referred to as the first display pattern and the second display pattern, respectively). This allows the display device 103 to time-division display the background image corresponding to the first main camera and the background image corresponding to the second main camera. It is also possible that there are periods within one frame in which no background image is displayed.

[0068] Referring to Figure 7, the first and second exposure periods will be explained. The first exposure period corresponds to the first display period and is allocated to the first half of a frame, similar to the example of the first display period. This allows the background image displayed during the first display period to be captured during the first exposure period. The second exposure period corresponds to the second display period and is allocated to the second half of a frame, similar to the example of the second display period. This allows the background image displayed during the second display period to be captured during the second exposure period. The first and second exposure periods are each repeatedly allocated at a frequency corresponding to the frame rate, similar to the display period (hereinafter referred to as the first exposure pattern and the second exposure pattern, respectively). Note that each exposure period only needs to include the corresponding display period and not include any uncorresponding display periods, and its length may differ from that of the corresponding display period. In this case as well, the background image of the corresponding display period can be captured without being superimposed on other background images.

[0069] Returning to the explanation of Figure 6, in step S603, the main camera selection device 104 sets an exposure pattern for the camera among cameras 101A to 101C that was set as the first main camera or the second main camera in the preceding step S602. Specifically, if the camera in question was set as the first main camera in step S602, the main camera selection device 104 sets itself to supply a synchronization signal to that camera for imaging with the first exposure pattern. If the camera in question was set as the second main camera in step S602, the main camera selection device 104 sets itself to supply a synchronization signal to that camera for imaging with the second exposure pattern. The synchronization signals for imaging with each exposure pattern will be described later.

[0070] Cameras 101A to 101C can perform imaging operations in accordance with the synchronization signal supplied from the main camera selection device 104, thereby enabling imaging using either the first or second exposure pattern. This allows the first and second main cameras to capture the corresponding background image even when the display pattern of the corresponding background image is changed by switching the main camera. Initially, at the start of imaging, the first exposure pattern is assumed to be set for each of the cameras 101A to 101C.

[0071] Next, in step S604, the main camera selection device 104 outputs data related to the first main camera and the second main camera to an external device. Specifically, the main camera selection device 104 outputs the position and orientation information of the first main camera and the position and orientation information of the second main camera from the position and orientation information of cameras 101A to 101C input in step S403 to the background image generation device 105. The background image generation device 105 then uses the position and orientation information of the first main camera and the position and orientation information of the second main camera input from the main camera selection device 104 to sequentially render CG of background images corresponding to their respective positions and orientations, and outputs them to the display control device 106. The display control device 106 then outputs the background image corresponding to the first main camera input from the background image generation device 105 to the display device 103 and controls it to display it in the first display pattern. Furthermore, the display control device 106 outputs the background image corresponding to the second main camera input from the background image generation device 105 to the display device 103 and controls it to display it in the second display pattern.

[0072] Furthermore, the main camera selection device 104 outputs and displays the image captured by the camera newly set as the first or second main camera in the final step S602, from among the images captured by cameras 101A to 101C acquired in step S403, to the display device 107. This allows the main camera selection device 104 to display the image captured by the camera specified by the user as the switching destination in the latest main camera switching instruction on the display device 107.

[0073] The synchronization control of the first display pattern and the second display pattern, and the first exposure pattern and the second exposure pattern, is performed according to synchronization signals that include the timing of the first and second halves of a frame, supplied by the clock generation unit 314 of the main camera selection device 104. Specifically, the clock generation unit 314 generates a synchronization signal indicating the timing of the first half of a frame (the start of the frame) and a synchronization signal indicating the timing of the second half of a frame (after half the time of the frame has elapsed from the start of the frame). The main camera selection device 104 also supplies the synchronization signal indicating the timing of the first half of a frame and the synchronization signal indicating the timing of the second half of a frame to the display control device 106 in a state that allows for determination.

[0074] The display control device 106 displays the background image corresponding to the position and orientation of the first main camera, which is output from the background image generation device 105, on the display device 103 at a timing corresponding to the synchronization signal indicating the timing of the first half of one frame. The display control device 106 also displays the background image corresponding to the position and orientation of the second main camera, which is output from the background image generation device 105, on the display device 103 at a timing corresponding to the synchronization signal indicating the timing of the second half of one frame. As a result, the display device 103 can display the background image corresponding to the position and orientation of the first main camera and the background image corresponding to the position and orientation of the second main camera in a first display pattern and a second display pattern, respectively.

[0075] Furthermore, the main camera selection device 104 supplies a synchronization signal to the first main camera that indicates the timing of the first half of one frame. The first main camera performs imaging operations based on the synchronization signal indicating the timing of the first half of one frame, thereby enabling imaging in the first exposure pattern. The main camera selection device 104 also supplies a synchronization signal to the second main camera that indicates the timing of the second half of one frame. The second main camera performs imaging operations based on the synchronization signal indicating the timing of the second half of one frame, thereby enabling imaging in the second exposure pattern.

[0076] Referring to Figure 8, an example of how the display on the display device 103 and the exposure patterns of cameras 101A to 101C change in response to a main camera switching instruction from the user is explained using the flow chart in Figure 6. Figure 8 shows examples of display patterns and exposure patterns. Note that in Figure 8, the expected frequency of main camera switching instructions and the frame rate scale do not match. The horizontal axis in Figure 8 shows the passage of time.

[0077] Time 801 is the time of the initial state after imaging has started, and indicates the time when camera 101A is set as the first main camera. Time 802 indicates the time when the main camera switching instruction from the user to switch the main camera to camera 101B was received. Time 803 indicates the time when the main camera switching instruction from the user to switch the main camera to camera 101C was received, following time 802. Time 804 indicates the time when the main camera switching instruction from the user to switch the main camera to camera 101A was received, following time 803.

[0078] Period 811A indicates the period during which the display device 103 displays a background image corresponding to the position and orientation of camera 101A in the first display pattern. Period 812A indicates the period during which the display device 103 displays a background image corresponding to the position and orientation of camera 101B in the second display pattern. Period 813A indicates the period during which the display device 103 displays a background image corresponding to the position and orientation of camera 101C in the first display pattern. Period 814A indicates the period during which the display device 103 displays a background image corresponding to the position and orientation of camera 101A in the second display pattern. Periods 811B, 812B, 813B, and 814B are the same periods as periods 811A, 812A, 813A, and 814A, respectively, and indicate the period during which each camera captures a background image corresponding to its own position and orientation.

[0079] Here, we will focus on period 812A and explain the changes in the display on the display device 103 and the exposure pattern of the camera 101B in chronological order. At time 802, the main camera selection device 104 starts displaying a background image corresponding to the position and orientation of the camera 101B using the second display pattern. The main camera selection device 104 also changes the exposure pattern of the camera 101B to the second exposure pattern corresponding to the second display pattern. This enables the camera 101B to capture a background image corresponding to its own position and orientation.

[0080] At time 803, the main camera selection device 104 starts displaying a background image corresponding to the position and orientation of camera 101C in the first display pattern. At this time, the background image corresponding to the position and orientation of camera 101B continues to be displayed in the second display pattern. At time 804, the main camera selection device 104 starts displaying a background image corresponding to the position and orientation of camera 101A in the second display pattern, and consequently ends the display of the background image corresponding to the position and orientation of camera 101B.

[0081] As a result, camera 101B can capture background images corresponding to its own position and orientation during period 815 of period 812A, until a main camera switching instruction is given to switch the main camera from camera 101B to another camera. Furthermore, it can continue to capture background images corresponding to its own position and orientation during the reserve period shown in period 816 of period 812A, which is after period 815.

[0082] As described above, this embodiment demonstrates an example in which the display of the background image corresponding to the originally set position and orientation of the main camera is not terminated after the main camera switching instruction. As a result, the main camera can capture a background image corresponding to its own position and orientation even after the time after the new main camera switching instruction. Furthermore, even if the time of the switch between the captured images (cuts) of the main camera before and after the main camera switching instruction is adjusted to be after the main camera switching instruction, it becomes less likely that a period of inconsistent background images will occur.

[0083] In this embodiment, we have shown an example where the background image corresponding to the position and orientation of a predetermined camera is displayed continuously from the start of the display until the next-next main camera switching instruction. However, the display can be continued until any time after the next main camera switching instruction. In this case as well, the period during which the predetermined camera can capture a background image corresponding to its own position and orientation can be extended compared to the case where the display continues until the next main camera switching instruction. For example, the display of the background image corresponding to the position and orientation of the predetermined camera may be continued for a certain period after the next main camera switching instruction. In this case, the main camera selection device 104 can continue displaying the background image corresponding to the position and orientation of the predetermined camera for a certain period by counting a predetermined time which is the length of the reserve period, as described in Embodiment 3 below.

[0084] In this embodiment, we have shown an example where the number of cameras used for imaging is 3, while the time division for displaying the background image is 2. However, any combination where the number of time divisions is less than the number of cameras used for imaging is applicable. In this case as well, the degree to which the exposure time of the cameras and the display time of the background image are shortened can be suppressed compared to the case where the number of time divisions for displaying the background image is the same as the number of cameras. Furthermore, even when increasing the number of time divisions from 2, the display period should be allocated so that each display period is included within one frame. [Examples]

[0085] Next, with reference to Figure 9, the switching operation of the main camera in Embodiment 3 of the present invention will be described. Figure 9 is a flowchart of the operation of the main camera selection device 104 in this embodiment.

[0086] Referring to Figure 6, in the main camera switching operation described in Example 2, even if the time of the cut change after the end of imaging is adjusted to be after the main camera switching instruction, it is possible to avoid a period of inconsistent background image display. On the other hand, in this embodiment, the main camera switching order is registered in advance, and accordingly, the display of the background image corresponding to the next main camera is started before the main camera switching instruction. This makes it possible to avoid a period of inconsistent background image display even if the time of the cut change is adjusted to be before the main camera switching instruction.

[0087] In Figure 9, the steps that perform the operations described with reference to Figure 4 or Figure 6 are denoted with the same reference numbers as in Figure 4 or Figure 6, and their explanations are omitted. Note that before the start of the flowchart in Figure 9, the main camera selection device 104 registers the information of the main camera switching order specified by the user. This information of the main camera switching order is, for example, the table shown in Figure 10, and includes an array that specifies the cameras in order. Figure 10 is a diagram showing an example of the main camera switching order information (table).

[0088] The operation of registering the main camera switching order information by the main camera selection device 104 will now be described. Before the start of imaging of the in-camera VFX image, the main camera selection device 104 receives a registration instruction for the main camera switching order information from the user via the instruction input unit 313. At this time, the main camera selection device 104 may display a UI on the display device 107 to display the main camera switching order specified by the user. When a registration instruction for the main camera switching order information is received from the user, the main camera selection device 104 stores and retains the main camera switching order information in the ROM 311 and RAM 312. This allows the main camera selection device 104 to register the main camera switching order information specified by the user.

[0089] Furthermore, since the destination of the main camera is specified by the information of the main camera switching order, in the example shown in the flowchart of Figure 9, the main camera switching instruction does not need to include information specifying which camera is the destination. In the following description, the processing of the main camera selection device 104 in steps S403 to S407 is performed every frame based on the synchronization signal supplied from the clock generation unit 314, similar to Embodiment 1.

[0090] First, in step S901, the main camera selection device 104 refers to the information on the switching order of the main cameras and sets the initial settings for the first and second main cameras. Specifically, it sets the camera designated first in the information on the switching order of the main cameras (camera 101A in the example of Figure 10) as the first main camera, and the camera designated second (camera 101B in the example of Figure 10) as the second main camera.

[0091] In step S404, the main camera selection device 104 may, as in Embodiment 1, notify the user of the main camera switching order. For example, the main camera selection device 104 displays information on the display device 107 regarding the next main camera to switch to, based on the main camera switching order information and the number of main camera switching instructions given since the start of imaging.

[0092] Specifically, when the first main camera switch command is received, the system indicates that the main camera will switch to the camera specified in the second item of the main camera switch order information when the user issues a main camera switch command. Similarly, when the second main camera switch command is received, the system indicates that the main camera will switch to the camera specified in the third item of the main camera switch order information when the user issues a main camera switch command. In the same manner, when the nth main camera switch command is received, the system indicates that the main camera will switch to the camera specified in the (n+1)th item of the main camera switch order information when the user issues a main camera switch command.

[0093] If it is determined in step S404 that a main camera switching instruction has been given, in step S902 the main camera selection device 104 sets a timer to count a predetermined time, which is the length of the buffer period after the cut. Specifically, the main camera selection device 104 adds a variable to RAM 312 that will be used to count the timer. It also sets a value to the variable that indicates the predetermined time. For example, if the predetermined time is 2 seconds and the frame rate of the display device 103, which does not consider time division, is 60 fps, then 120 is set.

[0094] In step S903, the main camera selection device 104 determines whether or not a timer is set. If it is determined that no timer is set, the process proceeds to step S604. On the other hand, if it is determined that a timer is set, the process proceeds to step S904.

[0095] In step S904, the main camera selection device 104 refers to a variable used for timer counting and determines whether a predetermined time has elapsed. Specifically, if the variable value is greater than 0, it is determined that the predetermined time has not elapsed. On the other hand, if the variable value is 0, it is determined that the predetermined time has elapsed. If it is determined that the predetermined time has not elapsed, the device proceeds to step S905. On the other hand, if it is determined that the predetermined time has elapsed, the device proceeds to step S906.

[0096] If it is determined in step S904 that the predetermined time has not elapsed, in step S905 the main camera selection device 104 starts the timer count. Specifically, it decreases the value of the variable used to count the timer by 1. On the other hand, if it is determined in step S904 that the predetermined time has elapsed, in step S906 the main camera selection device 104 deletes the timer. Specifically, it deletes the variable used to count the timer, resulting in a state where the timer is not set.

[0097] In step S907, the main camera selection device 104 selects the main camera according to the information on the main camera switching order (the start order of displaying the background images of each of the multiple imaging devices). Specifically, the main camera selection device 104 sets the camera specified in the item following the item set in the previous step S907 as the main camera. Since the first and second items of the main camera switching order information have already been set as the main camera in the process of step S901, the camera specified in the third item is set as the main camera in the initial process of step S907. Furthermore, when setting the main camera, the main camera selection device 104 sets it to either the first main camera or the second main camera. Specifically, it sets it to the main camera that was not set in the previous step S907 process, between the first and second main cameras. Note that the camera specified in the second item of the information regarding the switching order of the main camera is set as the second main camera in the process of step S901. Therefore, in the initial process of step S907, the camera specified in the third item is set as the first main camera.

[0098] Referring to Figure 11, the flow chart in Figure 9 illustrates how the display on the display device 103 and the exposure patterns of cameras 101A to 101C change in response to the information on the main camera switching sequence shown in the example in Figure 10 and the main camera switching instruction from the user. Figure 11 shows examples of display patterns and exposure patterns. Note that in Figure 11, the expected frequency of main camera switching instructions and the predetermined time set in the timer in step S902 do not match the frame rate scale. The horizontal axis in Figure 11 shows the passage of time.

[0099] Time 1101 is the time of the initial state after imaging has started, and indicates the time when camera 101A is set as the first main camera and camera 101B is set as the second main camera. Time 1102 indicates the time when the first main camera switching instruction from the user is received. Time 1103 indicates the time when the second main camera switching instruction from the user is received. Time 1104 indicates the time when the third main camera switching instruction from the user is received.

[0100] Period 1111A indicates the period during which the display device 103 displays a background image corresponding to the position and orientation of camera 101A in the first display pattern. Period 1112A indicates the period during which the display device 103 displays a background image corresponding to the position and orientation of camera 101B in the second display pattern. Period 1113A indicates the period during which the display device 103 displays a background image corresponding to the position and orientation of camera 101C in the first display pattern. Period 1114A indicates the period during which the display device 103 displays a background image corresponding to the position and orientation of camera 101A in the second display pattern. Period 1115A indicates the period during which the display device 103 displays a background image corresponding to the position and orientation of camera 101B in the first display pattern. Periods 1111B, 1112B, 1113B, 1114B, and 1115B correspond to periods 1111A, 1112A, 1113A, 1114A, and 1115A, respectively, and represent the periods during which each camera captures background images corresponding to its own position and orientation.

[0101] Here, we will focus on period 1113A and explain the display on the display device 103 and the changes in the exposure pattern of the camera 101C in chronological order. Time 1105 is the time when a predetermined amount of time has elapsed, which is the length of the buffer period after the cut, following the first main camera switching instruction at time 1102. At time 1105, the main camera selection device 104 refers to the third item of the information on the main camera switching order in Figure 10 and starts displaying the background image corresponding to the position and orientation of the camera 101C in the first display pattern. The main camera selection device 104 also changes the exposure pattern of the camera 101C to the first exposure pattern corresponding to the first display pattern. This enables the camera 101C to capture a background image corresponding to its own position and orientation.

[0102] Time 1103 is the time when the second main camera switching instruction is received. After a predetermined time has elapsed, the main camera selection device 104 refers to the fourth item of the information on the main camera switching order in Figure 10 and starts displaying the background image corresponding to the position and orientation of camera 101A in the second display pattern. At this time, the background image corresponding to the position and orientation of camera 101C continues to be displayed in the first display pattern. When the main camera selection device 104 receives the second main camera switching instruction, it indicates that if the user gives a main camera switching instruction, the main camera will be switched to camera 101C, which is the camera specified in the third item of the information on the main camera switching order. Therefore, the main camera selection device 104 can remind the user that the main camera will be switched to camera 101C with the main camera switching instruction at time 1103.

[0103] Time 1104 is the time when the third main camera switching instruction is received, and the main camera selection device 104 starts counting a predetermined time. When the main camera selection device 104 receives the third main camera switching instruction, it indicates that if the user gives a main camera switching instruction, the main camera will be switched to camera 101A, which is the camera specified in the fourth item of the main camera switching order information. Therefore, the main camera selection device 104 can remind the user that the main camera will be switched to camera 101A with the main camera switching instruction at time 1104.

[0104] Time 1106 is the time when a predetermined amount of time has elapsed since time 1104. The main camera selection device 104 refers to the fifth item in the information of the main camera switching order in Figure 10 and starts displaying the background image corresponding to the position and orientation of camera 101B in the first display pattern. Accordingly, the main camera selection device 104 ends the display of the background image corresponding to the position and orientation of camera 101C in the first display pattern.

[0105] As a result, camera 101C can capture a background image corresponding to its own position and orientation during period 1116 of period 1113A, which is the period during which the main camera selection device 104 reminds the user that camera 101C is the main camera. Furthermore, camera 101C can also capture a background image corresponding to its own position and orientation during the reserve period shown in period 1117 of period 1113A, which is before period 1116, prior to the main camera switching instruction at time 1103. In addition, camera 101C can continue to capture a background image corresponding to its own position and orientation during the reserve period shown in period 1118 of period 1113A, which is after period 1116.

[0106] As shown in the above operation, this embodiment demonstrates an example in which the switching order of the main cameras is registered in advance, and the display of the background image corresponding to the target main camera is started before the main camera switching instruction. As a result, a given camera can capture a background image corresponding to its position and orientation from a time before the main camera switching instruction time (the time that the user is reminded of when the camera is set as the main camera). Furthermore, even if the switching time between the captured images (cuts) of the main camera before and after the main camera switching instruction is adjusted to be before the main camera switching instruction, it becomes less likely that there will be a period in which the background images are not aligned.

[0107] In this embodiment, we have shown an example where a buffer period is provided before and after the period (cut) in which the user is led to assume that a certain camera is the main camera. However, the present invention can also be applied by providing a buffer period only before the cut. In this case, if the predetermined time is set to 0 seconds in the flowchart of Figure 9, a buffer period can be provided only before the cut.

[0108] In the flowchart of Figure 9, an example is shown where the main camera selection device 104 switches the main camera after a predetermined time has elapsed since receiving the main camera switching instruction, thereby providing a buffer period after the cut. However, the length of the predetermined time can be set by the user. In this case, the main camera selection device 104 receives the setting of the predetermined time from the user via the instruction input unit 313 before the start of in-camera VFX image capture. The information of the predetermined time specified by the user is stored in the ROM 311 and RAM 312 and is referenced during the processing of step S902. This allows the user to adjust the balance between the buffer period after the cut and the length of the buffer period before the cut, which becomes shorter the longer the buffer period after the cut is.

[0109] Furthermore, the flowchart in Figure 9 may include processing for when the next main camera switching instruction is received before a predetermined time has elapsed since the previous main camera switching instruction was received. Specifically, the main camera selection device 104 switches the main camera at the earlier of either the predetermined time having elapsed or the next main camera switching instruction being received. This makes it easier for the main camera selection device 104 to switch the main camera at the timing specified by the user, even if the next main camera switching instruction is received before the predetermined time has elapsed.

[0110] In this embodiment, we showed an example where the number of cameras used for imaging was 3, while the time-division ratio for displaying the background image was 2. However, any combination where the time-division ratio is less than the number of cameras used for imaging is applicable. In this case as well, the degree to which the exposure time of the cameras and the display time of the background image are shortened can be suppressed compared to the case where the time-division ratio for displaying the background image is the same as the number of cameras.

[0111] As described above, the control device (104) in each embodiment is a control device for an imaging system 100 that captures each background image displayed on the display device 103 using a plurality of imaging devices (cameras 101A to 101C), including a first imaging device and a second imaging device. The control device includes, but is not limited to, a main camera selection device 104, a background image generation device 105, a display control device 106, and a display device 107.

[0112] The control device includes a control unit (control unit 301) and an acquisition unit (instruction input unit 313). The control unit causes the display device to display a first background image corresponding to the first position and orientation of the first imaging device (one of a plurality of imaging devices) and a second background image corresponding to the second position and orientation of the second imaging device (the other of the plurality of imaging devices). The acquisition unit acquires a start instruction from the user to start displaying the second background image on the display device while the first background image is being displayed on the display device. In response to the start instruction, the control unit causes the display device to start displaying the second background image.

[0113] In each embodiment, the acquisition means acquires a start command from the user to start the display of the second imaging device via an operation unit for the user to select or identify one imaging device (second imaging device) from a plurality of imaging devices, but is not limited to this. For example, the user can pre-register a plurality of imaging devices that constitute the imaging system 100 and the order of display switching. In this case, the acquisition means can acquire a start command for the second imaging device in the registered start order each time the user operates the operation unit (for example, each time a specific button is pressed). The acquisition means can also acquire the start command via wired or wireless communication in response to user operation via an operation unit located physically separate from other components, instead of an operation unit located in the same location as other components such as the control unit 301.

[0114] Furthermore, in each embodiment, the start instruction from the user is not limited to an instruction obtained directly based on user operation via the control unit. For example, the user may pre-register (set) the timing (time) of the start instruction, and the control means may consider that a start instruction has been received when the registered time arrives and start displaying the second background image. Such a configuration is also included in the configuration in which the acquisition means obtains a start instruction from the user and the control means starts displaying the second background image in response to the start instruction.

[0115] Furthermore, in each embodiment, the control means may include an automatic mode for user start commands by, for example, utilizing AI (Artificial Intelligence). In this case, a start command acquired in automatic mode can be considered as a user start command. That is, when automatic mode is set, the acquisition means can automatically acquire a start command (user start command in automatic mode) at an appropriate timing according to various information such as the position and orientation of each of the multiple imaging devices, the time, or the displayed image. The control means can start displaying the second background image in response to the user start command in automatic mode.

[0116] Furthermore, in each embodiment, the control means starts displaying the second background image at the timing of the start instruction, but is not limited to this. A predetermined time difference may occur between the timing of the start instruction and the timing of the start of the display of the second background image, but this time difference can be of any length. In addition, this time difference may be configured to be arbitrarily set or changed by the user. This can improve convenience for each user. In other words, it is sufficient that the system is configured to display the second background image in response to a start instruction from the user, and the interval between the timing of the start instruction and the timing of the start of the display of the second background image can be set as appropriate.

[0117] Preferably, the control means terminates the display of the first background image in response to a start instruction. Also preferably, the control means acquires a first image captured by the first imaging device and a second image captured by the second imaging device. The control means then selects the first image captured at the first time while the first background image is displayed on the display device. The control means also selects the second image captured at the second time while the second background image is displayed on the display device. More preferably, the display device 107 displays the first or second image selected by the control means.

[0118] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0119] According to each embodiment, it is possible to provide a control device, an imaging system, a control method, and a program that can acquire visually appealing images regardless of the number of imaging devices.

[0120] Each embodiment's disclosure includes the following configuration and method. (Composition 1) A control device for an imaging system that captures background images displayed on a display device using a plurality of imaging devices, including a first imaging device and a second imaging device, Control means for displaying a first background image corresponding to the first position and orientation of the first imaging device and a second background image corresponding to the second position and orientation of the second imaging device on the display device, The device includes an acquisition means for acquiring a start command from the user to start displaying a second background image on the display device while the first background image is displayed on the display device, The control means is characterized in that, in response to the start instruction, it causes the display device to start displaying the second background image. (Configuration 2) The control device according to configuration 1, characterized in that the control means terminates the display of the first background image in response to the start instruction. (Composition 3) The control means is A first image captured by the first imaging device and a second image captured by the second imaging device are acquired. At a first time point in which the first background image is displayed on the display device, the first captured image captured at that first time point is selected. The control device according to configuration 1 or 2, characterized in that, at a second time when the second background image is displayed on the display device, the second captured image captured at that second time is selected. (Composition 4) The control device according to configuration 3, further comprising image display means for displaying the first captured image or the second captured image selected by the control means. (Composition 5) The control device according to any one of configurations 1 to 4, characterized in that the control means changes the information of the first position and orientation to the information of the second position and orientation in response to the start instruction, thereby causing the display device to start displaying the second background image. (Composition 6) The control device according to any one of configurations 1 to 5, characterized in that the control means changes the first imaging range of the first imaging device on the display surface of the display device to the second imaging range of the second imaging device in response to the start instruction. (Composition 7) The control device according to configuration 3 or 4, characterized in that the control means determines whether or not the second captured image was captured at the time the second background image was displayed. (Composition 8) The control device according to configuration 7, characterized in that the control means attaches to the second captured image information indicating whether or not the second captured image was captured at the time the second background image was displayed. (Composition 9) The control device according to configuration 7 or 8, characterized in that the control means extracts from the second captured images the captured image of the second background image at the time the second background image was displayed, using information indicating whether or not the second captured image was captured at the time the second background image was displayed. (Composition 10) The control device according to any one of configurations 1 to 9, characterized in that the control means sets a first display period for the first background image and a second display period for the second background image that do not overlap with each other on the display device. (Composition 11) The control device according to configuration 10, characterized in that the control means sets the first display period and the second display period within one frame period. (Composition 12) The control device according to configuration 10 or 11, characterized in that the number of display periods, including the first display period and the second display period, is less than the number of imaging devices. (Composition 13) The control device according to any one of configurations 10 to 12, characterized in that the control means controls the display device to display the first background image during the first display period and the second background image during the second display period. (Composition 14) The control device according to configuration 13, characterized in that the control means controls the second imaging device such that the second display period is included in the exposure period of the second imaging device and the first display period is not included in the exposure period. (Composition 15) The control device according to configuration 13 or 14, characterized in that the control means, in response to the start instruction, causes the display device to start displaying the second background image, and then continues displaying the first background image for a predetermined period of time. (Composition 16) The control device according to configuration 15, characterized in that the control means sets the length of the predetermined period. (Composition 17) The control device according to configuration 15 or 16, characterized in that the predetermined period is the period until the display of a third background image different from the first background image begins during the first display period. (Composition 18) The system further includes setting means that allows the user to pre-set the start order for displaying the background images of each of the multiple imaging devices. The control device according to any one of configurations 1 to 17, characterized in that the control means initiates the display of the background image in the order described above. (Method 1) A control method for an imaging system that captures background images displayed on a display device using a plurality of imaging devices, including a first imaging device and a second imaging device, A display control step of displaying a first background image corresponding to the first position and orientation of the first imaging device and a second background image corresponding to the second position and orientation of the second imaging device on the display device, The process includes an acquisition step of acquiring a start command from the user to start displaying a second background image on the display device while the first background image is displayed on the display device, A control method characterized in that, in the display control step, the display device starts displaying the second background image in response to the start instruction. (Composition 19) First imaging device, The second imaging device, Display device and An imaging system characterized by having a control device according to any one of configurations 1 to 18. (Composition 20) A program characterized by causing a computer to execute the control method described in Method 1.

[0121] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its essence. [Explanation of Symbols]

[0122] 100 Imaging Systems 101A, 101B, 101C Cameras (First Imaging Device, Second Imaging Device) 103 Display device 104 Main camera selection device (control device) 301 Control Unit (Control Means) 313 Instruction Input Unit (Acquisition Means)

Claims

1. A control device for an imaging system that captures background images displayed on a display device using a plurality of imaging devices, including a first imaging device and a second imaging device, Control means for displaying a first background image corresponding to the first position and orientation of the first imaging device and a second background image corresponding to the second position and orientation of the second imaging device on the display device, The device includes an acquisition means for acquiring a start command from the user to start displaying a second background image on the display device while the first background image is displayed on the display device, The control means is characterized in that, in response to the start instruction, it causes the display device to start displaying the second background image.

2. The control device according to claim 1, characterized in that the control means terminates the display of the first background image in response to the start instruction.

3. The control means is A first image captured by the first imaging device and a second image captured by the second imaging device are acquired. At a first time point in which the first background image is displayed on the display device, the first captured image captured at that first time point is selected. The control device according to claim 1 or 2, characterized in that, at a second time when the second background image is displayed on the display device, the second captured image captured at that second time is selected.

4. The control device according to claim 3, further comprising image display means for displaying the first or second captured image selected by the control means.

5. The control device according to claim 1 or 2, characterized in that the control means causes the display device to start displaying the second background image by changing the first position and orientation information to the second position and orientation information in response to the start instruction.

6. The control device according to claim 1 or 2, characterized in that the control means changes the first imaging range of the first imaging device on the display surface of the display device to the second imaging range of the second imaging device in response to the start instruction.

7. The control device according to claim 3, wherein the control means determines whether or not the second captured image was captured at the time the second background image was displayed.

8. The control device according to claim 7, characterized in that the control means attaches to the second captured image information indicating whether or not the second captured image was captured at the time the second background image was displayed.

9. The control device according to claim 7, characterized in that the control means extracts from the second captured images the captured image of the second background image at the time the second background image was displayed, using information indicating whether or not the second captured image was captured at the time the second background image was displayed.

10. The control device according to claim 1, wherein the control means sets a first display period for the first background image and a second display period for the second background image that do not overlap with each other on the display device.

11. The control device according to claim 10, characterized in that the control means sets the first display period and the second display period within one frame period.

12. The control device according to claim 10, characterized in that the number of display periods, including the first display period and the second display period, is less than the number of imaging devices.

13. The control device according to any one of claims 10 to 12, characterized in that the control means controls the display device to display the first background image during the first display period and the second background image during the second display period.

14. The control device according to claim 13, characterized in that the control means controls the second imaging device such that the second display period is included in the exposure period of the second imaging device and the first display period is not included in the exposure period.

15. The control device according to claim 13, characterized in that the control means, in response to the start instruction, causes the display device to start displaying the second background image, and then continues displaying the first background image for a predetermined period of time.

16. The control device according to claim 15, wherein the control means is characterized by setting the length of the predetermined period.

17. The control device according to claim 15, characterized in that the predetermined period is the period until the display of a third background image different from the first background image is started during the first display period.

18. The system further includes setting means that allows the user to pre-set the start order for displaying the background images of each of the multiple imaging devices. The control device according to any one of claims 1, 10 to 12, characterized in that the control means starts the display of the background image in the order described above.

19. A control method for an imaging system that captures background images displayed on a display device using a plurality of imaging devices, including a first imaging device and a second imaging device, A display control step of displaying a first background image corresponding to the first position and orientation of the first imaging device and a second background image corresponding to the second position and orientation of the second imaging device on the display device, The process includes an acquisition step of acquiring a start command from the user to start displaying a second background image on the display device while the first background image is displayed on the display device, A control method characterized in that, in the display control step, the display device starts displaying the second background image in response to the start instruction.

20. First imaging device, The second imaging device, Display device and An imaging system characterized by having the control device described in claim 1 or 2.

21. A program characterized by causing a computer to execute the control method described in claim 19.

Images