Control unit and control method for imaging system, and imaging system

Abstract

To provide a control unit and a control method for an imaging system that make it easier to shoot in-camera VFX images using a plurality of imaging apparatuses.SOLUTION: A control unit for an imaging system performs imaging with an image displayed on a display unit as a background. The control unit includes: allocation means that allocates imaging periods with different timings to individual imaging apparatuses; imaging control means that controls individual imaging apparatuses so that the individual imaging apparatuses perform imaging in the allocated imaging periods; and display control means that, in each of the imaging periods, causes the display unit to display an image according to the position and posture of the imaging apparatus to which the imaging period is allocated.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a control device and a control method for an imaging system. [Background technology]

[0002] VFX (Visual Effects) technology, which creates unrealistic scenes by compositing live-action footage with computer graphics (CG), is attracting attention. In addition, a method is known in which a VFX image is obtained without compositing processing by capturing an image of a subject against a display device or screen that displays CG (in-camera VFX).

[0003] To realize natural VFX, it is necessary to match the position and orientation of the virtual camera when rendering CG with the position and orientation of the camera capturing live-action footage. Therefore, it is necessary to detect the position and orientation of the camera and change the CG according to the detected position and orientation. Patent Document 1 discloses a technology for changing an image displayed on a head-mounted display worn by a user according to the position and orientation of the user. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2014-525049 A Summary of the Invention [Problem to be solved by the invention]

[0005] When shooting VFX footage of the same scene with multiple cameras, if the background image is changed according to the position and orientation of one camera, the position and orientation of the other cameras will not match the background, which is a problem. For this reason, in the past, it was not possible to shoot with one camera until shooting with another camera was completed, or it was necessary to recompose the background in the post-production process for footage from a camera whose position and orientation did not match the background.

[0006] In one aspect, the present invention provides a control device and control method for an imaging system that at least alleviates the problems associated with the conventional techniques and makes it easier to capture in-camera VFX footage using multiple imaging devices. [Means for solving the problem]

[0007] The above-mentioned object can be achieved by a control device for an imaging system that captures images with an image displayed on a display device as a background, the control device comprising: an assignment means for assigning imaging periods with different timing to each imaging device; an imaging control means for controlling each imaging device so that each imaging device captures images during its assigned imaging period; and a display control means for causing the display device to display, during each imaging period, an image corresponding to the position and orientation of the imaging device to which the imaging period is assigned. Effect of the Invention

[0008] According to the present invention, it is possible to provide a control device and a control method for an imaging system that can more easily capture in-camera VFX footage using a plurality of imaging devices. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of a functional configuration of an imaging apparatus according to an embodiment; [Diagram 2] FIG. 1 is a schematic diagram showing an example of the configuration of an imaging system according to an embodiment; [Diagram 3] FIG. 2 is a block diagram showing an example of the functional configuration of a control device of the imaging system according to the embodiment; [Figure 4] 1 is a flowchart showing the operation of a control device according to an embodiment of the present invention; [Diagram 5] FIG. 1 is a sequence diagram of an operation of an imaging system according to an embodiment of the present invention; [Figure 6] FIG. 1 is a diagram showing examples of exposure patterns and display patterns in the first embodiment; [Figure 7]FIG. 1 is a schematic diagram showing an example of a background image displayed in an embodiment; [Figure 8] FIG. 11 is a diagram showing an example of an exposure pattern and a display pattern in a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The present invention will be described in detail below based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. In addition, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated explanations are omitted.

[0011] <First embodiment> (Example of configuration of imaging device 100) 1 is a block diagram showing an example of the functional configuration of an imaging device 100 according to a first embodiment. The imaging lens 103 (imaging optical system) has multiple lenses including a movable lens. The movable lens includes at least a focus lens, and may further include one or more of a zoom lens and a shift lens. The imaging lens 103 forms an optical image of a subject on an imaging plane.

[0012] The aperture 101 has a variable aperture to adjust the amount of incident light. The aperture of the aperture 101 is controlled by a system control unit 50, which will be described later. An ND (Neutral Density) 104 is a neutral density filter that is inserted in the optical path as necessary.

[0013] The imaging unit 22 has an imaging element. The imaging element may be, for example, a known CCD or CMOS color image sensor having a primary color Bayer array color filter. The imaging element has a pixel array in which a plurality of pixels are arranged two-dimensionally, and a peripheral circuit for reading out signals from each pixel. Each pixel accumulates electric charge according to the amount of incident light by photoelectric conversion. By reading out a signal having a voltage according to the amount of electric charge accumulated during the exposure period from each pixel, a group of pixel signals (analog image signals) representing the optical storage of the subject formed on the imaging plane can be obtained.

[0014] The A / D converter 23 A / D converts the analog image signal read out from the imaging unit 22 and outputs a digital image signal (image data). Note that if the imaging unit 22 has an A / D conversion function, the A / D converter 23 is not necessary.

[0015] The image processing unit 24 applies predetermined image processing to image data output by the A / D converter 23 and the memory control unit 15, generates signals and image data according to the application, and acquires and / or generates various information. The image processing unit 24 may be a dedicated hardware circuit such as an ASIC (Application Specific Integrated Circuit) designed to realize a specific function. Alternatively, the image processing unit 24 may be configured such that a processor such as a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit) executes software to realize a specific function. The image processing unit 24 outputs the acquired or generated information and data to the system control unit 50, the memory control unit 15, etc. according to the application.

[0016] The image processing applied by the image processing unit 24 can include, for example, pre-processing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Pre-processing may include signal amplification, reference level adjustment, defective pixel correction, etc. Color interpolation processing is performed when an image sensor is provided with a color filter, and is a process for interpolating values ​​of color components that are not included in the individual pixel data that constitutes the image data. Color interpolation processing is also called demosaic processing. The correction processing can include white balance adjustment, gradation correction, correction of image degradation caused by optical aberration of the imaging lens 103 (image restoration), correction of the effects of peripheral light falloff of the imaging lens 103, color correction, and the like. The detection process may include detection of feature regions (for example, face regions or human body regions) and their movements, person recognition processing, and the like. Data processing may include processes such as area extraction (trimming), synthesis, scaling, encoding and decoding, header information generation (data file generation), etc. Data processing also includes the generation of image data for display or image data for recording. The evaluation value calculation process can include processes such as generation of a signal and evaluation value used in automatic focus detection (AF) and generation of an evaluation value used in automatic exposure control (AE). The special effects processing may include adding a blur effect, changing color tones, relighting, and the like. Note that these are merely examples of processes that the image processing unit 24 can apply, and do not limit the processes that the image processing unit 24 can apply.

[0017] The gyroscope 40 is a motion sensor that outputs a signal that indicates the motion of the imaging device 100. The gyroscope 40 is a three-axis angular velocity sensor that detects angular velocities around three axes, for example, an axis parallel to the optical axis of the imaging lens 103, an axis parallel to the direction of gravity, and an axis perpendicular to these two axes. Note that a sensor that detects acceleration in each axial direction may also be used in combination.

[0018] A system control unit 50, which will be described later, detects the movement of the imaging device 100 from the output of the gyro 40. The system control unit 50 realizes an image blur correction function by controlling the position of the shift lens of the imaging lens 103 and the position at which the image processing unit 24 trims the captured image according to the detected movement.

[0019] The system control unit 50 is, for example, a processor (CPU, MPU, microprocessor, etc.) capable of executing a program. The system control unit 50 loads a program stored in the non-volatile memory 56 into the system memory 52 and executes it, thereby controlling the operation of each unit of the imaging device 100 and realizing the functions of the imaging device 100. If the imaging lens 103 is replaceable, the system control unit 50 communicates with a controller included in the imaging lens 103 and controls the operation of the imaging lens 103.

[0020] The system control unit 50 also controls operations related to shooting and recording. Specifically, the system control unit 50 controls an automatic exposure control (AE) and an automatic focus detection (AF) operation based on an evaluation value generated by the image processing unit 24, an imaging operation by the imaging unit 22, an operation of the image processing unit 24, an operation of recording image data to the recording medium 150, and the like.

[0021] The non-volatile memory 56 is electrically rewritable and stores programs executed by the system control unit 50, various setting values ​​of the imaging device 100, GUI data, etc. The system memory 52 is used by the system control unit 50 to load programs to be executed and to temporarily store variables, etc. while the programs are being executed.

[0022] The image data output by the A / D converter 23 is supplied to the memory control unit 15 directly or via the image processing unit 24. The memory control unit 15 manages access to the memory 32. The memory control unit 15 stores the image data supplied from the A / D converter 23 or the image processing unit 24 in the memory 32. The memory control unit 15 also supplies the display image data stored in the video memory area of ​​the memory 32 to the D / A converter 13. The D / A converter 13 generates an analog video signal suitable for the display unit 28 based on the display image data.

[0023] The memory 32 is used as a data buffer memory, a working memory for the image processing unit 24, a video memory, and the like.

[0024] The display unit 28 is, for example, a liquid crystal display or an organic EL display. By continuously capturing moving images by the imaging unit 22 and displaying the captured moving images on the display unit 28, the display unit 28 can function as an electronic viewfinder (EVF). The moving images displayed to make the display unit 28 function as an EVF are called live view images. The display unit 28 may be a touch display.

[0025] The system timer 53 has a clock generator that outputs a reference clock and a built-in clock. The system timer 53 supplies a clock signal generated based on the reference clock to each functional block of the imaging device 100. The system timer 53 also uses the built-in clock to supply information on the current date and time and to measure a specific length of time.

[0026] In this embodiment, the clock signal generated by the system timer 53 can be supplied to an external device (for example, the external control device 180 and / or the external imaging device 200) through the control signal I / F 19. In this case, the clock signal can be used as a signal (external clock) for synchronizing the operation of the external device with the operation of the imaging device 100.

[0027] On the other hand, a clock signal (external clock) can also be supplied from an external device (e.g., external control device 180 and / or external imaging device 200) via control signal I / F 19. In this case, system control unit 50 controls system timer 53 to generate a clock signal based on the external clock. This makes it possible to synchronize the operation of imaging device 100 with the operation of a device that supplies the external clock (or another device that receives the same external clock).

[0028] The control signal I / F 19 is capable of communicating with the external control device 180 and the external imaging device 200. In addition, the system control unit 50 can control the operation of the imaging device 100 in response to a control signal supplied from the external device via the control signal I / F 19.

[0029] For example, the system control unit 50 can control the operation of the imaging unit 22 and the operation of the image processing unit 24 in response to a control signal supplied from an external device. Also, the system control unit 50 can transmit information about the imaging device 100 to the requesting external device in response to a request made by a control signal supplied from the external device. Furthermore, the system control unit 50 can output information about the timing of the operation (e.g., shooting) of the imaging device 100 to the external control device 180 and the external imaging device 200 via the control signal I / F 19.

[0030] By notifying the external imaging device 200 of information about the operation cycle of the imaging device 100 through the control signal I / F 19, it is possible to match the operation cycle of the imaging device 100 with the operation cycle of the external imaging device 200. Conversely, by receiving information about the operation cycle from the external imaging device 200 through the control signal I / F 19 and controlling the operation timing of the imaging device 100 with the system control unit 50, it is also possible to match the operation cycle of the imaging device 100 with the operation cycle of the imaging device 200.

[0031] The mode changeover switch 60, the recording switch 61, and the operation unit 70 are examples of operation members (switches, buttons, dials, levers, touch panels, etc.) that allow the user to input instructions to the imaging device 100 (system control unit 50).

[0032] The mode changeover switch 60 is a switch for switching (selecting) the operation mode of the imaging device 100. Representative operation modes include, but are not limited to, a moving image recording mode, a still image recording mode, and a playback mode. The operation modes may be hierarchical. For example, the moving image recording mode may include two or more of an auto shooting mode, an auto scene determination mode, a manual mode, a scene mode, a program AE mode, and a custom mode. Furthermore, the scene mode may include a plurality of modes (night view mode, starry sky mode, portrait mode, sports mode, etc.) according to the target scene.

[0033] The mode changeover switch 60 may be capable of selecting all operation modes, or may be capable of selecting only the operation modes of a specific hierarchy level. In the latter case, the operation modes of the other hierarchy levels are configured to be selected using a separate operating member.

[0034] Each time the recording switch 61 is operated, the imaging device 100 switches between a shooting standby state and a shooting state. Note that shooting here refers to shooting for the purpose of recording. Shooting for non-recording purposes (e.g. shooting for live view display) can also be performed in the shooting standby state.

[0035] The operation unit 70 is a general term for operation members other than the mode changeover switch 60 and the recording switch 61. The operation members constituting the operation unit 70 have names according to the functions assigned to them. Typical examples are directional keys, menu buttons, and a confirmation button. The functions assigned to the same input member may be variable. The input members may also be software buttons or keys using a touch display. The operation unit 70 may also have input members compatible with non-contact input methods such as voice input and eye-gaze input.

[0036] The power supply control unit 80 includes a battery detection circuit, a DC-DC converter, a switch circuit for switching between blocks to which electricity is applied, etc. The power supply control unit 80 detects whether a battery is installed, the type of battery, and the remaining battery power. The power supply control unit 80 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50 to supply appropriate voltages to each unit of the imaging device 100.

[0037] The power supply unit 30 may be any of a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li-ion battery, an AC adapter, or the like.

[0038] Recording moving image data (including audio data) and still image data obtained by imaging are recorded on a recording medium 150. The recording medium 150 is, for example, a semiconductor memory card or an external recording device (HDD, SSD, etc.).

[0039] The video signal I / F 18 is an interface with the recording medium 150 and the display control device 160. The video signal I / F 18 has hardware according to the specifications of the connected device and the recording medium 150. In this embodiment, the imaging device 100 can output to the outside (display control device 160) through the video signal I / F 18 based on image data for display.

[0040] The display control device 160 applies gradation characteristics (gamma curve) according to a display device (not shown) to the video signal output from the imaging device 100 via the video signal I / F 18, and outputs the signal to the display device.

[0041] (Imaging system) Fig. 2 is a schematic diagram showing an example of an imaging system 1000 for in-camera VFX video according to this embodiment. Note that the scale of the devices is not uniform in Fig. 2. In the following description, the external control device 180 is referred to as the control device 180, and the external imaging device 200 is referred to as the imaging device 200.

[0042] The imaging system 1000 has a plurality of imaging devices (here, imaging devices 100 and 200), a control device 180 which is a control device of the imaging system, and a display device 400. For ease of explanation and understanding, it is assumed here that the imaging device 200 has the functional configuration shown in FIG. 1, similar to the imaging device 100. Note that the functional configurations of the imaging devices included in the imaging system may be different as long as their operations can be controlled by the control device 180. Furthermore, the imaging system 1000 may have three or more imaging devices.

[0043] The display device 400 displays an image that is the background of the scene captured by the imaging devices 100 and 200. The display device 400 has, for example, many LED panels arranged vertically and horizontally, and is sometimes called an LED wall. For ease of explanation and understanding in this specification, it is assumed that the display device 400 has one LED panel, and the control device 180 directly controls the display of the display device 400. Note that in practice, a separate display control device can be used to divide and display the CG image rendered by the control device 180 on the many LED panels of the display device 400.

[0044] In the imaging system 1000, the control device 180 causes the display device 400 to display CG corresponding to the position and orientation of the imaging device performing imaging in accordance with the imaging timing of each connected imaging device. Alternatively, the control device 180 controls the operation timing of the display device 400 and each imaging device so that one imaging device performs imaging during a period in which CG corresponding to the position and orientation of one of the connected imaging devices is displayed on the display device 400.

[0045] For example, if one frame period is divided into imaging periods for each imaging device, all imaging devices can perform imaging with a CG background according to their own position and orientation within one frame period. Since multiple imaging devices can obtain captured images with a CG background according to their own position and orientation at a predetermined frame rate, this is essentially equivalent to multiple imaging devices capturing in-camera VFX video in parallel. In this way, the control device 180 enables multiple imaging devices to capture in-camera VFX video in parallel.

[0046] When rendering a CG image according to the position and orientation of the imaging device, the control device 180 can specify the range of the screen of the display device 400 that is actually captured, based on the information on the position and orientation and the information on the angle of view (focal length of the imaging lens) acquired from the imaging device. For example, the control device 180 may reduce the processing load of rendering by rendering the CG with lower image quality outside the imaging range (outer frustum) than within the imaging range (inner frustum).

[0047] 2, position and orientation detection devices 201 and 202 are attached to image capture devices 100 and 200, respectively, and output position and orientation information of the image capture devices 100 and 200 to a control device 180. Such position and orientation detection devices are known as camera tracking systems, and detailed description thereof will be omitted. For example, the position and orientation of the image capture devices can be detected by using images captured of multiple markers whose absolute positions are known, or by using signals received from multiple transmitters whose absolute positions are known.

[0048] 3 is a block diagram showing an example of the functional configuration of the control device 180. The control device 180 can be realized using, for example, a computer device. The control unit 181 is, for example, a CPU, and realizes the functions of the display control device 180 by loading a program (display control application program) stored in a ROM 187 into a RAM 188 and executing it.

[0049] The image processing circuit 182 is, for example, a graphic board equipped with a GPU. The image processing circuit 182 can execute image processing such as CG rendering at higher speeds than the control unit 181.

[0050] The first I / F 183, the second I / F 184, the third I / F 185, and the fourth I / F 186 are communication interfaces for connecting external devices. In this embodiment, the image capture device 100 is connected to the first I / F 183, the display device 400 is connected to the second I / F 184, the image capture device 200 is connected to the third I / F 185, and the position and orientation detection devices 201 and 202 are connected to the fourth I / F 186. In this embodiment, the first I / F 183 to the third I / F 185 can transmit video signals, synchronization signals, and control signals. Note that more communication interfaces may be provided to enable connection of more devices.

[0051] For convenience, the display control device 180 is illustrated as being connected to each of the imaging devices 100 and 200 and the display device 400 through one I / F, but they may be connected through multiple I / Fs depending on the signals to be communicated. The same applies to the connections between the display control device 180 and the display device 400 and the imaging device 200.

[0052] The control unit 181 acquires information and signals and controls the operations of the imaging device 100, the display device 400, and the imaging device 200 through communication via the first I / F 183 to the third I / F 185. Note that the display control device 180 may have four or more communication interfaces with external devices.

[0053] Furthermore, the control unit 181 continuously acquires the position and orientation information of the image capturing devices 100 and 200 from the position and orientation detection devices 201 and 202 through communication via the fourth I / F 186. Note that, for the image capturing devices 100 and 200 whose positions and shooting directions are fixed and known, it is not necessary to acquire the position and orientation information from the position and orientation detection device.

[0054] The ROM 187 stores programs (firmware, OS, application programs, etc.) executed by the control unit 181, GUI image data such as menu screens, setting values ​​for the display control device 180, and the like.

[0055] The RAM 188 is used as the main memory for the control unit 181 , as well as a working memory for the image processing circuit 125 and a video memory for the display unit 190 .

[0056] The storage unit 189 is a large-capacity storage device such as a hard disk or SSD. The storage unit 189 stores data (such as a three-dimensional model) required for rendering CG to be displayed on the display device 400, and applications for rendering CG. The OS and applications may be stored in the storage unit 189 instead of the ROM 187.

[0057] The display unit 190 is, for example, a liquid crystal display device. The display unit 190 may be a touch display. The display unit 190 displays a screen of an imaging control application, etc. The display unit 190 may also display a video signal output from the imaging devices 100 and 200 via the video signal I / F 18. The display unit 190 may also be an external display device.

[0058] The operation unit 191 has a plurality of input devices that can be operated by the user, such as a keyboard, a mouse, a touch pad, etc. When the display unit 190 is a touch display, the operation unit 191 includes a touch panel.

[0059] The clock generation circuit 192 generates a synchronization signal (clock) for synchronizing the operations of the imaging device 100, the display device 400, and the imaging device 200. The clock may be obtained from an external device (including the imaging devices 100 and 200).

[0060] (Operation of imaging system) Fig. 4 is a flowchart relating to the operation of the control device 180 in this embodiment. Fig. 5 is a sequence diagram relating to the operation of the control device 180 shown in Fig. 4 and the operations of the imaging devices 100, 200, and the display device 400 based on the operation.

[0061] Here, the control device 180 (control unit 181) executes an imaging control application to control the operations of the imaging device 100, the imaging device 200, and the display device 400. However, the control described as being performed by the control device 180 may be performed by the target device itself or another device (excluding the control device 180).

[0062] It is assumed that necessary pre-processing, such as setting each device and connecting the devices, has been completed before starting the following operations. Specifically, it is assumed that an imaging control application is running in the control device 180, and that the control device 180 has acquired information required for controlling the operations of the imaging devices 100, 200 and the display device 400 from each device. It is also assumed that the imaging devices 100, 200 and the display device 400 are set to operation modes in which their operations can be controlled by the control device 180.

[0063] In the following description, operations mainly performed by the imaging device 100 or 200 are performed by the respective system control units 50, and operations mainly performed by the control device 180 are performed by the control unit 181.

[0064] In S301, the control device 180 outputs instructions regarding operation timing control to the image capture device 100 and the image capture device 200 (FIG. 5, 501 and 502). Here, it is assumed that the control device 180 outputs an instruction to assign the image capture device 100 as the master device among the multiple image capture devices. The image capture devices other than the master device perform operations synchronized with the master device.

[0065] Based on the fact that the image capturing apparatus 100 has been designated as the master apparatus, the image capturing apparatus 100 supplies an external clock signal from the system timer 53 to the image capturing apparatus 200. In addition, the image capturing apparatus 100 supplies information relating to its own operation timing to the image capturing apparatus 200 (503).

[0066] When the imaging device 200 receives the information regarding the operation timing from the imaging device 100, the imaging device 200 executes an operation timing control operation to control its own operation in accordance with the external clock signal supplied from the imaging device 100 (504). Then, the imaging device 200 outputs the information regarding its own operation timing to the control device 180. The imaging device 100 also outputs the information regarding its own operation timing to the control device 180 (506).

[0067] When the control device 180 receives information about the operation timing from each of the image capture devices 100 and 200, it checks whether the operation timing is synchronized (507). Here, it is assumed that it is determined that the synchronization is successful.

[0068] Next, in S302, the control device 180 outputs instructions for the imaging periods (exposure periods) to the imaging devices 100 and 200 (508, 509). Here, the control device 180 allocates the imaging periods so that the individual imaging devices capture images at different timings. The control device 180 can allocate imaging periods that do not overlap with each other to the individual imaging devices, for example, as shown in FIG. 6.

[0069] The image capturing period is repeatedly allocated in a cycle according to the frame rate. The control device 180 notifies each image capturing device of information (exposure pattern) related to the timing of the allocated image capturing period. The exposure pattern enables each image capturing device to capture images in a time-division manner.

[0070] In the example shown in Fig. 6, the exposure patterns are designed so that the imaging periods of all the imaging devices fall within one frame period according to the frame rate. Then, exposure pattern 1 having an imaging period in the first half of one frame period is assigned to imaging device 100, and exposure pattern 2 having an imaging period in the second half of one frame period is assigned to imaging device 200. Note that the length of the imaging period may differ depending on the imaging device. Also, there may be a period within one frame period during which no imaging device captures an image.

[0071] In S303, the control device 180 renders a CG (background image) to be displayed on the display device 400 during the image capturing period of each image capturing device (512). The control device 180 renders the background image using the position and orientation information of the image capturing devices 100 and 200 obtained from the position and orientation detection devices 201 and 202 and information on the angle of view of the image capturing devices 100 and 200.

[0072] For example, consider a case where an in-camera VFX video is captured with a building as the background in an arrangement as shown in the plan view of Fig. 7(a). In this case, the control device 180 renders the background image shown in Fig. 7(b) for the imaging device 100, and the background image shown in Fig. 7(c) for the imaging device 200.

[0073] The imaging device 100 controls the operation of the imaging section 22 so as to accumulate charge during the exposure period of the instructed exposure pattern 1 (510). The imaging device 100 generates moving image data for recording using the image processing section 24, and records the data on the recording medium 150. Furthermore, the imaging device 100 outputs a video signal for live view display to the control device 180 via the video signal I / F 18.

[0074] Similarly, the imaging device 200 controls the operation of the imaging section 22 so as to accumulate charge during the exposure period of the instructed exposure pattern 2 (511). The imaging device 100 generates moving image data for recording by the image processing section 24, and records the data on the recording medium 150. Furthermore, the imaging device 200 outputs a video signal for live view display to the control device 180 via the video signal I / F 18.

[0075] In S304, the control device 180 causes the display device 400 to display the generated background image. As with imaging, the display of the background image must be repeatedly executed at a period corresponding to the frame rate. Therefore, the control device 180 determines the periodic display timing (display pattern) of the background image corresponding to each imaging device. In the example of Fig. 6, the control device 180 determines display pattern 1 for the imaging device 100 and display pattern 2 for the imaging device 200.

[0076] Then, the control device 180 instructs the display device 400 (513) to display a background image for the imaging device 100 at a timing (514) indicated according to the display pattern 1, and to display a background image for the imaging device 200 at a timing (515) indicated according to the display pattern 2.

[0077] In order to realize the timing of the exposure pattern and the display pattern as shown in Fig. 6, the control device 180 may change the period for rendering the background image from that shown in Fig. 5. For example, the control device 180 can render a background image for the imaging device 200 during the imaging period of the imaging device 100, and render a background image for the imaging device 100 during the imaging period of the imaging device 200.

[0078] When notifying the imaging devices of the exposure pattern, the control device 180 can identify the start timing of the imaging period of each imaging device. The control device 180 can also grasp the time difference from the display instruction to the display based on information acquired from the display device 400, for example. Therefore, by issuing a display instruction at a timing such that the background image displayed by the display device 400 is switched in accordance with the start timing of the imaging period of each imaging device, it is possible to realize switching of the background image synchronized with the switching of the imaging period as shown in FIG. 6.

[0079] In this manner, in this embodiment, imaging periods are assigned to the individual imaging devices so that they capture images at different times, and the display device is controlled so that, in each imaging period, a background image according to the position and orientation of the imaging device to which the imaging period is assigned is displayed. This makes it possible to capture in-camera VFX footage of the same scene in parallel using multiple imaging devices.

[0080] <Second embodiment> In the first embodiment, the exposure pattern is designed so that the imaging periods of all the imaging devices are within one frame period. Therefore, the length of the imaging period allocated to each imaging device is at most one frame period / total number of imaging devices. For example, in imaging at 24 frames / second and a shutter opening angle of 172.8 degrees, the exposure time per frame is 1 / 50 seconds. In this case, since (1 / 24) / (1 / 50)=2.08, if there are two imaging devices, the exposure time required for each imaging device can be secured. However, if there are three or more imaging devices, there will be insufficient exposure or the imaging periods allocated to the imaging devices will overlap.

[0081] This embodiment enables each imaging device to capture an appropriate background image even when the imaging periods overlap. This embodiment is basically the same as the first embodiment except for the exposure pattern and display pattern determined by the control device 180. Therefore, only the operation of the control device 180 of this embodiment will be described below using the flowchart in FIG.

[0082] In the following, a case will be described in which three imaging devices (imaging device A, imaging device B, and imaging device C) are connected to the control device 180. In this case, the control device 180 has more communication interfaces than the example shown in Fig. 3. The same concept can also be applied to the case in which four or more imaging devices are connected.

[0083] In this embodiment, the control device 180 assigns imaging periods with different start timings to the three imaging devices A, B, and C, each having a length determined from the frame rate and shutter open angle (1 / 50 seconds in the above example). Specifically, if the number of imaging devices is n, the control device 180 assigns imaging periods to each imaging device such that the start timings are shifted by 1 / n (=F / n) of one frame period F. Note that the length of the imaging period may differ depending on the imaging device.

[0084] Therefore, in S301 of Fig. 4, the control device 180 instructs each of the imaging devices A, B, and C to operate at timings shifted by F / 3. Then, the control device 180 acquires information on the operation timing of each imaging device and checks whether or not the imaging devices are operating at the instructed timing. Here, it is assumed that it has been confirmed that the imaging devices are operating at the instructed timing.

[0085] Next, in S302, the control device 180 determines and notifies each of the image capturing devices A, B, and C of exposure patterns, for example, as shown in Fig. 8. Here, it is assumed that exposure pattern 1 is determined for image capturing device A, exposure pattern 2 is determined for image capturing device B, and exposure pattern 3 is determined for image capturing device C.

[0086] In S303, the control device 180 generates a background image suitable for each imaging device, similarly to the first embodiment.

[0087] In S304, the control device 180 causes the generated background image to be displayed on the display device 400. At this time, the control device 180 controls the display on the display device 400 so that the background image for the target imaging device is displayed during a period that is included in the imaging period of the target imaging device and is not included in the imaging periods of the other imaging devices.

[0088] 8, for the imaging device A to which exposure pattern 1 is assigned, the control device 180 controls the display timing so that a background image for the imaging device A is displayed on the display device 400 according to display pattern 1. Similarly, for the imaging devices B and C, the control device 180 controls the display timing of the background images according to display patterns 2 and 3.

[0089] By displaying background images during periods when the imaging devices do not overlap, it is possible to prevent multiple imaging devices from capturing the same background image, while allowing each imaging device to capture an appropriate background image according to its position and orientation. Also, since the period during which each imaging device captures a background image falls within one frame period, in this embodiment as well, each imaging device can capture in-camera VFX video of the same scene in substantial parallel.

[0090] In addition, by setting the length of the imaging period based on the frame rate and the shutter opening angle, it is possible to capture images at a shutter speed that can suppress flicker, for example. However, since the background image is displayed only for a part of the imaging period, the exposure amount of the background image is reduced relative to the exposure amount of the foreground subject. For example, when the length of the imaging period is F / 2, the display period of the background image is F / 6, and the exposure amount of the background image is 1 / 3 of the exposure amount of the foreground subject.

[0091] Therefore, the brightness of the background image can be corrected so that the exposure amount of the background image is the same as that of the foreground subject. For example, if the exposure amount of the background image is 1 / 3 of the exposure amount of the foreground subject, the brightness of the background image generated in S303 or the standard display luminance of display device 400 can be corrected to three times, thereby eliminating the difference in the exposure amounts of the foreground and background images.

[0092] Furthermore, if the background image is a moving image, the exposure times of the foreground subject and the background image are different, which may cause the images to blur differently, creating an unnatural feeling. In such a case, the background image can be rendered in S303 in such a way that the difference in image blur caused by the difference in exposure times is reduced, thereby reducing the unnatural feeling.

[0093] In addition, when the shutter speed (length of the imaging period) can be changed, non-overlapping imaging periods may be assigned to each imaging device, as in the first embodiment. In this case, the imaging sensitivity may be increased to compensate for the underexposure. Alternatively, the underexposure may be corrected for an underexposed image during post-production. In these cases, when three or more imaging devices are used, the exposure pattern and display pattern can be determined in the same manner as in the first embodiment.

[0094] (Other embodiments) In the above-described embodiment, the control device 180 notifies each imaging device of the determined exposure pattern, and the system control unit of each imaging device controls the operation of the imaging device so that the imaging device performs imaging at a timing according to the notified exposure pattern.

[0095] However, the control device 180 may notify all determined exposure patterns to one specific imaging device (master device), and the system control unit of the master device may control the operation timing of the other imaging devices according to the corresponding exposure patterns. For example, the system control unit of the master device may control the other imaging devices to perform imaging operations at operation timings according to the exposure patterns by issuing instructions to the other imaging devices from the control signal I / F 19 at timings according to the exposure patterns of the other imaging devices. In this case, an external clock signal is supplied from the system timer to the other imaging devices, thereby synchronizing the operation timing of the other imaging devices with a clock signal common to the own device.

[0096] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0097] The disclosure of the present embodiment includes a control device, a control method, an imaging device, and a program for an imaging system as described below. (Item 1) A control device for an imaging system that captures an image with an image displayed on a display device as a background, an allocation means for allocating imaging periods with different timings to each imaging device; an imaging control means for controlling each of the imaging devices so that each of the imaging devices performs imaging during an assigned imaging period; a display control means for causing the display device to display, during each of the imaging periods, an image corresponding to a position and orientation of an imaging device to which the imaging period is assigned; A control device comprising: (Item 2) 2. The control device according to item 1, wherein the allocation unit allocates the imaging periods to the individual imaging devices so that the imaging periods are within one frame period. (Item 3) 3. The control device according to item 2, wherein the allocation means allocates imaging periods to the individual imaging devices so that the imaging periods do not overlap with each other. (Item 4) 3. The control device according to item 2, wherein the allocation means allocates imaging periods to the individual imaging devices such that the imaging periods partially overlap. (Item 5) 5. The control device according to item 4, wherein the display control means causes the display device to display the image during each of the imaging periods in a period that does not overlap with other imaging periods. (Item 6) 6. The control device according to item 4 or 5, further comprising a correction means for correcting a difference in exposure between the image and a foreground subject. (Item 7) 7. The control device according to item 6, wherein the correction means corrects the difference in the amount of exposure by correcting the brightness of the image. (Item 8) 7. The control device according to item 6, wherein the correction means corrects the difference in the amount of exposure by correcting the display luminance of the display device. (Item 9) 1. A control method executed by an imaging system that captures an image with an image displayed on a display device as a background, comprising: Allocating imaging periods with different timings to each imaging device; controlling each of the imaging devices so that the imaging devices perform imaging during the assigned imaging period; displaying, on the display device, an image corresponding to a position and orientation of an imaging device to which the imaging period is assigned, during each of the imaging periods; A control method comprising the steps of: (Item 10) an acquisition means for acquiring information regarding image capture timings of the image capture device and other image capture devices from an external device; a control means for controlling the image capturing timing of the other image capturing device based on the information; An imaging device comprising: (Item 11) 11. The imaging device according to item 10, further comprising a supplying means for supplying an external clock signal to the other imaging device. (Item 12) A program for causing a computer to function as various means possessed by the control device according to any one of items 1 to 8.

[0098] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention. [Explanation of symbols]

[0099] 50: system control unit, 100, 200: imaging device, 180: external control device, 400: display device

Claims

1. A control device for an imaging system that captures an image with an image displayed on a display device as a background, an allocation means for allocating imaging periods with different timings to each imaging device; an imaging control means for controlling each of the imaging devices so that each of the imaging devices performs imaging during an assigned imaging period; a display control means for causing the display device to display, during each of the imaging periods, an image corresponding to a position and orientation of an imaging device to which the imaging period is assigned; A control device comprising:

2. 2. The control device according to claim 1, wherein the allocating means allocates the imaging periods to the individual imaging devices so that the imaging periods are within one frame period.

3. 3. The control device according to claim 2, wherein the allocating means allocates image capture periods to the individual image capture devices so that the image capture periods do not overlap with each other.

4. 3. The control device according to claim 2, wherein the allocating means allocates the image capture periods to the individual image capture devices so that the image capture periods partially overlap each other.

5. 5. The control device according to claim 4, wherein the display control means causes the display device to display the image during each of the imaging periods, during a period that does not overlap with other imaging periods.

6. 5. The control device according to claim 4, further comprising a correction means for correcting a difference in exposure between the image and a foreground object.

7. 7. The control device according to claim 6, wherein the correction means corrects the difference in the amount of exposure by correcting brightness of the image.

8. 7. The control device according to claim 6, wherein the correction means corrects the difference in the amount of exposure by correcting the display luminance of the display device.

9. 1. A control method executed by an imaging system that captures an image with an image displayed on a display device as a background, comprising: Allocating imaging periods with different timings to each imaging device; controlling each of the imaging devices so that the imaging devices perform imaging during the assigned imaging period; displaying, on the display device, an image corresponding to a position and orientation of an imaging device to which the imaging period is assigned, during each of the imaging periods; A control method comprising the steps of:

10. an acquisition means for acquiring information regarding image capture timings of the image capture device and other image capture devices from an external device; a control means for controlling the image capturing timing of the other image capturing device based on the information; An imaging device comprising:

11. 11. The image pickup apparatus according to claim 10, further comprising a supplying means for supplying an external clock signal to said other image pickup apparatus.

12. A program for causing a computer to function as various means included in the control device according to any one of claims 1 to 8.

Images