Scene control device and scene control method, imaging apparatus, and virtual studio system

Abstract

To provide a scene control device capable of controlling the brightness of an imaging scene to suppress a change in exposure conditions of a camera for capturing a video as a background.SOLUTION: The scene control device determines whether it is necessary to change the brightness of at least one of a lighting device that illuminates an imaging scene and a video image displayed as a background of the imaging scene on the basis of information about the luminance of the imaging scene of an imaging apparatus. The scene control device changes the brightness of at least one of the lighting device that illuminates the imaging scene and the video displayed as the background according to the result of the determination.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a scene control device, a scene control method, an imaging device, and a virtual studio system. [Background technology]

[0002] A method (in-camera VFX) is known in which a subject is captured against a background image that corresponds to the position and orientation of the camera, thereby obtaining VFX (Visual Effects) footage without combining a background image with a live-action image (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7190594 Summary of the Invention [Problem to be solved by the invention]

[0004] During in-camera VFX footage (video), the automatic exposure (AE) function can change the camera's exposure conditions according to changes in the brightness of the scene. During video shooting, the amount of exposure is often controlled by the imaging sensitivity to avoid changes in the depth of field, etc.

[0005] Changes in imaging sensitivity affect the amount of noise in the image. Also, when AE control is performed to ensure proper exposure for a specific subject, changes in the brightness of the specific subject can cause the background to become overexposed or underexposed.

[0006] In light of these problems, in one aspect, the present invention provides a scene control device and a scene control method capable of controlling the brightness of an imaging scene so as to suppress changes in the exposure conditions of a camera that captures video with an image as the background. [Means for solving the problem]

[0007] In one aspect, the present invention provides a scene control device comprising: an acquisition means for acquiring information regarding the brightness of an imaging scene of an imaging device; a determination means for determining, based on the information, whether or not it is necessary to change the brightness of at least one of a lighting device that illuminates the imaging scene and an image displayed as the background of the imaging scene; and a control means for changing at least one of the brightness of the lighting device that illuminates the imaging scene and an image displayed as the background of the imaging scene depending on the result of the determination. Effect of the Invention

[0008] According to one aspect of the present invention, a scene control device and a scene control method can be provided that are capable of controlling the brightness of an imaging scene so as to suppress changes in the exposure conditions of a camera that captures video with an image as a background. [Brief description of the drawings]

[0009] [Figure 1] Schematic diagram of a virtual studio system according to an embodiment. [Diagram 2] A block diagram showing an example of the functional configuration of the camera in Figure 1 and the connections of each device. [Diagram 3] FIG. 1 is a block diagram showing an example of a functional configuration of a scene control device according to a first embodiment; [Figure 4] Flowchart of scene control operation in first mode [Diagram 5] Flowchart of scene control operation in second mode DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The present invention will be described in detail below based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. In addition, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated explanations are omitted.

[0011] ●(First embodiment) (Virtual Studio System Overview) 1 is a schematic diagram of a virtual studio system according to an embodiment. In virtual studio 100, camera 200 captures a VFX video by capturing an image of a real subject 400 against a background of images displayed on large display devices 310, 320, also known as LED walls. Captured range 311 is an example of a range of the background image captured by camera 200.

[0012] Viewpoint detection device 130 detects the viewpoint (position and orientation) of camera 200 from the absolute coordinates of marker 131 provided on the ceiling and the position of marker 131 included in an image of the ceiling captured by a viewpoint detection camera provided on camera 200. The position and orientation of camera 200 can be detected using any known method.

[0013] The scene control device 110 renders a preset three-dimensional model of a virtual space according to the viewpoint of the camera 200 detected by the viewpoint detection device 130, and generates a CG background image at a predetermined frame rate. When the shooting direction of the camera 200 is not directly facing the display devices 310 and 320, the scene control device 110 applies coordinate conversion (deformation processing) required to display the background image on the display devices 310 and 320. The scene control device 110 outputs the generated background image to the display control device 120. In addition, the scene control device 110 controls the lighting control device 140 based on information on the luminance of the captured scene obtained from the camera 200, thereby controlling the brightness of the captured scene, particularly the lighting 350 that illuminates the real subject 400.

[0014] Display control device 120 causes display devices 310 and 320 to display a background image in synchronization with the imaging timing of camera 200. When display devices 310 and 320 each include a plurality of display panels, display control device 120 divides the background image to fit each display panel before displaying the divided image.

[0015] 1, information on the luminance of the scene captured by the camera 200 is supplied from the camera 200 to the scene control device 110. However, information on the luminance of the scene captured by the camera 200 may be supplied to the scene control device 110 from an external exposure meter separate from the camera 200.

[0016] Furthermore, a synchronization signal is supplied from a reference clock generating device, also called a sync generator, to the camera 200, the viewpoint detection device 130, the scene control device 110, and the display control device 120. Each device controls its operation timing according to the reference clock, thereby realizing synchronization between the shooting period of the camera 200 and the display periods of the display devices 310 and 320. A technique for synchronizing the operations of devices based on a reference clock is well known, for example, as genlock (generator locking), and therefore a detailed description thereof will be omitted.

[0017] In this specification, a subject included in a background image displayed on display devices 310, 320 is called a virtual subject, in contrast to a real subject 400 that exists between display devices 310, 320 and camera 200. For convenience, this specification assumes that real subjects and virtual subjects are human subjects, but there is no limit to the type and number of types of subjects.

[0018] Fig. 2 is a block diagram showing the connection relationship of each device shown in Fig. 1 and an example of the functional configuration of camera 200. Among the functional blocks of camera 200, first optical system 210, first imaging section 220, image processing section 230, and recording section 250 realize a function of capturing and recording in-camera VFX video. In addition, second optical system 260, second imaging section 270, and A / D conversion section 280 realize a function of capturing an image for detecting the viewpoint of camera 200. Hereinafter, unless otherwise specified, first imaging section 220 and second imaging section 270 capture a video having a predetermined frame rate.

[0019] The control unit 240 has a processor (CPU, MPU, microprocessor, etc.) capable of executing a program, a ROM, and a RAM. The control unit 240 loads a program stored in the ROM into the RAM and executes it, thereby controlling the operation of each functional block of the camera 200 and realizing the operation of the camera 200 described below. Although not shown in the figure, the control unit 240 is connected to each functional block of the camera 200 so as to be able to communicate with them.

[0020] The angle of view and the optical axis direction of the second optical system 260 are determined so as to capture an image of the viewpoint detection marker 131 placed on the ceiling of the studio. The angle of view may be fixed or variable. The second imaging unit 270 has an imaging element and converts the optical image formed by the second optical system 260 into an analog image signal. Since it is sufficient for the image for viewpoint detection to be able to acquire the image coordinates of the marker 131, color information is not necessary, and the imaging element may be a monochrome image sensor.

[0021] The A / D conversion unit 280 generates a digital image signal by A / D converting the analog image signal output by the second imaging unit 270. The digital image signal is output to the viewpoint detection device .

[0022] The first optical system 210 is an optical system for capturing an in-camera VFX image. Therefore, the angle of view and the optical axis direction of the first optical system are determined so as to form an optical image of the real subject 400 against the background of the images displayed on the display devices 310 and 320. The angle of view of the first optical system 210 may be variable.

[0023] The first imaging unit 220 has an imaging element and converts the optical image formed by the first optical system 210 into an analog image signal. The imaging element of the first imaging unit 220 may be, for example, a known CCD or CMOS color image sensor having a primary color Bayer array color filter. The analog image signal output by the first imaging unit 220 is supplied to the image processing unit 230.

[0024] The image processing unit 230 applies predetermined image processing to the analog image signal output by the first imaging unit 220, generates signals and image data according to the application, and acquires and / or generates various information. The image processing unit 230 may be a dedicated hardware circuit such as an ASIC (Application Specific Integrated Circuit) designed to realize a specific function. Alternatively, the image processing unit 230 may be configured to realize a specific function by a processor such as a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit) executing software.

[0025] The image processing applied by the image processing unit 230 can include, for example, pre-processing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Pre-processing may include A / D conversion, signal amplification, reference level adjustment, defective pixel correction, etc. Color interpolation processing is performed when a color filter is provided on an image sensor, and is a process for interpolating values ​​of color components that are not included in the individual pixel data that constitutes the image data. Color interpolation processing is also called demosaic processing. The correction processing can include white balance adjustment, gradation correction, correction of image degradation caused by optical aberration of the first optical system 210 (image restoration), correction of the effect of peripheral light falloff of the first optical system 210, color correction, and the like. The detection process can include detection of a characteristic region or a region of a specific subject (for example, a face region or a human body region) and its movement, and person recognition processing. Data processing may include processes such as area extraction (trimming), synthesis, scaling, encoding and decoding, header information generation (data file generation), etc. Data processing also includes the generation of image data for display or image data for recording. The evaluation value calculation process may include processes such as generating a signal or evaluation value used in automatic focus detection (AF) and generating an evaluation value used in automatic exposure control (AE). The evaluation value used in AE is information about the luminance of the imaging scene, but the information about the luminance of which part of the imaging scene is different depending on, for example, the exposure mode that is set. For example, the information may reflect the luminance of the entire imaging scene, or it may be information about the luminance of an area of ​​a specific subject. The special effects processing may include adding a blur effect, changing color tones, relighting, and the like.

[0026] Note that these are examples of processes that the image processing unit 230 can apply, and do not limit the processes that the image processing unit 230 applies. The image processing unit 230 outputs acquired or generated information and data to a functional block according to the application. For example, the image processing unit 230 outputs image data for recording to the recording unit 250, and outputs information related to the luminance of the captured scene to the control unit 240.

[0027] The control unit 240 outputs information on the luminance of the imaging scene acquired from the image processing unit 230 to the scene control device 110. The control unit 240 also executes AE processing to determine exposure conditions based on information on the luminance of the imaging scene, and can control the operation of the first imaging unit 220 according to the determined exposure conditions. For example, the control unit 240 can determine exposure conditions such that the entire imaging scene is properly exposed, or determine exposure conditions such that a part of an area included in the shooting scene (for example, an area of ​​a specific subject) is properly exposed.

[0028] Generally, the exposure conditions are determined by a combination of the aperture value, the shutter speed (exposure time), and the imaging sensitivity. Therefore, the control unit 240 can determine a combination of the values ​​of these three parameters as the imaging conditions for obtaining the appropriate exposure. However, if the aperture value or the exposure time is changed during video shooting, the depth of field changes, and the moving distance of a moving subject between frames changes. Therefore, in the AE processing in this embodiment, the imaging sensitivity is determined basically without changing the aperture value and the shutter speed.

[0029] Furthermore, in this embodiment, the control unit 240 has an AE limit mode that limits the reflection of the exposure conditions determined in the AE process in the actual imaging, in addition to the normal AE mode in which imaging is performed under the imaging conditions determined in the AE process. Specifically, the control unit 240 has a first AE limit mode that does not reflect any of the exposure conditions determined in the AE process, and a second AE limit mode that limits the range in which the imaging sensitivity determined in the AE process is reflected. In the second AE limit mode, if the imaging sensitivity determined in the AE process is included in a predetermined imaging sensitivity range, it is reflected as is, if it exceeds the upper limit of the range, it is set to the upper limit, and if it falls below the lower limit of the range, it is set to the lower limit.

[0030] The focal distance of the first optical system 210 can be automatically adjusted by the control unit 240 executing AF processing based on the evaluation value generated by the image processing unit 230. Meanwhile, since the distance between the camera 200 and the ceiling is almost constant, the focal distance of the second optical system 260 is adjusted by manual focus before capturing an image, and does not need to be adjusted during capturing an image. By configuring the A / D conversion unit 280 in the same manner as the image processing unit 230, the control unit 240 may also automatically adjust the focal distance of the second optical system 260 by AF processing.

[0031] 3 is a block diagram showing an example of the functional configuration of the scene control device 110. The scene control device 110 can be realized using, for example, a computer device. The control unit 1101 is, for example, a CPU, and realizes the functions of the scene control device 110 by loading one or more application programs stored in a ROM 1107 into a RAM 1108 and executing them. The control unit 110 controls the operation timing of the scene control device 110 in accordance with a synchronization signal supplied from a reference clock generating device.

[0032] The image processing circuit 1102 is, for example, a graphics board equipped with a GPU, and is capable of high-speed image processing, such as CG rendering.

[0033] The first I / F 1103 to the fourth I / F 1106 are communication interfaces for connecting external devices. In this embodiment, the camera 200 or an external exposure meter is connected to the first I / F 1103, the display control device 120 is connected to the second I / F 1104, the viewpoint detection device 130 is connected to the third I / F 1105, and the lighting control device 140 is connected to the fourth I / F. Note that the first I / F 1103 to the fourth I / F 1106 are compliant with standards according to the type of external device to be connected and the type of signal to be communicated. For convenience, the scene control device 110 and the external device are illustrated as being connected through one I / F, but they may be connected using multiple I / Fs.

[0034] The control unit 1101 acquires captured image data and information related to the luminance of the captured scene from the camera 200 through the first I / F 1103. The control unit 1101 also acquires information related to the viewpoint of the camera 200 from the viewpoint detection device 130 by communication through the third I / F 1105. The control unit 1101 outputs image data for display (background image data) to the display control device through the second I / F 1104. The control unit 1101 also outputs a control signal to the lighting control device 140 through the fourth I / F 1106. Note that the scene control device 110 may have five or more communication interfaces with external devices.

[0035] The ROM 1107 stores some of the programs executed by the control unit 1101 (BIOS, bootstrap loader, firmware), setting values ​​for the scene control device 110, and the like.

[0036] The RAM 1108 is used as the main memory for the control unit 1101 , as well as a working memory for the image processing circuit 1102 and a video memory for the display unit 1110 .

[0037] The storage unit 1109 is a large-capacity storage device such as a hard disk or SSD. The storage unit 1109 stores operating system (OS), application programs, user data, etc. The storage unit 1109 also stores application programs (e.g., game engine applications) that generate background images according to the viewpoint of the camera 200, and data required to generate the background images (3D models of virtual space, textures, etc.).

[0038] The display unit 1110 is, for example, a liquid crystal display device. The display unit 1110 may be a touch display. The display unit 1110 displays a scene control application, a background image generation application (for example, a game engine), a GUI provided by an OS, and the like.

[0039] The operation unit 1111 has a plurality of input devices that can be operated by the user, such as a keyboard, a mouse, a touch pad, etc. When the display unit 1110 is a touch display, the operation unit 1111 includes a touch panel.

[0040] In this embodiment, in order to suppress changes in the exposure conditions of camera 200 or white blowout or black crush of the background image in the in-camera VFX image, the brightness of at least one of the background image and the lighting device is controlled in accordance with changes in the brightness of the scene captured by camera 200.

[0041] In the following, a scene control operation in which the brightness of at least one of the background image and the lighting device is controlled to suppress changes in the exposure conditions of camera 200 is referred to as a first mode scene control operation. Also, a scene control operation in which the brightness of at least one of the background image and the lighting device is controlled to suppress whiteout or blackout of the background image in the in-camera VFX video captured by camera 200 is referred to as a second mode scene control operation.

[0042] The scene control operations of the first mode and the second mode are not exclusive and can be executed as appropriate. For example, the scene control operation of the second mode can be executed when whiteout or blackout of the background image may occur, and the scene control operation of the first mode can be executed when the exposure conditions of the camera 200 need to be changed. Alternatively, the scene control operation of the first mode can be executed when the brightness of the entire image capture scene changes, and the scene control operation of the second mode can be executed when the brightness of the area of ​​a specific subject changes. Note that these are merely examples, and the type of scene control operation to be executed may be automatically determined according to other conditions. Alternatively, the scene control operation of the mode instructed or set by the user may be executed.

[0043] (Scene control operation in 1st mode) Next, the scene control operation in the first mode will be described with reference to the flowchart shown in FIG. 4. The following operation is executed while the camera 200 is capturing an in-camera VFX image (moving image). Here, the moving image may be for recording or for display. The moving image for display may be, for example, for performing a live view display when the camera 200 is in a capturing standby state. Also, it is assumed that the camera 200 is operating in the first or second AE limit mode.

[0044] Note that the following processes required for capturing in-camera VFX footage can be performed using known methods, and therefore detailed explanations of these processes will be omitted. A viewpoint (position and orientation) detection process of the camera 200 by the viewpoint detection device 130 using an image of the marker 131 Background image generation processing by the scene control device 110 according to the detected viewpoint of the camera 200 Background image display control process on the display devices 310 and 320 by the display control device 120

[0045] Furthermore, the scene control device 110 controls the brightness (including turning off) of the lighting device 350 through the lighting control device 140 according to a preset lighting pattern depending on the elapsed time (timeline) from the start of imaging. The position and irradiation direction of the lighting device 350 may be fixed, or the position and irradiation direction may change dynamically, for example, when the imaging staff holds the lighting device 350 by hand.

[0046] In the following description, the operations performed by the scene control device 110 are actually realized by the control unit 1101 executing an appropriate application program. It is also assumed that the scene control unit 110 acquires the exposure conditions (aperture value, shutter speed, and imaging sensitivity) from the camera 200 at least at the start of imaging. The scene control unit 110 regards the imaging conditions at the start of imaging in the camera 200 as imaging conditions set so that the camera 200 can obtain the appropriate exposure.

[0047] In S401, the scene control device 110 acquires information about the viewpoint (position and attitude) of the camera 200 detected by the viewpoint detection device .

[0048] In S403, the scene control device 110 acquires information about the luminance of the captured scene from the camera 200 or an external exposure meter. Here, it is assumed that the luminance of the entire captured scene (for example, the average luminance) is acquired.

[0049] In S405, the scene control device 110 determines, based on the information regarding the brightness of the imaging scene acquired in S403 and the exposure conditions acquired from the camera 200, whether the brightness of the imaging scene is appropriate under the exposure conditions of the camera 200 in the current AE limit mode.

[0050] For example, when the first AE limit mode is set, the scene control device 110 determines that the brightness of the imaging scene is appropriate if it is determined that a properly exposed VFX image can be captured without changing the exposure conditions of the camera 200. Also, when the second AE limit mode is set, the scene control device 110 determines that the brightness of the imaging scene is appropriate if the change amount of the exposure conditions (imaging sensitivity) of the camera 200 for obtaining a properly exposed image VFX image is within a predetermined range. The predetermined range may be, for example, ±2 stops based on the imaging conditions acquired from the camera 200.

[0051] The scene control device 110 can determine whether a properly exposed image VFX image can be captured under the exposure conditions of the camera 200, based on a pre-stored combination of scene luminance and imaging conditions that result in proper exposure. If it is determined that the brightness of the imaging scene is appropriate, the scene control device 110 executes S407, and if not, executes S413.

[0052] In S407, the scene control device 110 generates a CG background image by rendering a 3D model of the virtual space using the viewpoint and angle of view of the camera 200. When transitioning from S405 to S407, the scene control device 110 generates a background image without changing the rendering parameters related to brightness from the default values. The scene control device 110 outputs data of the generated background image to the display control device 120. Note that, when the imaging direction of the camera 200 is not directly facing the display devices 310 and 320, the scene control device 110 applies a process of transforming the image so that it becomes an image seen from a position directly facing the display devices 310 and 320, and then outputs the data of the background image to the display control device 120.

[0053] In S409, the display control device 120 causes the display devices 310 and 320 to display the background image data generated by the scene control unit 110.

[0054] In S411, the scene control unit 110 determines whether or not to end image capture. The scene control unit 110 can determine to end image capture when, for example, image capture according to a predetermined timeline is completed, or when an instruction to end image capture is given by a user via the operation unit 1111. If it is determined that image capture is to be ended, the scene control device 110 ends the scene control operation, and if it is not determined that image capture is to be ended, the scene control device 110 repeatedly executes the operation from S401.

[0055] If it is not determined in S405 that the brightness of the imaging scene is appropriate, the scene control device 110 determines in S413 whether or not the imaging scene is dark (underexposed) for the imaging conditions of the camera 200. If it is determined that the imaging scene is dark for the imaging conditions of the camera 200, the scene control device 110 executes S415, and if not, executes S421.

[0056] In S415, the scene control device 110 determines the amount of increase in brightness of the lighting device 350. When the imaging scene is dark, the brightness of the lighting device 350 and the background image is increased (brightened) so that a VFX image with proper exposure can be obtained under the imaging conditions of the camera 200 determined under the restricted AE mode. Specifically, the scene control device 110 determines the amount of increase in brightness equivalent to the insufficient amount of exposure. For example, when the exposure amount corresponding to n steps of the imaging sensitivity is insufficient, the amount of increase in brightness can be determined as 2 to the power of n. Note that the insufficient amount of exposure can be obtained by, for example, comparing the brightness of the imaging scene that is properly exposed under the imaging conditions of the camera 200 with the brightness indicated by the information on the luminance of the imaging scene acquired in S403.

[0057] Since the light emitted by the lighting device 350 attenuates depending on the distance, the amount of change in brightness of the area illuminated by the lighting device 350 is smaller than the amount of increase in brightness. Therefore, the amount of increase in brightness may be corrected depending on the distance between the lighting device 350 and the area illuminated by the lighting device 350. Since the amount of correction may depend on the type of light source and the light emission pattern used by the lighting device 350, the amount of increase in brightness and the amount of correction may be stored in advance in association with each other.

[0058] In S417, the scene control device 110 outputs the brightness increase amount to the lighting control device 140 to change (make brighter) the brightness of the lighting device 350.

[0059] In S419, the scene control device 110 determines the brightness increase amount of the background image. The brightness increase amount of the background image may also be a value corresponding to the exposure deficiency. Therefore, the brightness increase amount of the lighting device 350 may be used as the brightness increase amount of the background image.

[0060] In S407, the scene control unit 110 generates a CG background image by rendering a 3D model of the virtual space using the viewpoint and angle of view of the camera 200. When S407 is executed after S419, the scene control device 110 generates a background image with changed brightness by applying a brightness increase amount to the rendering parameters. The scene control device 110 outputs data of the generated background image to the display control device 120. The operations in S409 and S411 are as described above.

[0061] In S421, the scene control device 110 determines the amount of brightness reduction of the lighting device 350. When the imaging scene is bright, the brightness of the lighting device 350 and the background image is reduced (darkened) so that a VFX image with proper exposure is obtained under the imaging conditions of the camera 200 determined under the restricted AE mode. Specifically, the scene control device 110 determines the amount of brightness reduction corresponding to the excess exposure amount. For example, when the exposure amount of n steps of the imaging sensitivity is exceeded, the amount of brightness reduction can be determined as 2 to the power of -n. Note that the amount of excess exposure can be obtained by, for example, comparing the brightness of the imaging scene that is proper exposure under the imaging conditions of the camera 200 with the brightness indicated by the information on the luminance of the imaging scene acquired in S403.

[0062] The amount of brightness decrease may be corrected in the same way as when the illumination device 350 is brightened. Since the amount of correction may depend on the type of light source and the light emission pattern used by the illumination device 350, the amount of brightness decrease and the amount of correction may be stored in advance in association with each other.

[0063] In S423, the scene control device 110 outputs the brightness decrease amount to the lighting control device 140 to change the brightness of the lighting device 350 (to make it darker).

[0064] In S425, the scene control device 110 determines the brightness reduction amount of the background image. The brightness reduction amount of the background image may also be a value corresponding to the insufficient exposure amount. Therefore, the brightness reduction amount of the lighting device 350 may be used as the brightness reduction amount of the background image.

[0065] In S407, the scene control unit 110 generates a CG background image by rendering a 3D model of the virtual space using the viewpoint and angle of view of the camera 200. When S407 is executed after S425, the scene control device 110 generates a background image with changed brightness by applying a brightness fall amount to the rendering parameters. The scene control device 110 outputs data of the generated background image to the display control device 120. The operations in S409 and S411 are as described above.

[0066] Instead of changing the brightness of the background image by applying a brightness increase amount or a brightness decrease amount to the rendering parameters, the brightness of the background image may be changed by controlling the display devices 310 and 320. Specifically, in S407, the scene control device 110 generates a background image without changing the rendering parameters related to brightness from their default values. Then, data of the background image and the brightness increase amount or brightness decrease amount are output to the display control device 120.

[0067] When the display control device 120 receives an amount of brightness increase or decrease from the scene control device 110, the display control device 120 displays background image data by applying the amount of brightness increase or decrease to the reference brightness of the display devices 310 and 320. As a result, the display devices 310 and 320 display the background image at a brightness that reflects the amount of brightness increase or decrease.

[0068] (Scene control operation in 2nd mode) Next, the scene control operation in the second mode will be described with reference to the flowchart shown in Fig. 5. The scene control operation in the second mode is also executed while the camera 200 is capturing an in-camera VFX video (moving image). Other prerequisites are also the same as those in the scene control operation in the first mode.

[0069] In S501, the scene control device 110 acquires information about the viewpoint (position and attitude) of the camera 200 detected by the viewpoint detection device .

[0070] In S503, the scene control device 110 acquires information on the luminance of the captured scene from the camera 200 or an external exposure meter. Here, the luminance of the background image area of ​​the captured scene (for example, the average luminance value) and the luminance of the specific subject area are acquired. The specific subject area is an area for which the camera 200 performs exposure control so that the area has proper exposure. Here, it is assumed to be the area of ​​the real subject 400 (face area or human body area). When an external exposure meter is used, an external exposure meter that measures the luminance of the display device and an external exposure meter that measures the luminance of the real subject 400 may be used.

[0071] In S505, the scene control device 110 judges whether the brightness of the background image area and the specific subject area is appropriate under the current AE limit mode of the exposure conditions of the camera 200. The scene control device 110 judges whether the brightness of the background image area and the specific subject area is appropriate based on the information on the brightness of each area acquired in S503 and the exposure conditions acquired from the camera 200. The judgment method may be the same as that described in S405. If it is judged that the brightness of both the background image area and the specific subject area is appropriate, the scene control device 110 executes S507, and if not, executes S513.

[0072] In S507, the scene control device 110 generates a CG background image by rendering a 3D model of the virtual space using the viewpoint and angle of view of the camera 200. When transitioning from S505 to S507, the scene control device 110 generates a background image without changing the rendering parameters related to brightness from the default values. The scene control device 110 outputs data of the generated background image to the display control device 120. Note that, when the imaging direction of the camera 200 is not directly facing the display devices 310 and 320, the scene control device 110 applies a process of transforming the image so that it becomes an image seen from a position directly facing the display devices 310 and 320, and then outputs the data of the background image to the display control device 120.

[0073] In S509, the display control device 120 causes the data display devices 310 and 320 to display the background image generated by the scene control unit 110.

[0074] In S511, the scene control unit 110 judges whether or not to end image capture in the same manner as in S411. If it is judged that image capture should be ended, the scene control device 110 ends the scene control operation, and if not, it repeats the operations from S501.

[0075] If it is not determined in S505 that the brightness of at least one of the background image region and the specific subject region is appropriate, the scene control device 110 determines in S513 whether the background image region has a brightness that causes whiteout or blackout. If it is determined that the background image region has a brightness that causes whiteout or blackout, the scene control device 110 executes S515, and if not, executes S521.

[0076] The brightness at which whiteout or blackout occurs varies depending on the imaging conditions of the camera 200. As described above, imaging conditions other than the imaging sensitivity are generally fixed during video shooting. Therefore, a brightness range (brightness dynamic range) at which blackout or whiteout does not occur can be associated with each imaging sensitivity and stored in advance. Then, the scene control device 110 can determine whether or not the brightness of the background image area is at a brightness at which whiteout or blackout occurs, based on the brightness dynamic range corresponding to the exposure conditions of the camera 200 and the brightness of the background image area.

[0077] S521 is executed when the brightness of the background image area is appropriate and the brightness of the specific object area is not appropriate. In this case, the scene control device 110 changes the brightness of the lighting device so that the brightness of the specific object area becomes appropriate. Also, the scene control device 110 does not change the brightness of the background image.

[0078] Therefore, in S521, the scene control device 110 determines the amount of brightness change of the lighting device 350. The amount of brightness change may be the same as the amount of brightness increase or decrease described in S415 and S421 of the scene control operation of the first mode. The scene control device 110 determines the amount of brightness increase when the brightness of the area of ​​the specific object is insufficient, and determines the amount of brightness decrease when the brightness is too high.

[0079] In S523, the scene control device 110 changes the brightness of the lighting device 350 by outputting the brightness change amount to the lighting control device 140. The scene control device 110 may correct the brightness change amount and then output it to the lighting control device 140, similar to the scene control operation in the first mode.

[0080] When the process moves from S523 to S507, the brightness of the background image is not changed, and the subsequent operations are therefore executed in the same manner as when the process moves from S505 to S507.

[0081] In S515, the scene control device 110 judges whether the brightness of the area of ​​the specific object is appropriate or not, and if it is judged to be appropriate, executes S519, and if it is not judged to be appropriate, executes S517.

[0082] S517 is executed when the area of ​​the background image is overexposed or underexposed, and the brightness of the area of ​​the specific subject is also inappropriate. In this case, the scene control device 110 changes the brightness of the lighting device 350 so that the area of ​​the specific subject has appropriate brightness. The brightness of the background image is also changed so that it is included in the brightness dynamic range of the camera 200.

[0083] Specifically, the scene control device 110 changes the brightness of the lighting device 350 by the same processing as S521 and S523. The scene control device 110 also determines the amount of brightness change of the background image. The amount of brightness change of the background image may be the same as the amount of brightness increase or decrease described in S419 and S423 of the scene control operation of the first mode. The scene control device 110 determines the amount of brightness increase when the brightness of the background image is insufficient, and determines the amount of brightness decrease when the brightness is too high.

[0084] S519 is executed when the area of ​​the background image is overexposed or underexposed, but the brightness of the area of ​​the specific subject is appropriate. In this case, the scene control device 110 does not change the brightness of the lighting device 350, but changes the brightness of the background image so that the brightness range of the background image is included in the brightness dynamic range of the camera 200. The scene control device 110 determines the amount of change in brightness of the background image in the same manner as in S517.

[0085] When the second AE limit mode is set in the camera 200, the brightness of the lighting device 350 may be changed without changing the brightness of the background image in S519. By changing the brightness of the lighting device 350, the brightness of the area of ​​the specific subject changes. The camera 200 changes the imaging sensitivity so that the area of ​​the specific subject is properly exposed, and therefore the brightness dynamic range of the camera 200 shifts. As a result, when the brightness range of the background image is included in the brightness dynamic range of the camera 200, the blown-out highlights or crushed shadows of the background image are eliminated. The amount of change in the brightness of the lighting device must be determined so that the resulting change in the imaging sensitivity of the camera 200 falls within the change range of the imaging sensitivity allowed in the second AE limit mode. When the blown-out highlights or crushed shadows of the background image cannot be eliminated by the maximum change amount of the imaging sensitivity allowed in the second AE limit mode, the brightness of the background image is changed.

[0086] When the process transitions from S517 or S519 to S507, the scene control device 110 generates a background image with changed brightness by applying the brightness change amount to the rendering parameters in S507. The scene control device 110 outputs data of the generated background image to the display control device 120. The processes from S509 onwards are as described above.

[0087] In addition, in the scene control operation of the second mode, instead of changing the brightness of the background image by applying the brightness change amount to the rendering parameter, the brightness of the background image may be changed by controlling the display devices 310 and 320. Specifically, in S507, the scene control device 110 generates a background image without changing the rendering parameter related to brightness from the default value. Then, the scene control device 110 outputs the data of the background image and the brightness change amount to the display control device 120.

[0088] When the display control device 120 receives the brightness change amount from the scene control device 110, the display control device 120 applies the brightness change amount to the reference brightness of the display devices 310 and 320 and displays the background image data. As a result, the display devices 310 and 320 display the background image at a brightness that reflects the brightness change amount.

[0089] Here, the configurations that affect the brightness of the imaging scene are described as the display devices 310 and 320 and the lighting device 350. However, these are merely examples, and if other configurations are included, they can also be subject to control.

[0090] (Variation 1) When generating background image data with brightness changed, for background image data generated without changing brightness, (1) Change the brightness value uniformly (change the brightness value using a linear conversion characteristic) (2) Change the brightness value according to the brightness value (convert the brightness value using nonlinear conversion characteristics) By applying any of the above, the scene control device 110 can generate data of a background image with changed brightness.

[0091] In the case of (1), the brightness change amount is applied to the luminance value as a coefficient. In the case of (2), when making the background image darker, a gamma curve with a gradient of less than 1 in the high-luminance areas is applied, and when making the background image brighter, a gamma curve with a gradient of more than 1 in the low-luminance areas is applied to change the luminance of the background image. By preparing in advance a function of a gamma curve that has the amount of brightness change as a variable, it is possible to change the brightness of the background image using a gamma curve according to the determined amount of brightness change.

[0092] (Variation 2) When controlling the display devices 310 and 320 to change the brightness of the background image, (1) Changing the brightness value for the entire display device (2) The amount of change in brightness is varied depending on the brightness of the background image. Any of the above may be implemented by the display control device 120.

[0093] In the case of (1), the display luminance of the background image data is changed uniformly for the entire display devices 310, 320. For example, if the display devices 310, 320 have a backlight, the brightness change amount is applied uniformly to the luminance of the backlight. If the display devices 310, 320 are of a self-luminous type, the brightness change amount is applied uniformly to the gain that adjusts the luminance.

[0094] In the case of (2), the amount of change applied to the luminance of the display devices 310, 320 is varied for each adjustable unit. For example, when the display devices 310, 320 have an area-divided backlight, the amount of luminance adjustment is varied for each area, and when the display devices 310, 320 are configured with multiple display panels, the amount of luminance adjustment is varied for each panel.

[0095] Specifically, the average luminance of the background image data to be displayed is calculated for each unit in which the luminance can be adjusted, and the amount of adjustment of the luminance is varied according to the average luminance. For example, the display control device 120 divides the number of gradations of the luminance value of the image data into a plurality of ranges, and classifies the ranges to which the average luminance belongs by using the boundaries of the ranges as thresholds. Then, the display control device 120 assigns the amount of adjustment of the luminance to each classification.

[0096] When making the background image darker, a large amount of change is assigned to parts that belong to the high luminance range, and a small amount of change is assigned to parts that belong to the low to medium luminance range. This makes it possible to control the luminance with characteristics similar to applying a gamma curve whose gradient in the high luminance area is less than 1. Similarly, when making the background image brighter, a large amount of change is assigned to parts that belong to the low luminance range, and a small amount of change is assigned to parts that belong to the medium to high luminance range. This makes it possible to control the luminance with characteristics similar to applying a gamma curve whose gradient in the low luminance area is greater than 1.

[0097] 4 and 5 are executed within one frame period in camera 200. Specifically, in camera 200, operations from determining exposure conditions in the preparation stage for capturing the next frame to generating a background image are completed by the time capturing the next frame is started. This makes it possible to suppress the influence of changes in brightness of the captured scene on the in-camera VFX video captured by camera 200, and to avoid, for example, whiteout or blackout of the background image.

[0098] As described above, according to this embodiment, the brightness of the imaging scene of a camera capturing an image with an image as the background is controlled so as to suppress changes in the exposure conditions of the camera. Therefore, even if the brightness of the imaging scene changes during imaging, it is possible to suppress changes in the image quality and whiteout or blackout of the background.

[0099] (Other embodiments) In addition, when the entire scene or the area of ​​a specific subject cannot be properly exposed even if the brightness of the background image and the illumination image is changed to the maximum extent, the exposure of the camera 200 may be changed so that the proper exposure can be obtained. Specifically, even when the first AE limit mode is set, the exposure of the camera 200 may be changed. Also, when the second AE is set, the imaging sensitivity may be changed beyond the upper limit or the lower limit. For example, whether or not such exposure change is permitted may be set by the user.

[0100] For example, in the scene control operation of the first mode, the scene control unit 110 obtains the difference D=CB[EV] between the brightness B[EV] of the image capture scene acquired in S403 and the appropriate brightness C[EV]. If D>0, the scene control unit 110 calculates the upper limit Lupmax[EV] of the amount of increase in the brightness of the illumination, the upper limit Bupmax[EV] of the amount of increase in the brightness of the background image, and the difference D[EV]. D>Lupmax+Bupmax (Equation 1) If the relationship of Expression 1 is satisfied, the scene control device 110 instructs the control unit 240 of the camera 200 to increase the exposure by d=D-(Lupmax+Bupmax) [EV] and then capture an image.

[0101] Similarly, when D<0, the scene control unit 110 sets the upper limit Ldownmax[EV] of the amount of decrease in the brightness of the lighting, the upper limit Bdownmax[EV] of the amount of decrease in the brightness of the background image, and the difference D[EV]. |D|>Ldownmax+Bdownmax (Eq. 2) If the relationship in Expression 2 is satisfied, the scene control device 110 instructs the control unit 240 of the camera 200 to reduce the exposure by d=|D|-(Ldownmax+Bdownmax) [EV] and then capture an image.

[0102] In the case of scene control operation in the second mode, the scene control device 110 can allow a change in the exposure conditions of the camera 200 if the area of ​​the specific subject cannot be properly exposed with the amount of change in the illumination brightness determined in S517 or S521. For example, the scene control device 110 obtains a difference D=CB[EV] between the proper brightness C[EV] for the area of ​​the specific subject and the current brightness B[EV]. Then, when D>0, the scene control unit 110 determines whether the difference D[EV] is greater than the upper limit Lupmax[EV] of the amount of increase in the illumination brightness. D>Lupmax (Formula 3) The scene control device 110 determines whether the following relationship is satisfied. If the relationship of formula 3 is satisfied, the scene control device 110 instructs the control unit 240 of the camera 200 to increase the exposure amount by d=D-Lupmax[EV] before capturing an image. Furthermore, if the background image becomes overexposed due to the increase in the exposure amount, the scene control device 110 determines the amount of descent of the background image that corresponds to d.

[0103] Similarly, when D<0, the scene control unit 110 determines that the difference D[EV] between the upper limit Ldownmax[EV] of the amount of decrease in the brightness of the lighting is |D|>Ldownmax (Equation 4) The scene control device 110 determines whether the following relationship is satisfied. If the relationship of Equation 4 is satisfied, the scene control device 110 instructs the control unit 240 of the camera 200 to reduce the exposure by d=|D|-Ldownmax[EV] before capturing an image. Furthermore, if the background image becomes black due to the reduction in the exposure, the scene control device 110 determines the amount of rise of the background image corresponding to d.

[0104] In the above embodiment, the background image is a CG image. However, the background image may be an actual image. In this case, the viewpoint of the camera capturing the background image is synchronized with the viewpoint of the camera 200. The captured image of the background image is supplied from the camera to the display control device 120. The scene control device 110 outputs the brightness change amount to the camera capturing the background image, and the camera changes the imaging sensitivity according to the brightness change amount, thereby changing the brightness of the captured background image.

[0105] In the above embodiment, the scene control device 110, the display control device 120, and the lighting control device 140 are separate devices. However, the scene control device 100 may have the functions of the display control device 120 and the lighting control device 140.

[0106] The disclosure of the present embodiment includes the following scene control device, scene control method, imaging device, virtual studio system, and program. (Item 1) An acquisition means for acquiring information regarding the luminance of an imaging scene of the imaging device; a determination means for determining whether or not it is necessary to change the brightness of at least one of a lighting device that illuminates the image capture scene and an image that is displayed as a background of the image capture scene, based on the information; a control means for changing at least one of a lighting device for illuminating the image capture scene and a brightness of an image displayed as a background of the image capture scene according to a result of the determination; A scene control device comprising: (Item 2) The scene control device described in item 1, characterized in that when it is determined based on the information that the imaging sensitivity of the imaging device needs to be changed, the determination means determines that the brightness of at least one of the lighting device and the background needs to be changed. (Item 3) The scene control device according to item 2, characterized in that the determination means determines that the brightness of at least one of the lighting device and the background needs to be changed when the imaging sensitivity based on the information falls outside the luminance dynamic range of the imaging device. (Item 4) 2. The scene control device according to item 1, wherein the control means changes the brightness of the image by controlling a display device that displays the image. (Item 5) 2. The scene control device according to item 1, wherein the control means changes the brightness of the image by changing data representing the image. (Item 6) The scene control device according to any one of items 1 to 5, characterized in that the control means changes the brightness of the lighting device and the background so as to compensate for insufficient or excessive exposure in the imaging device. (Item 7) 6. The scene control device according to any one of claims 1 to 5, characterized in that the control means changes the brightness of the background so that it falls within a luminance dynamic range of the imaging device based on the information. (Item 8) The scene control device according to any one of items 1 to 5, characterized in that when the imaging device is performing exposure control so that a specific subject in the imaging scene is properly exposed, the control means controls the brightness of the background to be within a luminance dynamic range of the imaging device by changing the brightness of a lighting device illuminating the specific subject. (Item 9) 9. The scene control device according to any one of claims 1 to 8, wherein the control means changes the overall brightness of the background when changing the brightness of the background. (Item 10) 9. The scene control device according to any one of items 1 to 8, wherein the control means changes the brightness of the background by non-linearly converting the luminance of the data of the background. (Item 11) 11. The scene control device according to any one of items 1 to 10, characterized in that the process from the acquisition of the information by the acquisition means to the change of brightness by the control means is executed within one frame period of the imaging device. (Item 12) The scene control device according to any one of items 1 to 11, characterized in that the control means instructs the imaging device to change exposure conditions when the imaging scene or an area of ​​a specific subject in the imaging scene is not properly exposed even if the brightness of the lighting device and the image is changed. (Item 13) A generating means for generating information regarding the luminance of an imaging scene; an output means for outputting the information to an external device; A determination means for determining an exposure condition based on the information; an imaging means for performing imaging under the exposure conditions; when the exposure condition based on the information varies from the current exposure condition beyond a predetermined range, the determination means determines an exposure condition that does not vary from the current exposure condition beyond the predetermined range. 1. An imaging device comprising: (Item 14) Item 14. The imaging device according to item 13, A scene control device according to any one of items 1 to 12, a display device that displays a background of an image captured by the imaging device; an illumination device that illuminates a subject present in the imaging scene; A virtual studio system comprising: (Item 15) A scene control method executed by a scene control device, comprising: Obtaining information regarding the luminance of an imaging scene of an imaging device; determining whether or not it is necessary to change the brightness of at least one of an illumination device that illuminates the image capture scene and an image that is displayed as a background of the image capture scene based on the information; changing at least one of a lighting device that illuminates the imaging scene and a brightness of an image displayed as a background of the imaging scene according to a result of the determination; A scene control method comprising: (Item 16) A program for causing a computer to function as each of the means possessed by the scene control device according to any one of items 1 to 12.

[0107] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. [Explanation of symbols]

[0108] 110: scene control device, 120: display control device, 130: viewpoint detection device, 140: lighting control device, 200: camera, 310, 320: display device

Claims

1. An acquisition means for acquiring information regarding the luminance of an imaging scene of the imaging device; a determination means for determining whether or not it is necessary to change the brightness of at least one of a lighting device that illuminates the image capture scene and an image that is displayed as a background of the image capture scene, based on the information; a control means for changing at least one of a lighting device for illuminating the image capture scene and a brightness of an image displayed as a background of the image capture scene according to a result of the determination; A scene control device comprising:

2. The scene control device according to claim 1, characterized in that the determination means determines that the brightness of at least one of the lighting device and the background needs to be changed when it is determined based on the information that the imaging sensitivity of the imaging device needs to be changed.

3. The scene control device according to claim 2 , wherein the determining means determines that it is necessary to change the brightness of at least one of the lighting device and the background when the imaging sensitivity based on the information falls outside a luminance dynamic range of the imaging device.

4. 2. The scene control device according to claim 1, wherein the control means changes the brightness of the image by controlling a display device that displays the image.

5. 2. The scene control device according to claim 1, wherein said control means changes the brightness of said image by changing data representing said image.

6. 2. The scene control device according to claim 1, wherein said control means changes the brightness of said lighting device and said background so as to compensate for insufficient or excessive exposure in said image capture device.

7. 2. The scene control device according to claim 1, wherein said control means changes the brightness of said background so that the brightness falls within a luminance dynamic range of said imaging device based on said information.

8. The scene control device according to claim 1, characterized in that, when the imaging device is performing exposure control so that an area of ​​a specific subject present in the imaging scene is properly exposed, the control means controls the brightness of the background to fall within a luminance dynamic range of the imaging device by changing the brightness of a lighting device illuminating the specific subject.

9. 2. The scene control device according to claim 1, wherein said control means changes the overall brightness of said background when changing the brightness of said background.

10. 2. The scene control device according to claim 1, wherein said control means changes the brightness of said background by non-linearly converting the luminance of said background data.

11. 2. The scene control device according to claim 1, wherein the steps from acquisition of the information by the acquisition means to changing the brightness by the control means are executed within one frame period of the imaging device.

12. The scene control device according to claim 1, wherein the control means instructs the imaging device to change exposure conditions when the imaging scene or an area of ​​a specific subject in the imaging scene is not properly exposed even if the lighting device and the brightness of the image are changed.

13. A generating means for generating information regarding the luminance of an imaging scene; an output means for outputting the information to an external device; A determination means for determining an exposure condition based on the information; an imaging means for performing imaging under the exposure conditions; when the exposure condition based on the information varies from the current exposure condition beyond a predetermined range, the determination means determines an exposure condition that does not vary from the current exposure condition beyond the predetermined range.

1. An imaging device comprising:

14. The imaging device according to claim 13 ; A scene control device according to any one of claims 1 to 12; a display device that displays a background of an image captured by the imaging device; an illumination device that illuminates a subject present in the imaging scene; A virtual studio system comprising:

15. A scene control method executed by a scene control device, comprising: Obtaining information regarding the luminance of an imaging scene of an imaging device; determining whether or not it is necessary to change the brightness of at least one of an illumination device that illuminates the image capture scene and an image that is displayed as a background of the image capture scene based on the information; changing at least one of a lighting device that illuminates the imaging scene and a brightness of an image displayed as a background of the imaging scene according to a result of the determination; A scene control method comprising:

16. A program for causing a computer to function as each of the means included in the scene control device according to any one of claims 1 to 12.

Images