Image processing device and image processing method, and virtual studio system

Abstract

To provide an image processing device that can reflect the effect of a light source in a photographic scene in a background image of an in-camera VFX video.SOLUTION: The image processing device acquires information about a light source that exists in the real space that is captured by the imaging device, and generates an image to be displayed on a display device arranged in the real space. The image processing device generates an image that reflects the effect of light from the light source on the image on the basis of the information acquired.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, and a virtual studio system. [Background technology]

[0002] A method (in-camera VFX) is known in which a subject is captured against a background image that corresponds to the position and orientation of the camera, thereby obtaining VFX (Visual Effects) footage without combining a background image with a live-action image (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7190594 Summary of the Invention [Problem to be solved by the invention]

[0004] A background image is generated as an image of a virtual space seen from the viewpoint of a camera capturing an image of a subject. Meanwhile, the real space (captured scene) in which the subject exists may contain artificial and natural light sources such as vehicle headlights, handheld light sources held by the subject, and fire. Conventionally, even when light emitted from light sources in the real space affects the virtual space, the generated background image did not change, which could give an unnatural impression.

[0005] In consideration of such problems, one aspect of the present invention provides an image processing device and an image processing method capable of reflecting the influence of a light source in a shooting scene in a background image of an in-camera VFX video. [Means for solving the problem]

[0006] In one aspect, the present invention provides an image processing device having an acquisition means for acquiring information about a light source present in a real space to be captured by an imaging device, and a generation means for generating an image to be displayed on a display device arranged in the real space, wherein the generation means generates an image that reflects the effect of light from the light source on the image based on the information. Effect of the Invention

[0007] According to one aspect of the present invention, a scene control device and a control method therefor can be provided that are capable of controlling the brightness of an imaging scene so as to suppress changes in the exposure conditions of a camera that captures video with an image as a background. [Brief description of the drawings]

[0008] [Figure 1] Schematic diagram of a virtual studio system according to an embodiment. [Diagram 2] A block diagram showing an example of the functional configuration of the camera in Figure 1 and the connections of each device. [Diagram 3] A block diagram showing an example of the functional configuration of a scene control device [Figure 4] Flowchart for scene control operation [Diagram 5] FIG. 1 is a diagram showing an example of light source information stored in a scene control device; [Figure 6] Flowchart for a modified example of scene control operation DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The present invention will be described in detail below based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. In addition, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated explanations are omitted.

[0010] ●(First embodiment) (Virtual Studio System Overview) 1 is a schematic diagram of a virtual studio system according to an embodiment. In virtual studio 100, camera 200 captures in-camera VFX video by capturing images of automobiles 400, 401 as an example of a real subject against a background of images displayed on large display devices 310, 320 also known as LED walls. Captured range 311 is an example of the range of the background image captured by camera 200.

[0011] Viewpoint detection device 130 detects the viewpoint (position and orientation) of camera 200 from the absolute coordinates of marker 131 provided on the ceiling and the position of marker 131 included in an image of the ceiling captured by a viewpoint detection camera provided on camera 200. The position and orientation of camera 200 can be detected using any known method.

[0012] The scene control device 110 renders a preset three-dimensional model of a virtual space according to the viewpoint of the camera 200 detected by the viewpoint detection device 130, and generates a CG background image at a predetermined frame rate. If the shooting direction of the camera 200 is not directly facing the display device 310, the scene control device 110 applies coordinate transformation (deformation processing) required to display the background image on the display device 310. The same applies to the background image to be displayed on the display device 320. The scene control device 110 outputs the generated background image to the display control device 120.

[0013] As described later, the scene control device 110 acquires information about light sources (here, headlights of the automobiles 400 and 401) present in the captured scene, and reflects the effect of the light emitted by the light sources on the virtual space in the background image.

[0014] The display control device 120 causes the display devices 310 and 320 arranged in the real space to display a background image in accordance with the imaging timing of the camera 200. When each of the display devices 310 and 320 is configured with a plurality of display panels, the display control device 120 divides the background image to fit each display panel and then displays the divided image.

[0015] The lighting control device 140 controls the operation of a lighting device that illuminates an imaging scene, in this case, a lighting device 350 that illuminates automobiles 400 and 401 that are real subjects. The lighting control device 141 controls the operation of light sources present in the imaging scene. In this example, the lighting control device 141 controls the operation of the headlights of the automobiles 400, 401.

[0016] The type of operation (on, off, brightness, color, etc.) of the lighting device or light source controlled by the lighting control devices 140 and 141 may differ depending on the type of lighting device or light source. The lighting control devices 140 and 141 are operated according to a sequence predetermined by the scene control device 110. Therefore, the operation of the lighting device 350 and the headlights of the automobiles 400 and 401 is substantially controlled by the scene control device 110.

[0017] The light source information acquisition device 145 detects information on the presence or absence, position, brightness, and irradiation direction of a moving light source present in an image capture scene, and supplies the information to the scene control device 110. A moving light source is a light source whose position and irradiation direction cannot be known in advance, such as a light source held by a human subject. The light source information acquisition device 145 can detect information on the moving light source by a known method, for example, based on the output of a position and orientation sensor provided in the light source, brightness information of an image of the image capture scene, etc. Note that the image of the image capture scene can be acquired using multiple cameras including a camera (not shown) different from the camera 200. The light source information acquisition device 145 can also acquire information on the lighting device 350 controlled by the lighting control device 140 from the lighting control device 140.

[0018] Incidentally, information on light sources whose three-dimensional positions are fixed (fixed light sources) and static information (such as the type of light source and light diffusion pattern) on light sources whose three-dimensional positions are not fixed (moving light sources) can be stored in advance in the scene control device 110. In Fig. 1, information on the headlights of the automobiles 400 and 401 corresponds to the information on the fixed light sources. Incidentally, the light diffusion pattern is information indicating how the light irradiation range expands depending on the distance from the light source and how the light intensity is distributed on the irradiation surface.

[0019] Furthermore, a synchronization signal is supplied from a reference clock generating device, also called a sync generator, to the camera 200, the viewpoint detection device 130, the scene control device 110, and the display control device 120. Each device controls its operation timing according to the reference clock, thereby realizing synchronization between the shooting period of the camera 200 and the display periods of the display devices 310 and 320. A technique for synchronizing the operations of devices based on a reference clock is well known, for example, as genlock (generator locking), and therefore a detailed description thereof will be omitted.

[0020] In this specification, automobiles 400, 401, which are real objects existing between the display devices 310, 320 and the camera 200, are called real subjects, and subjects included in background images displayed on the display devices 310, 320 are called virtual subjects. Note that, for the sake of convenience, this specification assumes that the real subjects are two automobiles, but there is no limit to the type and number of types of subjects.

[0021] Fig. 2 is a block diagram showing the connection relationship of each device shown in Fig. 1 and an example of the functional configuration of camera 200. Among the functional blocks of camera 200, first optical system 210, first imaging section 220, image processing section 230, and recording section 250 realize a function of capturing and recording in-camera VFX video. In addition, second optical system 260, second imaging section 270, and A / D conversion section 280 realize a function of capturing an image for detecting the viewpoint of camera 200. Hereinafter, unless otherwise specified, first imaging section 220 and second imaging section 270 capture a video having a predetermined frame rate.

[0022] The control unit 240 has a processor (CPU, MPU, microprocessor, etc.) capable of executing a program, a ROM, and a RAM. The control unit 240 loads a program stored in the ROM into the RAM and executes it, thereby controlling the operation of each functional block of the camera 200 and realizing the operation of the camera 200 described below. Although not shown in the figure, the control unit 240 is connected to each functional block of the camera 200 so as to be able to communicate with them.

[0023] The angle of view and the optical axis direction of the second optical system 260 are determined so as to capture an image of the viewpoint detection marker 131 placed on the ceiling of the studio. The angle of view may be fixed or variable. The second imaging unit 270 has an imaging element and converts the optical image formed by the second optical system 260 into an analog image signal. Since it is sufficient for the image for viewpoint detection to be able to acquire the image coordinates of the marker 131, color information is not necessary, and the imaging element may be a monochrome image sensor.

[0024] The A / D conversion unit 280 generates a digital image signal by A / D converting the analog image signal output by the second imaging unit 270. The digital image signal is output to the viewpoint detection device .

[0025] The first optical system 210 is an optical system for capturing an in-camera VFX image. Therefore, the angle of view and the direction of the optical axis of the first optical system are determined so as to form optical images of the real subjects 400 and 401 against the background of the images displayed on the display devices 310 and 320. The angle of view of the first optical system 210 may be variable.

[0026] The first imaging unit 220 has an imaging element and converts the optical image formed by the first optical system 210 into an analog image signal. The imaging element of the first imaging unit 220 may be, for example, a known CCD or CMOS color image sensor having a primary color Bayer array color filter. The analog image signal output by the first imaging unit 220 is supplied to the image processing unit 230.

[0027] The image processing unit 230 applies predetermined image processing to the analog image signal output by the first imaging unit 220, generates signals and image data according to the application, and acquires and / or generates various information. The image processing unit 230 may be a dedicated hardware circuit such as an ASIC (Application Specific Integrated Circuit) designed to realize a specific function. Alternatively, the image processing unit 230 may be configured to realize a specific function by a processor such as a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit) executing software.

[0028] The image processing applied by the image processing unit 230 can include, for example, pre-processing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Pre-processing may include A / D conversion, signal amplification, reference level adjustment, defective pixel correction, etc. Color interpolation processing is performed when a color filter is provided on an image sensor, and is a process for interpolating values ​​of color components that are not included in the individual pixel data that constitutes the image data. Color interpolation processing is also called demosaic processing. The correction processing can include white balance adjustment, gradation correction, correction of image degradation caused by optical aberration of the first optical system 210 (image restoration), correction of the effect of peripheral light falloff of the first optical system 210, color correction, and the like. The detection process can include detection of a characteristic region or a region of a specific subject (for example, a face region or a human body region) and its movement, and person recognition processing. Data processing may include processes such as area extraction (trimming), synthesis, scaling, encoding and decoding, header information generation (data file generation), etc. Data processing also includes the generation of image data for display or image data for recording. The evaluation value calculation process may include processes such as generating a signal or evaluation value used in automatic focus detection (AF) and generating an evaluation value used in automatic exposure control (AE). The evaluation value used in AE is information about the luminance of the imaging scene, but the information about the luminance of which part of the imaging scene is different depending on, for example, the exposure mode that is set. For example, the information may reflect the luminance of the entire imaging scene, or it may be information about the luminance of an area of ​​a specific subject. The special effect processing may include processing such as adding a blur effect, changing color tones, relighting, etc. Processing that reflects the influence of a light source, which will be described later, is also included in the special effect processing.

[0029] Note that these are examples of processes that the image processing unit 230 can apply, and do not limit the processes that the image processing unit 230 applies. The image processing unit 230 outputs acquired or generated information and data to a functional block according to the application. For example, the image processing unit 230 outputs image data for recording to the recording unit 250, and outputs information related to the luminance of the captured scene to the control unit 240.

[0030] The control unit 240 outputs information on the luminance of the imaging scene acquired from the image processing unit 230 to the scene control device 110. The control unit 240 also executes AE processing to determine exposure conditions based on information on the luminance of the imaging scene, and can control the operation of the first imaging unit 220 according to the determined exposure conditions. For example, the control unit 240 can determine exposure conditions such that the entire imaging scene is properly exposed, or determine exposure conditions such that a part of an area included in the shooting scene (for example, an area of ​​a real subject) is properly exposed.

[0031] Generally, the exposure conditions are determined by a combination of the aperture value, the shutter speed (exposure time), and the imaging sensitivity. Therefore, the control unit 240 can determine a combination of the values ​​of these three parameters as the imaging conditions for obtaining the appropriate exposure. However, if the aperture value or the exposure time is changed during video shooting, the depth of field changes, and the moving distance of a moving subject between frames changes. Therefore, in the AE processing in this embodiment, the imaging sensitivity is determined basically without changing the aperture value and the shutter speed.

[0032] The focal distance of the first optical system 210 can be automatically adjusted by the control unit 240 executing AF processing based on the evaluation value generated by the image processing unit 230. Meanwhile, since the distance between the camera 200 and the ceiling is almost constant, the focal distance of the second optical system 260 is adjusted by manual focus before capturing an image, and does not need to be adjusted during capturing an image. By configuring the A / D conversion unit 280 in the same manner as the image processing unit 230, the control unit 240 may also automatically adjust the focal distance of the second optical system 260 by AF processing.

[0033] 3 is a block diagram showing an example of the functional configuration of the scene control device 110. The scene control device 110 can be realized using, for example, a computer device. The control unit 1101 is, for example, a CPU, and realizes the functions of the scene control device 110 by loading one or more application programs stored in a ROM 1108 into a RAM 1109 and executing them. The control unit 110 controls the operation timing of the scene control device 110 in accordance with a synchronization signal supplied from a reference clock generating device.

[0034] The image processing circuit 1102 is, for example, a graphics board equipped with a GPU, and is capable of high-speed image processing, such as CG rendering.

[0035] The first I / F 1103 to the sixth I / F 1108 are communication interfaces for connecting external devices. In this embodiment, the camera 200 is connected to the first I / F 1103, the display control device 120 is connected to the second I / F 1104, and the viewpoint detection device 130 is connected to the third I / F 1105. The lighting control device 140 is connected to the fourth I / F, the light source control device 141 is connected to the fifth I / F 1107, and the light source information acquisition device 145 is connected to the sixth I / F. Note that the first I / F 1103 to the sixth I / F 1108 are assumed to conform to standards according to the type of external device to be connected and the type of signal to be communicated. For convenience, the scene control device 110 and the external device are illustrated as being connected through one I / F, but they may be connected using multiple I / Fs.

[0036] The control unit 1101 acquires captured image data and information on the luminance of the captured scene from the camera 200 through the first I / F 1103. The control unit 1101 also acquires information on the viewpoint of the camera 200 from the viewpoint detection device 130 by communication through the third I / F 1105. The control unit 1101 outputs image data for display (background image data) to the display control device through the second I / F 1104. The control unit 1101 also outputs a control signal to the lighting control device 140 through the fourth I / F 1106 and to the light source control device 141 through the fifth I / F 1107. The control unit 1101 also acquires information on the light source existing in the real space from the light source information acquisition device 145 through the sixth I / F 1108. The scene control device 110 may have seven or more communication interfaces with external devices.

[0037] The ROM 1109 stores some of the programs executed by the control unit 1101 (BIOS, bootstrap loader, firmware), setting values ​​for the scene control device 110, and the like.

[0038] The RAM 1110 is used as the main memory for the control unit 1101 , as well as a working memory for the image processing circuit 1102 and a video memory for the display unit 1112 .

[0039] The storage unit 1111 is a large-capacity storage device such as a hard disk or SSD. The storage unit 1111 stores operating system (OS), application programs, user data, etc. The storage unit 1111 also stores application programs (e.g., game engine applications) that generate background images according to the viewpoint of the camera 200, and data required to generate the background images (3D models of virtual space, textures, etc.).

[0040] The display unit 1112 is, for example, a liquid crystal display device. The display unit 1112 may be a touch display. The display unit 1112 displays a scene control application, a background image generation application (for example, a game engine application), a GUI provided by the OS, and the like.

[0041] The operation unit 1113 has a plurality of input devices that can be operated by the user, such as a keyboard, a mouse, a touch pad, etc. When the display unit 1112 is a touch display, the operation unit 1113 includes a touch panel.

[0042] In this embodiment, the scene control device 110 generates a background image that reflects the influence of a light source present in an imaging scene (real space). The scene control operation by the scene control device 110 will be described below with reference to the flowchart shown in FIG.

[0043] The following operations are performed while the camera 200 is capturing an in-camera VFX image (moving image). Here, the moving image may be for recording or for display. The moving image for display may be, for example, for performing live view display when the camera 200 is in a shooting standby state.

[0044] Note that the following processes required for capturing in-camera VFX footage can be performed using known methods, and therefore detailed explanations of these processes will be omitted. A viewpoint (position and orientation) detection process of the camera 200 by the viewpoint detection device 130 using an image of the marker 131 A background image is generated by the scene control device 110 according to the detected viewpoint of the camera 200 without taking into account the influence of the light source in the captured scene. Background image display control process on the display devices 310 and 320 by the display control device 120

[0045] Furthermore, the scene control device 110 controls the brightness (including turning off) of the lighting device 350 through the lighting control device 140 according to a lighting pattern that is preset according to the elapsed time (timeline) from the start of imaging. Similarly, the scene control device 110 also controls the on / off of the headlights of the automobiles 400, 401 through the lighting control device 141. Note that it may also be possible to control switching between low beam and high beam, left and right blinkers, etc. of the automobiles 400, 401.

[0046] In the following description, the operations executed by the scene control device 110 are actually realized by the control unit 1101 executing an appropriate application program. In S401, the scene control device 110 acquires information about the viewpoint (position and attitude) of the camera 200 detected by the viewpoint detection device .

[0047] In S402, the scene control device 110 generates a CG background image by rendering a 3D model of the virtual space using the viewpoint and angle of view of the camera 200. The background image generated at this stage does not take into account the light source present in the real space (image capture scene). The scene control device 110 stores the generated background image in the RAM 1110.

[0048] In S403, the scene control device 110 acquires light source information from the light source information acquisition device 145. The light source information acquisition device 145 supplies the scene control device 110 with the number of light sources (real light sources) present in the imaging scene and, if a real light source exists, information for each light source.

[0049] The light source information acquired by the light source information acquisition device 145 is information about a real light source that cannot be acquired by the scene control device 110. Specifically, it is information about a real light source that is not or cannot be controlled by the scene control unit 110, and information that cannot be determined in advance.

[0050] Examples of real light sources that are not or cannot be controlled by the scene control unit 110 include, but are not limited to, light sources that are autonomously controlled by a real subject (for example, a light operated by a human subject). Information that cannot be determined in advance includes, but is not limited to, items of information about fixed light sources that can be dynamically changed (for example, posture or direction of illumination). Natural light sources such as a campfire are real light sources that are not and cannot be controlled by the scene control device 110, but if information such as the position and type is determined in advance, the light source information acquisition device 145 does not need to detect the information (it may detect it).

[0051] The light source information acquisition device 145 can detect information about a real light source in various ways. For example, information about a moving light source can be detected by communicating with the moving light source (or a sensor provided in the moving light source). For example, the three-dimensional position and orientation (illumination direction) of the moving light source can be detected by communicating with the moving light source or a position and orientation sensor provided in the moving light source. Note that the sensor is configured to transmit information in association with a unique ID so that it is possible to identify which real light source the acquired information relates to.

[0052] Furthermore, the light source information acquisition device 145 may detect information about a real light source using images of a captured scene. For example, the three-dimensional position and orientation are known, and an area having a brightness equal to or greater than a threshold is extracted from images of the captured scene captured by a plurality of cameras with different shooting directions, and the three-dimensional position and orientation (illumination direction) of the real light source can be detected based on the correspondence between the images.

[0053] The scene control unit 110 also acquires pre-stored static light source information by referencing the ROM 1109. Fig. 5 is a diagram showing an example of light source information stored in the ROM 1109. An ID is identification information assigned to each real light source. Here, the ID is a consecutive number, but it may be any unique information for the real light source. For each real light source, type information is included that specifies whether it is a moving light source or a fixed light source. Here, the real light source with ID2 is a moving light source, and the other real light sources are fixed light sources.

[0054] The type of light source is information used to distinguish between artificial and natural light sources. The position is indicated by three-dimensional coordinates. The origin of these three-dimensional coordinates is determined in advance. The attitude (illumination direction) is the xyz components of a vector that indicates the direction. Color temperature and brightness are general light source information. There are multiple types of diffusion patterns defined in advance, and the type that the pattern corresponds to is stored as light source information. Items that are left blank in Figure 5 indicate dynamic information or that no applicable information exists.

[0055] 5 is merely an example, and the types of items and the format of information stored for each item may be changed as appropriate. As long as information necessary for determining whether or not light emitted from each real light source affects the background image is obtained, there are no limitations on the items and contents of the light source information detected by the light source information acquisition device 145 and the light source information stored in the ROM 1109.

[0056] 4, in S405, the scene control unit 110 maps the light emitted by each real light source onto a virtual space based on the light source information acquired in S403. Specifically, the scene control device 110 calculates a three-dimensional range illuminated by the light emitted by each real light source. Then, the scene control device 110 regards the captured scene (real space) as a part of the virtual space represented by the three-dimensional model used to generate the background image, and maps the three-dimensional range onto the virtual space.

[0057] In this case, the travel distance of the light can be calculated as the distance that the light attenuates in the atmosphere until it reaches a predetermined brightness. The predetermined brightness may be, for example, a constant value, or may be the current brightness of the virtual studio. The current brightness of the virtual studio can be obtained as, for example, the average brightness of the captured scene obtained from the camera 200, or the average brightness obtained by the image processing unit 240 from the captured image of the camera 200.

[0058] In S407, the scene control device 110 determines whether or not there is a real light source that affects the background image based on the illumination range mapped in S405. Specifically, the scene control device 110 determines a real light source having an illumination range that intersects with the display surface of the display device 310, 320 as a real light source that affects the background image. Alternatively, the scene control device 110 determines a real light source that emits light that reaches the display surface of the display device 310, 320 as a real light source that affects the background image. If it is determined that there is a real light source that affects the background image, the scene control device 110 executes S409, and if not, executes S413.

[0059] In S409, the scene control device 110 calculates the range of the background image affected and the variation values ​​of saturation and luminance for each real light source that affects the background image. Specifically, the scene control device 110 calculates the area of ​​the display surface that intersects with the illumination range of the real light source as the range of the background image affected by the real light source. In addition, the scene control device 110 calculates the variation amounts of saturation and luminance as the influence of the real light source for each pixel of the background image from the luminance distribution in the area of ​​the display surface that intersects with the illumination range of the real light source and the color temperature of the real light source. Note that these calculation methods are merely examples, and calculations may be performed using other methods.

[0060] In S411, the scene control device 110 applies saturation and luminance variations to pixel values ​​in areas of the background image generated in S402 that are affected by the real light source, thereby reflecting the influence of the real light source in the background image.

[0061] In S413, the scene control device 110 outputs background image data to the display control device 120. When the imaging direction of the camera 200 is not directly facing the display devices 310, 320, the scene control device 110 applies a process to transform the image so that it becomes an image seen from a position directly facing the display devices 310, 320, and then outputs the background image data to the display control device 120. The display control device 120 causes the display devices 310, 320 to display the background image data generated by the scene control unit 110.

[0062] In S415, the scene control unit 110 determines whether or not to end image capture. The scene control unit 110 can determine to end image capture when, for example, image capture according to a predetermined timeline is completed, or when an instruction to end image capture is received from a user via the operation unit 1113. If it is determined that image capture is to be ended, the scene control device 110 ends the scene control operation, and if it is not determined that image capture is to be ended, the scene control device 110 repeatedly executes the operation from S401.

[0063] In this way, the scene control device 110 can reflect the influence of light emitted by light sources (real light sources) present in the captured scene on the background image, making it possible to capture more natural in-camera VFX footage and reducing the need for post-production corrections.

[0064] Here, the case where the background image generated in S402 is a CG image has been described. However, the case where the background image generated in S402 is a real-life image can also be applied. In this case, the viewpoint of the camera capturing the background image is synchronized with the viewpoint of the camera 200. Also, the video of the captured background image is supplied from the camera to the display control device 120. Then, the processes described in S409 and S411 can be applied to the real light source whose illumination range intersects with the display surface of the display devices 310 and 320.

[0065] (Modification) Next, a modified example of the scene control operation will be described with reference to the flowchart shown in Fig. 6. In the scene control operation described with reference to Fig. 4, the influence of a real light source is reflected in a background image generated without taking the real light source into consideration. In contrast, in the modified example, a real light source is added as a virtual light source to generate a background image.

[0066] In Fig. 6, the steps of executing the operations described in Fig. 4 are denoted by the same reference numerals as in Fig. 4, and the description thereof will be omitted. In the modified example, after executing S401, the scene control device 110 executes S403 without executing S402, and acquires light source information.

[0067] Then, in S404, the scene control device 110 adds a virtual light source based on the light source information to the virtual light source used when rendering the three-dimensional model in the virtual space. This corresponds to mapping a real light source to the virtual space. The scene control device 110 can add the real light source as a virtual light source by setting the parameters of the light source required by the application program that generates the background image based on the light source information acquired in S403.

[0068] In this modification, the 3D model of the virtual space includes not only virtual objects existing in the virtual space, but also virtual objects that are real objects existing in the real space mapped onto the virtual space. This is to reflect the shadows cast by the real objects that reach the display surfaces of the display devices 310 and 320 onto the background image.

[0069] In the modified example, it is not determined whether or not there is a real light source that affects the background image. This is because if there is an effect of a real light source, it is reflected in the background image by rendering, so there is no need to determine this. However, in order to reduce the load of rendering, it is also possible to execute the same processes as S405 and S407 in Fig. 4 and add only the real light sources that affect the background image as virtual light sources.

[0070] In S412, the scene control device 110 generates a background image. Since the real light source is added as a virtual light source in S404, if there is an effect of the real light source, a background image reflecting the effect is generated. The processing from S413 onwards is the same as that described with reference to FIG. 4, and therefore the description will be omitted.

[0071] In this modification, a background image is generated by adding a real light source as a virtual light source and rendering a three-dimensional model of a virtual space. Therefore, when light from a real light source hits an object (virtual object) in a virtual space, a shadow generated by the light from the real light source is also reflected in the background image.

[0072] Moreover, the background image is generated by rendering an image observed on the display screen of the display devices 310, 320 for a three-dimensional model in which the captured scene (real space) is also a part of the virtual space. Therefore, a background image is obtained that reflects the part of the shadow of the real object that reaches the display screen of the display devices 310, 320. For example, in a scene in the morning or evening where the light source is low and the shadow is long, it becomes possible to generate a more natural background image.

[0073] As described above, according to this embodiment, the influence of light from a light source existing in the imaging scene (real space) is reflected in the background image (virtual space image) used to capture the in-camera VFX video. Therefore, it is possible to capture a more natural in-camera VFX video.

[0074] (Other embodiments) In the above embodiment, the scene control device 110, the display control device 120, and the lighting control devices 140 and 141 are separate devices. However, the scene control device 100 may have the functions of the display control device 120 and the lighting control device 140.

[0075] The disclosure of the present embodiment includes the following image processing device, image processing method, virtual studio system, and program. (Item 1) An acquisition means for acquiring information about a light source existing in a real space captured by an imaging device; a generating means for generating an image to be displayed on a display device disposed in the real space, The image processing device according to claim 1, wherein the generating means generates an image that reflects the effect of light from the light source on the image based on the information. (Item 2) The image forming apparatus further includes a determining unit for determining whether or not the light emitted from the light source has an effect on the image, the generating means generates an image reflecting the influence of the light when it is determined that the light has an influence on the image. 2. The image processing device according to item 1, (Item 3) 3. The image processing device according to item 2, wherein the determining means determines that the light affects the image when it is determined that the light reaches a display surface of the display device. (Item 4) The image processing device according to any one of items 1 to 3, characterized in that the generating means generates the image reflecting the influence of the light by changing the brightness and saturation of the area of ​​the image illuminated by the light. (Item 5) the generating means generates the image by rendering a three-dimensional model of the virtual space using a viewpoint of the imaging device; the generating means generates an image reflecting the effect of the light on the image by mapping the light source into the virtual space and rendering the three-dimensional model. 4. The image processing device according to any one of items 1 to 3, (Item 6) 6. The image processing device according to item 5, wherein the effect includes a shadow cast on a virtual object in the virtual space by the light source. (Item 7) 7. The image processing device according to item 5 or 6, wherein the effects include a shadow cast by a real object due to the light source. (Item 8) 8. The image processing device according to any one of items 1 to 7, characterized in that the information about the light source includes at least the three-dimensional position of the light source, its attitude or irradiation direction, brightness, and type. (Item 9) 9. The image processing device according to item 8, wherein the acquisition means acquires the three-dimensional position of the light source based on an image of the real space captured by an imaging device different from the imaging device. (Item 10) 9. The image processing device according to item 8, wherein the acquisition means acquires the three-dimensional position and orientation of the light source based on an output of a sensor provided in the light source. (Item 11) 11. The image processing device according to any one of items 8 to 10, wherein the acquisition means acquires information about the light source from a control device that controls an operation of the light source. (Item 12) 12. The image processing device according to any one of items 8 to 11, characterized in that at least a part of the items included in the information about the light source is stored in advance. (Item 13) The imaging device; An image processing device according to any one of items 1 to 12, A display device for displaying the image; A virtual studio system comprising: (Item 14) An image processing method executed by an image processing device, comprising: Acquiring information about a light source present in a real space captured by an imaging device; generating an image to be displayed on a display device disposed in the real space; The image processing method, wherein the generating step comprises generating an image that reflects an effect of light from the light source on the image based on the information. (Item 15) 13. A program for causing a computer to function as each of the means possessed by the image processing device according to any one of items 1 to 12.

[0076] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. [Explanation of symbols]

[0077] 110: scene control device, 120: display control device, 130: viewpoint detection device, 140: lighting control device, 141: light source control device, 145: light source information acquisition device, 200: camera, 310, 320: display device

Claims

1. An acquisition means for acquiring information about a light source existing in a real space captured by an imaging device; a generating means for generating an image to be displayed on a display device disposed in the real space, The image processing device according to claim 1, wherein the generating means generates an image that reflects the effect of light from the light source on the image based on the information.

2. The image forming apparatus further includes a determining unit for determining whether or not the light emitted from the light source has an effect on the image, the generating means generates an image reflecting the influence of the light when it is determined that the light has an influence on the image.

2. The image processing device according to claim 1,

3. 3. The image processing device according to claim 2, wherein the determining means determines that the light affects the image when it is determined that the light reaches a display surface of the display device.

4. 2 . The image processing apparatus according to claim 1 , wherein the generating means generates the image reflecting the influence of the light by changing brightness and saturation of an area of ​​the image illuminated by the light.

5. the generating means generates the image by rendering a three-dimensional model of the virtual space using a viewpoint of the imaging device; the generating means generates an image reflecting the effect of the light on the image by mapping the light source into the virtual space and rendering the three-dimensional model.

2. The image processing device according to claim 1,

6. 6. The image processing apparatus according to claim 5, wherein the influence includes a shadow cast on a virtual object in the virtual space by the light source.

7. 6. The image processing apparatus according to claim 5, wherein the influence includes a shadow cast by a real object due to the light source.

8. 2 . The image processing apparatus according to claim 1 , wherein the information about the light source includes at least a three-dimensional position of the light source, a posture or irradiation direction, brightness, and a type.

9. The image processing apparatus according to claim 8 , wherein the acquisition means acquires the three-dimensional position of the light source based on an image of the real space captured by an imaging device different from the imaging device.

10. 9. The image processing apparatus according to claim 8, wherein the acquiring means acquires the three-dimensional position and orientation of the light source based on an output of a sensor provided in the light source.

11. 9. The image processing apparatus according to claim 8, wherein the acquiring means acquires information about the light source from a control device that controls an operation of the light source.

12. 9. The image processing device according to claim 8, wherein at least a part of the items included in the information about the light source is stored in advance.

13. The imaging device; An image processing device according to any one of claims 1 to 12, A display device for displaying the image; A virtual studio system comprising:

14. An image processing method executed by an image processing device, comprising: Acquiring information about a light source present in a real space captured by an imaging device; generating an image to be displayed on a display device disposed in the real space; The image processing method, wherein the generating step comprises generating an image that reflects an effect of light from the light source on the image based on the information.

15. A program for causing a computer to function as each of the means included in the image processing device according to any one of claims 1 to 12.

Images