Imaging system, method for controlling imaging system, and program

Abstract

To provide a technique that, when creating a composite image obtained by combining a real image and a CG image, allows creation of a more appropriate composite image.SOLUTION: An imaging system has: first light source control means that controls a first light source that irradiates a first object with light in a real space; second light source control means that controls a second light source that irradiates a second object with light in a virtual space; first image acquisition means that acquires a first image obtained by picking up an image of the first object irradiated with the light from the first light source that emits light with a first quantity of light; second image acquisition means that acquires a second image obtained by picking up an image of the second object irradiated with the light from the second light source that emits light with a second quantity of light; and creation means that creates a composite image obtained by combining the first image and the second image. The second light source control means controls the second quantity of light on the basis of the first quantity of light.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to an imaging system, a control method for the imaging system, and a program.

Background Art

[0002] There is a method of correcting CG according to the environment at the time of shooting a real image. In recent years, there is also a method using a technique such as light source estimation using a statistical learning method such as Deep Learning. However, in a statistical learning method such as Deep Learning, there is a problem that a huge amount of computer resources are required. Patent Document 1 discloses a technique for correcting CG according to the environment at the time of shooting.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There is a camera system that synthesizes a real image and virtual CG. The camera system performs camera control such as various auto exposures, auto white balances, and auto focuses using the evaluation value of the image sensor.

[0005] Here, when the synthesis is performed in a state where there is a difference in image quality due to the difference in the shooting environment between the real image and the CG image, an unnatural composite image in which the CG appears floating is generated.

[0006] Therefore, an object of the present invention is to provide a technique that enables generation of a more appropriate composite image when generating a composite image obtained by synthesizing a real image and a CG image.

Means for Solving the Problems

[0007] One aspect of the present invention is first light source control means for controlling a first light source that irradiates light onto a first object in the real space, second light source control means for controlling a second light source that irradiates light onto a second object in the virtual space, first image acquisition means for acquiring a first image obtained by imaging the first object irradiated with light from the first light source that emits light with a first light amount, second image acquisition means for acquiring a second image obtained by imaging the second object irradiated with light from the second light source that emits light with a second light amount, generation means for generating a composite image obtained by combining the first image and the second image, and the second light source control means controls the second light amount based on the first light amount. The imaging system is characterized by this.

[0008] One aspect of the present invention is first light source control means for controlling a first light source that irradiates light onto a first object in the real space, second light source control means for controlling a second light source that irradiates light onto a second object in the virtual space, first image acquisition means for acquiring a first image obtained by imaging the first object irradiated with light from the first light source that emits light with a first light amount, second image acquisition means for acquiring a second image obtained by imaging the second object irradiated with light from the second light source that emits light with a second light amount, generation means for generating a composite image obtained by combining the first image and the second image, and the first light source control means controls the first light amount based on the second light amount. The imaging system is characterized by this.

[0009] One aspect of the present invention is A first light source control step of controlling a first light source that irradiates light onto a first object in the real space, A second light source control step of controlling a second light source that irradiates light onto a second object in the virtual space, A first image acquisition step of acquiring a first image obtained by imaging the first object irradiated with light from the first light source that emits light with a first light amount, A second image acquisition step of acquiring a second image obtained by imaging the second object irradiated with light from the second light source that emits light with a second light amount, A generation step of generating a composite image obtained by combining the first image and the second image, and having, In the second light source control step, the second light amount is controlled based on the first light amount. A control method for an imaging system, characterized in that.

[0010] One aspect of the present invention is, A first light source control step of controlling a first light source that irradiates light onto a first object in the real space, A second light source control step of controlling a second light source that irradiates light onto a second object in the virtual space, A first image acquisition step of acquiring a first image obtained by imaging the first object irradiated with light from the first light source that emits light with a first light amount, A second image acquisition step of acquiring a second image obtained by imaging the second object irradiated with light from the second light source that emits light with a second light amount, A generation step of generating a composite image obtained by combining the first image and the second image, and having, In the first light source control step, the first light amount is controlled based on the second light amount. A control method for an imaging system, characterized in that.

Effects of the Invention

[0011] According to the present invention, when generating a composite image by combining a real image and a CG image, a more appropriate composite image can be generated.

Brief Description of the Drawings

[0012]

Figure 1A

Figure 1B

Figure 1C

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Embodiments for Carrying Out the Invention

[0013] Hereinafter, each embodiment of the present invention will be described in detail with reference to the accompanying drawings. First, matters common to each embodiment will be described.

[0014] FIG. 1A is a diagram showing an imaging system 1 according to an embodiment. The imaging system 1 is a digital camera system mainly for shooting still images and moving images. The imaging system 1 is not limited to a digital camera system, and may be various imaging systems including an HMD.

[0015] The imaging system 1 includes a lens device 10, a camera body unit 20, and a CG (computer graphics) unit 30.

[0016] As shown in FIG. 1B, the lens device 10 includes a zoom unit 101, a zoom drive control unit 102, a diaphragm unit 103, a diaphragm drive control unit 104, a correction unit 105, an optical control unit 106, and a focus unit 107. Further, the lens device 10 includes a focus drive control unit 108, a lens operation unit 109, a lens shake detection unit 110, a lens control unit 111, and a lens communication control unit 112.

[0017] The zoom unit 101 includes a zoom lens that performs zooming in and out. The zoom drive control unit 102 controls the drive of the zoom unit 101. The aperture unit 103 has an aperture function. The aperture drive control unit 104 controls the drive of the aperture unit 103.

[0018] The correction unit 105 includes an image stabilization lens such as a shift lens (hereinafter also referred to as a correction lens or OIS). The correction unit 105 performs first image stabilization. The optical control unit 106 controls the drive of the correction unit 105. The focus unit 107 includes a focus lens that performs focus adjustment to form a subject image. The focus drive control unit 108 controls the drive of the focus unit 107.

[0019] The lens operation unit 109 is an operation unit used for a user's operation on the lens device 10. The lens shake detection unit 110 detects the amount of shake applied to the lens device 10 and outputs a detection signal indicating the amount of shake to the lens control unit 111.

[0020] The lens control unit 111 comprehensively controls each component of the lens device 10. The lens control unit 111 communicates with the control unit of the camera body unit 20 via the lens communication control unit 112.

[0021] As shown in FIG. 1B, the camera body unit 20 includes a shutter unit 113, a shutter drive control unit 114, an imaging unit 115, an imaging plane control unit 116, an imaging plane correction unit 117, an imaging signal processing unit 118, and an image signal processing unit 119. The camera body unit 20 includes a power supply unit 122, a camera operation unit 123, a camera shake detection unit 124, an electronic control unit 125, a camera communication control unit 127, a strobe unit 128, and a strobe control unit 129.

[0022] The shutter drive control unit 114 controls the drive of the shutter unit 113. The imaging unit 115 includes an imaging device. The imaging unit 115 photoelectrically converts the optical image formed by passing through each lens group to generate an electrical signal. The imaging plane correction unit 117 includes an imaging plane correction unit (IIS) that moves the imaging unit 115 to correct image blur. The imaging plane correction unit 117 performs a second image blur correction. The imaging plane control unit 116 controls the drive of the imaging plane correction unit 117. The imaging signal processing unit 118 converts the electrical signal output from the imaging unit 115 into an image signal.

[0023] The image signal processing unit 119 processes the image signal output from the imaging signal processing unit 118 according to the application. For example, the image signal processing unit 119 changes the cut-out position of the image signal according to the correction amount determined by the electronic control unit 125. The electronic control unit 125 performs a third image blur correction. The electronic control unit 125 performs control to determine the correction amount of image blur correction by cutting out the image. Note that the third image blur correction is not limited to electronic image blur correction. For example, image blur may be corrected by driving an optical element in the camera body 20.

[0024] The power supply unit 122 supplies power (power supply) to the entire system according to the application. The camera operation unit 123 outputs an operation signal corresponding to the user's operation to the camera control unit 126. The camera shake detection unit 124 detects the amount of shake applied to the camera body 20 and outputs a detection signal indicating the amount of shake to the camera control unit 126.

[0025] The camera control unit 126 includes a CPU. The camera control unit 126 comprehensively controls the imaging system 1. The camera control unit 126 communicates with the lens communication control unit 112 of the lens device 10 via the camera communication control unit 127. That is, in a state where the lens device 10 is attached (electrically connected) to the camera body 20, the lens communication control unit 112, the camera communication control unit 127, the CG communication control unit 130, and the strobe control unit 129 communicate with each other.

[0026] The strobe control unit 129 controls the light emission of the strobe unit 128.

[0027] As shown in FIG. 1C, the CG unit 30 includes a display unit 120, a storage unit 121, a CG communication control unit 130, a virtual strobe control unit 131, a virtual light source control unit 132, a CG generation unit 133, and an image composition unit 134.

[0028] The display unit 120 displays a composite image as necessary based on the signal output from the image composition unit 134. The storage unit 121 stores various data such as image information.

[0029] The CG communication control unit 130 acquires the control information of the camera body unit 20 from the camera communication control unit 127. The CG communication control unit 130 outputs the control information of the camera body unit 20 to the virtual strobe control unit 131 and the virtual light source control unit 132.

[0030] The virtual strobe control unit 131 acquires the light adjustment result by the actual strobe and the color temperature information of the strobe among the camera control information. The virtual strobe control unit 131 determines the color temperature and light emission amount of the virtual strobe based on the light adjustment result. The virtual strobe control unit 131 acquires the light adjustment result by the actual strobe and the color temperature information of the strobe among the camera control information. The virtual strobe control unit 131 determines the color temperature and light emission amount of the virtual strobe based on the light adjustment result.

[0031] The virtual light source control unit 132 obtains the brightness and color temperature information of the real space (real environment) among the camera control information. The virtual light source control unit 132 sets the brightness and color temperature of the virtual light source based on the color temperature information.

[0032] The CG generation unit 133 generates an image of a CG (virtual object) based on the light source with the brightness and color temperature set by the virtual light source control unit 132 and the virtual strobe with the color temperature and light emission amount of the virtual strobe determined by the virtual strobe control unit 131.

[0033] The image composition unit 134 composes the CG image generated by the CG generation unit 133 and the image output from the image signal processing unit 119.

[0034] Next, the schematic operation of the imaging system 1 will be described.

[0035] The camera operation unit 123 includes a shutter release button. According to the amount of depression of the shutter release button, the first switch and the second switch sequentially transition to the ON state. When the user depresses the shutter release button by about half, the first switch transitions to the ON state. When the user depresses the shutter release button all the way, the second switch transitions to the ON state.

[0036] When the first switch transitions to the ON state, the focus drive control unit 108 drives the focus unit 107 to perform focus adjustment, and the aperture drive control unit 104 drives the aperture unit 103 to set an appropriate exposure amount. When the second switch transitions to the ON state, the image data obtained from the optical image exposed to the imaging unit 115 is sent to the imaging signal processing unit 118, and then sent to the image signal processing unit 119. The image composition unit 134 composes the image sent to the image signal processing unit 119 and the CG image generated by the CG generation unit 133 to generate a composite image. The composite image is stored in the display unit 120 or the storage unit 121 as needed.

[0037] Also, the camera operation unit 123 includes a video recording switch. The camera main body unit 20 starts video shooting after the video recording switch is pressed. The camera main body unit 20 ends recording when the video recording switch is pressed again during video shooting (recording). When the first switch and the second switch transition to the ON state by operating the shutter release button during video shooting, a process of acquiring and recording a still image during video recording is executed. Also, the camera operation unit 123 includes a playback mode selection switch that can select a playback mode.

[0038] <Embodiment 1> In Embodiment 1, the imaging system 1 synthesizes a real image based on the light emission of a strobe light source (hereinafter referred to as the "real light source"), which is an actual illumination unit (light emitting unit), and a CG image based on the light emission of a strobe light source (hereinafter referred to as the "virtual light source"), which is a virtual illumination unit. Note that it is assumed that the real light source performs preliminary light emission (hereinafter referred to as "pre-flash") before the light emission for shooting (hereinafter referred to as "main light emission") in order to determine the light emission amount of the main light emission.

[0039] FIG. 2 is a diagram simply showing the configuration of the imaging system 1 shown in FIG. 1 according to Embodiment 1. The imaging system 1 controls the dimming of the virtual light source based on the control information of the strobe dimming of the real light source, so that the virtual light source emits light with the same light emission amount as the real light source. This process is applicable not only to still images but also to strobe shooting in moving images. That is, the real image, CG image, and composite image described later may each be a moving image. By such a process, the imaging system 1 performs shooting using the light of the virtual light source linked to the real light source, and can obtain a photo-realistic composite image.

[0040] The imaging system 1 includes an optical system 201, an aperture control unit 202, an imaging sensor 203, an image processing unit 204, a composite unit 205, a camera control unit 207, a position estimation unit 208, and a CG generation unit 209. The imaging system 1 includes a dimming unit 210, a virtual strobe dimming unit 211, a virtual strobe control unit 212, a strobe module 213, and a strobe control unit 214. In Embodiment 1, for example, the optical system 201 and the aperture control unit 202 are included in the lens device 10. For example, the imaging sensor 203, the image processing unit 204, the camera control unit 207, the position estimation unit 208, the dimming unit 210, the strobe module 213, and the strobe control unit 214 are included in the camera body unit 20. For example, the composite unit 205, the CG generation unit 209, the virtual strobe dimming unit 211, and the virtual strobe control unit 212 are included in the CG unit 30.

[0041] The reflected light of pre-emission (prior emission) that has passed through the optical system 201 forms an image on the imaging sensor 203. The imaging sensor 203 is an image acquisition unit that acquires the formed image as a real image. The aperture control unit 202 controls the driving of the optical system 201. The aperture control unit 202 corresponds to, for example, at least any one of the zoom drive control unit 102, the aperture drive control unit 104, the optical control unit 106, the focus drive control unit 108, and the lens control unit 111. The imaging sensor 203 corresponds to, for example, the imaging unit 115.

[0042] The image processing unit 204 performs image processing on the real image (the image obtained by the camera body unit 20 imaging the real space) acquired by the imaging sensor 203. The image processing unit 204 corresponds to, for example, the imaging signal processing unit 118 and the image signal processing unit 119.

[0043] The synthesis unit 205 synthesizes the real image subjected to image processing by the image processing unit 204 and the CG image generated by the CG generation unit 209 to generate a synthesized image.

[0044] The camera control unit 207 generates a camera evaluation value necessary for camera control based on the real image acquired from the imaging sensor 203. In Embodiment 1, the camera evaluation value is, for example, an evaluation value representing the brightness of the real image (real space).

[0045] The position estimation unit 208 estimates the position of the imaging sensor 203 (camera body unit 20) based on the real image. For the estimation of the position of the imaging sensor 203, for example, techniques such as SLAM (Simultaneous Localization and Mapping) can be used. The position estimation unit 208 may estimate the position of the imaging sensor 203 based on the measured values of the gyro sensor (inertial sensor).

[0046] The CG generation unit 209 arranges CG, which is a virtual subject (virtual object), in the virtual space. The CG generation unit 209 is also an image acquisition unit that generates an image (CG image) of the CG illuminated by the virtual light source.

[0047] The dimming unit 210 calculates the amount of light emission required for the actual light source to emit light based on the camera evaluation value obtained from the camera control unit 207. Since the method of calculating the amount of light emission required for the actual light source to emit light based on the brightness of the real image can apply existing technologies, detailed descriptions are omitted. The dimming unit 210 outputs information on the amount of light emission required for actual light emission (strobe light emission amount) to the virtual strobe dimming unit 211. The dimming unit 210 can also grasp the approximate distance from the actual light source to the subject according to the light emission amount of the actual light source, and thus calculates that distance (hereinafter referred to as the "subject distance").

[0048] The virtual strobe dimming unit 211 replaces the light emission amount of the actual light source with the light emission amount of the virtual light source.

[0049] The virtual strobe control unit 212 arranges the virtual light source at the position in the "virtual space" corresponding to the "position of the actual light source in the real space". For example, the virtual strobe control unit 212 arranges the virtual light source at a position away from the real subject by the same distance as the distance between the real subject and the real light source in the virtual space (= the position where the real light source is estimated to be arranged).

[0050] The strobe control unit 214 controls the strobe module 213. The strobe control unit 214 gives an execution instruction for pre-emission for strobe dimming to the strobe module 213. The strobe module 213 has an actual light source. The strobe module 213 irradiates the subject with light by performing pre-emission and actual light emission of the actual light source.

[0051] Figure 3 shows a composite image obtained by combining two images (a real person's image 301 and a virtual character's image 302) taken in an environment with appropriate brightness. Since there is no significant difference in brightness between image 301 and image 302, a good composite image can be obtained by combining image 301 and image 302.

[0052] On the other hand, when similar shooting is performed at night or in a dark environment, as shown in Figure 4, a composite image in which both the real person's image 401 and the virtual character's image 402 are in a dark state is obtained.

[0053] Also, as shown in FIG. 5, even when the actual light source emits light so as to adjust the image 501 of an actual person to an image with an appropriate brightness, the image 502 of the virtual character remains completely dark. For this reason, a composite image with a difference in brightness between objects is generated.

[0054] Therefore, in Embodiment 1, as shown in FIG. 6, when the actual light source emits light so as to adjust the image 601 of an actual person to an appropriate brightness, the virtual light source in the virtual space emits light in the same manner as the actual light source based on the dimming information of the actual light source or the like. As a result, the image 602 of the virtual character can also be adjusted to an appropriate brightness (since a CG reflecting the effect of stroboscopic light emission equivalent to the real image can be obtained), and a photo-realistic composite image can be obtained.

[0055] Referring to the flowchart of FIG. 7, the processing according to Embodiment 1 will be described. When the flowchart of FIG. 7 starts, shooting has started.

[0056] In step S702, the strobe module 213 performs pre-emission of the actual light source with a predetermined emission amount.

[0057] In step S703, the camera control unit 207 calculates a camera evaluation value indicating the brightness of the real image or the like based on the real image (the image formed on the imaging sensor 203 by the light of the pre-emission being reflected by the subject). Then, the dimming unit 210 calculates the emission amount required for the main emission based on the camera evaluation value. The dimming unit 210 calculates the subject distance based on the emission amount required for the main emission.

[0058] In step S704, the strobe module 213 performs the main emission of the actual light source according to the emission amount required for the main emission.

[0059] In step S705, the imaging sensor 203 acquires a real image of the subject irradiated with light by the main emission of the actual light source.

[0060] In step S706, the virtual strobe control unit 212 calculates the emission amount of the virtual light source and the arrangement position of the virtual light source based on the strobe emission amount calculated in step S703 and the subject distance (the distance between the actual light source and the subject).

[0061] In step S707, the virtual strobe control unit 212 arranges the virtual light source at the calculated arrangement position in the virtual space. Then, the virtual strobe control unit 212 causes the virtual light source to emit light with the calculated emission amount of the virtual light source.

[0062] In step S708, the CG generation unit 209 acquires a CG image (CG image) irradiated with light by the emission of the virtual light source.

[0063] In step S709, the synthesis unit 205 synthesizes the real image (the real image acquired in step S705) and the CG image to generate a synthesized image.

[0064] In the flowchart of FIG. 7, when the processes of steps S704 and S705 are completed, the processes of steps S706 to S708 are executed. On the other hand, when the processes of steps S706 to S708 are completed, the processes of steps S704 and S705 may be executed. Also, it is desirable that the shooting of the actual subject and the shooting of the CG are executed simultaneously. Therefore, the processes of steps S704 and S705 and the processes of steps S706 to S708 may be executed simultaneously.

[0065] FIG. 8 shows a graph 801 representing the relationship between the emission amount of the present light emission and the subject distance (the distance between the actual light source and the subject). According to FIG. 8, the imaging system 1 can calculate the subject distance 803 according to the emission amount 802 of the actual light source.

[0066] ​According to Embodiment 1, when photographing an actual person and a virtual character, the imaging system 1 synthesizes a real image of a person irradiated with light from an actual light source and an image of a virtual character irradiated with light from a virtual light source based on the emission amount of the actual light source's original light emission. As a result, the imaging system 1 can obtain a photo-realistic composite image.

[0067] <Embodiment 2> In Embodiment 2, when photographing a virtual main subject simultaneously with an actual background, etc., the emission amount of the actual light source is determined based on the emission information of the virtual light source. The imaging system 2 according to Embodiment 2 causes the virtual light source to emit light based on the determined emission amount to generate a CG image. The imaging system 2 synthesizes the real image and the CG image. As a result, the imaging system 2 can obtain a photo-realistic composite image. Note that in Embodiment 2, it is assumed that the distance from the virtual light source to the CG is known.

[0068] Referring to FIG. 9, the imaging system 2 according to Embodiment 2 will be described. The imaging system 2 includes an optical system 201, an aperture control unit 202, an imaging sensor 203, an image processing unit 204, a synthesis unit 205, a camera control unit 907, a position estimation unit 208, and a CG generation unit 209. The imaging system 2 includes a light adjustment unit 910, a virtual strobe light adjustment unit 911, a virtual strobe control unit 912, a strobe module 213, and a strobe control unit 914. Note that the components having the same names as those in Embodiment 1 in FIG. 9 perform the same processing as in Embodiment 1.

[0069] The camera control unit 907 generates a camera evaluation value based on the real image. The camera control unit 907 determines whether strobe light emission is necessary based on the camera evaluation value. The camera control unit 907 outputs the result of determining whether strobe light emission is necessary and the camera evaluation value to the virtual strobe light adjustment unit 911.

[0070] The light adjustment unit 910 measures the brightness of the environment in the real space.

[0071] When it is determined that it is necessary to perform strobe light emission, the virtual strobe dimming unit 911 controls the brightness of the CG to match the brightness of the real image before light emission.

[0072] Since the virtual strobe control unit 912 knows the distance from the virtual light source to the CG, it calculates the light emission amount of the virtual light source to make the CG located at the known distance have an appropriate brightness. The virtual strobe control unit 912 outputs the calculated light emission amount of the virtual light source to the CG generation unit 209 and the strobe control unit 914.

[0073] The strobe control unit 914 controls the strobe module 213 to perform the main light emission of the real light source with the same (identical) light emission amount as that of the virtual light source. The light reflected by hitting the object by the main light emission passes through the optical system 201 and forms an image on the imaging sensor 203.

[0074] FIG. 10 shows a composite image in which two images (the image 1002 of the virtual character and the image 1001 of the real background) taken in an environment with an appropriate brightness are combined. There is no significant difference in the brightness between the image 1001 and the image 1002, and a good composite image can be obtained.

[0075] However, when shooting in the same way at night as shown in FIG. 11, in the composite image, both the image 1102 of the virtual character and the image 1101 of the real background become completely dark.

[0076] Also, in the real space as shown in FIG. 12, when trying to emit light from the real light source, referring to the brightness of the real background 1201 based on the pre-emission, the light emission amount of the main light emission is controlled. Since the real background is in the distance, the light emission amount of the main light emission is set large. When the light of that light emission amount irradiates the virtual character 1302, as shown in FIG. 13, the virtual character 1302 is photographed with overexposure and is displayed as whitewashed in the composite image.

[0077] On the other hand, in Embodiment 2, as shown in FIG. 14, the imaging system 2 emits a virtual light source so that the image 1402 of the virtual character has an appropriate brightness and then performs imaging. After that, the imaging system 2 emits an actual light source with a light emission amount equivalent to that of the virtual light source. In this way, the imaging system 2 can image the virtual character with an appropriate brightness. Further, the imaging system 2 images the actual background also with a light emission amount equivalent to that of the virtual light source. In this way, the imaging system 2 can acquire the image 1402 which is a CG image and the image 1401 which is a background image with a more appropriate image quality.

[0078] Referring to the flowchart of FIG. 15, the processing according to Embodiment 2 will be described.

[0079] In step S1502, the light control unit 910 measures the brightness of the environment in the real space.

[0080] In step S1503, the virtual strobe control unit 912 calculates the light emission amount of the virtual light source for making the CG have an appropriate brightness based on the brightness of the environment in the real space.

[0081] In step S1504, the virtual strobe control unit 912 arranges the virtual light source in the virtual space and emits the virtual light source with the calculated light emission amount.

[0082] In step S1505, the CG generation unit 209 acquires an image of the CG irradiated with light from the virtual light source.

[0083] In step S1506, the strobe control unit 914 sets the light emission amount of the main emission of the actual light source to be equivalent to the light emission amount of the virtual light source calculated in step S1503.

[0084] In step S1507, the strobe module 213 performs the main emission of the actual light source with the light emission amount set in step S1506. performed.

[0085] In step S1508, the imaging sensor 203 captures a subject or background irradiated with light from the actual light source's original emission to obtain a real image.

[0086] In step S1509, the synthesizing unit 205 synthesizes the real image and the CG image to generate a synthesized image.

[0087] FIG. 16 shows a graph 1601 representing the relationship between the emission amount and the distance to the subject. When a virtual light source emits light to a virtual character located at a subject distance 1604 shorter than the subject distance 1605 with an appropriate emission amount 1602 in relation to the background located at the subject distance 1605, the virtual character will be photographed in an overexposed or whitewashed state. On the other hand, by emitting a virtual light source to the virtual character with an appropriate emission amount 1603 in relation to the virtual character located at the subject distance 1604, it becomes possible to obtain an image of the virtual character with an appropriate brightness. Also, at this time, the actual light source is emitted with an emission amount similar to that of the virtual light source, and the background in the real space is photographed. For this reason, it becomes possible to obtain a synthesized image with less discomfort for the user.

[0088] According to Embodiment 2, when photographing a virtual character and a real background at night or in a dark environment, an image of the virtual character photographed by the emission of the virtual light source and an image of the background photographed by the emission of an actual light source with an emission amount similar to that of the virtual light source are synthesized. As a result, it becomes possible to obtain a photo-realistic synthesized image.

[0089] Note that the processing according to Embodiment 1 and the processing according to Embodiment 2 may be switchable. Specifically, in the first case, the imaging system determines the light emission amount of the virtual light source based on the light emission amount of the real light source as in Embodiment 1, and in the second case (a case other than the first case), the imaging system may determine the light emission amount of the real light source based on the light emission amount of the virtual light source as in Embodiment 2. Here, in the second case, for example, the distance between the real light source and the main subject is farther than a specific distance. In this case, the imaging system can obtain a more appropriate composite image by determining the light emission amount of the real light source based on the light emission amount of the virtual light source as described above. Alternatively, in the second case, for example, it may be a case where the ambient light in the real space is dark, or a case where the user performs a specific operation.

[0090] <Embodiment 3> In Embodiment 3, when the imaging system 3 photographs a virtual object (an object in the virtual space), the imaging system 3 controls the virtual light source according to the light emission of the real light source so that, for example, the color temperature of the real light source and the color temperature of the virtual light source become the same. By doing so, a photo-realistic composite image can be obtained.

[0091] FIG. 17 shows the imaging system 3 according to Embodiment 3. The imaging system 3 includes an optical system 201, a diaphragm control unit 202, an imaging sensor 203, an image processing unit 204, a composite unit 205, a camera control unit 1707, and a position estimation unit 1708. The imaging system 3 includes a CG generation unit 1709, a color temperature determination unit 1710, a virtual strobe dimming unit 1711, a virtual strobe control unit 1712, a dimming unit 1713, a strobe module 213, and a strobe control unit 1715.

[0092] The camera control unit 1707 generates a camera evaluation value based on the real image. In Embodiment 3, the camera evaluation value is, for example, a value related to the brightness of the real space (the captured image) and the color temperature of the real space (the captured image). The camera control unit 1707 determines whether it is necessary to execute strobe light emission based on the camera evaluation value. When it is necessary to execute strobe light emission (for example, when the brightness of the real space is darker than a specific brightness), the camera control unit 170 7 outputs the information of the camera evaluation value to the color temperature determination unit 1710 and the light control unit 1713.

[0093] The color temperature determination unit 1710 determines the color temperature of the real space (ambient light) and outputs the determined color information to the virtual strobe light control unit 1711. The light control unit 1713 calculates the brightness of the real space (ambient light) and outputs the calculated brightness information to the virtual strobe light control unit 1711.

[0094] The strobe control unit 1715 gives an instruction for pre-emission to the strobe module 213. The strobe module 213 performs pre-emission of the real light source. The light from the pre-emission hits the subject and is reflected, and then passes through the optical system 201. The light passing through the optical system 201 forms an image on the imaging sensor 203. The camera control unit 1707 generates a camera evaluation value again based on the formed image (real image).

[0095] Also, the light control unit 1713 calculates the emission amount of the main emission based on the camera evaluation value and outputs it to the strobe control unit 1715. Then, the strobe control unit 1715 gives an instruction for the main emission to the strobe module 213. The strobe module 213 executes the main emission of the real light source.

[0096] The virtual strobe light control unit 1711 determines the color temperature and emission amount of the virtual light source based on the color temperature of the real light source and the emission amount of the main emission. The virtual strobe control unit 1712 matches the color temperature and brightness of the light illuminating the CG with the color temperature and brightness of the real space. The CG generation unit 1709 sets the color temperature and emission amount of the virtual light source and arranges the virtual light source. The CG generation unit 1709 generates a CG illuminated by the ambient light and the light from the virtual light source. The synthesis unit 205 synthesizes the image-processed real image and the CG image to obtain a synthesized image.

[0097] This method is applicable not only to still images but also to strobe shooting in videos. In this way, it is possible to obtain a photo-realistic synthesized image taken with a virtual light source linked to the real light source.

[0098] Figure 18 shows the ratio of ambient light to strobe light. As the first pattern, the light 1803 combining ambient light and strobe light (light of an actual light source) is a mixed light of ambient light α1 and strobe light β1. As the second pattern, the light 1804 combining ambient light and strobe light is a mixed light of ambient light α2 and strobe light β2.

[0099] Figure 19 shows the blackbody radiation curve on the xy color space. For example, assume that the color temperature of the ambient light is 4500K (see point 1901), and the color temperature of the actual light source (strobe light source) is 3000K (see point 1902). In this case, the color temperature of the light 1803, which is a mixed light of ambient light and strobe light in the first pattern of Figure 18, can be calculated by the following formula. By using this formula, it is possible to match the color temperature of the mixed light of the strobe light and ambient light of the virtual light source with the mixed light of the strobe light and ambient light of the actual light source.

Equation

[0100] Referring to the flowchart of Figure 20, the process according to Embodiment 3 will be described.

[0101] In step S2002, the strobe module 213 performs pre-emission of the actual light source.

[0102] In step S2003, the camera control unit 1707 calculates a camera evaluation value based on the real image (an image formed on the imaging sensor 203 by the light of the pre-emission being reflected by the subject). Based on the camera evaluation value, the dimming unit 1713 calculates the emission amount required for the main emission of the actual light source, and the intensity and color temperature of the ambient light.

[0103] In step S2004, the strobe module 213 performs the main emission of the actual light source with the emission amount calculated in step S2003.

[0104] In step S2005, the imaging sensor 203 acquires a real image of a subject irradiated with light by the main emission of a real light source.

[0105] In step S2006, the virtual strobe dimming unit 1711 calculates the emission amount and color temperature of a virtual light source based on "the emission amount and color temperature of the main emission of the real light source" and "the intensity and color temperature of the ambient light". At this time, the virtual strobe control unit 1712 adjusts the CG to the same brightness and color temperature as when irradiated with ambient light based on the intensity and color temperature of the ambient light.

[0106] In step S2007, the virtual strobe control unit 1712 causes the virtual light source to emit light with the calculated emission amount and color temperature of the virtual light source.

[0107] In step S2008, the CG generation unit 1709 acquires an image of a CG (virtual object) irradiated with light from the virtual light source.

[0108] In step S2009, the synthesizing unit 205 synthesizes the real image and the CG image to generate a synthesized image.

[0109] According to Embodiment 3, when photographing a virtual object and a real subject at night or in a dark place, it is possible to more appropriately control not only the brightness but also the color temperature of the subject photographed using the real light source and the virtual object photographed using the virtual light source. Therefore, it is possible to obtain a photo-realistic synthesized image.

[0110] Although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope not departing from the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely shows one embodiment of the present invention, and it is also possible to appropriately combine the embodiments.

[0111] Further, the present invention is applicable not only to the imaging device main body which is a camera, but also to a control device that communicates with an imaging device (including a network camera) via wired or wireless communication and remotely controls the imaging device. Examples of the device for remotely controlling the imaging device include devices such as smartphones, tablet PCs, and desktop PCs. Based on the operations performed on the control device side or the processes performed on the control device side, the imaging device can be remotely controlled by notifying the imaging device with commands for performing various operations and settings from the control device side. Also, the live view image captured by the imaging device may be received via wired or wireless communication and displayed on the control device side.

[0112] Also, in the above, “If A is greater than or equal to B, proceed to step S1; if A is smaller (lower) than B, proceed to step S2” may be read as “If A is greater (higher) than B, proceed to step S1; if A is less than or equal to B, proceed to step S2”. Conversely, “If A is greater (higher) than B, proceed to step S1; if A is less than or equal to B, proceed to step S2” may be read as “If A is greater than or equal to B, proceed to step S1; if A is smaller (lower) than B, proceed to step S2”. Therefore, as long as there is no contradiction, “A or more” may be read as “greater (higher; longer; more) than A”, and “A or less” may be read as “smaller (lower; shorter; less) than A”. And “greater (higher; longer; more) than A” may be read as “A or more”, and “smaller (lower; shorter; less) than A” may be read as “A or less”. As long as there is no contradiction, “A or more” may be read as “greater (higher; longer; more) than A”, and “A or less” may be read as “smaller (lower; shorter; less) than A”. And “greater (higher; longer; more) than A” may be read as “A or more”, and “smaller (lower; shorter; less) than A” may be read as “A or less”.

[0113] Note that each functional unit in the above embodiments (each modification) may be individual hardware, or may not be. The functions of two or more functional units may be realized by common hardware. Each of the multiple functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Also, each functional unit may be realized by hardware such as an ASIC, FPGA, or DSP, or may not be. For example, the device may have a processor and a memory (storage medium) storing a control program. And the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0114] (Other Embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

[0115] The disclosure of the above embodiments includes the following configurations, methods, and programs. (Configuration 1) First light source control means for controlling a first light source that irradiates light on a first object in the real space, Second light source control means for controlling a second light source that irradiates light on a second object in the virtual space, First image acquisition means for acquiring a first image obtained by imaging the first object irradiated with light from the first light source that emits light with a first light amount, Second image acquisition means for acquiring a second image obtained by imaging the second object irradiated with light from the second light source that emits light with a second light amount, Generation means for generating a composite image by combining the first image and the second image, having The second light source control means controls the second light quantity based on the first light quantity. An imaging system characterized by this. (Configuration 2) The second light source control means controls the color temperature of the second light source based on the color temperature of the first light source. The imaging system according to Configuration 1, characterized by this. (Configuration 3) The second light source control means controls the color temperature of the second light source based on the color temperature of the first light source and the color temperature of the ambient light in the real space. The imaging system according to Configuration 2, characterized by this. (Configuration 4) The first light source control means controls the first light quantity based on a third image obtained by imaging the first object irradiated with light from the first light source that emits light with a predetermined light quantity. The imaging system according to any one of Configurations 1 to 3, characterized by this. (Configuration 5) The imaging system further includes arrangement means for arranging the second light source at a position in the virtual space corresponding to the position of the first light source in the real space. The imaging system according to any one of Configurations 1 to 4, characterized by this. (Configuration 6) In a specific case, the second light source control means does not control the second light quantity based on the first light quantity, and the first light source control means controls the first light quantity based on the second light quantity. The imaging system according to any one of Configurations 1 to 5, characterized by this. (Configuration 7) First light source control means for controlling a first light source that irradiates light on a first object in the real space; Second light source control means for controlling a second light source that irradiates light on a second object in the virtual space; First image acquisition means for acquiring a first image obtained by imaging the first object irradiated with light from the first light source that emits light with a first light quantity; Second image acquisition means for acquiring a second image obtained by imaging the second object irradiated with light from the second light source that emits light with a second light quantity; generation means for generating a composite image obtained by combining the first image and the second image; and the first light source control means controls the first light quantity based on the second light quantity. An imaging system characterized by the above. (Configuration 8) The second light source control means controls the second light quantity based on the brightness of the real space. The imaging system according to Configuration 7, characterized by the above. (Configuration 9) The first image, the second image, and the composite image are each a moving image. The imaging system according to any one of Configurations 1 to 8, characterized by the above. (Configuration 10) The second light quantity is the same as the first light quantity. The imaging system according to any one of Configurations 1 to 9, characterized by the above. (Method 1) A first light source control step of controlling a first light source that irradiates a first object with light in the real space; A second light source control step of controlling a second light source that irradiates a second object with light in the virtual space; A first image acquisition step of acquiring a first image obtained by imaging the first object irradiated with light from the first light source that emits light with a first light quantity; A second image acquisition step of acquiring a second image obtained by imaging the second object irradiated with light from the second light source that emits light with a second light quantity; A generation step of generating a composite image obtained by combining the first image and the second image; and In the second light source control step, the second light quantity is controlled based on the first light quantity. A control method for an imaging system, characterized by the above. (Method 2) A first light source control step of controlling a first light source that irradiates light onto a first object in the real space; A second light source control step of controlling a second light source that irradiates light onto a second object in the virtual space; A first image acquisition step of acquiring a first image obtained by imaging the first object irradiated with light from the first light source that emits light with a first light amount; A second image acquisition step of acquiring a second image obtained by imaging the second object irradiated with light from the second light source that emits light with a second light amount; A generation step of generating a composite image by combining the first image and the second image; comprising In the first light source control step, the first light amount is controlled based on the second light amount. A control method for an imaging system, characterized by the above. (Program) A program for causing a computer to function as each means of the imaging system according to any one of Configurations 1 to 10.

Explanation of Signs

[0116] 1: Imaging system, 203: Imaging sensor, 209: CG generation unit, 212: Virtual strobe control unit, 213: Strobe module, 214: Strobe control unit

Claims

1. First light source control means for controlling a first light source that irradiates light onto a first object in the real space; Second light source control means for controlling a second light source that irradiates light onto a second object in the virtual space; First image acquisition means for acquiring a first image obtained by imaging the first object irradiated with light from the first light source that emits light with a first light quantity; Second image acquisition means for acquiring a second image obtained by imaging the second object irradiated with light from the second light source that emits light with a second light quantity; Generation means for generating a composite image obtained by combining the first image and the second image; comprising The second light source control means controls the second light quantity based on the first light quantity. An imaging system characterized by this.

2. The second light source control means controls the color temperature of the second light source based on the color temperature of the first light source. The imaging system according to Claim 1, characterized by this.

3. The second light source control means controls the color temperature of the second light source based on the color temperature of the first light source and the color temperature of the ambient light in the real space. The imaging system according to Claim 2, characterized by this.

4. The first light source control means controls the first light quantity based on a third image obtained by imaging the first object irradiated with light from the first light source that emits light with a predetermined light quantity. The imaging system according to Claim 1, characterized by this.

5. Further comprising arrangement means for arranging the second light source at a position in the virtual space corresponding to the position of the first light source in the real space. The imaging system according to Claim 1, characterized by this.

6. In a specific case, the second light source control means does not control the second light amount based on the first light amount, and the first light source control means controls the first light amount based on the second light amount. The imaging system according to claim 1, characterized in that.

7. First light source control means for controlling a first light source that irradiates light on a first object in the real space; Second light source control means for controlling a second light source that irradiates light on a second object in the virtual space; First image acquisition means for acquiring a first image obtained by imaging the first object irradiated with light from the first light source that emits light with a first light amount; Second image acquisition means for acquiring a second image obtained by imaging the second object irradiated with light from the second light source that emits light with a second light amount; Generating means for generating a composite image by combining the first image and the second image; Having The first light source control means controls the first light amount based on the second light amount. An imaging system characterized in that.

8. The second light source control means controls the second light amount based on the brightness of the real space. , The imaging system according to claim 7, characterized in that.

9. The first image, the second image, and the composite image are each a moving image. The imaging system according to any one of claims 1 to 8, characterized in that.

10. The second light amount is the same as the first light amount. The imaging system according to any one of claims 1 to 8, characterized in that.

11. A first light source control step of controlling a first light source that irradiates light on a first object in the real space; A second light source control step of controlling a second light source that irradiates light onto a second object in a virtual space; A first image acquisition step of acquiring a first image obtained by imaging the first object irradiated with light from the first light source that emits light with a first light quantity; A second image acquisition step of acquiring a second image obtained by imaging the second object irradiated with light from the second light source that emits light with a second light quantity; A generation step of generating a composite image obtained by combining the first image and the second image; comprising; In the second light source control step, the second light quantity is controlled based on the first light quantity. A control method for an imaging system, characterized by the above.

12. A first light source control step of controlling a first light source that irradiates light onto a first object in the real space; A second light source control step of controlling a second light source that irradiates light onto a second object in the virtual space; A first image acquisition step of acquiring a first image obtained by imaging the first object irradiated with light from the first light source that emits light with a first light quantity; A second image acquisition step of acquiring a second image obtained by imaging the second object irradiated with light from the second light source that emits light with a second light quantity; A generation step of generating a composite image obtained by combining the first image and the second image; comprising; In the first light source control step, the first light quantity is controlled based on the second light quantity. A control method for an imaging system, characterized by the above.

13. A program for causing a computer to function as each means of the imaging system according to any one of Claims 1 to 8.

Images