Control device, control method, and imaging apparatus

Abstract

To provide a control device that can appropriately control the focal distance of an imaging apparatus according to the focal distance of a background image.SOLUTION: When a target focal distance is larger than a first distance being the distance between an imaging apparatus and a display provided within an imaging range of the imaging apparatus, a control device controls the focal distance of the imaging apparatus to be the first distance. When the target focal distance is equal to or less than the first distance, the control device controls the focal distance of the imaging apparatus to be the target focal distance.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a control device, a control method, and an imaging device, and in particular to a technique for controlling a focal distance. [Background technology]

[0002] There is known a method (in-camera VFX) for capturing an image of a VFX (Visual Effects) video without compositing a background image and a live-action video by capturing an image of a subject against a background of a display device that displays an image according to the position and orientation of the camera. Patent Document 1 discloses a virtual studio system that generates a background image that reflects the focus and zoom operations of the camera in addition to the position and orientation of the camera. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-102923 A Summary of the Invention [Problem to be solved by the invention]

[0004] The technique of Patent Document 1 does not anticipate a focus operation such as focusing on an object farther than the display device.

[0005] In view of the problems with the conventional techniques, one aspect of the present invention provides a control device and a control method capable of appropriately controlling the focal distance of an imaging device in accordance with the focal distance of a background image. [Means for solving the problem]

[0006] According to one aspect of the present invention, a control device is provided which comprises: an acquisition means for acquiring a target focus distance; and a control means for controlling the focus distance of the imaging device to the first distance when the target focus distance is greater than a first distance which is the distance between the imaging device and a display device provided in the imaging range of the imaging device; and for controlling the focus distance of the imaging device to the target focus distance when the target focus distance is equal to or less than the first distance. Effect of the Invention

[0007] According to one aspect of the present invention, it is possible to provide a control device and a control method capable of appropriately controlling the focal distance of an imaging device in accordance with the focal distance of a background image. [Brief description of the drawings]

[0008] [Figure 1] Schematic diagram of a virtual studio according to an embodiment. [Diagram 2] FIG. 1 is a block diagram showing an example of the functional arrangement of devices that realize a virtual studio according to a first embodiment. [Diagram 3] FIG. 11 is a block diagram showing another example of the functional configuration of the device that realizes the virtual studio according to the first embodiment. [Figure 4] FIG. 1 is a diagram showing an example of an in-camera VFX image obtained in a virtual studio according to an embodiment. [Diagram 5] Flowchart for controlling the focus distance according to the embodiment [Figure 6] Flowchart of blurring operation according to the embodiment [Figure 7] Flowchart for a modified example of the control operation of the focal distance according to the embodiment [Figure 8] FIG. 11 is a block diagram showing an example of the functional arrangement of devices that realize a virtual studio according to a second embodiment. [Figure 9] FIG. 11 is a block diagram showing another example of the functional configuration of the device that realizes the virtual studio according to the second embodiment. [Figure 10] Flowchart for controlling the focal distance according to the second embodiment DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The present invention will be described in detail below based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. In addition, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated explanations are omitted.

[0010] In the following embodiment, the present invention will be described with respect to a case where the present invention is implemented in a digital camera. However, the present invention can be implemented in any electronic device having a camera capable of controlling the focal distance. Such electronic devices include not only imaging devices but also computer devices (personal computers, tablet computers, media players, PDAs, etc.), smartphones, game consoles, robots, drones, etc. These are merely examples, and the present invention can be implemented in other electronic devices.

[0011] ●(First embodiment) In the method of Patent Document 1, for example, when a focus operation is performed to focus on a point farther than the display device, a background image reflecting the focus operation is generated. However, in this case, the camera will not focus on the display device that displays the background image, so there is a problem that an appropriate VFX image cannot be obtained. Therefore, in this embodiment, a system is provided that can obtain an appropriate VFX image even when imaging is performed with a focus on a point farther than the display device by appropriately controlling the focus distance of the background image and the focus distance of the imaging device.

[0012] (Virtual Studio Overview) FIG. 1 is a schematic diagram of a virtual studio according to an embodiment. FIG. 1(a) shows a case where a background image is generated by CG, and FIG. 1(b) shows a case where a background image is generated by imaging. In either case, the virtual studio has at least a camera 11 that captures VFX footage, and a display device 12 that is provided within the imaging range of the camera 11 and displays a background image. In the following, a subject 13 existing between the display device 12 and the camera 11 is referred to as a real subject, and a subject 14 included in the background image displayed on the display device 12 is referred to as a virtual subject. In the following description, a human subject is assumed for convenience, but there is no restriction on the type of subject.

[0013] In the configuration of FIG. 1(a), an image generating device 20 generates a background image (CG) according to the position and attitude of a camera 11. The image generating device 20 may be, for example, a computer on which real-time 3D image generating software called a game engine runs. A display control device 21 causes a display device 12 to display the background image generated by the image generating device 20. The display control device 21 causes the background image to be displayed in accordance with the image capturing timing of the camera 11. In addition, when the display device 12 is composed of multiple display panels, the background image may be divided to match each display panel. In FIG. 1(a), a virtual subject 14 is a CG image generated by the image generating device 20 based on, for example, a 3D model.

[0014] When generating a background image using CG, a background image focused at the set distance can be generated by setting the focal distance as a parameter of the virtual camera that captures the virtual space. The blur of the background image is expressed according to the optical parameters and imaging parameters (e.g., aperture value) of the virtual lens used by the virtual camera.

[0015] In the configuration of Fig. 1(b), camera 16 captures a background image according to the position and attitude of camera 11. Display control device 21 causes display device 12 to display the background image output by camera 16. Here, it is assumed that camera 16 captures a background image including real subject 17. Therefore, virtual subject 14 in Fig. 1(b) is a real image of real subject 17.

[0016] When a background image is actually captured, the background image is obtained in focus at the focal distance of camera 16 that captures the background image. The blur of the background image depends on the optical parameters of the lens used by camera 16 and the imaging parameters (for example, aperture value).

[0017] (Equipment configuration) Fig. 2 is a block diagram showing an example of the functional configuration of the device shown in Fig. 1(a). For ease of explanation and understanding, in Fig. 2, it is assumed that the display device 12 has functions corresponding to the image generating device 20 and the display control device 21. It is also assumed that the camera 11 is fixed to a known position and orientation.

[0018] Camera 11 has lens unit 160 that generates an optical image of a subject on an imaging surface. Lens unit 160 may be an interchangeable lens that is detachable from camera 11. When lens unit 160 is an interchangeable lens, lens unit 160 and camera 11 are communicatively connected via contacts provided on a lens mount.

[0019] In the lens unit 160, the aperture 103 has a variable aperture size and adjusts the amount of light incident on the imaging lens 104. The aperture drive circuit 105 drives the aperture 103 under the control of the lens control unit 107, thereby making it possible to adjust the aperture size of the aperture 103. Note that the aperture size (aperture value) of the aperture 103 can be set, for example, by operating an aperture ring provided on the lens unit 160 or as a result of automatic exposure control (AE) processing executed by the image processing unit 109.

[0020] The imaging lens 104 is composed of multiple lenses including a movable lens, and generates an optical image of a subject on an imaging plane. The movable lenses include at least a focus lens, and may also include a zoom lens, an image stabilization lens, and the like. The focal distance of the imaging lens 104 (lens unit 160) is determined by the position of the focus lens. The position of the focus lens is detected by the lens unit 160, and can be notified to the control device 150, for example, upon request.

[0021] The position of the focus lens can be adjusted by the AF drive circuit 106 driving the focus lens under the control of the lens control unit 107. The position of the focus lens can be automatically adjusted by automatic focus detection (autofocus: AF). The position of the focus lens can also be adjusted according to the amount of operation of a focus ring 101 provided on the lens unit (manual focus: MF).

[0022] Here, manual focusing is achieved by a by-wire system in which lens control unit 107 controls the driving of the focus lens in accordance with the amount and direction of operation of focus ring 101 detected by rotary encoder 102. However, a configuration in which focus ring 101 mechanically drives the focus lens may also be used.

[0023] When the imaging lens 104 includes a movable lens other than a focus lens, such as a zoom lens or an image stabilization lens, the lens unit 160 has a drive circuit for each movable lens. The operation of the drive circuit is controlled by the lens control unit 107.

[0024] The imaging unit 108 includes an imaging element and converts the subject image generated by the imaging lens 104 (lens unit 160) into an image signal. The imaging element may be, for example, a known CCD or CMOS color image sensor having a primary color Bayer array color filter. The imaging element has a pixel array in which a plurality of pixels are arranged two-dimensionally, and a peripheral circuit for reading out signals from each pixel. Each pixel accumulates electric charge according to the amount of incident light by photoelectric conversion. A pixel signal group (analog image signal) representing the subject image formed on the imaging surface is obtained by reading out from each pixel a signal having a voltage according to the amount of electric charge accumulated during the exposure period.

[0025] The image processing unit 109 applies predetermined signal processing and image processing to the analog image signal output by the imaging unit 108, generates signals and image data according to the application, and acquires and / or generates various information. The image processing unit 109 may be a dedicated hardware circuit such as an ASIC (Application Specific Integrated Circuit) designed to realize a specific function. Alternatively, the image processing unit 109 may be configured such that a processor such as a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit) executes software to realize a specific function. The image processing unit 109 outputs the acquired or generated information and data to the control device 150 according to the application.

[0026] The image processing applied by the image processing unit 109 can include, for example, pre-processing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Pre-processing may include signal amplification, reference level adjustment, defective pixel correction, A / D conversion, etc. Color interpolation processing is performed when a color filter is provided on an image sensor, and is a process for interpolating values ​​of color components that are not included in the individual pixel data that constitutes the image data. Color interpolation processing is also called demosaic processing. The correction processing can include white balance adjustment, gradation correction, correction of image degradation caused by optical aberration of the imaging lens 104 (image restoration), correction of the effect of peripheral light falloff of the imaging lens 104, color correction, and the like. The detection process may include detection of feature regions (for example, face regions or human body regions) and their movements, person recognition processing, and the like. Data processing may include processes such as area extraction (trimming), synthesis, scaling, encoding and decoding, header information generation (data file generation), etc. Data processing also includes the generation of image data for display or image data for recording. The evaluation value calculation process can include processes such as generation of a signal and evaluation value used in automatic focus detection (AF) and generation of an evaluation value used in automatic exposure control (AE). The special effects processing may include adding a blur effect, changing color tones, relighting, and the like. It should be noted that these are merely examples of processes that the image processing unit 109 can apply, and do not limit the processes that the image processing unit 109 can apply.

[0027] In this embodiment, the imaging element of the imaging unit 108 can output an image signal for performing imaging surface phase difference AF, and the image processing unit 109 calculates the defocus amount and defocus direction as the evaluation value calculation process. Note that the image processing unit 109 may obtain the defocus amount and defocus direction according to other AF methods, such as phase difference AF or contrast AF using a dedicated AF sensor. The defocus amount and defocus direction can be converted into the drive amount and drive direction of the focus lens, respectively.

[0028] Furthermore, the image processing unit 109 executes AE processing based on the AE evaluation value and determines the exposure conditions (imaging parameters). Note that if the aperture value changes during video capture, the depth of field changes. Therefore, when the brightness of the imaging range changes, the image processing unit 109 changes at least one of the imaging sensitivity and the exposure time in the AE processing without changing the aperture value.

[0029] Control device 150 is, for example, a processor (CPU, MPU, microprocessor, etc.) capable of executing a program. Control device 150 reads a program stored in ROM 152 into RAM 151 and executes it to control the operation of camera 11 including lens unit 160 and realize the functions of camera 11.

[0030] 2, the functional blocks included in the control device 150 are functional blocks that represent the main functions in this embodiment among various functions realized by the control device 150. Therefore, the operations of the target focus distance designation unit 110, the image focus control unit 112, the camera focus control unit 113, and the background distance acquisition unit 119 are actually executed by the control device 150. Note that one or more of the target focus distance designation unit 110, the image focus control unit 112, the camera focus control unit 113, and the background distance acquisition unit 119 may be implemented by a hardware circuit.

[0031] ROM 152 is a rewritable non-volatile memory, and stores programs executed by control device 150, various setting values ​​of camera 11, GUI data, etc. RAM 151 is used to read programs executed by control device 150 and to temporarily store variables and the like when executing the programs. RAM 151 may be used as a buffer memory for image data, etc., or as a video memory for storing image data for display.

[0032] Target focus distance designation unit 110 determines a target focus distance (a target distance between camera 11 and a subject at which a focused image can be obtained) of camera 11 (lens unit 160). When operation of focus ring 101 is enabled, target focus distance designation unit 110 determines the target focus distance according to the defocus amount calculated by image processing unit 109 and / or the operation of focus ring 101. Camera 11 is capable of setting an autofocus mode or a manual focus mode. Also, it is capable of setting whether operation of focus ring 101 in autofocus mode is enabled or disabled.

[0033] When the manual focus mode is set, the target focus distance designation unit 110 determines, for example, a focus distance that reflects the operation of the focus ring 101 on the current focus distance as the target focus distance. On the other hand, when the autofocus mode (operation of the focus ring 101 is valid) is set, the target focus distance designation unit 110 determines, for example, a focus distance corresponding to a defocus amount calculated by the image processing unit 109 as the target focus distance. When the operation of the focus ring 101 is detected, the target focus distance is adjusted according to the operation of the focus ring 101. When the autofocus mode (operation of the focus ring 101 is invalid) is set, the target focus distance designation unit 110 determines a target focus distance corresponding to the defocus amount calculated by the image processing unit 109. At this time, for example, a specific subject area (for example, a face area) may be detected, and a distance corresponding to the defocus amount for the detected area may be set as the target focus distance.

[0034] In this case, in either the autofocus mode or the manual focus mode, the focal distance of the camera 11 at the time when the user clearly gives an instruction, for example, by operating a confirmation button included in the operation unit of the camera 11, is determined as the target focal distance. However, the target focal distance may be determined based on other conditions. For example, if the focal distance does not change for a certain period of time after the operation of the focus ring 101 is stopped, the focal distance of the camera 11 at that time may be determined as the target focal distance. Alternatively, the target focal distance may be changed in real time following the change in the focal distance.

[0035] Video focus control unit 112 and camera focus control unit 113 constitute focus control unit 111. Video focus control unit 112 determines the focus distance of the background image based on the target focus distance from target focus distance designation unit 110 and the background distance from background distance acquisition unit 119, and transmits it to video generation unit 201 of display device 12. Video generation unit 201 has the functions of image generation device 20 and display control device 21 in FIG. 1. Video generation unit 201 generates a background image (CG) using the focus distance of the background image supplied from video focus control unit 112.

[0036] The camera focus control unit 113 determines the focus distance of the lens unit 160 based on the target focus distance from the target focus distance designation unit 110 and the background distance from the background distance acquisition unit 119. The camera focus control unit 113 indicates the determined focus distance to the lens control unit 107. Note that the aperture value of the aperture 103 can also be changed by notifying the lens control unit 107 of the aperture value from the camera focus control unit 113.

[0037] The background distance acquisition unit 119 outputs the background distance to the focus control unit 111. The background distance is the distance from the camera 11 to the display device 12 (display screen). In this embodiment, the distance between the camera 11 and the display device 12 is assumed to be fixed. Therefore, the background distance acquisition unit 119 outputs a fixed distance that is set in advance based on the installation positions of the camera 11 and the display device 12. Note that, if the distance between the camera 11 and the display device 12 is variable, the background distance acquisition unit 119 can measure the background distance by any known method.

[0038] For example, when the image processing unit 109 generates a defocus map, the background distance acquisition unit 119 can obtain the background distance based on the defocus map. The defocus map has a format of a two-dimensional image, and each pixel value represents the subject distance at a corresponding position. The image background distance acquisition unit 119 can detect the defocus amount of the display device from the defocus map, and obtain the background distance based on the detected defocus amount and the current focus distance. The defocus amount of the display device can be detected as the most frequently occurring defocus amount, for example. The current focus distance can be known by obtaining the focus lens position from the lens unit 160.

[0039] The display device 12 has an image generating unit 201 and an image display unit 202. The image generating unit 201 generates a CG image of a 3D model of a preset scene viewed from a viewpoint having the same position and orientation as the camera 11. When generating the CG image, the focus distance of the background image is used as a virtual focus distance, so that a CG image of a scene focused on the target focus distance determined by the target focus distance designation unit 110 can be generated. When generating the CG image, a blur amount according to the distance from the target focus distance is given to a 3D object present in the scene. The image display unit 202 is a display panel, and displays the CG image (background image) generated by the image generating unit 201.

[0040] Fig. 3 is a block diagram showing an example of the functional configuration of the device shown in Fig. 1(b). For ease of explanation and understanding, in Fig. 3, it is assumed that the display device 12 has a function equivalent to the display control device 21. It is also assumed that the cameras 11 and 16 are fixed at known positions and orientations. In order to capture an image that does not look unnatural as the background of the scene captured by the camera 11, the camera 16 is installed, for example, so that its orientation and imaging position height are the same as those of the camera 11.

[0041] 3, camera 16 that captures background video has the same functional configuration as camera 11. Therefore, the description of the functional blocks of camera 16 will be given by citing the description of the functional blocks of the same names in camera 11.

[0042] 3, the image focus control unit 112 of the camera 11 transmits the focus distance of the background image to the target focus distance designation unit 310 of the camera 16, instead of transmitting it to the image generation unit 201 of the display device 12. Also, in the camera 16, the image processing unit 309 generates background image data to be displayed on the display device 12, and outputs the background image data to the display device 12.

[0043] In the configuration of Fig. 3, since there is no need to generate CG, display device 12 has a display control unit 203 having the same function as display control device 21. Note that display device 12 may have an image generation unit 201 as in Fig. 2. In this case, image generation unit 210 displays the received image on image display unit 202 without generating CG.

[0044] When camera 11 is connected, camera 16 operates with the focus distance of the background image received from camera 11 as the target focus distance. Therefore, when camera 11 is connected, operation of focus ring 101 and AF operation in camera 16 may basically be disabled. Note that control device 150 can detect whether camera 11 is connected or not by a known method according to the connection method. For example, in the case of a wired connection, it is possible to detect whether a device is connected or not based on the voltage of a specified terminal of a connector to which a cable is connected.

[0045] Fig. 4 is a schematic diagram of a video expression that can be realized in this embodiment. Fig. 4(a) is a schematic diagram of the virtual studio shown in Fig. 1 as seen from the side. Virtual subject 15 is a schematic representation of the position in the virtual space of virtual subject 14 displayed on display device 12, and does not actually exist behind display device 12.

[0046] 4(b) to (d) show schematic diagrams of how real subject 13 and virtual subject 14 are blurred in an image captured by camera 11. Fig. 4(b) shows an image in which camera 11 focuses on both real subject 13 and virtual subject 14. For example, such an image can be obtained when the distance between display device 12 and real subject 13 is short or when the aperture value of lens unit 160 is large.

[0047] Fig. 4(c) shows an image in which real subject 13 is in focus, but virtual subject 14 is out of focus. For example, such an image is obtained when the distance between display device 12 and real subject 13 is long, or when the aperture value of lens unit 160 is small. In this embodiment, it is assumed that virtual subject 14 is displayed on display device 12, but when the background image is a cityscape or the like, an image like that shown in Fig. 4(c) may be captured.

[0048] Fig. 4(d) shows an image in which virtual subject 14 is in focus, but real subject 13 is out of focus. The image shown in Fig. 4(d) has been difficult to capture in the past. This is because virtual subject 15 is located behind display device 12 in the virtual space, and when camera 11 is focused on the position of virtual subject 15 in the virtual space (virtual distance), virtual subject 14 displayed on display device 12 also becomes out of focus.

[0049] To obtain an image such as that shown in Fig. 4(d), it is necessary to generate a background image focused on virtual subject 15 while focusing camera 11 on display device 12. The method disclosed in the above-mentioned Patent Document 1 synchronizes the focal distance of camera 11 with the focal distance of the background image, and therefore it is not possible to obtain an image such as that shown in Fig. 4(d), which requires the focal distance of camera 11 to be different from the focal distance of the background image.

[0050] (Focal distance control operation) Next, the control operation of the focal distance in this embodiment will be described with reference to the flowchart shown in Fig. 5. The operation described below can be realized by the control device 150 executing a program, but for convenience, it will be described as an operation mainly performed by the focus control unit 111. The following operation can be executed, for example, in response to the target focal distance designation unit 110 determining the target focal distance.

[0051] In S 11 , the focus control unit 111 acquires the background distance from the background distance acquisition unit 119 .

[0052] In S12, the focus control unit 111 acquires a target focus distance from the target focus distance designation unit 110. The target focus distance is the distance between the camera 11 and a subject from which a focused image is to be acquired.

[0053] In S13, the focus control unit 111 determines whether the acquired target focus distance is equal to or shorter than the background distance (whether to focus on the display device 12 or a subject in front of the display device 12). If it is determined that the target focus distance is equal to or shorter than the background distance, the focus control unit 111 executes S14, and if not, executes S15.

[0054] In S14, camera focus control unit 113 outputs the target focus distance as the camera focus distance to lens control unit 107. In this way, when the subject to be focused is located at the same distance as display device 12 or in front of it, camera focus control unit 113 controls camera 11 to focus at the target focus distance.

[0055] In S15, camera focus control unit 113 outputs the background distance as the camera focusing distance to lens control unit 107. In this way, when the subject to be focused on is located behind display device 12, camera focus control unit 113 controls camera 11 to focus on display device 12.

[0056] In S16, the image focus control unit 112 outputs the target focus distance as the focus distance of the background image to the display device 12. The image generation unit 201 of the display device 12 generates CG of the background image by rendering the 3D model so as to focus at the target focus distance. Since the position and orientation of the virtual camera used when rendering the 3D model are equal to the position and orientation of the camera 11, the CG can be generated using the target focus distance as is.

[0057] 5 can be similarly applied to the manual focus mode. In S12, the focus control unit 111 acquires a target focus distance determined by the operation of the focus ring 101. At this time, even if the target focus distance is set to be behind the display device 12 (farther than the background distance), the background distance is output to the lens control unit 107 as the camera focus distance. This makes it possible to control the camera focus distance so that it does not exceed the background distance.

[0058] 3, image focus control unit 112 outputs the target focus distance to camera 16. Then, camera focus control unit 313 of camera 16 outputs the target focus distance to lens control unit 307, and controls the focus distance of camera 16 to become the target focus distance.

[0059] In this way, the background image is controlled to be focused at the target focus distance, and the focus distance of the camera 11 capturing the subject is made different depending on the magnitude relationship between the target focus distance and the background distance. Specifically, if the target focus distance is greater than the background distance, the focus distance of the camera is set as the background distance, and if not, the focus distance of the camera is set as the target focus distance. This makes it possible to realize a background image focused on a virtual subject that exists farther away than the display device, and a blur of a real subject that exists in front of the display device.

[0060] (Variation 1) When the target focus distance is greater than the background distance, and the focus distance of camera 11 is set as the background distance, if the distance between display device 12 and real subject 13 is short, the amount of blur for the image of real subject 13 may be insufficient. In addition, for a target focus distance greater than the background distance, the amount of blur for real subject 13 is constant regardless of the target focus distance, which may give an unnatural impression.

[0061] Therefore, when the target focus distance is greater than the background distance, an amount of blur according to the target focus distance can be added to the image of real subject 13. This process will be described with reference to the flowchart shown in Fig. 6. This process can be executed as part of the process in S15 in Fig. 5, for example.

[0062] In S22, the control device 150 detects the real subject. There is no particular limitation on the method of detecting the real subject, but the image processing unit 109 may generate a defocus map, and an area in the defocus map where the focus distance is shorter than the background distance may be detected as the image area of ​​the real subject. Alternatively, a specific subject area (e.g., a human body area) may be detected by the image processing unit 109, and a subject area that exists in front of the background distance may be detected as the area of ​​the real subject based on the defocus amount of the detected subject area.

[0063] In S23, the control device 150 acquires the actual defocus amount of the real subject. This defocus amount is the defocus amount when the focus lens is at the reference position. When the defocus map generated by the image processing unit 109 indicates the defocus amount when the focus lens is at the reference position, it can be used as the actual defocus amount as it is. On the other hand, when the defocus map indicates the relative defocus amount for a focus lens position other than the reference position, the control device 150 converts the defocus amount of the defocus map into the actual defocus amount for the reference position.

[0064] In S24, the control device 150 calculates the necessary defocus amount for the real subject. The necessary defocus amount is the defocus amount of the real subject area when the camera 11 is focused on the target focus distance. The necessary defocus amount can be calculated from the distance obtained by subtracting the distance between the camera 11 and the real subject from the target focus distance.

[0065] In S25, the control device 150 calculates a corrected defocus amount for the real object. The insufficient defocus amount is obtained by subtracting the actual defocus amount from the required defocus amount.

[0066] In S26, the control device 150 controls the image processing unit 109 to impart a bokeh effect based on the corrected defocus amount to the real object region in the image captured by the imaging unit 108. Although bokeh can also be added by reducing the aperture value of the aperture 103, the bokeh effect is added by the image processing unit 109 when it is desired that no change occurs in the imaging parameters.

[0067] When blur is provided by opening the aperture 103, if the corrected defocus amount is twice the actual defocus amount, the aperture value of the aperture 103 can be halved. For example, the aperture is opened from F5.6 to F2.8 at first. This is because the relationship between the blur diameter d, the defocus amount def, and the aperture value F is expressed as blur diameter d = defocus amount def / aperture value F. Therefore, if blur equivalent to twice the defocus amount def is required, the F value must be halved.

[0068] In addition, when blurring is performed by image processing, the blur diameter d obtained from the above formula is divided by the pixel pitch to convert it into the number of pixels, and the parameters of the Gaussian filter that imparts the blurring effect can be determined. By applying Gaussian filter processing to the real object region, the blurring effect can be imparted to the real object region.

[0069] (Variation 2) If the background image's focal distance and the camera's focal distance are controlled separately, the timing of the change in focal distance may be shifted, resulting in an unnatural impression of the in-camera VFX image. Therefore, in this modified example, the background image's focal distance and the camera's focal distance are controlled in coordination. This makes it possible to obtain an in-camera VFX video in which the background image's focal distance and the camera's focal distance change naturally.

[0070] Fig. 7(a) shows a case where the target focus distance changes from a state where it is closer than the background distance to a state where it is farther than the background distance, and Fig. 7(b) shows a case where the target focus distance changes from a state where it is farther than the background distance to a state where it is closer than the background distance. Fig. 7(a) can be executed as S15 in Fig. 5 and S16 following S15, and Fig. 7(b) can be executed as S14 in Fig. 5 and S16 following S14.

[0071] When the target focus distance is changed from a state equal to or less than the background distance to a state greater than the background distance, the video focus control unit 112 and the camera focus control unit 113 cooperate to control the focus distance from the target focus distance before the change to the background distance in S33 in Fig. 7(a). Specifically, the video focus control unit 112 and the camera focus control unit 113 control the focus distance of the background image and the focus distance of the camera 11 to change at the same speed.

[0072] For example, the image focus control unit 112 and the camera focus control unit 113 divide the difference between the target focus distance before the change and the background distance into a plurality of sections, and change the focus distance one section at a time. At this time, the image focus control unit 112 and the camera focus control unit 113 may synchronize the timing of notifying the focus distance.

[0073] When the background image focus distance and the camera focus distance reach the background distance, in S34, the video focus control unit 112 changes the background image focus distance to the changed target focus distance. In this case, as in S33, the difference between the background distance and the changed target focus distance may be divided into multiple intervals to change the background image focus distance in stages. Also, in order to match the change in the background image focus distance with the change in the focus distance in S33, the size of one interval may be set to the same as that of one interval in S33.

[0074] Also, when the target focus distance is changed from a state where it is larger than the background distance to a state where it is equal to or smaller than the background distance, in S43 of Fig. 7(b), the image focus control unit 112 changes the focus distance of the background image from the focus distance before the change to the background distance. Because the target focus distance before the change is larger than the background distance, the camera 11 is focused on the background distance. Therefore, the focus distance of the camera 11 is not changed in S43.

[0075] When the focus distance of the background image reaches the background distance, in S44, the video focus control unit 112 and the camera focus control unit 113 cooperate to control the focus distance from the background distance to the changed target focus distance. Specifically, the video focus control unit 112 and the camera focus control unit 113 control the focus distance of the background image and the focus distance of the camera 11 to change at the same speed. The method of controlling the focus distance here may be the same as that in S33.

[0076] In S43, the difference between the target focus distance before the change and the background distance may be divided into a plurality of intervals to change the focus distance of the background image in stages, as in S44. Also, in order to match the change in the focus distance of the background image with the change in the focus distance in S44, the size of one interval may be set to the same as that of one interval in S44.

[0077] Note that, although it is assumed here that the target focus distance is changed discontinuously across the background distance, similar control can also be implemented in cases where the target focus distance changes continuously following the operation of focus ring 101. In this case, when the target focus distance is within the background distance, image focus control unit 112 and camera focus control unit 113 cooperate to perform control, and when the target focus distance exceeds the background distance, image focus control unit 112 only controls the focus distance of the background image. This makes it possible to obtain VFX images with natural changes in focus distance.

[0078] According to this embodiment, when the focus distance of the background image is changed according to the setting of the focus distance of the imaging device that captures the in-camera VFX image, an appropriate VFX image can be obtained even when the focus distance is set behind the display device that displays the background image. Also, by adding a bokeh effect to the image of the real subject according to the target focus distance, a VFX image in which the real subject has a natural foreground blur can be obtained when the focus distance is set behind the display device that displays the background image.

[0079] ●(Second embodiment) Next, a second embodiment of the present invention will be described. In the first embodiment, focus control for a target focus distance specified by a user has been described. In this embodiment, focus control is performed when the distance at which the camera 11 focuses on a detected subject is automatically determined as the target focus distance.

[0080] When determining the target focus distance according to the detected subject, it is necessary to switch focus control depending on whether the detected subject is a real subject or a virtual subject. In addition, the distance of the virtual subject obtained from the camera is only the background distance, and it is not possible to obtain a distance corresponding to the position of the virtual subject in the virtual space. Therefore, the camera 11 cannot obtain a distance to focus on the virtual subject 15 shown in FIG. 4(a).

[0081] The focal distance of virtual subject 15 can be obtained from a device that generates the CG of the background image (image generation device 20 or image generation unit 201), or from a camera that captures the background image.

[0082] 8 and 9 are block diagrams showing examples of the functional configuration of the camera and display device constituting the virtual studio system according to this embodiment. The same components as those in the camera and display device explained in the first embodiment are given the same reference numerals as in Fig. 2 and Fig. 3, and duplicated explanations will be omitted.

[0083] In the configuration of Fig. 8 for generating a background image (CG), a virtual subject information transmission unit 204 is added to the display device 12'. In both Figs. 8 and 9, the configuration is such that virtual subject information is transmitted from the image processing unit 309 of the camera 16' to the image processing unit 109 of the camera 11'. In this specification, communication between cameras and between the camera and the display device is performed through an interface that complies with any standard suitable for the signals to be communicated. Representative standards that can be used for communication include, but are not limited to, serial digital interface (SDI), HDMI (registered trademark), USB, and wireless LAN.

[0084] In order to enable image processing unit 109 to determine whether a detected subject is a real subject or a virtual subject and to obtain the focal distance if the subject is a virtual subject, virtual subject information is supplied to camera 11. The virtual subject information includes information on the position (two-dimensional position) in the background image and the distance (focal distance) in the virtual space for each specific subject existing in the virtual space. Here, the specific subject is assumed to be a human subject, but may be any one or more types of subjects that can be detected by image processing unit 109. When there can be multiple types of virtual subjects, the virtual subject information includes information on the type for each virtual subject.

[0085] 8, the background image includes a virtual subject within a three-dimensional model of a virtual space rendered by the image generating unit 201. Therefore, the image generating unit 201 can obtain information regarding the position and distance of the virtual subject. The image generating unit 201 outputs virtual subject information regarding a predetermined type of subject to the camera 11' via the virtual subject information transmitting unit 204.

[0086] The camera 11' supplies the virtual subject information received from the display device 12' to the image processing unit 109. The image processing unit 109 executes a subject detection process for, for example, a preset type of subject, and obtains detection results such as the position, size, and detection reliability for each detected subject region. Then, based on the supplied virtual subject information, the image processing unit 109 can determine whether the detected subject region is a region of a real subject or a region of a virtual subject. Furthermore, based on the virtual subject information, the image processing unit 109 can also obtain a focus distance in the virtual space for the region of the virtual subject.

[0087] In the configuration of FIG. 9 in which a background image is captured by camera 16', for example, image processor 309 of camera 16' executes subject detection processing similar to that of image processor 109 of camera 11 to obtain subject detection results. Then, the defocus amount is obtained as information on the focal distance for each detected subject region. The defocus amount may be converted into distance. Then, image processor 309 outputs the subject detection result including information on the focal distance for each subject region to camera 11'. The processing on the camera 11' side is as described in FIG. 8.

[0088] A focus control operation for setting the focal distance for the subject area detected by camera 11' as the target focal distance will be described using the flowchart shown in Fig. 10. Note that, although the operation of camera 11' will be described here, it is assumed that virtual subject information is periodically (for example, every predetermined number of frames) supplied to camera 11' from display device 12' (Fig. 8) and camera 16' (Fig. 9).

[0089] In S51, the image processing unit 109 applies subject detection processing to, for example, frame images of a moving image for live view display obtained from the imaging unit 108. Here, the subject detection processing detects a human face area, but as described above, the type of subject to be detected is arbitrary. In the configuration shown in Fig. 9, it is sufficient that the subject is one that can be detected in common by the image processing unit 109 and the image processing unit 309 of the camera 16'.

[0090] In S52, the image processing unit 109 selects a main subject region from the subject regions detected in S51. If there is one subject region detected in S51, it is set as the main subject region as it is. If multiple subject regions are detected, the image processing unit 109 can select the main subject region according to a predetermined condition based on the size and position of the region. For example, the image processing unit 109 can select the subject region closest to the center of the image or the largest subject region as the main subject region. In addition, in the case of a face region, the face region of a specific individual may be selected as the main subject region. In addition, when the main subject region is switched at a specific timing, such as in movie shooting, the main subject region to be selected may be switched depending on the elapsed time from the start of shooting.

[0091] In S53, the image processing unit 109 acquires virtual subject information. The virtual subject information may be acquired by requesting it from the camera 11' to the display device 12' or the camera 16', or may be periodically transmitted from the display device 12' or the camera 16' to the camera 11'. Here, it is assumed that the most recently received virtual subject information is stored in the RAM 151 or a memory included in the image processing unit 109.

[0092] In S54, the image processing unit 109 refers to the virtual subject information and determines whether the selected main subject region is a region of a real subject or a region of a virtual subject. If the virtual subject information includes information of a virtual subject region corresponding to the main subject region, the image processing unit 109 can determine that the main subject region is a region of a virtual subject.

[0093] If it is determined in S54 that the main subject region is the region of the virtual subject, the image processing unit 109 executes S55, and if not, executes S59.

[0094] In S55, the image processing unit outputs the focus distance of the main subject region (the focus distance in the virtual space) to the target focus distance designation unit 110. Then, the target focus distance designation unit 110 determines the focus distance output by the image processing unit 109 as the target focus distance.

[0095] In S56, the focus control unit 111 determines whether the current focus distance is equal to or shorter than the background distance, and if it is determined that it is equal to or shorter than the background distance, executes S57, and if not, executes S58.

[0096] In S57, the focus control unit 111 controls the background image and the focal distance of the camera 11', for example, as described with reference to FIG. 7(a). When S58 is executed, the current focus distance of the camera 11' is the background distance. Therefore, the image focus control unit 112 controls the focus distance of the background image to the target focus distance. The camera focus control unit 113 does not change the focus distance.

[0097] In S59, image processing unit 109 acquires a focus distance for a main subject region, which is a region of a real subject. Image processing unit 109 acquires a defocus amount relative to the current focus distance, and converts it into an actual defocus amount corresponding to an absolute distance between camera 11' and the main subject. Image processing unit 109 further converts the actual defocus amount into a distance, and outputs it to target focus distance designation unit 110 as a focus distance.

[0098] In S60, the focus control unit 111 determines whether the current focus distance is equal to or shorter than the background distance, and if it is determined that it is equal to or shorter than the background distance, executes S62, and if not, executes S61.

[0099] In S61, the focus control unit 111 controls the background image and the focal distance of the camera 11', for example, as described with reference to FIG. 7(b).

[0100] In S62, the focus control unit 111 controls the focus distance of the background image and the focus distance of the camera 11' to the target focus distance. The camera focus control unit 113 converts the actual defocus amount into a defocus amount based on the current focus distance and outputs it to the focus control unit 107.

[0101] According to this embodiment, in addition to the effects of the first embodiment, it becomes possible to obtain an in-camera VFX image by automatically setting a target focus distance using a subject detection function in a virtual studio system as well.

[0102] (Other embodiments) In the above embodiment, a method of controlling the focal distance for both the camera and the background image has been described, but it is also possible to control only the focal distance of the camera. When the target focal distance is greater than the background distance, the problem of the conventional technology that the background image becomes blurred can be solved simply by fixing the focal distance of the camera to the background distance. In this case, for example, the execution of S16 in the operation shown in the flowchart of FIG. 5 can be skipped.

[0103] In the above embodiment, the control device 150 has been described as being included in the camera 11, but the control device 150 may be a device independent of the camera 11 and the display device 12. The control device 150 may also be provided in the display device 12. Some of the components of the control device 150 may also be provided independent of the camera 11 and the display device 12. For example, only the focus control unit 111 may be provided independent of the camera 11 and the display device 12.

[0104] In this embodiment, the method of predetermining the background distance and the method of detecting the background distance from a defocus map have been described, but the background distance may be detected using other methods. For example, when the camera 11 is hung by a crane, the background distance may be acquired based on the control parameters of the crane. Also, the attitude of the camera 11 may be detected, and the background distance may be acquired based on the attitude information of the camera 11. Also, the background distance may be acquired using an active ranging method such as LiDAR.

[0105] The disclosure of the present embodiment includes the following control device, control method, imaging device, and program. (Item 1) An acquisition means for acquiring a target focusing distance; A control means, When the target focus distance is greater than a first distance, which is a distance between an imaging device and a display device provided in an imaging range of the imaging device, the focus distance of the imaging device is controlled to the first distance; When the target focus distance is equal to or shorter than the first distance, the focus distance of the imaging device is controlled to the target focus distance. A control means; A control device comprising: (Item 2) 2. The control device according to item 1, wherein the control means controls a generating means that generates an image to be displayed on the display device using the target focus distance, thereby controlling the focus distance of the image. (Item 3) The control device described in item 2, characterized in that when the control means controls both the focal distance of the imaging device and the focal distance of the image, the control means controls so that the focal distance of the imaging device and the focal distance of the image change synchronously. (Item 4) The control device described in item 2 or 3, characterized in that when the current focus distance of the imaging device is greater than the first distance and the target focus distance is equal to or less than the first distance, the control means controls the focus distance of the image to the first distance, and then controls the focus distance of the imaging device and the focus distance of the image to the target focus distance. (Item 5) The control device described in any one of items 2 to 4, characterized in that when the current focus distance of the imaging device is equal to or less than the first distance and the target focus distance is greater than the first distance, the control means controls the focus distance of the imaging device and the focus distance of the image to the first distance, and then controls the focus distance of the image to the target focus distance. (Item 6) The control device described in any one of items 1 to 5, characterized in that, when the target focus distance is greater than the first distance, the control means further controls the imaging device to impart a bokeh effect corresponding to the difference from the target focus distance to an area of ​​a subject in an image captured by the imaging device that is closer than the first distance. (Item 7) 7. The control device according to item 6, wherein the control means controls the imaging device to impart the bokeh effect by changing imaging parameters of the imaging device. (Item 8) 7. The control device according to item 6, wherein the control means controls an image processing means included in the imaging device so that the imaging device applies the bokeh effect. (Item 9) 9. The control device according to any one of items 1 to 8, wherein the target focus distance is set by a user of the imaging device. (Item 10) 9. The control device according to any one of items 1 to 8, wherein the target focus distance is automatically set by the imaging device. (Item 11) a control means for controlling imaging by an imaging device and display by a display device provided within an imaging range of the imaging device; The control means controls the focal distance in the imaging so as not to exceed a first distance according to a distance between the imaging device and the display device. A control device comprising: (Item 12) An acquisition means for acquiring a target focusing distance; A control device having a control means for controlling imaging by an imaging device and display by a display device provided within an imaging range of the imaging device, The control means a generating means for generating an image to be displayed on the display device using the target focus distance, a focus distance of the imaging device, a bokeh effect on an image to be displayed on the display device, and a bokeh effect on an area of ​​a subject other than the display device in the image captured by the imaging device are controlled according to the target focus distance; A control device comprising: (Item 13) An imaging means; an image processing means for processing the image captured by the imaging means; A control device according to any one of items 1 to 12, which controls a focal distance of the imaging means; An imaging device comprising: (Item 14) 14. The imaging apparatus according to item 13, wherein the focal distance of the subject area detected by the image processing means is set as the target focal distance. (Item 15) 15. The imaging device according to item 14, characterized in that, when the subject area is a subject included in the image, the target focusing distance is set based on information of the subject included in the image. (Item 16) Obtaining a target focus distance; When the target focus distance is greater than a first distance, which is a distance between an imaging device and a display device provided in an imaging range of the imaging device, controlling the focus distance of the imaging device to the first distance; When the target focus distance is equal to or less than the first distance, controlling a focus distance of the imaging device to the target focus distance; A control method comprising the steps of: (Item 17) A control method for controlling imaging by an imaging device and display by a display device provided within an imaging range of the imaging device, comprising: Controlling a focal distance in the imaging so as not to exceed a first distance corresponding to a distance between the imaging device and the display device; A control method comprising the steps of: (Item 18) Obtaining a target focus distance; A control method comprising: capturing an image by an imaging device; and controlling a display by a display device provided within an imaging range of the imaging device, The controlling comprises: generating an image to be displayed on the display device using the target focus distance; controlling a focus distance of the imaging device, a bokeh effect on an image to be displayed on the display device, and a bokeh effect on an area of ​​a subject other than the display device in the image captured by the imaging device according to the target focus distance; A control method comprising the steps of: (Item 19) A program for causing a computer to function as each of the means possessed by the control device according to any one of items 1 to 12.

[0106] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. [Explanation of symbols]

[0107] 11, 16... camera, 12... display device, 13... real subject, 14, 15... virtual subject, 111... focus control unit, 112... video focus control unit, 113... camera focus control unit, 150... control device

Claims

1. An acquisition means for acquiring a target focusing distance; A control means, When the target focus distance is greater than a first distance, which is a distance between an imaging device and a display device provided in an imaging range of the imaging device, the focus distance of the imaging device is controlled to the first distance; When the target focus distance is equal to or shorter than the first distance, the focus distance of the imaging device is controlled to the target focus distance. A control means; A control device comprising:

2. 2. The control device according to claim 1, wherein the control means controls a generating means for generating an image to be displayed on the display device using the target focusing distance, thereby controlling the focusing distance of the image.

3. The control device according to claim 2, characterized in that, when the control means controls both the focal distance of the imaging device and the focal distance of the image, the control means controls so that the focal distance of the imaging device and the focal distance of the image change synchronously.

4. The control device according to claim 2, characterized in that, when the current focus distance of the imaging device is greater than the first distance and the target focus distance is equal to or less than the first distance, the control means controls the focus distance of the image to the first distance, and then controls the focus distance of the imaging device and the focus distance of the image to the target focus distance.

5. The control device described in claim 2, characterized in that when the current focus distance of the imaging device is equal to or less than the first distance and the target focus distance is greater than the first distance, the control means controls the focus distance of the imaging device and the focus distance of the image to the first distance, and then controls the focus distance of the image to the target focus distance.

6. The control device according to claim 1, characterized in that, when the target focus distance is greater than the first distance, the control means further controls the imaging device to impart a bokeh effect to areas of subjects in the image captured by the imaging device that are closer than the first distance, according to the difference from the target focus distance.

7. 7. The control device according to claim 6, wherein the control means controls the imaging device to impart the bokeh effect by changing an imaging parameter of the imaging device.

8. 7. The control device according to claim 6, wherein the control means controls an image processing means included in the imaging device so that the imaging device applies the bokeh effect.

9. The control device according to claim 1 , wherein the target focus distance is set by a user of the imaging device.

10. 2. The control device according to claim 1, wherein the target focus distance is automatically set by the imaging device.

11. a control means for controlling imaging by an imaging device and display by a display device provided within an imaging range of the imaging device; The control means controls a focal distance in the imaging so as not to exceed a first distance corresponding to a distance between the imaging device and the display device. A control device comprising:

12. An acquisition means for acquiring a target focusing distance; A control device having a control means for controlling imaging by an imaging device and display by a display device provided within an imaging range of the imaging device, The control means a generating means for generating an image to be displayed on the display device using the target focus distance, a focus distance of the imaging device, a bokeh effect on an image to be displayed on the display device, and a bokeh effect on an area of ​​a subject other than the display device in the image captured by the imaging device are controlled according to the target focus distance; A control device comprising:

13. An imaging means; an image processing means for processing the image captured by the imaging means; A control device according to any one of claims 1 to 12, which controls a focal distance of the imaging means; An imaging device comprising:

14. 14. The imaging apparatus according to claim 13, wherein the focus distance of the subject area detected by the image processing means is set as the target focus distance.

15. The imaging apparatus according to claim 14 , wherein, when the subject area is a subject included in the video, the target focusing distance is set based on information about the subject included in the video.

16. Obtaining a target focus distance; When the target focus distance is greater than a first distance, which is a distance between an imaging device and a display device provided in an imaging range of the imaging device, controlling the focus distance of the imaging device to the first distance; When the target focus distance is equal to or less than the first distance, controlling a focus distance of the imaging device to the target focus distance; A control method comprising the steps of:

17. A control method for controlling imaging by an imaging device and display by a display device provided within an imaging range of the imaging device, comprising: Controlling a focal distance in the imaging so as not to exceed a first distance corresponding to a distance between the imaging device and the display device; A control method comprising the steps of:

18. Obtaining a target focus distance; A control method comprising: capturing an image by an imaging device; and controlling a display by a display device provided within an imaging range of the imaging device, The controlling comprises: generating an image to be displayed on the display device using the target focus distance; controlling a focus distance of the imaging device, a bokeh effect on an image to be displayed on the display device, and a bokeh effect on an area of ​​a subject other than the display device in the image captured by the imaging device according to the target focus distance; A control method comprising the steps of:

19. A program for causing a computer to function as each of the means included in the control device according to any one of claims 1 to 12.

Images