Camera focus control

Abstract

A control device for a video camera, comprising: an input for receiving a focus command indicating a focal length; and one or more outputs for transmitting (i) a first signal for controlling a focus of a lens of the video camera; and (ii) a second signal indicating the focal length indicated by the focus command, the control device being configured to determine a focus limit and to transmit the first signal instructing the video camera to adjust the focus to the focal length indicated by the focus command only if the focal length indicated by the focus command does not exceed the focus limit.

Description

Technical Field

[0001] The present invention relates to controlling one or more of (i) the focus operation of a camera and (ii) the operation of a controllable background.

Background Art

[0002] It is known to shoot a video of a subject against a known color, generally a green background, and then edit the video to replace the original background with a still image or video. This can give the impression that the subject is in a different environment. For example, a subject can be photographed against a green background in a studio, and the video can be edited to replace the green background with a seaside landscape. In this approach, post-processing of the captured video is required, and the amount of data processing may increase. In particular, when the subject includes the same color as the background, inaccuracies may occur in the replacement process.

[0003] An alternative approach is to use a screen as the background, display a still image or video on the screen, and video-record the subject against the screen with a stationary or moving camera. The screen may be a plain screen on which a still image or video is projected, or a display device or a plurality of display devices arranged side by side. This approach can reduce the amount of post-processing required, but a display system (generally a graphics engine) that presents the background on the display is required at the location where the video is recorded.

[0004] It is desirable to be able to enhance or better control the quality that enables video recording against a display background.

Summary of the Invention

[0005] According to one approach, an apparatus and method as described in the appended claims are provided.

Brief Description of the Drawings

[0006] Herein, the present invention will be described as an example with reference to the attached drawings.

[0007] [Figure 1] Figure 1 shows a device for capturing video of a subject against a dynamic background. [Modes for carrying out the invention]

[0008] In the system shown in Figure 1, camera 1 is connected to the screen display wall 3. as a screen The system is positioned to capture video of subject 2 with the background. The system is configured so that the camera's focus is controlled depending on the relative position of the camera to the display wall. The system is also configured so that the image on the display can be blurred depending on the camera's focus setting. The image may be a still image or a frame from a video.

[0009] Next, we will explain this system in more detail.

[0010] Camera 1 can capture video. Camera 1 has a lens 4 whose focus is adjustable by a motor 5. It also has an image sensor (not shown) for sensing images. Images sensed by the sensor can be stored in the camera or transmitted by the camera for storage or display elsewhere. The camera may have another motor for adjusting the depth of field of the lens. The field of view and / or aperture of the lens may optionally be independently adjustable using other motors.

[0011] The camera's focus may be controlled by the camera operator. This option is described in more detail below. Alternatively, the camera operator controls the camera's direction, and the focus puller is located away from the camera, providing a follow focus 6. Focus control deviceThe camera's focus may be controlled using a follow focus device. The follow focus device includes a knob 7 or other input device, which can provide an input for the focus puller to control the camera's focus.

[0012] The display control device 8 controls the still images or videos displayed on the screen 3. The display control device is linked to a data store 9 that stores the still image and / or video data to be displayed (which may be photographs or computer-generated in each case).

[0013] In this example, the display wall 3 is composed of multiple display screens 3a arranged in a row. Alternatively, the display wall 3 may be a screen onto which images are projected.

[0014] Conventionally, the focus control device 6 is directly connected to the camera 1 by a wired or wireless link. In the system of Figure 1, the follow focus communicates with the camera via a link device 10. The link device receives focus commands from the follow focus via link 11 and forwards at least some of those commands to both the camera 1 and the display control device 8 via links 12 and 13, respectively. Each of links 11-13 may independently be a wired or wireless link. In one convenient embodiment, the link device presents the follow focus with the same logical and / or physical interface presented by the camera 1. In one convenient embodiment, the link device presents the camera with the same logical and / or physical interface presented by the follow focus. One or both of these features may mean that the link device can be easily interposed between an existing camera and follow focus. Alternatively, the follow focus may be adapted to work in conjunction with the link device.

[0015] The link device 10 and the display control device 8 work together to provide a mechanism for controlling the mode of the image displayed on the camera 1 and the display wall 3. The link device 10 may include a processor 14 and a memory 15 that stores, in a non-temporary manner, code executable by the processor to cause the link device to perform the set of operations described herein. The display control device 8 may include a processor 16 and a memory 17 that stores, in a non-temporary manner, code executable by the processor 16 to cause the display control device to perform the set of operations described herein. The division of functions between the link device and the display control device can be selected in any convenient manner and can be shared with other devices not shown in Figure 1.

[0016] Next, we will explain the operation of the system shown in Figure 1.

[0017] Subject 2 performs between display wall 3 and camera 1. The camera is operated so that the subject is within its field of view. The camera then captures a video of the subject with the exterior of the display wall as the background.

[0018] In a simple example of focus control, it may be desirable for the captured video to be in focus on a subject. This can happen if the subject or the camera moves, but when the subject moves relative to the camera, the focus puller adjusts the follow focus 6. The follow focus sends a control signal to the camera's motor 5, causing the camera's lens 4 to adjust so that the camera maintains focus on the subject. These signals are transmitted to the camera unchanged by the link device 10. In a second example, it may be desirable for the focus of the video captured by the camera to appear to scan from the subject to an object displayed on the display wall 3. The focus puller adjusts the follow focus, causing it to send a signal that commands the camera to shift its focus from the subject to the object. The subject is positioned closer to the display wall 3 than the camera. The object is displayed on the display wall and appears to be further away from the camera than the display wall. The focus command signal from the follow focus is processed by the link device 10. If the focus command signal from the follow focus commands the camera to focus on the area from the camera to the display wall, the link device passes that signal to the camera. Motor 5 changes the camera's focus depending on the signal received by the camera. If the focus command signal from the follow focus instructs the camera to focus further away from the display wall, the link device does not pass that signal to the camera, and the display control device does not pass it. 8 The desired focus position is transmitted as a signal. Next, the display control device performs image processing operations on the image displayed on the display wall 3 to adapt its appearance depending on the desired focus position. For example, blurring may be applied to parts of the image other than those intended to appear at the distance from the camera corresponding to the desired focus position. Since the camera's focus is maintained on the display wall, the camera can accurately capture the appearance of the display wall.

[0019] If a series of commands is generated to move the camera's focus from a position beyond the display wall to a position between the display wall and the camera, the reverse can be done.

[0020] The same principle described above may be implemented in the following alternative ways. Camera 1 may be adapted to set a focus limit or upper limit for the distance at which the camera lens focuses. Link device 10 may determine this limit or upper limit depending on the distance between the camera and the display wall. Link device may also transmit this limit, which acts as an upper limit for camera motor 5, to camera 1 so that the camera lens focus can only be adjusted up to that upper limit or upper limit. For example, if the camera is located 10 m away from the display wall, the upper limit for the focal length of the camera lens is 10 m. The camera motor adjusts the focus of the lens according to the distance required by any command from follow focus, which is either the upper limit or less than the limit. If the follow focus command exceeds the limit, the camera lens focus stops when it reaches the 10 m limit and remains focused on 10 m on the display wall. Link device may also transmit the same command from follow focus to the display control device, which adjusts the screen accordingly, thereby focusing on the virtual object. In this configuration, the link device sends a follow focus command to the camera along with the determined limit. Assuming camera 1 is stationary and the display wall is still, the distance between the camera and the display wall remains constant, so it is not necessary to resend the limit with each new follow focus command. The limit sent to the camera by the link device depends on the camera's position, which can be calculated in any appropriate way. If the position of camera 1 changes, the limit may need to be adjusted accordingly.

[0021] If the focus is commanded to extend beyond the display wall, the display control device 8 may, in practice, select a range of virtual distances for display objects that should not be blurred. For example, if the focus is set to 20m, objects at virtual distances between 18m and 22m may not be blurred, while objects outside that range may be blurred. The depth of the non-blurred range may depend on the current depth of field of the lens 4, which may be transmitted as a signal to the link device 10.

[0022] To control the blurring of the display wall depending on a desired focus position, the display control device 8 can use information indicating the distance of an object from the camera within the display area. This distance information may be pre-stored along with the data in the data store 9. Alternatively, the distance information may be estimated by the display control device 8, for example, using a learned distance estimation algorithm. The blurring of the display wall by the display control device 8 may depend on information indicating the distance of camera 1 from the display wall. The position of camera 1 may be sensed by a position sensor 18 on the camera and transmitted as a signal to the display control device. The display control device may pre-store the position of the display wall 3 so that it can determine the distance of the camera from the display wall 3. Alternatively, since in many situations the distance of the camera from the display wall may be small compared to the virtual distance from the camera to the object shown on the display wall, satisfactory results may be obtained by assuming a distance from the camera to the display wall (e.g., 10m).

[0023] The camera position detected by the position sensor 18 may also be transmitted as a signal to the link device 10 and used in conjunction with the known position of the display wall 3 to determine the depth at which the camera's focus should be limited.

[0024] The position of the camera can be estimated in any convenient way. For example, there may be sensors on the camera that sense reference markers, beacons, or other features within the environment of the camera, and a processing circuit on or remote from the camera can use the data from those sensors to estimate the position of the camera. Alternatively, the camera may include markers or beacons sensed by one or more sensors within the environment, and the processing circuit could use the output of such sensors to estimate the position of the camera. The absolute position of the camera may be estimated. Alternatively, the relative position of the camera with respect to the screen may be estimated.

[0025] Depending on the direction of the principal axis of the camera with respect to the surface of the display wall, whether the display wall is flat (e.g., it may be a curved surface or a faceted surface), and the field of view of the camera, some portions of the display wall within the field of view of the camera may be at different distances from the camera. If the set focus of the camera is farther than the display wall, it is preferable for all those portions of the display wall to be in focus. For this purpose, the link device 10 may receive information indicating the depth of field of the camera from the camera. The information may directly indicate the depth of field, or may indicate the aperture setting of the lens 4 or the type of the lens 4, and the link device 10 may infer the depth of field from it using pre-stored information regarding the characteristics of the lens 4. The link device may receive the position and orientation of the camera 1 from the position sensor 18. The link device may receive the zoom setting of the lens 4 from the camera. The link device may perform geometric calculations and estimate the range of distances from the camera to those portions of the display wall within the field of view of the camera. Next, when it is intended that the focus of the camera is on the display wall, the link device may send a signal to the camera to adjust the depth of field of the camera (e.g., by expanding the depth of field of the camera and / or adjusting the aperture of the lens) to maintain a state where the entire display wall within its field of view is in focus.

[0026] When the display wall is pixelated and displayed, for example, when the display wall is composed of a plurality of display screens arranged side by side, due to the interaction between the apparent interval of the pixel grid on the display screen and the pixels on the image sensor of the camera, moiré patterns may be captured by the camera. The link device may be configured to mitigate such effects. The link device may be configured to estimate the focus configuration of the camera where such effects are likely to occur. The focus configuration may be estimated depending on information regarding the relative position between the camera and the display wall and the lens settings of the camera (which can all be obtained as described above), and information regarding the pixel intervals between the camera and the display wall (which may be provided to the link device in advance). Then, the link device can avoid the camera staying at a focus configuration where moiré is likely to occur. This can be done by instructing the camera to adopt a focus configuration that is longer or shorter than that configuration instead of instructing the camera to adopt such a configuration when the follow focus commands the camera to do so.

[0027] In the above-described embodiment, the follow focus is remote from the camera. Instead, the focus command may be provided by a focus control device integrated with the camera. The link device 10 may be integrated with the camera or a part thereof, such as a lens.

[0028] The devices 1, 10, 6, 8 may be connected by dedicated or shared interfaces on each device. For example, the output from the link device 10 to the display control device 8 may be configured by a dedicated physical interface on the link device 10 or by the link device transmitting a signal addressed to the display control device 8.

[0029] The link device 10 may forward all focus commands received from the follow focus 6 to the display control device, or it may forward such commands only when instructed to bring the camera's focus to or beyond the screen 3.

[0030] The embodiments described above may be convenient to implement because they may not require the adaptation of the focus control device and / or camera. The link device 10 can simply be interposed between the two. Alternatively, if the camera is adapted to receive a focus limit, the control of the camera's motor can be uppered by this limit.

[0031] The link device 10 can control the lens focus in several ways. Once the focus limit is determined, commands from the focus control device 6 to focus the camera at distances exceeding that limit may simply not be forwarded to the camera. Alternatively, such commands may be adapted by the link device 10 to focus the camera at that limit and then forwarded to the camera. Alternatively, the link device 10 may signal the limit to the camera, and the camera itself may then block focusing beyond that limit. The link device 10 may forward both the limit and the distance commanded by the focus puller, and the camera 1 may use the smaller of the two distances to focus. In each case, the camera function may be performed by any applicable part of the camera, including the lens.

[0032] The camera lens focus should ideally be adjustable in response to both new focus control device commands and the movement of the camera itself. As mentioned earlier, the camera's position in front of the display wall is monitored by the camera position sensor 18. There may be situations where the focus command from the focus control device remains fixed on a virtual object on the display wall while the camera moves around in front of the display wall. For example, the camera is 20m away from the display wall, and the focus puller is focused on an object 30m away, so the focus command extends 10m beyond the display wall, and the camera is set to focus on 20m. Even if the camera moves forward 5m, the focus puller's command is still to focus on a virtual image 10m away from the stationary display wall, so the camera may not receive an instruction to adjust the lens focus. In this case, since the camera has moved forward 5m, it will no longer be in focus on the display wall. For the camera's focus to remain on the display wall, the camera's position must be recognized so that when a change in the camera's position is detected, the camera lens can be adjusted to maintain focus on the display wall.

[0033] In the example described above, the screen is capable of adaptively displaying information, and this information is captureable by the camera during shooting. The information may be displayed by being emitted from a light source on the screen (as in the case of an LED wall), or by being projected onto a wall and reflected off the wall. Alternatively, the wall may be a screen suitable for chroma keying, where a portion of the image including the screen is replaced with other information. Such a chroma key screen may be a color separable from subject 2, for example, a so-called green screen or a so-called blue screen. The screen may also be of a single hue.

[0034] The applicant individually discloses the individual features described herein and any combination of two or more such features, but such features or combinations are disclosed only to the extent that they can be implemented in whole under this specification in light of common generalities. It is irrelevant whether such features or combinations of features solve the problems disclosed herein and does not limit the scope of the claims. The applicant has shown that aspects of the invention may consist of such individual features or combinations of features. In consideration of the foregoing, it will be apparent to those skilled in the art that various modifications can be made within the scope of the invention.

Claims

1. A focus control device for a video camera pointed at a screen, The focus control device is connected to a display control device that displays an image on the screen. The video camera has a lens that allows for focus adjustment over an area extending on both sides of the screen, The lens is configured to limit the focus to the region between the video camera and the screen, depending on the relative position between the video camera and the screen. The aforementioned screen is a display screen or a chroma key screen. When adjusting the focus of the lens within the range between the video camera and the screen, a signal is sent to the video camera to adjust the focus of the lens. When adjusting the focus of the lens outside the range between the video camera and the screen, if the video camera reaches the focus limit, instead of sending a signal to the video camera to adjust the focus of the lens, the display control device is sent a signal to apply blur to the portion of the image other than that intended to appear at the distance from the video camera corresponding to the desired focus position. Focus control device.

2. It is a system, The focus control device according to claim 1, A user input device for receiving focus control input from the user, Includes a display control device that displays an image on a screen, The focus control device, The lens is configured to adjust its focus in response to the user input device, and to limit the focus of the lens to the area between the video camera and the screen. The focus control device responds to the user input device, When the user input device commands the adjustment of the focus of the lens within the range between the video camera and the screen, it sends a signal to the video camera to adjust the focus of the lens. When the user input device commands the adjustment of the focus of the lens outside the range between the video camera and the screen, and the video camera reaches the focus limit, it sends a signal to the display control device to apply blur to the portion of the image other than the portion that is intended to appear at the distance from the video camera corresponding to the desired focus position. system.

3. A control device for a video camera, The control device is connected to a display control device that displays images on a screen. An input for receiving focus commands indicating the focal length, (i) a first signal for controlling the focus of the lens of the video camera, and (ii) one or more outputs for transmitting a second signal indicating the focal length indicated by the focus command, The system determines that the focus limit is restricted to the area between the video camera and the screen, and transmits the first signal instructing the video camera to adjust the focus of the lens to the focal length only if the focal length indicated by the focus command does not exceed the focus limit. If the focal length indicated by the focus command exceeds the focus limit, when the video camera reaches the focus limit, it does not transmit the first signal to the video camera, but transmits a signal to the display control device to apply blur to the portion of the image other than that intended to appear at the distance from the video camera corresponding to the desired focus position. A control device configured in such a way.

4. The control device according to claim 3, configured to transmit the second signal when the focal length indicated by the focus command exceeds the focus limit.

5. The control device according to claim 3, which is configured to transmit a signal to the video camera to adjust the focus of the lens to the distance indicated by the focus limit when the focal length indicated by the focus command exceeds the focus limit.

6. The control device according to claim 3 or 4, comprising a wireless data interface for receiving the focus command and transmitting the first signal.

7. The control device according to claim 6, wherein the focus command and the first signal are in the same format.

8. A video capture system including a control system implemented by one or more processors configured to execute pre-stored program code, wherein the program code is A step of receiving a focus command indicating the focal length, The steps include deciding to limit the focus limit to the area between the video camera and the screen, The steps include: transmitting a signal to a video camera having a lens, and adjusting the focus of the lens to the focal length, only if the focal length is less than or equal to the focus limit; If the focal length is greater than or equal to the focus limit, when the video camera reaches the focus limit, the video camera is not instructed to adjust the focus of the lens, but rather a blur is applied to the portion of the image other than the portion intended to appear at the distance from the video camera corresponding to the desired focus position to form a blurred image, and the blurred image is transmitted to the display screen. A video capture system is a program code that performs this task.

9. The video capture system according to claim 8, comprising the video camera and the display screen, wherein the display screen is a display background.

10. The step of applying blur to a portion of the aforementioned image is A video capture system according to claim 8 or 9, comprising estimating a portion of the image containing the object that is not at the focal length, depending on depth information representing the depth of the object in the image and the focal length, and applying blur to that portion of the image.

11. The step of applying blur to a portion of the aforementioned image is Receiving information representing the depth of field of the aforementioned video camera, Depending on the depth information, focal length, and depth of field, the portion of the image containing the object outside the depth of field is estimated for the focal length, and blur is applied to that portion of the image. The video capture system according to claim 10, including the following:

12. The step of applying blur to a portion of the aforementioned image is Receiving information representing the relative position between the video camera and the display screen, Depending on the depth information, focal length, depth of field, and relative position, the portion of the image containing the object outside the depth of field is estimated relative to the focal length, and blur is applied to that portion of the image. The video capture system according to claim 11, including the following:

13. A method for controlling a video camera and a display screen, Receiving a focus command for the aforementioned video camera, Deciding to limit the focus limit to the area between the video camera and the screen, If the focus command is to adopt the focal length between the video camera and the display screen, then causing the video camera to adopt the focal length, If the focus command involves adopting a focal length beyond the display screen, when the video camera reaches the focus limit, the video camera is instructed to adopt the focal length on the display screen instead of the aforementioned focal length, and blurring is applied to the portion of the image other than the one intended to appear at a distance from the video camera corresponding to the desired focus position of the image displayed on the display screen. A method that includes this.

14. The method according to claim 13, comprising selecting the portion of the image based on depth information representing the depth of an object included in the portion of the image.

15. A device configured to determine the focus limit of a video camera pointed at a screen, The device is connected to a display control device that displays images on the screen, The video camera has a lens that allows for focus adjustment over an area extending on both sides of the screen, The focus limit is determined to be limited to the area between the video camera and the screen, depending on the position of the video camera relative to the screen. Only if the focal length indicated by the focus command does not exceed the focus limit, a signal is sent to the video camera instructing it to adjust the focus of the lens to that focal length. If the focal length indicated by the focus command exceeds the focus limit, the device, upon reaching the focus limit, does not transmit a signal to the video camera to adjust the focus, but instead transmits a signal to the display control device to apply blur to portions of the image other than those intended to appear at a distance from the video camera corresponding to the desired focus position.

Images