Image blur control device, image blur control method, and imaging apparatus

Abstract

Image blur control device and image blur control method capable of appropriately controlling image blur correction in camera that captures video for detecting viewpoint of camera that captures VFX video SOLUTION: An image blur control device detects a position attitude of a first imaging device for imaging a subject with a video displayed on a display device as a background by using a video obtained from a second imaging device different from the first imaging device. The image blur control apparatus also generates a video to be displayed on a display device in accordance with the position and orientation. When the image blur correction of the first imaging device and the second imaging device is effective, the image blur control device controls the image blur correction of the second imaging device on the basis of the image blur correction of the first imaging device.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an image blur control device, an image blur control method, and an imaging device. [Background technology]

[0002] Conventionally, VFX (Visual Effects) images have been images synthesized by combining separately generated live-action images and background images. In recent years, a method (in-camera VFX) has become known in which a VFX image is directly captured by a camera by displaying a background image corresponding to the viewpoint of the camera capturing the real subject in real time on a large display device placed behind the real subject (Patent Document 1). In-camera VFX eliminates the need for a post-process to composite background images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7190594 Summary of the Invention [Problem to be solved by the invention]

[0004] To create VFX images, it is necessary to detect the viewpoint (position and orientation) of the camera used for capturing images (called the VFX camera).One method for detecting the viewpoint position of the VFX camera is to use images captured by a separate camera (called a viewpoint detection camera) attached to the VFX camera.

[0005] Conventionally, VFX cameras have been fixed to large tripods or cranes installed in the studio, so there was no need to consider image blur caused by camera movement for either the VFX camera or the viewpoint detection camera.

[0006] However, to increase the degree of freedom of the VFX camera's viewpoint, it is conceivable that the VFX camera will be held by a cameraman. In this case, it is desirable to consider image blur caused by body movement in the images from the VFX camera and the viewpoint detection camera. However, no technology is known for appropriately correcting image blur, especially for the images from the viewpoint detection camera.

[0007] In consideration of these problems, one aspect of the present invention provides an image blur control device and an image blur control method that can appropriately control image blur correction in a camera that captures video for detecting the viewpoint of a camera that captures VFX video. [Means for solving the problem]

[0008] In one aspect, the present invention provides an image blur control device comprising: a detection means for detecting the position and orientation of a first imaging device that captures an image of a subject against a background of an image displayed on a display device, using an image obtained from a second imaging device different from the first imaging device; a generation means for generating an image to be displayed on the display device according to the position and orientation; and a control means for controlling the image blur correction of the second imaging device based on the image blur correction of the first imaging device, when image blur correction of the first imaging device and the second imaging device is effective. [Effects of the Invention]

[0009] According to one aspect of the present invention, it is possible to provide an image blur control device and an image blur control method that can appropriately control image blur correction in a camera that captures video for detecting the viewpoint of a camera that captures VFX video. [Brief explanation of the drawings]

[0010] [Figure 1] Schematic diagram of a virtual studio system according to an embodiment. [Figure 2] Figure 1 shows an example of the camera's functional configuration and a block diagram showing the connections between the various devices. [Figure 3] A block diagram showing an example of the functional configuration of a viewpoint detection device. [Figure 4]A diagram for explaining coordinate transformation between the imaging camera and the viewpoint detection camera. [Figure 5] Block diagram showing an example of the functional configuration of a scene control device [Figure 6] Flowchart for image blur correction operation [Figure 7] Flowchart for viewpoint detection operation DETAILED DESCRIPTION OF THE INVENTION

[0011] ●(First embodiment) (Virtual Studio System Overview) FIG. 1 is a schematic diagram of a virtual studio system according to an embodiment. The virtual studio is a studio capable of capturing in-camera VFX video, equipped with a large display device that displays background video and equipment for detecting the position and orientation of the camera. The virtual studio and other equipment required for capturing in-camera VFX video (such as imaging equipment such as cameras and equipment for generating background video) are collectively referred to as a virtual studio system. In this specification, "video" refers to video, a signal representing video, or data representing video.

[0012] Large display devices 310 and 320, also known as LED walls, are provided on the walls of virtual studio 100. Display devices 310 and 320 may be movable. Here, the imaging range 311 of imaging camera 200 that captures the in-camera VFX video is within the display screen of display device 310, and the background video is displayed on display device 310. On the other hand, display device 320 is not captured by imaging camera 200, and is used to display an image to be reflected in real subject 400. Display device 320 is not essential.

[0013] Viewpoint detection markers 131 are provided on the ceiling of the virtual studio 100 to be captured by the viewpoint detection camera 205. The viewpoint detection markers 131 are, for example, a plurality of markers arranged in a specific pattern. The three-dimensional coordinates (position in the world coordinate system) of each marker are known.

[0014] The position and orientation of the imaging camera 200 is detected from the image coordinates of the markers and the three-dimensional coordinates of the markers that appear in an image captured in the direction of the ceiling from the viewpoint detection camera 205. The position and orientation of the imaging camera 200 is detected as a combination of the three-dimensional coordinates and rotation angles around three axes (tilt, pan, and roll).

[0015] The coordinate system of the imaging camera 200 and the viewpoint detection camera 205 is an orthogonal coordinate system with the intersection of the optical axis and the imaging surface as the origin, an axis extending from the optical axis, and two axes perpendicular to the optical axis. The two axes perpendicular to the optical axis are parallel to two sides of the imaging surface that do not face each other.

[0016] Here, the state in which the optical axis of the imaging camera 200 is perpendicular to the direction of gravity and the rotation angle around the optical axis of the imaging camera 200 is 0 degrees is defined as the reference attitude of the imaging camera 200. The viewpoint detection camera 205 is attached to a rig or the like that holds the imaging camera 200 so that the optical axis of the viewpoint detection camera 205 points vertically upward when the imaging camera 200 is in the reference attitude.

[0017] Here, the position and orientation of the imaging camera 200 is detected from an image of a marker whose three-dimensional coordinates are known. However, the position and orientation of the imaging camera 200 may be detected by any other known method, such as a method that does not use a marker.

[0018] Capturing camera 200 records the in-camera VFX video by capturing an image of real subject 400 against the background of the image displayed on display device 310. Capturing range 311 is an example of the range captured by capturing camera 200 within the background video displayed on display device 310.

[0019] The viewpoint detecting device 130 detects the viewpoint (position and orientation) of the imaging camera 200 from the coordinates of a viewpoint detecting marker 131 provided on the ceiling and the position of the viewpoint detecting marker 131 in the image acquired by the viewpoint detecting camera 205. The viewpoint detecting device 130 assumes that the three-dimensional positional deviation between the origin of the coordinate system of the imaging camera 200 and the origin of the coordinate system of the viewpoint detecting camera 205 in the coordinate system of the virtual studio 100 is known through a calibration operation or the like. Therefore, the viewpoint detecting device 130 can detect the viewpoint of the imaging camera 200 using the image acquired from the viewpoint detecting camera 205.

[0020] As will be described later, when image blur correction is enabled in both the imaging camera 200 and the viewpoint detection camera 205, the viewpoint detection device 130 controls the image blur correction of the viewpoint detection camera 205 based on image blur correction information in the imaging camera 200. The viewpoint detection device 130 also detects a viewpoint that reflects the image blur correction of the imaging camera 200 and the viewpoint detection camera 205. The viewpoint detection device 130 continuously detects the viewpoint of the imaging camera 200. The viewpoint detection device 130 also continuously supplies information about the detected viewpoint to the scene control device 110. Details of the operation of the viewpoint detection device 130 will be described later.

[0021] The scene control device 110 renders a preset three-dimensional model of a virtual space from the viewpoint of the imaging camera 200 detected by the viewpoint detection device 130, and generates a virtual space image at a predetermined frame rate. The scene control device 110 generates a background image to be displayed on the display device 310 based on the virtual space image. When generating the background image from the virtual space image, the scene control device 110 applies coordinate transformation (deformation processing) to the background image according to the angle between the imaging direction (optical axis direction) of the imaging camera 200 and the display surface of the display device 310. The scene control device 110 supplies the generated background image to the display control device 120. The scene control device 110 also supplies an image to be displayed on the display device 320 to the display control device 120. The image to be displayed on the display device 320 may be a predetermined still image, or may be an image composed of frames having the same content as the predetermined still image.

[0022] Display control device 120 causes display device 310 to display a background image at a timing that allows imaging camera 200 to appropriately capture the background image. Display control device 120 also causes display device 320 to display an image at the same timing as display device 310. Display control device 120 controls the supply of images according to the configurations of display devices 310 and 320. For example, if display device 310 and / or 320 is configured with multiple independent display panels, display control device 120 divides the image into areas to be displayed on each display panel. Then, display control device 120 supplies each display panel with the image of the area to be displayed on that display panel.

[0023] Furthermore, a synchronization signal is supplied from a reference clock generating device, also called a sync generator, to the imaging camera 200, viewpoint detection device 130, scene control device 110, and display control device 120. Each device controls its operation timing in accordance with the reference clock, thereby achieving synchronization between the imaging period of the imaging camera 200 and the display periods of the display devices 310 and 320. The technology for synchronizing the operations of devices based on a reference clock is well known, for example, as genlock (generator locking), and therefore a detailed description thereof will be omitted.

[0024] For convenience, this specification assumes that the real subject 400 is a human subject, but there is no limit to the type and number of real subjects.

[0025] Fig. 2 is a block diagram showing the connection relationships of the devices shown in Fig. 1, and an example of the functional configuration of the imaging camera 200 and the viewpoint detection camera 205. Unless otherwise specified below, the first imaging section 220 and the second imaging section 270 capture moving images at a predetermined frame rate (for example, 30 frames per second).

[0026] The first control unit 240 has a processor (CPU, MPU, microprocessor, etc.) capable of executing a program, a ROM, and a RAM. The first control unit 240 loads a program stored in the ROM into the RAM and executes it, thereby controlling the operation of each functional block of the imaging camera 200 and realizing the operation of the imaging camera 200 described below. Although not shown in the figure, the first control unit 240 is connected to each functional block of the imaging camera 200 so as to be able to communicate with them.

[0027] The first optical system 210 is an optical system for capturing in-camera VFX video. If the first optical system 210 has an image blur correction function, the first optical system 210 has an image blur correction lens (shift lens), a shift lens movement mechanism, and a position detection mechanism. The shift lens movement mechanism is capable of moving the shift lens in a direction perpendicular to the optical axis of the first optical system 210. The shift lens position detection mechanism detects the position of the shift lens or the amount of movement from a reference position. The angle of view of the first optical system 210 may be variable. This embodiment assumes a situation where there are few physical constraints on the capturing direction of the capturing camera 200, such as handheld capturing, and the capturing range 311 of the first optical system 210 does not exceed the display range of the display device 310.

[0028] The first imaging unit 220 has an imaging element and converts the optical image formed by the first optical system 210 into an analog image signal. The imaging element of the first imaging unit 220 may be, for example, a known CCD or CMOS color image sensor with a primary color Bayer array color filter. If the first imaging unit 220 has an image stabilization function, the first imaging unit 220 has an imaging element movement mechanism and a position detection mechanism. The imaging element movement mechanism is capable of moving the imaging element in a direction perpendicular to the optical axis of the first optical system 210. The imaging element movement mechanism may also be capable of rotating the imaging element around the optical axis of the first optical system 210. The imaging element position detection mechanism detects the position of the imaging element or the amount of movement from a reference position. The analog image signal output by the first imaging unit 220 is supplied to the first image processing unit 230.

[0029] The first image processing unit 230 applies predetermined image processing to the analog image signal output by the first imaging unit 220 to generate signals and image data according to the intended use, and acquires and / or generates various types of information. The first image processing unit 230 may be a dedicated hardware circuit such as an ASIC (Application Specific Integrated Circuit) designed to realize a specific function. Alternatively, the first image processing unit 230 may be configured to realize a specific function by a processor such as a DSP (Digital Signal Processor) or GPU (Graphics Processing Unit) executing software.

[0030] The image processing applied by the first image processing unit 230 can include, for example, pre-processing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Pre-processing may include A / D conversion, signal amplification, reference level adjustment, defective pixel correction, and the like. Color interpolation, also known as demosaicing, is performed when the image sensor is equipped with a color filter, and is a process of interpolating the values ​​of color components that are not included in the individual pixel data that make up the image data. The correction processing may include white balance adjustment, tone correction, correction of image degradation caused by optical aberration of the first optical system 210 (image restoration), correction of the effects of peripheral light falloff of the first optical system 210, color correction, and the like. The detection process can include detection of a characteristic region or a region of a specific subject (for example, a face region or a human body region) and its movement, and person recognition processing. Data processing can include processes such as area extraction (trimming), compositing, scaling, encoding and decoding, header information generation (data file generation), etc. Data processing also includes the generation of image data for display or image data for recording. The evaluation value calculation process may include processes such as generating a signal or evaluation value used for autofocus detection (AF) and generating an evaluation value used for automatic exposure control (AE). The evaluation value used for AE is information about the luminance of the captured scene, but the information about the luminance of which part of the captured scene may differ depending on, for example, the exposure mode that is set. For example, the information may reflect the luminance of the entire captured scene, or it may be information about the luminance of an area of ​​a specific subject. Special effect processing may include adding a blur effect, changing color tones, relighting, etc. Special effect processing also includes processing that reflects the influence of a light source, which will be described later.

[0031] Note that these are examples of processes that the first image processing unit 230 can apply, and do not limit the processes that the first image processing unit 230 applies. The first image processing unit 230 outputs acquired or generated information and data to a functional block according to the application. For example, the first image processing unit 230 outputs image data for recording to the recording unit 250, and outputs information related to the luminance of the captured scene to the first control unit 240. The first image processing unit 230 also outputs video data for display to the scene control device 110.

[0032] The first control unit 240 executes AE processing to determine exposure conditions based on information about the luminance of the imaging scene acquired from the first image processing unit 230, and can control the operation of the first imaging unit 220 in accordance with the determined exposure conditions. The first control unit 240 can, for example, determine exposure conditions such that the entire imaging scene is properly exposed, or determine exposure conditions such that a portion of the imaging scene (for example, the area of ​​the real subject) is properly exposed.

[0033] Generally, exposure conditions are determined by a combination of aperture value, shutter speed (exposure time), and imaging sensitivity. Therefore, the first control unit 240 can determine a combination of these three parameter values ​​as imaging conditions for obtaining proper exposure. However, if the aperture value or exposure time is changed during video capture, the depth of field may change, or the movement distance of a moving subject between frames may change. Therefore, in the AE processing of this embodiment, the imaging sensitivity is determined without basically changing the aperture value or shutter speed.

[0034] The focal length of the first optical system 210 can be automatically adjusted by the first control unit 240 performing AF processing based on the evaluation value generated by the first image processing unit 230.

[0035] The first motion detection device 235 is a sensor that detects the motion of the imaging camera 200. The first motion detection device 235 may be, for example, a known six-axis sensor that detects acceleration in each axis direction and angular velocity around each axis of the coordinate system of the imaging camera 200. The first motion detection device 235 supplies the detection values ​​to the first control unit 240 at a predetermined cycle.

[0036] In this embodiment, the imaging camera 200 has an optical and / or electronic image blur correction function. The optical image blur correction function is realized by the first control unit 240 controlling the shift lens of the first optical system 210 and / or the movement mechanism of the imaging element of the first imaging unit 220 based on the movement detected by the first motion detection device 235. The electronic image blur correction function is realized by the first control unit 240 controlling the position at which a frame image obtained by the first imaging unit 220 is cut out based on the movement detected by the first motion detection device 235.

[0037] The first control unit 240 calculates an amount of image blur correction according to the type of image blur correction method, the type of image blur correction member (shift lens and / or image sensor), etc., based on the movement detected by the first movement detection device 235. Then, the first control unit 240 controls the position of the image blur correction member and the clipping position of the frame image according to the calculated amount of image blur correction.

[0038] The second control unit 290 has a processor (CPU, MPU, microprocessor, etc.) capable of executing a program, a ROM, and a RAM. The second control unit 290 loads a program stored in the ROM into the RAM and executes it, thereby controlling the operation of each functional block of the viewpoint detection camera 205 and realizing the operation of the viewpoint detection camera 205. Although not shown in the figure, the second control unit 290 is connected to each functional block of the viewpoint detection camera 205 so as to be able to communicate with them.

[0039] In this embodiment, a viewpoint detection marker 131 is placed on the ceiling of the virtual studio 100. Therefore, the viewpoint detection camera 205 is attached to the imaging camera 200 so that the optical axis of the second optical system 260 faces vertically upward when the imaging camera 200 is in the reference attitude. The angle of view of the second optical system may be fixed or variable.

[0040] The second imaging unit 270 has an imaging element and converts the optical image formed by the second optical system 260 into an analog image signal. The imaging element may be a monochrome image sensor, as it is sufficient for the image for viewpoint detection to be able to acquire the image coordinates of the viewpoint detection marker 131. Furthermore, the number of pixels of the imaging element of the second imaging unit 270 may be less than the number of pixels of the imaging element of the first imaging unit 220.

[0041] The second image processing unit 280 generates a viewpoint detection image by applying image processing to the analog image signal output by the second imaging unit 270. The second image processing unit 280 may have the same functions as the first image processing unit 230, or may have only the functions of the first image processing unit 230 that are necessary for generating the viewpoint detection image. The viewpoint detection image may be a display image similar to that generated by the first image processing unit 230. When the second imaging unit 270 outputs a monochrome image, the second image processing unit may generate the viewpoint detection image by A / D converting the monochrome image. The second image processing unit 280 outputs the viewpoint detection image to the viewpoint detection device 130.

[0042] The second motion detection device 285 is a sensor that detects the motion of the viewpoint detection camera 205. The second motion detection device 285 may be, for example, a known six-axis sensor that detects acceleration in each axis direction and angular velocity around each axis of the coordinate system of the viewpoint detection camera 205. The second motion detection device 285 supplies the detection values ​​to the second control unit 290 at a predetermined cycle. Note that if an external device (for example, the viewpoint detection device 130 (control unit 1301)) determines the drive amount of the image blur correction member of the viewpoint detection camera 205, the second motion detection device 285 may not be necessary.

[0043] In this embodiment, the viewpoint detection camera 205 has an optical and / or electronic image blur correction function. The optical image blur correction function is realized by the second control unit 290 controlling the movement mechanism of the shift lens of the second optical system 260 and / or the image sensor of the second imaging unit 270 based on the movement detected by the second motion detection device 285. The electronic image blur correction function is realized by the second control unit 290 controlling the position at which a frame image obtained by the second imaging unit 270 is cut out based on the movement detected by the second motion detection device 285.

[0044] The second control unit 290 may control the movement mechanism of the shift lens of the second optical system 260 and / or the image sensor of the second imaging unit 270 based on a drive amount supplied from an external device. In this case, the second control unit 290 does not need to use the movement detected by the second movement detection unit 285.

[0045] The second control unit 290 calculates an amount of image blur correction according to the type of image blur correction member (shift lens and / or image sensor), etc., based on the image blur control information supplied from the viewpoint detection device 130 and the motion detected by the second motion detection device 285. Then, the second control unit 290 controls the position of the image blur correction member and the clipping position of the frame image according to the calculated amount of image blur correction.

[0046] The second control unit 290 controls one or more of the drive range (maximum movement amount), drive direction and drive amount, drive cycle, and drive timing of the image blur correction member in accordance with the image blur control information supplied from the viewpoint detection device 130. The second control unit 290 also controls the cut-out position of the frame image in the same way as it controls the drive of the image blur correction member.

[0047] 3 is a block diagram showing an example of the functional configuration of the viewpoint detecting device 130. The viewpoint detecting device 130 can be realized using, for example, a computer device. The control unit 1301 is, for example, a CPU, and realizes the functions of the viewpoint detection device 130 by loading one or more application programs stored in a ROM 1308 into a RAM 1309 and executing them. The control unit 1301 controls the operation timing of the viewpoint detection device 130 in accordance with a synchronization signal supplied from a reference clock generation device.

[0048] The image processing circuit 1302 is, for example, a graphics board equipped with a GPU. The image processing circuit 1302 is capable of high-speed image processing, such as processing for detecting the image coordinates of the viewpoint detection marker 113 from the viewpoint detection video supplied from the viewpoint detection camera 205.

[0049] The first I / F 1303 to the third I / F 1305 are communication interfaces for connecting external devices. In this embodiment, the imaging camera 200 is connected to the first I / F 1303, the viewpoint detection camera 205 is connected to the second I / F 1304, and the scene control device 110 is connected to the third I / F 1305. It should be noted that the first I / F 1303 to the third I / F 1305 comply with standards according to the external devices to be connected and the types of signals to be communicated. For convenience, the illustration shows the viewpoint detection device 130 and the external device connected via one I / F, but they may be connected using multiple I / Fs. It should be noted that the viewpoint detection device 130 may have four or more communication interfaces with external devices.

[0050] The control unit 1301 acquires one or more of motion data, an image blur correction amount, and a position of an image blur correction member as image blur correction information from the imaging camera 200 via a first I / F 1303. The control unit 1301 also acquires a viewpoint detection image from the viewpoint detection camera 205 via a second I / F 1304. The control unit 1301 outputs the viewpoint (position and orientation) of the imaging camera 200 to the scene control device 110 via a third I / F 1304.

[0051] The ROM 1309 is, for example, an electrically rewritable nonvolatile memory. The ROM 1309 stores part of the program executed by the control unit 1301 (BIOS, bootstrap loader, firmware), setting values ​​of the viewpoint detection device 130, etc. The deviation between the origin of the coordinate system of the imaging camera 200 and the origin of the coordinate system of the viewpoint detection camera 205 (the difference in three-dimensional coordinates in the coordinate system of the virtual studio 100) is also stored in the ROM 1309. For example, when the viewpoint detection camera 205 is installed, the control unit 1301 executes a calibration application to detect the deviation of the origin of the coordinate system and stores it in the ROM 1309. The three-dimensional position of the viewpoint detection marker 113 (the three-dimensional coordinates in the coordinate system of the virtual studio 100) is also stored in the ROM 1309.

[0052] The RAM 1310 is used as the main memory for the control unit 1301 , as well as a working memory for the image processing circuit 1302 and a video memory for the display unit 1312 .

[0053] The storage unit 1311 is a large-capacity storage device such as a hard disk or SSD. The storage unit 1311 stores operating system (OS), application programs, user data, etc. The application used for detecting the viewpoint position, as exemplified below, is stored in the storage unit 1311. An application that calculates the viewpoint (position and orientation) of the imaging camera 200 from the viewpoint detection image supplied from the viewpoint detection camera 205 and the three-dimensional position of the viewpoint detection marker 113 stored in the ROM 1309. A program for controlling the operation of the viewpoint detection camera 205 Calibration application for the gaze detection device 130 Position registration application for viewpoint detection marker 113 It should be noted that these are merely examples, and not all of them are necessarily required, and other applications may also be stored.

[0054] The control unit 1301 executes necessary applications using the image processing circuit 1302 as necessary, detects the viewpoint of the imaging camera 200 for each frame of the viewpoint detection video, for example, and supplies it to the scene control device 110. As described above, the viewpoint of the imaging camera 200 is detected as a combination of the three-dimensional coordinates of the origin of the coordinate system of the imaging camera 200 and the rotation angle around each axis (tilt, pan, roll) of the coordinate system of the imaging camera 200.

[0055] The display unit 1312 is, for example, a liquid crystal display device. The display unit 1312 may also be a touch display. The display unit 1312 displays an application used for detecting the viewpoint position, a GUI provided by the OS, and the like.

[0056] The operation unit 1313 has a plurality of input devices that can be operated by the user, such as a keyboard, a mouse, a touch pad, etc. If the display unit 1312 is a touch display, the operation unit 1313 includes a touch panel.

[0057] In this embodiment, the control unit 1301 controls the image blur correction in the viewpoint detection camera 205 based on the image blur correction information acquired from the imaging camera 200 .

[0058] 4 is a diagram showing a schematic relationship between the coordinate system of the imaging camera 200 and the coordinate system of the viewpoint detection camera 205. The bottom of FIG. 4 also shows, for each coordinate axis of the coordinate system of the imaging camera 200, which coordinate axis of the coordinate system of the viewpoint detection camera 205 the movement along that coordinate axis corresponds to.

[0059] Here, a state in which the imaging camera 200 is in the reference attitude is shown. For ease of explanation and understanding, the coordinate system of the imaging camera 200 and the coordinate system of the viewpoint detection camera 205 have the same axis directions, and the yaw axis of the imaging camera 200 and the roll axis of the viewpoint detection camera 205 are shown to coincide. There are no particular limitations on the axis directions and positional relationship of the origins of the coordinate system of the imaging camera 200 and the coordinate system of the viewpoint detection camera 205, as long as they are known and fixed.

[0060] Reference numeral 220 schematically indicates the imaging element of the first imaging unit, and 270 the imaging element of the second imaging unit. Furthermore, "up," "down," "left," and "right" indicate the top, bottom, left, and right directions of the imaging elements of the imaging camera 200 and the viewpoint detection camera 205 when the imaging camera 200 is in the reference attitude.

[0061] The coordinate system of the imaging camera 200 and the coordinate system of the viewpoint detection camera 205 can be converted to each other using known coordinate conversion techniques. Specifically, a coordinate conversion matrix that converts from the coordinate system of the imaging camera 200 to the coordinate system of the viewpoint detection camera 205 is generated in advance based on the relationship between known coordinate systems. This makes it possible to convert, for example, the drive vector of the image blur correction member represented by the image blur correction amount in the imaging camera 200 into the drive vector of the image blur correction member in the coordinate system of the viewpoint detection camera 205 by applying the coordinate conversion matrix.

[0062] In this embodiment, the viewpoint detecting device 130 (control unit 1301 ) controls the image blur correction of the viewpoint detecting camera 205 based on the image blur correction information of the imaging camera 200 . The viewpoint detecting device 130 can perform one or more of the following controls so that the image blur correction of the viewpoint detecting camera 205 has the same capability as the image blur correction of the imaging camera 200, for example. The image blur correction amount in each direction of the viewpoint detection camera 205 is controlled to be equal to the image blur correction amount in each direction of the imaging camera 200. The image blur correction range in each direction of the viewpoint detection camera 205 is controlled so as not to exceed the image blur correction range in each direction of the imaging camera 200. For movements in a direction in which the imaging camera 200 cannot perform image blur correction, the viewpoint detection camera 205 is also controlled not to perform image blur correction. The cycle of image blur correction in the viewpoint detection camera 205 is controlled to be equal to the cycle of image blur correction in the imaging camera 200. The start timing of image blur correction in the viewpoint detection camera 205 is controlled to be synchronized with the start timing of image blur correction in the imaging camera 200.

[0063] Note that these are merely examples, and other control may be performed. Furthermore, the type of control to be performed may be set by, for example, the user in the viewpoint detection device 130. By performing such control, it is possible to reduce the unnaturalness of the in-camera VFX video caused by a mismatch between the movement of the viewpoint detected based on the viewpoint detection video and the movement of the video of the real subject.

[0064] In these controls, image blur correction information (correction amount, drive vector, or position of the image blur correction member) of the imaging camera 200 is converted into the coordinate system of the viewpoint detection camera 205 and then used for image blur correction control of the viewpoint position detection camera 205. When the start timing of image blur correction is synchronized, it is assumed that the imaging camera 200 and the viewpoint detection camera 205 operate using a common clock.

[0065] The control unit 1301 generates image blur control information indicating the control content, for example, each time it receives image blur control information from the imaging camera 200. Then, each time it generates image blur control information, the control unit 1301 transmits the image blur control information to the viewpoint detecting camera 205 (second control unit 290) via the second I / F 1304. The viewpoint detecting camera 205 (second control unit 290) corrects the amount of image blur correction calculated based on the movement detected by the second motion detection device 285, for example, based on the image blur control information. Then, the second control unit 290 drives the image blur correction member based on the corrected amount of image blur correction.

[0066] The control unit 1301 can know the movable range of the image blur correction member based on information about the image blur correction function and capability of the viewpoint detection camera 205. Therefore, the control unit 1301 can calculate the image blur correction amount to be used in the viewpoint detection camera 205. In this way, when the control unit 1301 includes the image blur correction amount (the drive amount and drive direction of the image blur correction member) in the image blur control information, the second control unit 290 may drive the image blur correction member in accordance with the image blur control information.

[0067] Note that when controlling static items that do not change during shooting, such as the image blur compensation range and image blur compensation cycle, the control unit 1301 only needs to transmit the image blur control information once to the viewpoint detecting camera 205. For example, the control unit 1301 generates static image blur control information based on information about the image blur compensation function and capacity of the imaging camera 200, and transmits it to the viewpoint detecting camera 205. The viewpoint detecting camera 205 (second control unit 290) sets an upper limit for the image blur compensation amount to be calculated in accordance with the compensation range indicated in the image blur control information. The second control unit 290 then calculates the image blur compensation amount based on the movement detected by the second motion detection device 285 and the set upper limit, and drives the image blur compensation member. Furthermore, if the image blur compensation cycle is specified in the image blur control information, the control unit 1301 calculates the image blur compensation amount at the specified cycle.

[0068] The control unit 1301 may acquire information about the functions and capabilities of the image blur correction of the imaging camera 200 and the viewpoint detection camera 205, for example, at the time of initial setup, and store the information in, for example, the ROM 1309. In this case, the control unit 1301 can generate static image blur control information without acquiring image blur correction information from the imaging camera 200.

[0069] Note that optical image blur correction that drives the image sensor is assumed here. However, in the following explanation, the drive amount of the image sensor can be replaced with the movement amount of the cropping position in the case of electronic image blur correction, or with the drive amount of the image blur correction lens in the case of optical image blur correction that drives the image blur correction lens.

[0070] 5 is a block diagram showing an example of the functional configuration of the scene control device 110. The scene control device 110 can be realized using, for example, a computer device. The control unit 1101 is, for example, a CPU, and realizes the functions of the scene control device 110 by loading one or more application programs stored in the ROM 1108 into the RAM 1110 and executing them. The control unit 1101 controls the operation timing of the scene control device 110 in accordance with a synchronization signal supplied from a reference clock generation device.

[0071] The image processing circuit 1102 is, for example, a graphics board equipped with a GPU. The image processing circuit 1102 is capable of high-speed image processing such as CG rendering using a 3D model and the viewpoint of a virtual camera. For example, the image processing circuit 1102 can generate one frame of background video in a time that is shorter than the frame cycle of the in-camera VFX video.

[0072] The first I / F 1103 to the third I / F 1105 are communication interfaces for connecting external devices. In this embodiment, the imaging camera 200 is connected to the first I / F 1103, the display control device 120 is connected to the second I / F 1104, and the viewpoint detection device 130 is connected to the third I / F 1105. It should be noted that the first I / F 1103 to the third I / F 1105 comply with standards according to the external devices to be connected and the types of signals to be communicated. For convenience, the scene control device 110 and the external device are shown connected via one I / F, but they may be connected using multiple I / Fs. It should be noted that the scene control device 110 may have four or more communication interfaces with external devices.

[0073] The control unit 1101 acquires captured image data and information about the brightness of the captured scene from the capturing camera 200 via the first I / F 1103. The control unit 1101 also acquires information about the viewpoint of the capturing camera 200 from the viewpoint detection device 130 by communication via the third I / F 1105. The control unit 1101 outputs image data for display (background video data) to the display control device via the second I / F 1104.

[0074] The ROM 1109 is, for example, an electrically rewritable nonvolatile memory, and stores some of the programs executed by the control unit 1101 (BIOS, bootstrap loader, firmware), setting values ​​of the scene control device 110, and the like.

[0075] The RAM 1110 is used as the main memory for the control unit 1101 , as well as a working memory for the image processing circuit 1102 and a video memory for the display unit 1112 .

[0076] The storage unit 1111 is a large-capacity storage device such as a hard disk or SSD. The storage unit 1111 stores operating system (OS), application programs, user data, etc. The storage unit 1111 also stores an application program (e.g., a game engine application) that generates a background image according to the viewpoint of the imaging camera 200, and data required to generate the background image (e.g., a 3D model of the virtual space, textures, etc.).

[0077] The display unit 1112 is, for example, a liquid crystal display device. The display unit 1112 may also be a touch display. The display unit 1112 displays a scene control application, a background image generation application (for example, a game engine application), a GUI provided by the OS, and the like.

[0078] The operation unit 1113 has a plurality of input devices that can be operated by the user, such as a keyboard, a mouse, a touch pad, etc. If the display unit 1112 is a touch display, the operation unit 1113 includes a touch panel.

[0079] Next, the image blur correction operation by the first control unit 240 will be described using the flowchart shown in Fig. 6. The image blur correction operation is performed by the first control unit 240 executing a program. The image blur correction operation is performed when image blur correction is enabled in the imaging camera 200. The user may be able to set whether image blur correction is enabled or disabled, or the imaging camera 200 (first control unit 240) may automatically determine this in accordance with one or more predetermined conditions.

[0080] Here, it is assumed that the first motion detection device 235 is continuously supplying the first control unit 240 with detection values ​​indicating the movement of the imaging camera 200. It is also assumed that the imaging camera 200 is currently capturing video. Furthermore, when optical image stabilization is performed, it is assumed that the first control unit 240 is continuously acquiring position information of the image stabilization members (shift lens and imaging element). It is to be noted that optical image stabilization and electronic image stabilization are not exclusive and may be used together.

[0081] In S501, the first control unit 240 acquires a motion detection value supplied from the first motion detection device 235. The first control unit 240 stores the acquired detection value in RAM. The RAM stores detection values ​​for the most recent predetermined period.

[0082] In S503, the first control unit 240 calculates the amount of image blur correction using the detection value acquired in S501. The first control unit 240 calculates the amount of image blur correction according to the type of image blur correction (optical and / or electronic), and in the case of an optical type, the image blur correction member used (one or both of the shift lens and the image sensor). The first control unit 240 can calculate the amount of image blur correction using any known method. Therefore, a detailed explanation of the calculation of the amount of image blur correction will be omitted.

[0083] When optical image blur correction is performed, the image blur correction amount includes, for example, a combination of the movement direction and movement amount of the image blur correction member. When electronic image blur correction is performed, the image blur correction amount includes, for example, the image coordinates (absolute coordinates or amounts of change) of multiple vertices that specify the clipping position of the frame image.

[0084] In S505, the first control unit 240 performs image blur correction based on the amount of image blur correction calculated in S503. That is, the first control unit 240 moves the image blur correction member and / or the clipping position of the frame image based on the amount of image blur correction.

[0085] In S507, the first control unit 240 calculates the actual movement amount of the image blur correction member moved in S505. After issuing an instruction to move the image blur correction member in S505, the first control unit 240 obtains the actual movement amount of the image blur correction member via the position detection mechanism. Note that if only electronic image blur correction is performed in S505, S507 may be skipped.

[0086] In S509, the first control unit 240 outputs the image blur correction information to the viewpoint detection device 130. The image blur correction information is The detected value of the movement of the imaging camera 200 acquired in S501, The image stabilization amount calculated in S503, The position of the image stabilization member obtained in S507, From the viewpoint of improving the viewpoint detection accuracy of the imaging camera 200 in the viewpoint detection device 130, the position of the image blur correction member is most preferable, followed by the amount of image blur correction. This is because the position of the correction member is closest to the actual amount of image blur correction.

[0087] In addition, when there are multiple image blur correction amounts, such as when optical and electronic image blur correction are used in combination, or when optical image blur correction is performed using a shift lens and an image sensor in combination, the image blur correction amount output in S509 is a composite value of the multiple image blur correction amounts.

[0088] In S511, the first control unit 240 determines whether or not to end image capture by the imaging camera 200. If it is determined that image capture should be ended, the first control unit 240 ends the image blur correction operation shown in Fig. 6, and if not, the first control unit 240 repeats the processing from S501.

[0089] Next, the operation of the viewpoint detection device 130 will be described using the flowchart shown in Fig. 7. The operation described here is performed by the control unit 1301 executing a program at least when viewpoint detection video is being supplied from the viewpoint detection camera 205.

[0090] In S601, the control unit 1301 acquires image blur correction information from the imaging camera 200 via the first I / F 1303 and stores it in the RAM. The RAM stores image blur correction information acquired during the most recent predetermined period.

[0091] In S603, the control unit 1301 generates image blur control information according to the image blur correction information acquired in S601 and / or acquired from the ROM 1309 and the content of the image blur control to be performed, and transmits the image blur control information to the viewpoint detection camera 205 via the second I / F 1304. As described above, if the image blur control information does not need to be updated, S603 needs to be executed only once.

[0092] Note that if the amount of image blur correction in the imaging camera 200 is small and has little effect on the image blur correction of the viewpoint detecting camera 205, for example, if the amount of image blur correction is less than a predetermined threshold, the image blur control information does not need to be transmitted. The threshold may be determined experimentally or may be determined based on the viewpoint detection resolution (the smallest unit of detectable coordinates and orientation) of the viewpoint detection system (the viewpoint detecting camera 205 and the viewpoint detecting device 130). If the amount of change in position and orientation due to image blur correction is less than the viewpoint detection resolution, the image blur control information does not need to be transmitted. Alternatively, the image blur control information may be transmitted, but the second control unit 290 of the viewpoint detecting camera 205 may ignore the image blur control information.

[0093] In S605, the control unit 1301 acquires a predetermined amount (for example, one frame) of viewpoint detection video from the viewpoint detection camera 205 via the second I / F 1304, and stores it in the RAM.

[0094] In S607, the control unit 1301 detects the viewpoint (position and orientation) of the imaging camera 200 from the three-dimensional position of the viewpoint detection marker 113 shown in the viewpoint detection image. The control unit 1301 can detect the position and orientation of the camera that captured the marker based on the image capturing the marker using any known method. The control unit 1301 stores the detected position and orientation in RAM.

[0095] In S609, the control unit 1301 outputs the position and orientation detected in S607 to the scene control device 110 via the third I / F 1305.

[0096] In S611, the control unit 1301 determines whether or not to end the viewpoint detection operation. If it is determined that the viewpoint detection operation should be ended, the control unit 1301 ends the viewpoint detection operation shown in Fig. 7, and if not, the process repeats from S601.

[0097] The image blur correction operation in the viewpoint detection camera 205 may be the same as the image blur correction operation in the imaging camera 200 described using Figure 6, except that the image blur control information is taken into account when calculating the image blur correction amount in S503, so detailed description will be omitted.

[0098] In this embodiment, when image blur correction is enabled for both the imaging camera 200 and the viewpoint detection camera 205, the image blur correction for the viewpoint detection camera 205 is controlled based on the image blur correction for the imaging camera 200. This makes it possible to suppress unnaturalness caused by a mismatch in shaking between the background image portion and the real subject portion of the in-camera VFX video, which occurs when the imaging camera 200 and the viewpoint detection camera 205 perform image blur correction independently.

[0099] (Other embodiments) In the above-described embodiment, the viewpoint detection device 130 controls the image blur correction of the viewpoint detection camera 205. However, the function of generating image blur control information implemented by the control unit 1301 may be included in the second control unit 290 of the viewpoint detection camera 205. In this case, the image blur correction information is input from the imaging camera 200 to the viewpoint detection camera 205.

[0100] In the above-described embodiment, the viewpoint detection device 130, the scene control device 110, and the display control device 120 are separate devices. However, the scene control device 110 may have the functions of the display control device 120 and the viewpoint detection device 130.

[0101] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0102] The disclosure of the present embodiment includes the following image blur control device, image blur control method, program, and imaging device. (Item 1) a detection means for detecting the position and orientation of a first image capturing device that captures an image of a subject against a background of an image displayed on a display device, using an image obtained from a second image capturing device different from the first image capturing device; a generating means for generating an image to be displayed on the display device in accordance with the position and orientation; and a control unit that, when image blur compensation of the first imaging device and the second imaging device is enabled, controls image blur compensation of the second imaging device based on image blur compensation of the first imaging device. (Item 2) 2. The image blur control device according to item 1, wherein the control means controls the image blur correction of the second imaging device based on the image blur correction amount or the position of the image blur correction member acquired from the first imaging device. (Item 3) Item 3. The image blur control device according to item 2, wherein the control means converts the image blur compensation amount or the position of the image blur compensation member of the first imaging device into the second image blur compensation amount or the position of the image blur compensation member, and uses the second image blur compensation amount or the position of the image blur compensation member to control the image blur compensation of the second imaging device. (Item 4) 4. The image blur control device according to item 2 or 3, wherein the image blur correction member is one or more of a shift lens and an imaging element. (Item 5) 2. The image blur control device according to item 1, wherein the detection means calculates an amount of image blur correction by the first imaging device based on information regarding the movement of the first imaging device acquired from the first imaging device, and controls image blur correction of the second imaging device based on the amount of image blur correction. (Item 6) 6. The image blur control device according to any one of items 1 to 5, wherein the second imaging device is attached to the first imaging device. (Item 7) 7. The image blur control device according to any one of items 1 to 6, wherein the second image capturing device captures an image of an area in which a marker whose three-dimensional position is known is arranged. (Item 8) The control means controls the image blur correction capability of the first imaging device so as to be equivalent to that of the first imaging device. 8. The image blur control device according to any one of items 1 to 7, characterized in that it controls image blur correction of the second imaging device. (Item 9) 2. The image blur control device according to item 1, wherein the generating means generates an image of a virtual space captured by a virtual camera based on the position and orientation. (Item 10) An image blur control method executed by an image blur control device, comprising: Detecting the position and orientation of a first image capturing device that captures an image of a subject against a background of an image displayed on a display device, using an image obtained from a second image capturing device different from the first image capturing device; generating an image to be displayed on the display device according to the position and orientation; and when image blur compensation of the first imaging device and the second imaging device is enabled, controlling the image blur compensation of the second imaging device based on the image blur compensation of the first imaging device. (Item 11) 10. A program for causing a computer to function as each of the means included in the image blur control device according to any one of items 1 to 9. (Item 12) An imaging device for capturing an image for detecting the position and orientation of another imaging device that captures an object against a background of an image displayed on a display device, a detection means for detecting the movement of the imaging device; an image blur correction unit that performs image blur correction based on the movement; a control unit that controls the operation of the image blur correction unit based on information about image blur correction in the other imaging device, the information being supplied from an external device; An imaging device comprising:

[0103] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to clarify the scope of the invention. [Explanation of symbols]

[0104] 110...Scene control device, 120...Display control device, 130...Viewpoint detection device, 200...Image capture camera, 205...Viewpoint detection camera, 310, 320...Display device

Claims

1. a detection means for detecting the position and orientation of a first image capturing device that captures an image of a subject against a background of an image displayed on a display device, using an image obtained from a second image capturing device different from the first image capturing device; a generating means for generating an image to be displayed on the display device in accordance with the position and orientation; and a control unit that, when image blur compensation of the first imaging device and the second imaging device is effective, controls image blur compensation of the second imaging device based on image blur compensation of the first imaging device.

2. 2. The image blur control device according to claim 1, wherein the control means controls the image blur correction of the second image capture device based on the image blur correction amount or the position of an image blur correction member acquired from the first image capture device.

3. 3. The image blur control device according to claim 2, wherein the control means converts the image blur compensation amount or the position of the image blur compensation member of the first imaging device into the second image blur compensation amount or the position of the image blur compensation member, and uses the second image blur compensation amount or the position of the image blur compensation member to control the image blur compensation of the second imaging device.

4. 3. An image blur control device according to claim 2, wherein the image blur correction member is at least one of a shift lens and an image pickup element.

5. 2. The image blur control device according to claim 1, wherein the detection means calculates an amount of image blur correction by the first image capturing device based on information regarding the movement of the first image capturing device acquired from the first image capturing device, and controls image blur correction of the second image capturing device based on the amount of image blur correction.

6. 2. An image blur control device according to claim 1, wherein the second image pickup device is attached to the first image pickup device.

7. 2. The image blur control device according to claim 1, wherein the second image capturing device captures an image of an area in which a marker whose three-dimensional position is known is arranged.

8. The control means controls the image blur correction capability of the first imaging device so as to be equivalent to that of the first imaging device.

2. An image blur control device according to claim 1, further comprising: a control unit for controlling image blur correction of said second image pickup device;

9. 2. The image blur control device according to claim 1, wherein the generating means generates an image of a virtual space captured by a virtual camera based on the position and orientation.

10. An image blur control method executed by an image blur control device, comprising: Detecting the position and orientation of a first image capturing device that captures an image of a subject against a background of an image displayed on a display device, using an image obtained from a second image capturing device different from the first image capturing device; generating an image to be displayed on the display device according to the position and orientation; and when image blur compensation of the first imaging device and the second imaging device is effective, controlling the image blur compensation of the second imaging device based on the image blur compensation of the first imaging device.

11. A program for causing a computer to function as each of the means included in the image blur control device according to any one of claims 1 to 9.

12. An imaging device for capturing an image for detecting the position and orientation of another imaging device that captures an object against a background of an image displayed on a display device, a detection means for detecting the movement of the imaging device; an image blur correction unit that performs image blur correction based on the movement; a control unit that controls the operation of the image blur correction unit based on information about image blur correction in the other imaging device, the information being supplied from an external device; An imaging device comprising:

Images