Image processing apparatus, image processing method, and virtual studio system

Abstract

To provide an image processing device capable of generating an appropriate background video image even when image blur correction is effective in a camera that captures a live-action video image used for a VFX video image SOLUTION: An image processing apparatus detects the position and orientation of an image capturing apparatus which captures an image of an object using a video displayed on a display device as a background. The image processing apparatus also generates a video to be displayed on a display apparatus in accordance with the position and orientation. When the image blur correction of the imaging device is effective, the image processing device corrects the position and posture so that the motion of the video reflects the image blur correction.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a virtual studio system. [Background technology]

[0002] Conventionally, VFX (Visual Effects) images have been images synthesized by combining separately generated live-action images and background images. In recent years, a method (in-camera VFX) has become known in which a VFX image is directly captured by a camera by displaying a background image corresponding to the viewpoint of the camera capturing the real subject in real time on a large display device placed behind the real subject (Patent Document 1). In-camera VFX eliminates the need for a post-process to composite background images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7190594 Summary of the Invention [Problem to be solved by the invention]

[0004] Traditionally, in-camera VFX footage has been shot with a camera fixed to a large tripod or crane. This meant that there was no need to consider image shake caused by camera movement. However, when a cameraman holds the camera and shoots in-camera VFX footage, image shake must be taken into consideration.

[0005] Image blur in live-action footage can be suppressed by enabling the camera's built-in image stabilization. However, implementing image stabilization creates a discrepancy between the camera movement and the image movement, resulting in unnatural in-camera VFX background images generated using the camera's position and orientation.

[0006] In light of these issues, one aspect of the present invention provides an image processing device and an image processing method that can generate an appropriate background image even when image stabilization is enabled on a camera that captures live-action footage used in VFX images. [Means for solving the problem]

[0007] In one aspect, the present invention provides an image processing device including: a detection means for detecting a position and orientation of an imaging device that captures an image of a subject with an image displayed on a display device as a background; and a generation means for generating an image to be displayed on the display device in accordance with the position and orientation, wherein, when image blur correction of the imaging device is enabled, the detection means corrects the position and orientation so that movement of the image reflects the image blur correction. [Effects of the Invention]

[0008] According to one aspect of the present invention, it is possible to provide an image processing device and an image processing method that can generate an appropriate background image even when image stabilization is enabled on a camera that captures live-action footage used in VFX footage. [Brief explanation of the drawings]

[0009] [Figure 1] Schematic diagram of a virtual studio system according to an embodiment. [Figure 2] Figure 1 shows an example of the camera's functional configuration and a block diagram showing the connections between the various devices. [Figure 3] A block diagram showing an example of the functional configuration of a viewpoint detection device. [Figure 4] Block diagram showing an example of the functional configuration of a scene control device [Figure 5] Flowchart for image blur correction operation [Figure 6] Flowchart for viewpoint detection operation DETAILED DESCRIPTION OF THE INVENTION

[0010] ●(First embodiment) (Virtual Studio System Overview) FIG. 1 is a schematic diagram of a virtual studio system according to an embodiment. The virtual studio is a studio capable of capturing in-camera VFX video, equipped with a large display device that displays background video and equipment for detecting the position and orientation of the camera. The virtual studio and other equipment required for capturing in-camera VFX video (such as imaging equipment such as cameras and equipment for generating background video) are collectively referred to as a virtual studio system. In this specification, "video" refers to video, a signal representing video, or data representing video.

[0011] Large display devices 310 and 320, also known as LED walls, are provided on the walls of virtual studio 100. Display devices 310 and 320 may be movable. Here, the imaging range 311 of imaging camera 200 that captures the in-camera VFX video is within the display screen of display device 310, and the background video is displayed on display device 310. On the other hand, display device 320 is not captured by imaging camera 200, and is used to display an image to be reflected in real subject 400. Display device 320 is not essential.

[0012] Viewpoint detection markers 131 are provided on the ceiling of the virtual studio 100 to be captured by the viewpoint detection camera 205. The viewpoint detection markers 131 are, for example, a plurality of markers arranged in a specific pattern. The three-dimensional coordinates (position in the world coordinate system) of each marker are known.

[0013] The position and orientation of the imaging camera 200 is detected from the image coordinates of the markers and the three-dimensional coordinates of the markers that appear in an image captured in the direction of the ceiling from the viewpoint detection camera 205. The position and orientation of the imaging camera 200 is detected as a combination of the three-dimensional coordinates and rotation angles around three axes (tilt, pan, and roll).

[0014] The coordinate system of the imaging camera 200 and the viewpoint detection camera 205 is an orthogonal coordinate system with the intersection of the optical axis and the imaging surface as the origin, an axis extending from the optical axis, and two axes perpendicular to the optical axis. The two axes perpendicular to the optical axis are parallel to two sides of the imaging surface that do not face each other.

[0015] Here, the state in which the optical axis of the imaging camera 200 is perpendicular to the direction of gravity and the rotation angle around the optical axis of the imaging camera 200 is 0 degrees is defined as the reference attitude of the imaging camera 200. The viewpoint detection camera 205 is attached to a rig or the like that holds the imaging camera 200 so that the optical axis of the viewpoint detection camera 205 points vertically upward when the imaging camera 200 is in the reference attitude.

[0016] Here, the position and orientation of the imaging camera 200 is detected from an image of a marker whose three-dimensional coordinates are known. However, the position and orientation of the imaging camera 200 may be detected by any other known method, such as a method that does not use a marker.

[0017] Capturing camera 200 records the in-camera VFX video by capturing an image of real subject 400 against the background of the image displayed on display device 310. Capturing range 311 is an example of the range captured by capturing camera 200 within the background video displayed on display device 310.

[0018] The viewpoint detecting device 130 detects the viewpoint (position and orientation) of the imaging camera 200 from the coordinates of a viewpoint detecting marker 131 provided on the ceiling and the position of the viewpoint detecting marker 131 in the image acquired by the viewpoint detecting camera 205. The viewpoint detecting device 130 assumes that the three-dimensional positional deviation between the origin of the coordinate system of the imaging camera 200 and the origin of the coordinate system of the viewpoint detecting camera 205 in the coordinate system of the virtual studio 100 is known through a calibration operation or the like. Therefore, the viewpoint detecting device 130 can detect the viewpoint of the imaging camera 200 using the image acquired from the viewpoint detecting camera 205.

[0019] As will be described later, when image blur correction of the imaging camera 200 is enabled, the viewpoint detection device 130 detects a viewpoint that reflects the image blur correction of the imaging camera 200. The viewpoint detection device 130 continuously detects the viewpoint of the imaging camera 200. The viewpoint detection device 130 also continuously supplies information about the detected viewpoint to the scene control device 110.

[0020] The scene control device 110 renders a preset three-dimensional model of a virtual space from the viewpoint of the imaging camera 200 detected by the viewpoint detection device 130, and generates a virtual space image at a predetermined frame rate. The scene control device 110 generates a background image to be displayed on the display device 310 based on the virtual space image. When generating the background image from the virtual space image, the scene control device 110 applies coordinate transformation (deformation processing) to the background image according to the angle between the imaging direction (optical axis direction) of the imaging camera 200 and the display surface of the display device 310. The scene control device 110 supplies the generated background image to the display control device 120. The scene control device 110 also supplies an image to be displayed on the display device 320 to the display control device 120. The image to be displayed on the display device 320 may be a predetermined still image, or may be an image composed of frames having the same content as the predetermined still image.

[0021] Display control device 120 causes display device 310 to display a background image at a timing that allows imaging camera 200 to appropriately capture the background image. Display control device 120 also causes display device 320 to display an image at the same timing as display device 310. Display control device 120 controls the supply of images according to the configurations of display devices 310 and 320. For example, if display device 310 and / or 320 is configured with multiple independent display panels, display control device 120 divides the image into areas to be displayed on each display panel. Then, display control device 120 supplies each display panel with the image of the area to be displayed on that display panel.

[0022] Furthermore, a synchronization signal is supplied from a reference clock generating device, also called a sync generator, to the imaging camera 200, viewpoint detection device 130, scene control device 110, and display control device 120. Each device controls its operation timing in accordance with the reference clock, thereby achieving synchronization between the imaging period of the imaging camera 200 and the display periods of the display devices 310 and 320. The technology for synchronizing the operations of devices based on a reference clock is well known, for example, as genlock (generator locking), and therefore a detailed description thereof will be omitted.

[0023] For convenience, this specification assumes that the real subject 400 is a human subject, but there is no limit to the type and number of real subjects.

[0024] Fig. 2 is a block diagram showing the connection relationships of the devices shown in Fig. 1, and an example of the functional configuration of the imaging camera 200 and the viewpoint detection camera 205. Unless otherwise specified below, the first imaging section 220 and the second imaging section 270 capture moving images at a predetermined frame rate (for example, 30 frames per second).

[0025] The first control unit 240 has a processor (CPU, MPU, microprocessor, etc.) capable of executing a program, a ROM, and a RAM. The first control unit 240 loads a program stored in the ROM into the RAM and executes it, thereby controlling the operation of each functional block of the imaging camera 200 and realizing the operation of the imaging camera 200 described below. Although not shown in the figure, the first control unit 240 is connected to each functional block of the imaging camera 200 so as to be able to communicate with them.

[0026] The first optical system 210 is an optical system for capturing in-camera VFX video. If the first optical system 210 has an image blur correction function, the first optical system 210 has an image blur correction lens (shift lens), a shift lens movement mechanism, and a position detection mechanism. The shift lens movement mechanism is capable of moving the shift lens in a direction perpendicular to the optical axis of the first optical system 210. The shift lens position detection mechanism detects the position of the shift lens or the amount of movement from a reference position. The angle of view of the first optical system 210 may be variable. This embodiment assumes a situation where there are few physical constraints on the capturing direction of the capturing camera 200, such as handheld capturing, and the capturing range 311 of the first optical system 210 does not exceed the display range of the display device 310.

[0027] The first imaging unit 220 has an imaging element and converts the optical image formed by the first optical system 210 into an analog image signal. The imaging element of the first imaging unit 220 may be, for example, a known CCD or CMOS color image sensor with a primary color Bayer array color filter. If the first imaging unit 220 has an image stabilization function, the first imaging unit 220 has an imaging element movement mechanism and a position detection mechanism. The imaging element movement mechanism is capable of moving the imaging element in a direction perpendicular to the optical axis of the first optical system 210. The imaging element movement mechanism may also be capable of rotating the imaging element around the optical axis of the first optical system 210. The imaging element position detection mechanism detects the position of the imaging element or the amount of movement from a reference position. The analog image signal output by the first imaging unit 220 is supplied to the first image processing unit 230.

[0028] The first image processing unit 230 applies predetermined image processing to the analog image signal output by the first imaging unit 220 to generate signals and image data according to the intended use, and acquires and / or generates various types of information. The first image processing unit 230 may be a dedicated hardware circuit such as an ASIC (Application Specific Integrated Circuit) designed to realize a specific function. Alternatively, the first image processing unit 230 may be configured to realize a specific function by a processor such as a DSP (Digital Signal Processor) or GPU (Graphics Processing Unit) executing software.

[0029] The image processing applied by the first image processing unit 230 can include, for example, pre-processing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Pre-processing may include A / D conversion, signal amplification, reference level adjustment, defective pixel correction, and the like. Color interpolation, also known as demosaicing, is performed when the image sensor is equipped with a color filter, and is a process of interpolating the values ​​of color components that are not included in the individual pixel data that make up the image data. The correction processing may include white balance adjustment, tone correction, correction of image degradation caused by optical aberration of the first optical system 210 (image restoration), correction of the effects of peripheral light falloff of the first optical system 210, color correction, and the like. The detection process can include detection of a characteristic region or a region of a specific subject (for example, a face region or a human body region) and its movement, and person recognition processing. Data processing can include processes such as area extraction (trimming), compositing, scaling, encoding and decoding, header information generation (data file generation), etc. Data processing also includes the generation of image data for display or image data for recording. The evaluation value calculation process may include processes such as generating a signal or evaluation value used for autofocus detection (AF) and generating an evaluation value used for automatic exposure control (AE). The evaluation value used for AE is information about the luminance of the captured scene, but the information about the luminance of which part of the captured scene may differ depending on, for example, the exposure mode that is set. For example, the information may reflect the luminance of the entire captured scene, or it may be information about the luminance of an area of ​​a specific subject. Special effect processing may include adding a blur effect, changing color tones, relighting, etc. Special effect processing also includes processing that reflects the influence of a light source, which will be described later.

[0030] Note that these are examples of processes that the first image processing unit 230 can apply, and do not limit the processes that the first image processing unit 230 applies. The first image processing unit 230 outputs acquired or generated information and data to a functional block according to the application. For example, the first image processing unit 230 outputs image data for recording to the recording unit 250, and outputs information related to the luminance of the captured scene to the first control unit 240. The first image processing unit 230 also outputs video data for display to the scene control device 110.

[0031] The first control unit 240 executes AE processing to determine exposure conditions based on information about the luminance of the imaging scene acquired from the first image processing unit 230, and can control the operation of the first imaging unit 220 in accordance with the determined exposure conditions. The first control unit 240 can, for example, determine exposure conditions such that the entire imaging scene is properly exposed, or determine exposure conditions such that a portion of the imaging scene (for example, the area of ​​the real subject) is properly exposed.

[0032] Generally, exposure conditions are determined by a combination of aperture value, shutter speed (exposure time), and imaging sensitivity. Therefore, the first control unit 240 can determine a combination of these three parameter values ​​as imaging conditions for obtaining proper exposure. However, if the aperture value or exposure time is changed during video capture, the depth of field may change, or the movement distance of a moving subject between frames may change. Therefore, in the AE processing of this embodiment, the imaging sensitivity is determined without basically changing the aperture value or shutter speed.

[0033] The focal length of the first optical system 210 can be automatically adjusted by the first control unit 240 performing AF processing based on the evaluation value generated by the first image processing unit 230.

[0034] The motion detection device 235 is a sensor that detects the motion of the imaging camera 200. The motion detection device 235 may be, for example, a known six-axis sensor that detects acceleration in each axis direction and angular velocity around each axis of the coordinate system of the imaging camera 200. The motion detection device 235 supplies the detected values ​​to the first control unit 240 at a predetermined cycle.

[0035] In this embodiment, the imaging camera 200 has an optical and / or electronic image blur correction function. The optical image blur correction function is realized by the first control unit 240 controlling the shift lens of the first optical system 210 and / or the movement mechanism of the imaging element of the first imaging unit 220 based on the movement detected by the motion detection device 235. The electronic image blur correction function is realized by the first control unit 240 controlling the position at which a frame image obtained by the first imaging unit 220 is cut out based on the movement detected by the motion detection device 235.

[0036] The first control unit 240 calculates the amount of image blur correction according to the type of image blur correction method, the type of image blur correction member (shift lens and / or image sensor), etc., based on the movement detected by the movement detection device 235. Then, the first control unit 240 controls the position of the image blur correction member and the clipping position of the frame image according to the calculated amount of image blur correction.

[0037] The second control unit 290 has a processor (CPU, MPU, microprocessor, etc.) capable of executing a program, a ROM, and a RAM. The second control unit 290 loads a program stored in the ROM into the RAM and executes it, thereby controlling the operation of each functional block of the viewpoint detection camera 205 and realizing the operation of the viewpoint detection camera 205. Although not shown in the figure, the second control unit 290 is connected to each functional block of the viewpoint detection camera 205 so as to be able to communicate with them.

[0038] In this embodiment, a viewpoint detection marker 131 is placed on the ceiling of the virtual studio 100. Therefore, the viewpoint detection camera 205 is attached to the imaging camera 200 so that the optical axis of the second optical system 260 faces vertically upward when the imaging camera 200 is in the reference attitude. The angle of view of the second optical system may be fixed or variable.

[0039] The second imaging unit 270 has an imaging element and converts the optical image formed by the second optical system 260 into an analog image signal. The imaging element may be a monochrome image sensor, as it is sufficient for the image for viewpoint detection to be able to acquire the image coordinates of the viewpoint detection marker 131. Furthermore, the number of pixels of the imaging element of the second imaging unit 270 may be less than the number of pixels of the imaging element of the first imaging unit 220.

[0040] The second image processing unit 280 generates a viewpoint detection image by applying image processing to the analog image signal output by the second imaging unit 270. The second image processing unit 280 may have the same functions as the first image processing unit 230, or may have only the functions of the first image processing unit 230 that are necessary for generating the viewpoint detection image. The viewpoint detection image may be a display image similar to that generated by the first image processing unit 230. When the second imaging unit 270 outputs a monochrome image, the second image processing unit may generate the viewpoint detection image by A / D converting the monochrome image. The second image processing unit 280 outputs the viewpoint detection image to the viewpoint detection device 130.

[0041] 3 is a block diagram showing an example of the functional configuration of the viewpoint detecting device 130. The viewpoint detecting device 130 can be realized using, for example, a computer device. The control unit 1301 is, for example, a CPU, and realizes the functions of the viewpoint detection device 130 by loading one or more application programs stored in a ROM 1308 into a RAM 1309 and executing them. The control unit 1301 controls the operation timing of the viewpoint detection device 130 in accordance with a synchronization signal supplied from a reference clock generation device.

[0042] The image processing circuit 1302 is, for example, a graphics board equipped with a GPU. The image processing circuit 1302 is capable of high-speed image processing, such as processing for detecting the image coordinates of the viewpoint detection marker 113 from the viewpoint detection video supplied from the viewpoint detection camera 205.

[0043] The first I / F 1303 to the third I / F 1305 are communication interfaces for connecting external devices. In this embodiment, the imaging camera 200 is connected to the first I / F 1303, the viewpoint detection camera 205 is connected to the second I / F 1304, and the scene control device 110 is connected to the third I / F 1305. It should be noted that the first I / F 1303 to the third I / F 1305 comply with standards according to the external devices to be connected and the types of signals to be communicated. For convenience, the illustration shows the viewpoint detection device 130 and the external device connected via one I / F, but they may be connected using multiple I / Fs. It should be noted that the viewpoint detection device 130 may have four or more communication interfaces with external devices.

[0044] The control unit 1301 acquires one or more of motion data, an image blur correction amount, and a position of an image blur correction member as image blur correction information from the imaging camera 200 via a first I / F 1303. The control unit 1301 also acquires a viewpoint detection image from the viewpoint detection camera 205 via a second I / F 1304. The control unit 1301 outputs the viewpoint (position and orientation) of the imaging camera 200 to the scene control device 110 via a third I / F 1304.

[0045] The ROM 1309 is, for example, an electrically rewritable nonvolatile memory. The ROM 1309 stores part of the program executed by the control unit 1301 (BIOS, bootstrap loader, firmware), setting values ​​of the viewpoint detection device 130, etc. The deviation between the origin of the coordinate system of the imaging camera 200 and the origin of the coordinate system of the viewpoint detection camera 205 (the difference in three-dimensional coordinates in the coordinate system of the virtual studio 100) is also stored in the ROM 1309. For example, when the viewpoint detection camera 205 is installed, the control unit 1301 executes a calibration application to detect the deviation of the origin of the coordinate system and stores it in the ROM 1309. The three-dimensional position of the viewpoint detection marker 113 (the three-dimensional coordinates in the coordinate system of the virtual studio 100) is also stored in the ROM 1309.

[0046] The RAM 1310 is used as the main memory for the control unit 1301 , as well as a working memory for the image processing circuit 1302 and a video memory for the display unit 1312 .

[0047] The storage unit 1311 is a large-capacity storage device such as a hard disk or SSD. The storage unit 1311 stores operating system (OS), application programs, user data, etc. The application used for detecting the viewpoint position, as exemplified below, is stored in the storage unit 1311. An application that calculates the viewpoint (position and orientation) of the imaging camera 200 from the viewpoint detection image supplied from the viewpoint detection camera 205 and the three-dimensional position of the viewpoint detection marker 113 stored in the ROM 1309. A program for controlling the operation of the viewpoint detection camera 205 Calibration application for the gaze detection device 130 Position registration application for viewpoint detection marker 113 It should be noted that these are merely examples, and not all of them are necessarily required, and other applications may also be stored.

[0048] The control unit 1301 executes necessary applications using the image processing circuit 1302 as necessary, detects the viewpoint of the imaging camera 200 for each frame of the viewpoint detection video, for example, and supplies it to the scene control device 110. As described above, the viewpoint of the imaging camera 200 is detected as a combination of the three-dimensional coordinates of the origin of the coordinate system of the imaging camera 200 and the rotation angle around each axis (tilt, pan, roll) of the coordinate system of the imaging camera 200.

[0049] The display unit 1312 is, for example, a liquid crystal display device. The display unit 1312 may also be a touch display. The display unit 1312 displays an application used for detecting the viewpoint position, a GUI provided by the OS, and the like.

[0050] The operation unit 1313 has a plurality of input devices that can be operated by the user, such as a keyboard, a mouse, a touch pad, etc. If the display unit 1312 is a touch display, the operation unit 1313 includes a touch panel.

[0051] 4 is a block diagram showing an example of the functional configuration of the scene control device 110. The scene control device 110 can be realized using, for example, a computer device. The control unit 1101 is, for example, a CPU, and realizes the functions of the scene control device 110 by loading one or more application programs stored in the ROM 1108 into the RAM 1110 and executing them. The control unit 1101 controls the operation timing of the scene control device 110 in accordance with a synchronization signal supplied from a reference clock generation device.

[0052] The image processing circuit 1102 is, for example, a graphics board equipped with a GPU. The image processing circuit 1102 is capable of high-speed image processing such as CG rendering using a 3D model and the viewpoint of a virtual camera. For example, the image processing circuit 1102 can generate one frame of background video in a time that is shorter than the frame cycle of the in-camera VFX video.

[0053] The first I / F 1103 to the third I / F 1105 are communication interfaces for connecting external devices. In this embodiment, the imaging camera 200 is connected to the first I / F 1103, the display control device 120 is connected to the second I / F 1104, and the viewpoint detection device 130 is connected to the third I / F 1105. It should be noted that the first I / F 1103 to the third I / F 1105 comply with standards according to the external devices to be connected and the types of signals to be communicated. For convenience, the scene control device 110 and the external device are shown connected via one I / F, but they may be connected using multiple I / Fs. It should be noted that the scene control device 110 may have four or more communication interfaces with external devices.

[0054] The control unit 1101 acquires captured image data and information about the brightness of the captured scene from the capturing camera 200 via the first I / F 1103. The control unit 1101 also acquires information about the viewpoint of the capturing camera 200 from the viewpoint detection device 130 by communication via the third I / F 1105. The control unit 1101 outputs image data for display (background video data) to the display control device via the second I / F 1104.

[0055] The ROM 1109 is, for example, an electrically rewritable nonvolatile memory, and stores some of the programs executed by the control unit 1101 (BIOS, bootstrap loader, firmware), setting values ​​of the scene control device 110, and the like.

[0056] The RAM 1110 is used as the main memory for the control unit 1101 , as well as a working memory for the image processing circuit 1102 and a video memory for the display unit 1112 .

[0057] The storage unit 1111 is a large-capacity storage device such as a hard disk or SSD. The storage unit 1111 stores operating system (OS), application programs, user data, etc. The storage unit 1111 also stores an application program (e.g., a game engine application) that generates a background image according to the viewpoint of the imaging camera 200, and data required to generate the background image (e.g., a 3D model of the virtual space, textures, etc.).

[0058] The display unit 1112 is, for example, a liquid crystal display device. The display unit 1112 may also be a touch display. The display unit 1112 displays a scene control application, a background image generation application (for example, a game engine application), a GUI provided by the OS, and the like.

[0059] The operation unit 1113 has a plurality of input devices that can be operated by the user, such as a keyboard, a mouse, a touch pad, etc. If the display unit 1112 is a touch display, the operation unit 1113 includes a touch panel.

[0060] Next, the image blur correction operation by the first control unit 240 will be described using the flowchart shown in Fig. 5. The image blur correction operation is performed by the first control unit 240 executing a program. The image blur correction operation is performed when image blur correction is enabled in the imaging camera 200. The user may be able to set whether image blur correction is enabled or disabled, or the imaging camera 200 (first control unit 240) may automatically determine this in accordance with one or more predetermined conditions.

[0061] Here, it is assumed that the motion detection device 235 is continuously supplying the first control unit 240 with detection values ​​indicating the movement of the imaging camera 200. It is also assumed that the imaging camera 200 is currently capturing video. Furthermore, when optical image stabilization is performed, it is assumed that the first control unit 240 is continuously acquiring position information of the image stabilization members (shift lens and imaging element). It is to be noted that optical image stabilization and electronic image stabilization are not exclusive and may be used together.

[0062] In S501, the first control unit 240 acquires a motion detection value supplied from the motion detection device 235. The first control unit 240 stores the acquired detection value in RAM. The RAM stores detection values ​​for the most recent predetermined period.

[0063] In S503, the first control unit 240 calculates the amount of image blur correction using the detection value acquired in S501. The first control unit 240 calculates the amount of image blur correction according to the type of image blur correction (optical and / or electronic), and in the case of an optical type, the image blur correction member used (one or both of the shift lens and the image sensor). The first control unit 240 can calculate the amount of image blur correction using any known method. Therefore, a detailed explanation of the calculation of the amount of image blur correction will be omitted.

[0064] When optical image blur correction is performed, the image blur correction amount includes, for example, a combination of the movement direction and movement amount of the image blur correction member. When electronic image blur correction is performed, the image blur correction amount includes, for example, the image coordinates (absolute coordinates or amounts of change) of multiple vertices that specify the clipping position of the frame image.

[0065] In S505, the first control unit 240 performs image blur correction based on the amount of image blur correction calculated in S503. That is, the first control unit 240 moves the image blur correction member and / or the clipping position of the frame image based on the amount of image blur correction.

[0066] In S507, the first control unit 240 calculates the actual movement amount of the image blur correction member moved in S505. After issuing an instruction to move the image blur correction member in S505, the first control unit 240 obtains the actual movement amount of the image blur correction member via the position detection mechanism. Note that if only electronic image blur correction is performed in S505, S507 may be skipped.

[0067] In S509, the first control unit 240 outputs the image blur correction information to the viewpoint detection device 130. The image blur correction information is The detected value of the movement of the imaging camera 200 acquired in S501, The image stabilization amount calculated in S503, The position of the image stabilization member obtained in S507, From the viewpoint of improving the viewpoint detection accuracy of the imaging camera 200 in the viewpoint detection device 130, the position of the image blur correction member is most preferable, followed by the amount of image blur correction. This is because the position of the correction member is closest to the actual amount of image blur correction.

[0068] In addition, when there are multiple image blur correction amounts, such as when optical and electronic image blur correction are used in combination, or when optical image blur correction is performed using a shift lens and an image sensor in combination, the image blur correction amount output in S509 is a composite value of the multiple image blur correction amounts.

[0069] In S511, the first control unit 240 determines whether or not to end image capture by the imaging camera 200. If it is determined that image capture should be ended, the first control unit 240 ends the image blur correction operation shown in Fig. 5, and if not, the process repeats from S501.

[0070] Next, the viewpoint detection operation by the viewpoint detection device 130 will be described with reference to the flowchart shown in Fig. 6. The viewpoint detection operation is performed by the control unit 1301 executing a program at least when viewpoint detection video is being supplied from the viewpoint detection camera 205.

[0071] In S601, the control unit 1301 acquires image blur correction information from the imaging camera 200 via the first I / F 1303 and stores it in the RAM. The RAM stores image blur correction information acquired during the most recent predetermined period.

[0072] In S603, the control unit 1301 acquires a predetermined amount (for example, one frame) of viewpoint detection video from the viewpoint detection camera 205 via the second I / F 1304, and stores it in the RAM.

[0073] In S605, the control unit 1301 detects the viewpoint (position and orientation) of the imaging camera 200 from the three-dimensional position of the viewpoint detection marker 113 shown in the viewpoint detection video. The control unit 1301 can detect the position and orientation of the camera that captured the marker based on the video of the marker using any known method. The control unit 1301 stores the detected position and orientation in RAM.

[0074] In S607, the control unit 1301 corrects the position and orientation detected in S605 using the image blur correction information acquired in S601. This is equivalent to reflecting the difference between the movement of the in-camera VFX video caused by image blur correction and the movement of the capturing camera 200 in the position and orientation detected in S605, and applying virtual blur correction to the position and orientation detected in S605. Therefore, the control unit 1301 corrects the value of the position and orientation detected in S605 based on the image blur correction information acquired in S601 so as to reduce the amount of change in the position and orientation of the capturing camera 200. The control unit 1301 stores the corrected position and orientation in RAM.

[0075] Note that if the amount of image blur correction is small and its effect on the detected position and orientation is small, for example, if it is less than a predetermined threshold, the position and orientation detected in S605 may not need to be corrected. The threshold may be determined experimentally or may be determined based on the viewpoint detection resolution (smallest unit of detectable coordinates and orientation) of the viewpoint detection system (the viewpoint detection camera 205 and the viewpoint detection device 130). If the amount of change in the position and orientation due to image blur correction is less than the viewpoint detection resolution, the detected position and orientation may not need to be corrected.

[0076] In S609, the control unit 1301 outputs the position and orientation corrected in S607 to the scene control device 110 via the third I / F 1305. The scene control device 110 generates a background video using the corrected position and orientation, so that the movement of the background video reflects the image blur correction of the imaging camera 200.

[0077] In S611, the control unit 1301 determines whether or not to end the viewpoint detection operation. If it is determined that the viewpoint detection operation should be ended, the control unit 1301 ends the viewpoint detection operation shown in Fig. 6, and if not, the process repeats from S601.

[0078] In this embodiment, when image stabilization is enabled in imaging camera 200 capturing in-camera VFX video, the effect of imaging camera 200 movement, which is suppressed by image stabilization, on the in-camera VFX video is reflected in the viewpoint detection result of imaging camera 200. Therefore, it is possible to suppress unnaturalness caused by a mismatch between the shaking of the background image portion and the actual subject portion in the image stabilized in-camera VFX video.

[0079] (Other embodiments) In the above-described embodiment, the viewpoint detection device 130, the scene control device 110, and the display control device 120 are separate devices. However, the scene control device 110 may have the functions of the display control device 120 and the viewpoint detection device 130.

[0080] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0081] The disclosure of the present embodiment includes the following image processing device, image processing method, and program. (Item 1) a detection means for detecting the position and orientation of an imaging device that captures an image of a subject against the background of an image displayed on a display device; a generating means for generating an image to be displayed on the display device in accordance with the position and orientation, The image processing device is characterized in that, when image stabilization of the imaging device is enabled, the detection means corrects the position and orientation so that the movement of the image reflects the image stabilization. (Item 2) 2. The image processing device according to item 1, wherein the detection means corrects the position and orientation based on an image blur correction amount or a position of an image blur correction member acquired from the imaging device. (Item 3) 3. The image processing device according to item 2, wherein the image blur correction member is one or more of a shift lens and an imaging element. (Item 4) 2. The image processing device according to item 1, wherein the detection means corrects the position and orientation based on information regarding the movement of the imaging device acquired from the imaging device. (Item 5) 2. The image processing device according to item 1, wherein the detection means calculates an amount of image blur correction by the imaging device based on information regarding the movement of the imaging device acquired from the imaging device, and corrects the position and orientation based on the amount of image blur correction. (Item 6) 6. The image processing device according to item 5, wherein the detection means does not correct the position and orientation when the amount of image blur correction is less than a threshold value. (Item 7) 7. The image processing device according to item 6, wherein the threshold value is determined based on the detection resolution of the detection means. (Item 8) 8. The image processing device according to any one of items 1 to 7, wherein the detection means detects the position and orientation using a member attached to the imaging device. (Item 9) Item 9. The image processing device according to item 8, wherein the member is a second imaging device that captures an image of an area in which a marker whose three-dimensional position is known is placed, and the detection means detects the position and orientation using an image captured by the second imaging device. (Item 10) 10. The image processing device according to item 9, wherein the second imaging device does not perform image blur correction. (Item 11) 11. The image processing device according to any one of items 1 to 10, wherein the generating means generates an image of a virtual space captured by a virtual camera based on the position and orientation. (Item 12) An image processing method executed by an image processing device, Detecting the position and orientation of an imaging device that captures an image of a subject against the background of an image displayed on a display device; generating an image to be displayed on the display device according to the position and orientation; the detecting includes, when image stabilization of the imaging device is enabled, correcting the position and orientation so that the movement of the image reflects the image stabilization. (Item 13) 12. A program for causing a computer to function as each of the means possessed by the image processing device according to any one of items 1 to 11.

[0082] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to clarify the scope of the invention. [Explanation of symbols]

[0083] 110...Scene control device, 120...Display control device, 130...Viewpoint detection device, 200...Image capture camera, 205...Viewpoint detection camera, 310, 320...Display device

Claims

1. a detection means for detecting the position and orientation of an imaging device that captures an image of a subject against the background of an image displayed on a display device; a generating means for generating an image to be displayed on the display device in accordance with the position and orientation, The image processing device is characterized in that, when image stabilization of the imaging device is enabled, the detection means corrects the position and orientation so that the movement of the image reflects the image stabilization.

2. 2. The image processing apparatus according to claim 1, wherein the detection means corrects the position and orientation based on an image blur correction amount or a position of an image blur correction member obtained from the imaging device.

3. 3. The image processing device according to claim 2, wherein the image blur correction member is at least one of a shift lens and an image pickup element.

4. The image processing apparatus according to claim 1 , wherein the detection means corrects the position and orientation based on information about the movement of the image capturing device acquired from the image capturing device.

5. 2. The image processing device according to claim 1, wherein the detection means calculates an amount of image blur correction by the imaging device based on information regarding the movement of the imaging device acquired from the imaging device, and corrects the position and orientation based on the amount of image blur correction.

6. 6. The image processing apparatus according to claim 5, wherein the detection means does not correct the position and orientation when the image blur correction amount is less than a threshold value.

7. 7. The image processing apparatus according to claim 6, wherein the threshold value is determined based on the detection resolution of the detection means.

8. 2. The image processing apparatus according to claim 1, wherein the detecting means detects the position and orientation using a member attached to the image capturing device.

9. 9. The image processing device according to claim 8, wherein the member is a second imaging device that captures an image of an area in which a marker whose three-dimensional position is known is placed, and the detection means detects the position and orientation using an image captured by the second imaging device.

10. 10. The image processing apparatus according to claim 9, wherein the second image capturing device does not perform image blur correction.

11. The image processing apparatus according to claim 1 , wherein the generating means generates an image of a virtual space captured by a virtual camera based on the position and orientation.

12. An image processing method executed by an image processing device, Detecting the position and orientation of an imaging device that captures an image of a subject against the background of an image displayed on a display device; generating an image to be displayed on the display device according to the position and orientation; the detecting includes, when image stabilization of the imaging device is enabled, correcting the position and orientation so that the movement of the image reflects the image stabilization.

13. A program for causing a computer to function as each of the means included in the image processing device according to any one of claims 1 to 11.

Images