Image processing apparatus, image processing method, virtual studio system, and image capturing apparatus

Abstract

Disclosed is an image processing device capable of assisting in capturing a VFX video in a camera while monitoring a background image outside an imaging range. An image processing apparatus causes a display device to display a portion of a first image corresponding to a viewpoint of an imaging device as a background image of a space captured by the imaging device. In addition, the image processing apparatus outputs, to an external device different from the display device, a second image including an imaging range of the imaging device in the first image and having a field of view larger than the imaging range.

Description

Technical Field

[0001] The present invention relates to an image processing device, an image processing method, a virtual studio system and an image capturing device. Background Art

[0002] A method is known for obtaining visual effects (VFX) video by displaying an image based on the position and orientation of a camera on a large-format display device and capturing an image of a subject using the displayed image as a background, without combining the background image with live-action video (Patent Document (PTL) 1). Video obtained using this method is referred to as "in-camera VFX video."

[0003] Reference List

[0004] Patent Literature

[0005] PTL 1: Japanese Patent No.7190594 Summary of the Invention

[0006] Technical issues

[0007] When shooting in-camera VFX video, the background image displayed on the display device changes depending on the camera's position and orientation. In other words, the content displayed on the display device is a portion of the background image. On the other hand, the content displayed on the display device installed in the camera is an image of the capture area, which is an even smaller portion of the background image. Therefore, it is not easy to capture the image while identifying the background image outside the capture area. Furthermore, it is impossible to identify the content of the background image in areas not displayed on the display device.

[0008] Therefore, the present invention, in one aspect, provides an image processing apparatus and an image processing method capable of assisting in the capture of in-camera VFX video while recognizing what a background image outside a shooting area is.

[0009] Solution to the problem

[0010] In one aspect, the present invention provides an image processing device, characterized in that it includes: a generating component for generating a first image based on the viewpoint of an image capture device; a display control component for displaying a portion of the first image as a background image in a space captured by the image capture device on a display device; and an output component for outputting a second image as a portion of the first image to an external device different from the display device, the second image including a portion of the first image corresponding to a capture area captured by the image capture device and having a field of view larger than the capture area.

[0011] Advantageous Effects of the Invention

[0012] According to one aspect of the present invention, it is possible to provide an image processing apparatus and an image processing method capable of assisting in capturing in-camera VFX video while recognizing what a background image outside a shooting area is.

[0013] Other features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings. Note that throughout the drawings, like reference numerals represent like or similar components. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0015] Figure 1 is a block diagram illustrating an example of a functional configuration of an image capturing apparatus according to an embodiment.

[0016] Figure 2 is a diagram schematically illustrating a virtual studio system according to the first embodiment.

[0017] Figure 3 is a block diagram illustrating an example of a functional configuration of a display control device according to an embodiment.

[0018] Figure 4A is a flowchart related to operations performed by the display control device according to the first embodiment.

[0019] Figure 4B is a flowchart related to operations performed by the display control device according to the first embodiment.

[0020] Figure 5 is a flowchart related to operations performed by the image capturing apparatus according to the first embodiment.

[0021] Figure 6A are diagrams illustrating operations performed by the display control device and the image capturing device according to the first embodiment.

[0022] Figure 6B are diagrams illustrating operations performed by the display control device and the image capturing device according to the first embodiment.

[0023] Figure 6C are diagrams illustrating operations performed by the display control device and the image capturing device according to the first embodiment.

[0024] Figure 7A are diagrams illustrating operations performed by the display control device and the image capturing device according to the first embodiment.

[0025] Figure 7Bare diagrams illustrating operations performed by the display control device and the image capturing device according to the first embodiment.

[0026] Figure 8 is a schematic diagram showing that the image capture direction has changed from Figure 2 Schematic diagram of the state of direction change shown in FIG.

[0027] Figure 9A are diagrams illustrating operations performed by the display control device and the image capturing device according to the first embodiment.

[0028] Figure 9B are diagrams illustrating operations performed by the display control device and the image capturing device according to the first embodiment.

[0029] Figure 9C are diagrams illustrating operations performed by the display control device and the image capturing device according to the first embodiment.

[0030] Figure 10A are diagrams illustrating operations performed by the display control device and the image capturing device according to the first embodiment.

[0031] Figure 10B are diagrams illustrating operations performed by the display control device and the image capturing device according to the first embodiment.

[0032] Figure 11A is a flowchart related to operations performed by the display control device according to the second embodiment.

[0033] Figure 11B is a flowchart related to operations performed by the display control device according to the second embodiment.

[0034] Figure 12 is a diagram schematically illustrating a virtual studio system according to a second embodiment.

[0035] Figure 13A are diagrams illustrating operations performed by the display control device and the image capturing device according to the second embodiment.

[0036] Figure 13B are diagrams illustrating operations performed by the display control device and the image capturing device according to the second embodiment.

[0037] Figure 13C are diagrams illustrating operations performed by the display control device and the image capturing device according to the second embodiment.

[0038] Figure 14A 2 are diagrams illustrating operations performed by the display control device and the image capturing device according to the second embodiment.

[0039] Figure 14B are diagrams illustrating operations performed by the display control device and the image capturing device according to the second embodiment.

[0040] Figure 15 It schematically illustrates that the viewing direction of the HMD has changed from Figure 12 Schematic diagram of a state where the viewing direction is changed.

[0041] Figure 16A are diagrams illustrating operations performed by the display control device and the image capturing device according to the second embodiment.

[0042] Figure 16B are diagrams illustrating operations performed by the display control device and the image capturing device according to the second embodiment.

[0043] Figure 17 is a schematic diagram illustrating that the image capture direction and the viewing direction of the HMD have changed from Figure 15 Schematic diagram of a state in which the image capturing direction and the viewing direction are changed.

[0044] Figure 18A are diagrams illustrating operations performed by the display control device and the image capturing device according to the second embodiment.

[0045] Figure 18B are diagrams illustrating operations performed by the display control device and the image capturing device according to the second embodiment.

[0046] Figure 18C are diagrams illustrating operations performed by the display control device and the image capturing device according to the second embodiment.

[0047] Figure 19A are diagrams illustrating operations performed by the display control device and the image capturing device according to the second embodiment.

[0048] Figure 19B are diagrams illustrating operations performed by the display control device and the image capturing device according to the second embodiment. DETAILED DESCRIPTION

[0049] Hereinafter, the present invention will be described in detail based on exemplary embodiments of the present invention with reference to the accompanying drawings. Note that the following embodiments do not limit the present invention as set forth in the scope of the patent claims. In addition, although several features are described in the embodiments, not all of these features are required for the present invention, and multiple features can be combined as needed. In addition, in the drawings, the same or similar configurations are given the same reference numerals, and their redundant descriptions will be omitted.

[0050] The following embodiments will describe the application of the present invention to an image capture device as an example of an image processing device. However, the image capture function is not essential to the present invention, and the present invention can be implemented in any electronic device. Examples of such electronic devices include computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, and game consoles. However, these are merely examples, and the present invention can also be applied to other electronic devices.

[0051] First embodiment

[0052] Figure 1 This is a block diagram illustrating an example configuration of a virtual studio system for capturing in-camera VFX video, focusing on an example functional configuration of the image capture device 100. The virtual studio system includes at least the image capture device 100, a display control device 300 that generates a background image based on the three-dimensional position and orientation of the image capture device 100, and a display device 400 that displays the background image in the space captured by the image capture device 100. The following configuration is illustrated: a viewpoint detection device 500 detects the three-dimensional position and orientation of the image capture device 100 and supplies this information to the display control device 300. However, this configuration may be such that the image capture device 100 supplies its own three-dimensional position and orientation to the display control device 300. A head-mounted display device (HMD) 600 is optional.

[0053] The image capture device 100 may be any device capable of capturing moving images, and is assumed here to be a digital video camera. The image capture lens 103 is a lens group including movable lenses such as a zoom lens, a focus lens, and a shift lens, and forms a subject image on an image capture surface. The system control unit 50 (described later) is capable of obtaining and adjusting the position of the movable lens of the image capture lens 103. The baffle 102 is capable of opening and closing, and protects the image capture lens 103. The aperture stop 101 is capable of opening to a variable value, and adjusts the amount of light entering from the image capture lens 103. A neutral density (ND) filter 104 can be selectively inserted into the optical path.

[0054] The image capture unit 22 includes an image sensor and converts the subject image formed by the image capture lens 103 into an analog image signal. The image sensor can be a well-known CCD or CMOS color image sensor having, for example, a primary color Bayer array filter. The image sensor includes a pixel array having a plurality of pixels arranged two-dimensionally and a peripheral circuit for controlling the operation of each pixel. Each pixel accumulates a charge corresponding to the amount of incident light through photoelectric conversion. By reading out a signal having a voltage corresponding to the amount of charge accumulated during the exposure period from each pixel, a group of pixel signals (analog image signals) representing the subject image formed on the image capture surface is obtained. The A / D converter 23 converts the analog image signal obtained by the image capture unit 22 into a digital image signal (image data). The A / D converter 23 outputs the image data to the image processing unit 24 or the memory control unit 15.

[0055] The image processing unit 24 applies predetermined image processing to the image data output by the A / D converter 23 or the memory control unit 15 to generate signals and image data for various purposes, and to obtain and / or generate various types of information. For example, the image processing unit 24 may be a dedicated hardware circuit, such as an application-specific integrated circuit (ASIC), designed to implement specific functions. Alternatively, the image processing unit 24 may be composed of a processor, such as a digital signal processor (DSP) or a graphics processing unit (GPU), that executes software to implement specific functions. The image processing unit 24 outputs the information and data obtained or generated to the system control unit 50, the memory control unit 15, etc., depending on the intended use.

[0056] For example, the image processing applied by the image processing unit 24 may include pre-processing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like.

[0057] Preprocessing includes signal amplification, reference level adjustment, defective pixel correction, etc.

[0058] Color interpolation processing is performed when the image sensor is provided with a color filter, and interpolates the values ​​of color components not included in the individual pixel data constituting the image data. Color interpolation processing is also called "demosaicing."

[0059] The correction processing may include white balance adjustment, color tone adjustment, correction of image degradation caused by optical aberration in the image capturing lens 103 (image restoration), correction of vignetting effects in the image capturing lens 103, color correction, and the like.

[0060] The detection process includes detecting a feature area (eg, a face area or a human body area) or motion in such an area, processing for recognizing a person, and the like.

[0061] Data processing may include cropping regions (trimming), combining, scaling, encoding and decoding, and header information generation (data file generation). Generation of display image data and recording image data is also included in data processing.

[0062] The evaluation value calculation process may include processes such as generating a signal, an evaluation value, etc. for automatic focus detection (AF), generating an evaluation value for automatic exposure control (AE), and the like.

[0063] Special effects processing includes adding bokeh effects, changing color tones, relighting, etc.

[0064] Note that these are examples of processing that can be applied by the image processing unit 24 and are not intended to limit the processing applied by the image processing unit 24 .

[0065] The system control unit 50 is, for example, a processor (CPU, MPU, microprocessor, etc.) capable of executing programs. The system control unit 50 controls the operation of each unit of the image capture device 100 by loading programs stored in the nonvolatile memory 56 into the system memory 52 and executing these programs, thereby realizing the functions of the image capture device 100.

[0066] The nonvolatile memory 56 is electrically rewritable. The nonvolatile memory 56 stores programs executed by the system control unit 50, values ​​of settings in the image capture apparatus 100, GUI data, and the like. The system memory 52 is used to load programs executed by the system control unit 50. Note that the memory 32 (described later) and the system memory 52 may be different areas in the same storage space.

[0067] The gyroscope 40 is a motion sensor that outputs a signal based on the angular velocity about each axis of a Cartesian coordinate system that includes the optical axis direction of the image capture device 100 and the direction of gravity. Note that an accelerometer that outputs a signal based on the motion in each axis direction may be used in conjunction with the gyroscope. The system control unit 50 can perform optical image stabilization by driving the shift lens based on the output of the gyroscope 40 to offset the motion of the image capture device 100. Note that the system control unit 50 may also perform electronic image stabilization.

[0068] The memory 32 is used as a buffer for display image data, recorded image data, intermediate image data generated during image processing by the image processing unit 24, image data read from the recording medium 200, audio data accompanying moving images, etc. A portion of the memory 32 is also used as a video memory for the display unit 28.

[0069] The D / A converter 13 converts the image data for display stored in the video area of ​​the memory 32 into an analog signal that can be displayed by the display unit 28, and supplies the analog signal to the display unit 28. The display image data is displayed by the display unit 28 in this manner. The display unit 28 can be, for example, a liquid crystal display (LCD) or an organic EL display. For example, the display unit 28 is provided on the surface of the housing of the image capture device 100 or inside thereof.

[0070] By continuously capturing moving images and displaying the captured moving images, the display unit 28 can be used as an electronic viewfinder (EVF). The moving images when the display unit 28 is used as an EVF are referred to as "live view images."

[0071] The system timer 53 measures time using an internal clock. The system control unit 50 can use the time measured by the system timer 53 for various types of control. The mode change switch 60, the recording switch 61, and the operation unit 70 are input devices through which the user of the image capture apparatus 100 inputs various types of instructions.

[0072] For example, the mode change switch 60 sets the image capture device 100 to one of predetermined operating modes, such as a moving image recording mode, a playback mode, and the like. The record switch 61 switches between an image capture standby state for moving images and an image capture state. When the system control unit 50 detects that the record switch 61 has been operated in the image capture standby state, it controls various units to execute operations involved in moving image recording. Then, when the system control unit 50 detects that the record switch 61 has been operated while a moving image is being recorded, it controls various units to stop operations involved in moving image recording.

[0073] The "operation unit 70" is a general term for input devices (buttons, switches, dials, etc.) provided in the image capture apparatus 100. The input devices constituting the operation unit 70 are named according to the functions assigned thereto. For example, the operation unit 70 includes a menu button, a direction key, a setting key, etc. Note that the functions assigned to the same input device may be variable. In addition, the input device may include software buttons or keys using a touch screen. The operation unit 70 may also include input devices corresponding to non-contact input methods such as voice input, gaze input, etc.

[0074] The power supply control unit 80 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching the blocks to be energized, and the like. The power supply control unit 80 detects whether a battery is attached, the type of battery, and the remaining battery charge. The power supply control unit 80 also controls the DC-DC converter based on the detection result and instructions from the system control unit 50, and supplies the required voltage for the required period to various units including the recording medium 200.

[0075] The power supply unit 30 may be a battery or an AC adapter. The interface (I / F) 18 connects the image capture device 100 (system control unit 50) to the recording medium 200, which may be, for example, a memory card or an external device such as the display control device 300, in a communicative manner. The I / F 18 has a configuration that conforms to the standard of the connected device. The system control unit 50 can transmit information from the image capture device 100 to an external device such as the display control device 300, and receive information and instructions from the external device, via the I / F 18.

[0076] The viewpoint detection device 500 is a device that detects the viewpoint (three-dimensional position, orientation, or image capturing direction) of the image capture device 100. The viewpoint detection device 500 can detect the viewpoint of the image capture device 100 using any well-known method. For example, the viewpoint detection device 500 can detect the viewpoint of the image capture device 100 based on the image coordinates of a marker arranged at a known three-dimensional position in the shooting space. Note that the orientation of the image capture device 100 can be detected using the output of the gyroscope 40. The viewpoint detection device 500 is attached to the image capture device 100 so that its position relative to the image capture device 100 does not change. The viewpoint detection device 500 continuously detects the viewpoint and supplies the detection result to the display control unit 300 through the image capture device 100. Note that the viewpoint detection device 500 can supply the detection result directly to the display control unit 300.

[0077] Based on the viewpoint information of the image capture device 100 and the installation position of the display device 400 that displays the background image in the image capture space, the display control device 300 generates a background image to be displayed on the display device 400. The background image is an image of the virtual space at the position of the display surface of the display device 400 when the virtual space is viewed from the viewpoint of the image capture device 100. The background image can be generated by projecting an image of a three-dimensional model of the virtual space that has been rendered using the viewpoint of the image capture device 100 onto the display surface of the display device 400.

[0078] Figure 2 It is a diagram showing the use of Figure 1 Schematic diagram of in-camera VFX video capture of a virtual studio system illustrated in The figure illustrates the virtual studio from above.

[0079] The image capture apparatus 100 held by the photographer 800 records an in-camera VFX video by capturing an image of a person 210 as a main subject using the image displayed on the display device 400 as a background. A dotted line indicates the horizontal field of view of the image capture apparatus 100.

[0080] The display control device 300 is connected to the image capture device 100 and the display device 400, and generates a background image, an image for assisting in the capture of in-camera VFX video by the image capture device 100, and the like. The viewpoint detection device 500 is attached to the image capture device 100, and detects the viewpoint of the image capture device 100. The detection result from the viewpoint detection device 500 is supplied to the display control device 300 via the image capture device 100 or directly.

[0081] Note that a synchronization signal is supplied from a reference clock generating device (also referred to as a synchronization generator) to the image capture device 100, the viewpoint detection device 500, the display device 400, and the display control device 300. The image capture period of the image capture device 100 and the display period of the display device 400 are synchronized because each of these devices controls its operation timing based on the reference clock. A technique for synchronizing operations between devices based on the reference clock is called generator locking, for example, and therefore these operations will not be described in detail.

[0082] Figure 3 3 is a block diagram illustrating an example of a functional configuration of the display control apparatus 300. The display control apparatus 300 can be realized using a computer device, for example.

[0083] The control unit 301 is, for example, a CPU, and implements the functions of the display control device 300 by loading one or more application programs stored in the ROM 307 into the RAM 308 and executing these programs. Note that the control unit 301 controls the timing of operations performed by the display control device 300 based on a synchronization signal supplied from a reference clock generating device.

[0084] The image processing circuit 302 is, for example, a graphics board including a GPU. The image processing circuit 302 can perform image processing such as rendering a computer graphic image (CG image) at high speed.

[0085] The first through fourth I / Fs 303 through 306 are communication interfaces for connecting external devices. This embodiment assumes that the image capture device 100 is connected to the first I / F 303, the display device 400 is connected to the second I / F 304, the viewpoint detection device 500 is connected to the third I / F 305, and the optional head-mounted display device 600 is connected to the fourth I / F. The first through fourth I / Fs 303 through 306 are assumed to conform to standards based on the external devices to be connected and the types of signals to be communicated. While the display control device 300 and the external devices are illustrated as being connected via a single I / F for simplicity, these devices may be connected using multiple I / Fs.

[0086] The control unit 301 obtains captured image data from the image capture device 100 via the first I / F 303. When obtaining information about the viewpoint of the image capture device 100 from the image capture device 100, the control unit 301 obtains this information via the first I / F 303. However, the control unit 301 may obtain information about the viewpoint of the image capture device 100 from the viewpoint detection device 500 through communication via the third I / F 305. The control unit 301 outputs image data for display (background image data) to the display device 400 via the second I / F 304. The control unit 301 may also output captured display image data of the auxiliary in-camera VFX video to the image capture device 100 via the first I / F 303, or output it to the HMD 600 via the fourth I / F 306. Note that the display control device 300 may have at least five interfaces for communicating with external devices.

[0087] The ROM 307 stores some programs (BIOS, boot loader, firmware, etc.) executed by the control unit 301 , setting values ​​of the display control device 300 , and the like.

[0088] The RAM 308 is used as a main memory of the control unit 301 as well as a work memory for the image processing circuit 302 and a video memory for the display unit 310 .

[0089] The storage unit 309 is a large-capacity storage device such as a hard disk, SSD, etc. The storage unit 309 stores basic software (OS), application programs, user data, etc. An application program (e.g., a game engine application) that generates a background image based on the viewpoint of the image capture device 100 and data required for generating the background image (three-dimensional model of the virtual space, textures, etc.) are also stored in the storage unit 309.

[0090] For example, the display unit 310 is a liquid crystal display device. The display unit 310 may be a touch screen. The display unit 310 displays a background image generation application (eg, a game engine), a GUI provided by an OS, and the like.

[0091] The operation unit 311 includes a plurality of input devices that can be operated by a user, such as a keyboard, a mouse, a touch panel, etc. If the display unit 310 is a touch screen, the touch panel is a constituent element of the operation unit 311 .

[0092] Will refer to Figure 4A and Figure 4B The flowchart illustrated in describes operations performed by the display control apparatus 300. Unless otherwise specified, the operations described below are realized by the control unit 301 of the display control apparatus 300 executing programs stored in the ROM 307 and / or the storage unit 309.

[0093] In step S301, the control unit 301 instructs the image capture device 100 and the display device 400 to operate in the in-camera VFX mode. This instruction can be performed using any known method, such as sending a command, changing the voltage at a specific terminal, etc. In the in-camera VFX mode, the image capture device 100 and the display device 400 operate at a timing based on a synchronization control signal (genlock signal) sent from the display control device 300. If the viewpoint detection device 500 is independent of the image capture device 100, the control unit 301 also instructs the viewpoint detection device 500 to operate in the in-camera VFX mode.

[0094] In step S302, the control unit 301 obtains information about the shooting area (field of view of the image capture lens 103) and the viewpoint (three-dimensional position and orientation) of the image capture device 100. Unless otherwise specified, it is assumed that the field of view is expressed using the lens focal length (unit: mm) converted to 35 mm. If the image capture lens 103 is a fixed focal length lens, the information about the field of view can be set in advance and therefore does not need to be obtained. As a detection result of the viewpoint detection device 500, the information about the viewpoint of the image capture device 100 can be obtained directly from the viewpoint detection device 500 or through the image capture device 100.

[0095] In step S303, the control unit 301 uses the image processing circuit 302 to set a virtual camera with the viewpoint of the image capture device 100, and generates a virtual space image (first image) by rendering the three-dimensional model of the virtual space stored in advance in the storage unit 309. As a result, a virtual space image is generated in which the virtual space represented by the three-dimensional model is viewed from the viewpoint of the image capture device 100. The three-dimensional model is information representing the three-dimensional shape of a virtual object that exists farther from the viewpoint of the image capture device 100 than the display device 400, and can be written in any known format.

[0096] Note that the size (the number of pixels in the horizontal and vertical directions) of the virtual space image generated in step S303 is larger than the display resolution in the horizontal and vertical directions of the display device 400. In other words, the control unit 301 generates a virtual space image having an area wider than the area displayed as the background image on the display device 400.

[0097] In step S304, using the image processing circuit 302, the control unit 301 generates a background image to be displayed on the display device 400 from the virtual space image generated in step S303, based on the relationship between the position where the display surface of the display device 400 exists and the viewpoint of the image capture device 100. For example, when the image capture direction of the image capture device 100 is orthogonal to the display surface of the display device 400, the image processing circuit 302 can generate the background image by cropping an image the size of the display area from the virtual space image based on the viewpoint of the image capture device 100. If the image capture direction of the image capture device 100 is not orthogonal to the display surface of the display device 400, the image processing circuit 302 generates the background image by applying a geometric transformation as needed so that the image is viewed from a position directly opposite the display device 400.

[0098] In step S305 , the control unit 301 sends the background image to the display device 400 .

[0099] In step S306, the control unit 301 determines whether the display of the wide-angle background image is enabled, and if so, performs step S313, and if not, performs step S310. For example, whether the display of the wide-angle background image is enabled can be determined by confirming the settings of the application, but any other method can be used.

[0100] In step S313 , the control unit 301 obtains the field of view setting of the wide-angle background image from, for example, the ROM 307 or the storage unit 309 .

[0101] In step S314, the control unit 301 obtains the image of the previous frame captured by the image capture device 100 from the image capture device 100. Note that this step does not need to be performed when the captured image is not combined with the wide-angle background image, or when the combination is performed by the image capture device 100. Alternatively, the image may be obtained at another time, such as in step S320, and stored in the RAM 309.

[0102] In step S315, the control unit 301 generates a wide-angle background image using the image processing circuit 302. The wide-angle background image is an image of the virtual space viewed from the viewpoint of the image capture device 100. Unlike when generating a background image, when generating a wide-angle background image, the distance or angle between the display device 400 and the image capture device 100 does not need to be considered.

[0103] In step S316, the control unit 301 generates a combined image in which the captured image from the previous frame or the indicator indicating the capture area captured by the image capture device 100 has been combined with the wide-angle background image using the image processing circuit 302. Note that when the combination is performed by the image capture device 100, step S316 does not need to be performed.

[0104] In step S317, the control unit 301 sends the combined image (or wide-angle background image) to the image capture device 100. Note that the destination need not be the image capture device 100, and may instead be another device that displays images (eg, the HMD 600).

[0105] In step S310, control unit 301 transmits a genlock signal as an instruction for capturing and displaying an image to image capture apparatus 100 and display device 400. As a result, updating of the background image in display device 400 and capturing of the next frame by image capture apparatus 100 are synchronously performed.

[0106] In step S319, the control unit 301 determines whether to continue capturing in-camera VFX video, and if so, repeats the operation starting from step S302, and if not, ends the process. Figure 4A and Figure 4B Whether to continue capturing images may be determined based on any desired condition, such as whether a predetermined image capture sequence has ended, whether image capture device 100 has made a notification indicating that capture has ended, or the like.

[0107] Next, we will refer to Figure 5 The flowchart in describes operations performed in the in-camera VFX mode by the image capture apparatus 100. Unless otherwise mentioned, the operations described below are implemented by the system control unit 50 of the image capture apparatus 100 executing a program stored in the nonvolatile memory 56.

[0108] In S401, the system control unit 50 determines whether a genlock signal has been received from the display control device 300. If so, S403 is executed. If not, S401 is repeated. In in-camera VFX mode, the system control unit 50 captures one frame of imagery using the genlock signal as an image capture start instruction. This allows image capture to be synchronized with the display in the display device 400, which also operates based on the genlock signal.

[0109] In S403, the system control unit 50 captures an image for one frame using the image capture unit 22. The system control unit 50 controls the image processing unit 24 and the like to generate one frame's worth of image data, which is then stored in the memory 32. Note that the image data generated in step S403 is image data for recording and image data for display if recording is in progress, and is image data for display when in the standby state.

[0110] In S405, the system control unit 50 transmits information about the state of the image capture device 100 during image capture to the display control device 300 via the I / F 18. It is assumed that the information transmitted here includes at least information about the field of view (the focal length of the image capture lens 103). Other information such as whether the display of the wide-angle background image is enabled or not, the field of view setting of the wide-angle background image, information about exposure, and information about the three-dimensional position and orientation of the image capture device 100 detected by the viewpoint detection device 500 may also be included as needed.

[0111] If the display control apparatus 300 generates a combined image of the wide-angle background image and the captured image, the data of the captured image generated in S403 may be transmitted to the display control apparatus 300, for example, between S405 and S407 in response to a request from the display control apparatus 300. Alternatively, the data may be transmitted together with information about the field of view at the time of image capture in step S405.

[0112] When the captured image data is transmitted to the display control device 300, the captured image data may be transmitted after being reduced in size according to the display size of the combined image or the wide-angle background image by the image processing unit 24. The reduction ratio of the captured image may be determined based on the difference between the field of view of the capture area and the field of view of the wide-angle background image.

[0113] In S407 , the system control unit 50 receives the wide-angle background image from the display control device 300 through the I / F 18 , and stores the wide-angle background image in the memory 32 .

[0114] In S409, using the image processing unit 24, the system control unit 50 generates a combined image in which the image indicating the current capture area is combined with the wide-angle background image. For example, the system control unit 50 generates a combined image in which the image of the frame captured in S403, the indicator indicating the capture area, etc. are combined at corresponding positions in the wide-angle background image. When combining the captured images, the images are reduced in the same manner as when they are sent to the display control device 300, and aligned with the wide-angle background image. As shown in FIG. Figure 4A and Figure 4BAs described in the flowchart in , if the display control apparatus 300 is to generate a similar combined image, the combined image has already been received in S407 , and therefore there is no need to perform generation in S409 .

[0115] In step S411, the system control unit 50 stores the combined image in the video storage area of ​​the memory 32, and displays the combined image on the display unit 28 through the D / A converter 13. Note that the combined image may be displayed on an external display device connected to the image capturing apparatus 100, or on a display device different from the display unit 28 provided in the image capturing apparatus 100.

[0116] In S413, the system control unit 50 determines whether to continue capturing in-camera VFX video, and if so, repeats the operation from S401, and if not, ends the process. Figure 5 Whether to continue capturing video can be determined based on any desired conditions, such as whether an instruction to end capturing an image has been given by operating the operation unit 70 or the record switch 61, whether the display control device 300 has made a notification indicating that capturing has ended, and the like.

[0117] Will refer to Figures 6A to 6C To simplify the description and facilitate understanding, the following examples are given: Figure 6A As illustrated in FIG, it is assumed here that the virtual space is a virtual space in which a virtual object such as a building exists, and that the image capture device 100 is facing the virtual object. It is also assumed that the virtual object appears larger than the display surface at a distance from the display device 400 of the image capture device 100. In this manner, only a portion of the virtual object is displayed on the display device 400, and the photographer cannot recognize the entire virtual object simply by observing the display device 400.

[0118] Figure 6B This corresponds to a region displayed in the display device 400 of a virtual space image generated by the display control apparatus 300 and to be observed from the viewpoint of the image capture apparatus 100 at the position of the display surface of the display device 400 .

[0119] Figure 6C Illustrated is an example of the relationship between a background image and a capture area captured by the image capture device 100. The area surrounded by a dotted line is the capture area, and an in-camera VFX video including a real object such as the illustrated real object is captured.

[0120] Typically, image capture device 100 generates a live view image based on captured images, and therefore does not include areas outside the capture area. Therefore, based on the display of display unit 28, photographer 800 is unable to identify not only the portion of the virtual space image not displayed on display device 400, but also the background image outside the capture area. While capturing images while viewing both display device 400 and display unit 28 does allow for identification of the background image outside the capture area, it still makes it impossible to grasp the portion of the virtual space image not displayed as the background image.

[0121] Figure 7A The diagram shows Figures 6A to 6C An example of the relationship between the background image, wide-angle background image, and capture area generated according to this embodiment under the same assumptions. Figure 7A In the example illustrated in , the wide-angle background image is an area in the virtual space image that includes the background image and is larger than the background image. The size (field of view) of the wide-angle background image may be set in advance, or may be dynamically changeable in response to an instruction made through the operating unit 70 of the image capture device 100. Alternatively, the size may be automatically set using the current field of view of the image capture device 100 as a reference. The field of view of the wide-angle background image is set to be no less than the current field of view of the image capture device 100. The wide-angle background image is an image whose center (optical axis position) is the same as the center of the captured image but whose field of view is different. Therefore, the wide-angle background image is an image that includes the surrounding area outside the captured area.

[0122] Figure 7B The diagram shows Figure 7A An example of a combined image in which the captured image is combined with the wide-angle background image in the state illustrated in FIG. Note that, instead of the captured image, the combined image may be an indicator indicating the captured area (e.g., Figure 7A or Figure 7B In order to naturally combine the wide-angle background image with the captured image, image processing such as geometric transformation may be applied to the captured image as needed before combination, and image processing may be applied to the boundary portion of the image after combination. The photographer 800 can set an appropriate capture area while confirming the state of the capture area's surroundings by, for example, operating the image capture device 100 while viewing the combined image displayed on the display unit 28.

[0123] Figure 8 The diagram schematically shows that the photographer 800 has Figure 2 The state illustrated in FIG is moved to the right and the image capture device 100 no longer faces the display device 400. In this case, the angle of the virtual object viewed from the image capture device 100 is as follows: Figure 9AIn addition, because the image capture device 100 no longer faces the display apparatus 400, the background image viewed from the image capture device 100 has a different image quality than that of the image capture device 100. Figure 9B The relationship between the background image and the capture area becomes as follows Figure 9C Middle picture.

[0124] Figure 10A The diagram shows Figure 8 An example of the relationship between the background image, the wide-angle background image, and the capture area in the state illustrated in FIG. Figure 10B The diagram shows Figure 10A An example of a combined image in the state illustrated in . In this way, a wide-angle background image and a combined image that follow the change of the viewpoint of the image capturing device 100 are obtained.

[0125] First Modification

[0126] This embodiment has described a case where a portion of an image generated by the display control device based on the viewpoint of the image capture device is displayed on the display device 400. However, the capture area of ​​in-camera VFX video is typically set to an area smaller than the display area of ​​the display device 400. Therefore, the wide-angle background image can be any image that includes the capture area and is larger than the capture area. Therefore, for example, a background image can be used as a wide-angle background image.

[0127] Second Modification

[0128] In this embodiment, a combined image is displayed in which the captured image or an indicator indicating the captured area is combined with the wide-angle background image. However, even if the wide-angle background image is displayed as is, it is still possible to identify areas that are not visible in the background image. In particular, a configuration in which the wide-angle background image is displayed as is is suitable for a situation in which the display device displaying the live view image and the display device displaying the wide-angle background image are separate devices.

[0129] Third Modification

[0130] Instead of a virtual space image as a computer graphics image, an image captured by a separate image capture device having a viewpoint synchronized with that of the image capture device 100 may be used. In this case, the field of view of the separate image capture device is set so as to capture an area larger than the background image. The field of view of the separate image capture device may be changed according to the field of view setting of the wide-angle background image.

[0131] As described above, according to this embodiment, a wide-angle background image is generated to assist image capture by using an image based on the image capture device's viewpoint as a background image. The wide-angle background image is an image based on the image capture device's viewpoint that includes the capture area and has a field of view larger than that of the capture area. As a result, the photographer can operate the image capture device to appropriately change the capture area of ​​the in-camera VFX video while using the wide-angle background image to identify the state of the area surrounding the capture area.

[0132] Second embodiment

[0133] Next, a second embodiment of the present invention will be described. In the first embodiment, the viewpoint of the wide-angle background image is the viewpoint of the image capture device 100. However, in this embodiment, the wide-angle background image (second image) is generated from a viewpoint (second viewpoint) different from the viewpoint of the image capture device 100.

[0134] Figure 12 So with Figure 2 , a schematic diagram of capturing in-camera VFX video in a virtual studio system according to the present embodiment is illustrated in the same manner as in . The present embodiment assumes that a wide-angle background image is generated from the viewpoint of a head-mounted display (HMD) 600 worn by a photographer 800. Because the user wears the HMD 600 on their head, the viewpoint of the HMD 600 can also be regarded as the viewpoint of the photographer 800. However, another viewpoint such as the viewpoint of another member operated by the photographer may be used. In Figure 12 , two two-dot chain lines extending from the HMD 600 indicate the horizontal field of view of the wide-angle background image displayed in the HMD 600.

[0135] The HMD 600 has a function for detecting the viewpoint (three-dimensional position and position / orientation) of the HMD 600. In response to a request from the display control device 300 (or automatically at a predetermined timing), information about the detected viewpoint is output to the display control device 300. Note that, like the viewpoint of the image capture device 100, a separate viewpoint detection device may be used to detect the viewpoint of the HMD 600. In this case, information about the detected viewpoint is output from the viewpoint detection device to the display control device 300. Figure 12 It is assumed that the viewing direction of the HMD 600 is parallel to the image capturing direction (optical axis direction) of the image capturing device 100 .

[0136] Figure 11A and Figure 11B300 is a flowchart related to the operation performed by the display control device 300 according to the present embodiment. Unless otherwise specified, the operation described below is realized by the control unit 301 of the display control device 300 executing the program stored in the ROM 307 and / or the storage unit 309. In addition, the steps for performing the same operation as in the first embodiment are given the same Figure 4A and Figure 4B The same reference numerals as in the reference numerals in FIG.

[0137] The following will focus on the differences from the first embodiment. When the display of the wide-angle background image is not enabled, the operation is the same as in the first embodiment and will not be described here. Even when the display of the wide-angle background image is enabled, the operations of steps S301 to S307, S313, and S314 are the same as in the first embodiment. If the captured image is not to be combined with the wide-angle background image, step S314 does not need to be performed. Note that the field of view of the wide-angle background image is set within the horizontal and vertical fields of view of the HMD 600.

[0138] In step S1001, the control unit 301 obtains information about the viewpoint of the HMD 600. This information may be obtained from the HMD 600 or from a separate viewpoint detection device.

[0139] In step S1003, using the image processing circuit 302, the control unit 301 generates a wide-angle background image of the virtual space viewed from the viewpoint of the HMD 600. This is the same as the generation of the wide-angle background image performed in step S315 in the first embodiment, except that the viewpoint is different.

[0140] In step S316, the control unit 301 generates a combined image in which the captured image from the previous frame or the indicator indicating the capture area captured by the image capture device 100 is combined with the wide-angle background image using the image processing circuit 302. Note that the resolution of the combined image can be adjusted according to the display resolution of the HMD 600.

[0141] In step S1005, the control unit 301 transmits the combined image to the HMD 600. The processing thereafter is the same as in the first embodiment.

[0142] According to the present embodiment, the image capturing device 100 does not display the wide-angle background image, and therefore does not perform the operations related to the wide-angle background image described in the first embodiment ( Figure 5 S407 to S411 in ).

[0143] The HMD 600 repeatedly transmits information about the viewpoint, and displays the combined image received from the display control device 300 .

[0144] Hereinafter, a specific example of display will be described. In order to simplify the description and facilitate understanding, as shown in FIG. Figure 13A As illustrated in FIG, it is assumed here that the virtual space is a virtual space in which a virtual object of a building exists, and the image capture device 100 and the HMD 600 are facing the virtual object. As in the first embodiment, it is also assumed that the virtual object appears larger than the display surface at a distance from the display device 400 of the image capture device 100. In this way, a portion of the virtual object is displayed on the display device 400. Figure 13B , and the photographer cannot recognize what the entirety of the virtual object is simply by observing the display device 400 . Figure 13C Illustrated is an example of the relationship between a background image and a capture area captured by the image capture device 100. The area surrounded by a dotted line is the capture area, and an in-camera VFX video including a real object such as the illustrated real object is captured.

[0145] Figure 14A The diagram shows 13A to 13C An example of the relationship between the background image, the wide-angle background image, and the capture area generated according to this embodiment under the same assumptions. Figure 12 As illustrated in , the image capturing apparatus 100 and the HMD 600 are facing the display device 400, so the background image, the wide-angle background image, and the capture area are represented as rectangular areas having different centers.

[0146] In this embodiment, the viewpoint of the wide-angle background image is the viewpoint of the HMD 600 rather than the viewpoint of the image capture device 100, so the wide-angle background image does not coincide with the center of the capture area. The wide-angle background image has a wider field of view than the capture area and is Figure 14A The example shown in includes a background image.

[0147] Figure 14B The diagram shows Figure 14A An example of a combined image in which the captured image is combined with the wide-angle background image in the state illustrated in FIG. Note that, instead of the captured image, the combined image may be an indicator indicating the captured area (e.g., Figure 14A or Figure 14B The photographer 800 wearing the HMD 600 can set an appropriate capture area while confirming the state around the capture area by operating the image capture device 100 while viewing the combined image.

[0148] Figure 15 Schematically illustrates the Figure 12 The state illustrated in FIG is a state in which the photographer 800 moves their sight line to the right without moving the image capture device 100. In this case, the angle and visible area of ​​the virtual object viewed from the photographer 800 change.

[0149] Figure 16A The diagram shows Figure 15 An example of the relationship between the background image, wide-angle background image, and capture area in the state shown in FIG. Note that, Figure 16A The wide-angle background image in indicates that the area of ​​the wide-angle background image displayed in the HMD 600 is moving to the right relative to the background image and the capture area, and to make the figure easier to see, the change in the angle of the virtual object is not indicated.

[0150] Figure 16B The diagram shows Figure 16A , an example of a combined image in the state illustrated in . In practice, before combining, the captured image can be transformed according to the angle of the virtual object in the wide-angle background image. Because the viewing direction of HMD 600 is rotating rightward relative to the image capture direction of image capture device 100, the left side of the capture area is outside the field of view of the wide-angle background image. However, photographer 800 can recognize the state of the virtual space to the right of the capture area over a wide range without expanding the field of view of the wide-angle background image.

[0151] Figure 17 The following state is schematically illustrated: for example, based on the state of the virtual space recognized from the wide-angle background image, the photographer 800 has changed the position and image capturing direction from Figure 15 The state illustrated in FIG is changed so that the subject 210 is captured from the right at a certain angle. Here, the viewing direction of the HMD 600 is also changed to be approximately parallel to the image capturing direction of the image capturing device 100.

[0152] In this case, the angle of the virtual object viewed from the image capture device 100 is as follows: Figure 18A In addition, because the image capture device 100 no longer faces the display apparatus 400, the background image viewed from the image capture device 100 has a different image quality than that of the image capture device 100. Figure 18B The relationship between the background image and the capture area becomes as follows Figure 18C Middle picture.

[0153] Figure 19A The diagram shows Figure 17 An example of the relationship between the background image, wide-angle background image, and capture area in the state shown in FIG. Figure 19A and Figure 19B , the difference in shape between the wide-angle background image and the captured image (capture area) caused by the parallax between the image capture device 100 and the HMD 600 is also not illustrated.

[0154] Figure 19B The diagram shows Figure 19AAn example of a combined image in the state illustrated in . The parallax between the image capture device 100 and the HMD 600 is reduced, so the capture area is now included in the wide-angle background image.

[0155] According to this embodiment, in addition to the effects of the first embodiment, generating a wide-angle background image from a viewpoint independent of the viewpoint of the image capture device makes it possible to recognize the state of the virtual space over a wider range than in the first embodiment. The modification described in the first embodiment can also be applied to this embodiment.

[0156] Note that in this embodiment, when the parallax between the HMD 600 and the image capture device 100 increases, a situation may arise where the capture area is no longer included in the wide-angle background image. In this case, the position of the capture area cannot be grasped from the wide-angle background image. Therefore, if the capture area is not included in the wide-angle background image at all, an indicator (e.g., an arrow) indicating the direction in which the capture area exists may be displayed on the HMD 600 as a combined image obtained by combining it with the wide-angle background image.

[0157] Other embodiments

[0158] The present invention can be implemented by supplying a program for implementing one or more functions of the above-described embodiments to a system or device via a network or storage medium, and causing one or more processors in a computer of the system or device to read and execute the program. The present invention can also be implemented by a circuit (e.g., an ASIC) for implementing one or more functions.

[0159] The present invention is not limited to the contents of the aforementioned embodiments, and many changes and modifications can be made without departing from the spirit and scope of the present invention. Therefore, the accompanying claims are intended to inform the public of the scope of the present invention.

[0160] This application claims the benefit of Japanese Patent Application No. 2023-28845, filed on February 27, 2023, which is hereby incorporated by reference herein.

Claims

1. An image processing device, characterized in that include: a generating component for generating a first image based on a viewpoint of an image capturing device; a display control unit for displaying a portion of the first image on a display device as a background image in the space captured by the image capture device; as well as An output component for outputting a second image as part of the first image to an external device different from the display device, the second image including a part of the first image corresponding to the capture area captured by the image capture device and having a larger field of view than the capture area.

2. An image processing device, characterized in that include: generating means for generating a first image based on a first viewpoint as a viewpoint of an image capturing device and a second image based on a second viewpoint different from the first viewpoint; a display control unit for displaying a portion of the first image on a display device as a background image in the space captured by the image capture device; as well as an output component for outputting the second image to an external device different from the display device, The second image has a larger field of view than the first image.

3. The image processing device according to claim 2, wherein The second viewpoint is a viewpoint of a user operating the image capturing device.

4. The image processing device according to claim 2 or 3, characterized in that In a case where the second image does not include the capture area captured by the image capture device, the output section outputs the second image in which an indicator indicating a direction in which the capture area exists is combined with the second image.

5. The image processing device according to any one of claims 1 to 4, characterized in that The external device is the image capturing apparatus.

6. The image processing device according to any one of claims 1 to 4, characterized in that The external device is a head mounted display (HMD) worn by a user who operates the image capturing apparatus.

7. The image processing device according to any one of claims 1 to 6, characterized in that The output section outputs the second image in which an indicator indicating the capture area captured by the image capture device is combined with the second image.

8. The image processing device according to any one of claims 1 to 6, characterized in that The output section outputs the second image in which the image captured by the image capture device is combined with the second image.

9. The image processing device according to any one of claims 1 to 8, characterized in that The second image includes a portion that is not included in the background image.

10. The image processing device according to any one of claims 1 to 9, characterized in that The first image is a computer graphics image generated based on a three-dimensional model of a virtual space.

11. An image capturing device, characterized in that include: an obtaining component for obtaining a second image from the image processing apparatus according to any one of claims 1 to 10; as well as The display component is configured to display the second image.

12. The image capture device according to claim 11, wherein A display component different from the display component is further included for displaying a live view image.

13. A virtual studio system, characterized in that include: The image processing device according to any one of claims 1 to 10; The image capture device according to claim 11 or 12; as well as A display device displays a background image.

14. An image processing method performed by an image processing device, the image processing method comprising: generating a first image based on a viewpoint of an image capture device; displaying, on a display device, a portion of the first image as a background image in a space captured by the image capture device; as well as A second image is output as a portion of the first image to an external device different from the display device, the second image including a portion of the first image corresponding to a capture area captured by the image capture device and having a larger field of view than the capture area. 15 . A program for causing a computer to function as each component in the image processing apparatus according to claim 1 .

Images