Imaging system, image processing method, imaging method, program, and storage medium

Abstract

The goal is to record footage that is easy to edit later, when capturing real-world subjects with a displayed image as the background. [Solution] The system comprises a display means, an imaging means for capturing and outputting an image that includes at least a portion of the display area of ​​the display means in its field of view, a determination means for determining whether the image includes an image of an area outside the display area, and a processing means for replacing the image of the external area with an image of a predetermined color if it is determined that the image includes an image of the external area.

Description

Technical Field

[0001] The present invention relates to an imaging system, an image processing method, an imaging method, a program, and a storage medium, and particularly relates to in-camera VFX technology.

Background Art

[0002] A method (in-camera VFX) is known that obtains VFX (Visual Effects) footage without compositing background footage and real-shot footage, by imaging a subject with footage corresponding to the position and orientation of the camera as the background (Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of Invention

Problems to be Solved by the Invention

[0004] When using footage displayed on an LED wall as a background, depending on the shooting distance from the camera to the LED wall and the zoom state of the camera lens, an area outside the LED wall may fall within the imaging range of the camera.

[0005] In that case, possible countermeasures include replacing the area outside the LED wall with CG through post-editing, or cropping and resizing the footage to exclude the area outside the LED wall. However, performing such processes manually is extremely labor-intensive, and even if such processes are performed automatically using image processing technology, there is a possibility that clipping errors may occur at the boundary between the inside and outside of the LED wall.

[0006] The present invention has been made in view of the above problems, and an object of the present invention is to enable recording of footage that is easy to edit later when a real subject is imaged using displayed footage as a background. [Means for solving the problem]

[0007] To achieve the above objective, the imaging system of the present invention comprises: a display means; an imaging means for capturing and outputting an image whose field of view includes at least a part of the display area of ​​the display means; a determination means for determining whether the image includes an image of an area outside the display area; and a processing means for replacing the image of the external area with an image of a predetermined color if it is determined that the image includes an image of the external area. [Effects of the Invention]

[0008] According to the present invention, when capturing images of a real subject with a displayed image as the background, it is possible to record images that are easy to edit later. [Brief explanation of the drawing]

[0009] [Figure 1] A schematic diagram of a virtual studio system according to the first embodiment of the present invention. [Figure 2] A block diagram showing the functional configuration of the camera and the connection relationships of each device in the first embodiment. [Figure 3] A block diagram showing the functional configuration of the imaging system control unit and the connection relationships of each device in the first embodiment. [Figure 4] A flowchart illustrating the operation during image capture of in-camera VFX footage in the first embodiment. [Figure 5] A diagram showing an example of the relationship between the camera's imaging range and the display device in the first embodiment. [Figure 6] A diagram showing an example of in-camera VFX footage captured in the first embodiment. [Figure 7] A diagram showing the video area to be overwritten with green screen footage in the first embodiment. [Figure 8] A flowchart illustrating the operation during image capture of in-camera VFX footage in the second embodiment. [Figure 9]A diagram showing the video area to be overwritten with green screen footage in the second embodiment. [Figure 10] A flowchart illustrating the switching of images displayed on the display device in the third embodiment. [Figure 11] A diagram showing an example of in-camera VFX footage recorded in the third embodiment. [Modes for carrying out the invention]

[0010] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0011] Figure 1 is a schematic diagram of the virtual studio system in this embodiment. In the virtual studio 100, the camera 200 captures VFX images by imaging a real subject 180 with the image displayed on a large display device 160, also known as an LED wall, as the background.

[0012] The imaging range 170 is an example of the range (angle of view) captured by camera 200 within the background image. Since camera 200's position, orientation, and zoom level can be freely changed within the virtual studio system, the imaging range 170 will change accordingly.

[0013] The viewpoint detection device 120 calculates the viewpoint (position and orientation) of camera 200 from the absolute coordinates of the viewpoint detection marker 130 installed on the ceiling and the position of the viewpoint detection marker 130 included in the image of the ceiling captured by the viewpoint detection camera installed on camera 200.

[0014] The display device position acquisition markers 191 to 194 are installed on the outer peripheries of the top, bottom, left, and right of the display area of the display device 160. The display device position acquisition markers 191 to 194 may use colors having predetermined hue, saturation, and lightness, such as green or fluorescent colors that are easily distinguishable from the display device 160 and the background, or may be light sources capable of emitting light in any color. Further, the display device position acquisition markers 191 to 194 do not all need to be the same color, and different colors may be used respectively to facilitate the distinction between the top, bottom, left, and right of the display device 160.

[0015] The imaging system control device 300 renders a preset three-dimensional model of a virtual space according to the viewpoint of the camera 200 detected by the viewpoint detection device 120, and generates a CG background video at a predetermined frame rate. Note that when the imaging direction of the camera 200 does not directly face the display device 160, the imaging system control device 300 applies coordinate transformation (deformation processing) necessary for displaying the background video on the display device 160. The imaging system control device 300 outputs the generated background video to the display control device 110. In addition, the imaging system control device 300 controls the illumination control device 140 based on the information related to the luminance of the imaging scene obtained from the camera 200, thereby controlling the brightness of the illumination 150 that illuminates the imaging scene, particularly the real subject 180.

[0016] The display control device 110 uses the synchronization signal received from the imaging system control device 300 to cause the display device 160 to display the background video received from the imaging system control device 300 in synchronization with the imaging timing of the camera 200. When the display device 160 is composed of a plurality of display panels, the display control device 110 divides the background video to match each individual display panel before displaying it. Note that when the display device 160 is composed of a plurality of display panels, the display device position acquisition markers 191 to 194 are installed on the outer periphery of one display area constituted by the plurality of display panels.

[0017] A synchronization signal is supplied from an imaging system control device 300 to the camera 200, the display control device 110, the viewpoint detection device 120, and the illumination control device 140. Each device controls its operation timing in accordance with a reference clock, whereby synchronization between the imaging period of the camera 200 and the display period of the display device 160, and the like, are achieved. A technique for synchronizing operations between devices based on a reference clock is known as, for example, Genlock (Generator Locking), and thus detailed description thereof will be omitted.

[0018] Figure 2 is a block diagram illustrating a connection relationship of each device shown in Figure 1 and an example of a functional configuration of the camera 200. Among the functional blocks included in the camera 200, a first optical system 210, a first imaging unit 220, an image processing unit 230, and a recording unit 250 implement functions for capturing and recording in-camera VFX video. Further, a second optical system 260 and a second imaging unit 270 implement a function of capturing an image for detecting the viewpoint of the camera 200. Hereinafter, unless otherwise specified, the first imaging unit 220 and the second imaging unit 270 capture moving images having a predetermined frame rate.

[0019] The control unit 240 includes a processor (CPU, MPU, microprocessor, etc.) capable of executing programs, a ROM, and a RAM. The control unit 240 reads a program stored in the ROM into the RAM and executes the program, thereby controlling the operation of each functional block of the camera 200 and realizing the operation of the camera 200 to be described later. Although not illustrated, the control unit 240 is communicably connected to each functional block included in the camera 200.

[0020] The angle of view and the optical axis direction of the second optical system 260 are determined so as to image the viewpoint detection marker 130 arranged on the ceiling of the studio. The angle of view may be fixed or variable. The second imaging unit 270 has an image sensor and converts the optical image formed by the second optical system 260 into an analog image signal, and further converts the analog image signal to a digital signal. The digital image signal is output to the viewpoint detection device 120. Note that for viewpoint detection, it is sufficient to obtain the image coordinates of the viewpoint detection marker 130, so color information is not necessary, and the image sensor may be a monochrome image sensor.

[0021] The first optical system 210 is an optical system for capturing in-camera VFX images. Therefore, the field of view and optical axis direction of the first optical system are determined so as to form an optical image of a real subject 180 with the image displayed on the display device 160 as the background. Furthermore, the field of view of the first optical system 210 may be variable. The first imaging unit 220 has an image sensor and converts the optical image formed by the first optical system 210 into an analog image signal, and further converts the analog image signal to a digital signal by A / D conversion. The digital image signal is output to the image processing unit 230.

[0022] The image processing applied by the image processing unit 230 may include, for example, preprocessing, color interpolation, development, correction, detection, data processing, evaluation value calculation, and special effects processing. These are merely examples of processing that the image processing unit 230 can apply and do not limit the processing that the image processing unit 230 can apply.

[0023] The image processing unit 230 outputs acquired or generated information and data to functional blocks according to their intended use. For example, the image processing unit 230 outputs the in-camera VFX video after development processing to the recording unit 250 and the control unit 240, and outputs shooting parameters (information regarding shooting mode, focal length, focal position, etc.) to the control unit 240. The control unit 240 outputs the in-camera VFX video and shooting parameters from the image processing unit 230 to the imaging system control device 300.

[0024] Figure 3 is a block diagram showing an example of the functional configuration of the imaging system control device 300. The imaging system control device 300 can be implemented, for example, using computer equipment. The control unit 301 is, for example, a CPU, and it realizes the functions of the imaging system control device 300 by loading one or more application programs stored in the ROM 308 into the RAM 309 and executing them. The control unit 301 also controls the operating timing of the imaging system control device 300 according to a synchronization signal supplied from a reference clock generation device (not shown).

[0025] The image processing circuit 302 is a circuit that can perform various image processing tasks, for example, using a graphics card equipped with a GPU. The image processing circuit 302 can perform image processing, such as CG rendering, at high speed.

[0026] The imaging range detection unit 303 receives in-camera VFX video from camera 200. It also uses information from the display device position acquisition markers 191-194 contained in this in-camera VFX video to determine whether the imaging range 170 of camera 200 exceeds the display area of ​​display device 160 (i.e., whether the video includes areas outside the display area).

[0027] The first I / F 304 to the fourth I / F 307 are communication interfaces for connecting external devices. In this embodiment, the camera 200 is connected to the first I / F 304, the display control device 110 to the second I / F 305, the viewpoint detection device 120 to the third I / F 306, and the lighting control device 140 to the fourth I / F 307. The first I / F 304 to the fourth I / F 307 shall conform to standards corresponding to the external devices to be connected and the types of signals to be communicated. For convenience, the shooting system control device 300 and the external devices are shown to be connected through a single I / F, but they may be connected using multiple I / Fs.

[0028] The control unit 301 acquires video data and shooting parameters (information regarding shooting mode, focal length, focal position, etc.) from the camera 200 via the first interface 304. The control unit 301 also acquires shooting parameters (the viewpoint (position and orientation) of the camera 200) from the viewpoint detection device 120 via communication through the third interface 306. The control unit 301 also outputs background video data for display to the display control device 110 via the second interface 305. The control unit 301 also outputs a control signal to the lighting control device 140 via the fourth interface 307. The shooting system control device 300 may have five or more communication interfaces with external devices.

[0029] ROM308 stores parts of the program executed by the control unit 301 (BIOS, bootstrap loader, firmware), settings of the imaging system control device 300, and the like. RAM309 is used as the main memory for the control unit 301, as well as for the working memory of the image processing circuit 302 and the video memory of the display unit 312.

[0030] The memory unit 310 is a large-capacity storage device such as a hard disk or SSD. The memory unit 310 stores the operating system (OS), application programs, user data, etc. The memory unit 310 also stores application programs that generate background images according to the viewpoint of the camera 200 (for example, a game engine application), as well as the data necessary to generate the background images (3D models of the virtual space, textures, etc.).

[0031] The recording unit 311 is a large-capacity storage device such as a hard disk or SSD. The recording unit 311 records the in-camera VFX video itself received from the camera 200, as well as the video processed by the image processing circuit 302.

[0032] The display unit 312 is, for example, a liquid crystal display device. The display unit 312 may also be a touch display. The display unit 312 displays scene control applications, background video generation applications (e.g., game engines), GUIs provided by the OS, etc.

[0033] The operation unit 313 has multiple input devices that the user can operate, such as a keyboard, mouse, and touchpad. If the display unit 312 is a touch display, the touch panel is a component of the operation unit 313.

[0034] The clock generation circuit 314 generates a synchronization signal (clock) to synchronize the operation of the camera 200, display device 160, viewpoint detection device 120, and lighting control device 140. The clock may also be obtained from an external device (including the camera 200, etc.).

[0035] Next, using the flowchart shown in Figure 4, we will explain the operation of the virtual studio 100 during imaging, which is characteristic of this embodiment. The following operations are performed while the in-camera VFX image is being captured. Furthermore, since the following processes necessary for capturing in-camera VFX footage can be performed using publicly known methods, a detailed explanation of these processes will be omitted. • Detection process of the camera 200's viewpoint (position and orientation) and the distance from the camera 200 to the display device 160 using the image of the camera viewpoint detection marker 130 by the viewpoint detection device 120. • Background image generation process by the shooting system control device 300 according to the viewpoint of the detected camera 200. • Display control processing of background video on the display device 160 by the display control device 110 • Control processing of the lighting control device 140 and lighting 150 by the shooting system control device 300

[0036] In S401, the imaging range detection unit 303 of the shooting system control device 300 acquires in-camera VFX video from the camera 200.

[0037] In S402, the imaging range detection unit 303 of the shooting system control device 300 detects the display position acquisition markers 191 to 194 included in the in-camera VFX image acquired in S401.

[0038] In S403, it is determined whether the imaging range of the camera 200 exceeds the display area of ​​the display device 160. If it does not exceed, the process proceeds to S404; if it does exceed, the process proceeds to S405. Details of the processing in S403 will be described later.

[0039] In S404, the shooting system control device 300 records the in-camera VFX video received from the camera 200 directly to the recording unit 311.

[0040] Meanwhile, in S405, the imaging range detection unit 303 of the shooting system control device 300 identifies the video area in the in-camera VFX video received from the camera 200 that corresponds to the area outside the display area of ​​the display device 160, and the image processing circuit 302 overwrites that area with the green screen video. Details of the processing in S405 will be described later.

[0041] In S406, the in-camera VFX video, which was overwritten with green screen footage in S405, is recorded to the recording unit 311.

[0042] In S407, the imaging system control device 300 determines whether or not to terminate imaging. The imaging system control device 300 can determine to terminate imaging, for example, when imaging according to a predetermined timeline is completed, or when the user instructs to terminate imaging through the operation unit 313. If the imaging system control device 300 determines that imaging should be terminated, it terminates the imaging system control operation; otherwise, it repeats the operations from S401.

[0043] Next, we will explain the characteristic parts of the flowchart in Figure 4 using Figures 5 to 7.

[0044] Figure 5 shows the positional relationship between the camera 200, the display device 160, the real-world subject 180, and the markers 191-194 for acquiring the display device's position, as viewed from above in the virtual studio 100. Figure 5(a) shows an example where the camera 200 is close to the display device 160, and the imaging range (angle of view) of the camera 200 fits within the display area of ​​the display device 160. Figure 5(b) shows an example where the camera 200 is far from the display device 160, and the imaging range (angle of view) of the camera 200 exceeds the display area of ​​the display device 160.

[0045] Figure 6 shows an example of an in-camera VFX image acquired from the camera 200 by the imaging range detection unit 303. Figure 6(a) shows the image when the imaging range of camera 200 is within the display area of ​​display device 160, as shown in Figure 5(a), while Figure 6(b) shows the image when the imaging range of camera 200 exceeds the display area of ​​display device 160, as shown in Figure 5(b).

[0046] In Figure 6, 600 represents the entire image area of ​​the in-camera VFX video, 610 represents the real subject 180 in the in-camera VFX video, and 620 represents the area in the in-camera VFX video where the display device 160 is captured. Also, 631 to 634 represent the markers 191 to 194 used to acquire the position of the display device in the in-camera VFX video, respectively. The area 640 of the horizontal line represents the area outside the display device 160 in the in-camera VFX video.

[0047] As mentioned above, the display device position acquisition markers 191-194 are colors with predetermined hues, saturations, and brightness, such as green and fluorescent colors, or light sources that can emit light of any color. Therefore, in S402, by using the pixel value information in the in-camera VFX video, it is possible to easily detect which region (position) in the video contains the display device position acquisition markers 631-634. As described above, the pixel region information of the display device position acquisition markers 631 to 634 is acquired.

[0048] As shown in Figure 6(a), if the imaging range of camera 200 is within the range of display device 160, the display device position acquisition markers 631-634 are not included in the in-camera VFX image. On the other hand, if the imaging range of the camera 200 exceeds the range of the display device 160, as shown in Figure 6(b), at least one of the display device position acquisition markers 631 to 634 will be included in the in-camera VFX image.

[0049] In S403, it is determined whether the imaging range of camera 200 exceeds the display area of ​​display device 160. If any of the display device position acquisition markers 631 to 634 are present in the in-camera VFX video, it is determined that the range "overflows". If none of the display device position acquisition markers 631 to 634 are present in the in-camera VFX video, it is determined that the range "does not overflow".

[0050] Next, in S405, a method for acquiring the video area that falls outside the display area of ​​the display device 160 within the in-camera VFX video will be described.

[0051] Figure 7(a), similar to Figure 6(b), shows the in-camera VFX image when the imaging range of camera 200 exceeds the display area of ​​display device 160.

[0052] 701 to 704 each indicate the position (2D coordinate) of a pixel in the in-camera VFX image. By detecting this position information, the image area enclosed by these points can be divided into the display area of ​​the display device 160, and the remaining area can be divided into the image area outside the display area of ​​the display device 160.

[0053] As coordinate 701, the coordinates of the lower left corner of the display device position acquisition marker 631 detected in S402, or the coordinates of the upper right corner of the display device position acquisition marker 633, are set. For coordinate 702, the coordinates of the lower right corner of the display device position acquisition marker 631 detected in S402, or the coordinates of the upper left corner of the display device position acquisition marker 634, are set. Coordinate 703 is set to the coordinate of the lower right corner of the display device position acquisition marker 633 detected in S402, or the coordinate of the upper left corner of the display device position acquisition marker 632. For coordinate 704, the coordinates of the lower left corner of the display device position acquisition marker 634 detected in S402, or the coordinates of the upper right corner of the display device position acquisition marker 632, are set.

[0054] Then, the video area outside the video area 620 enclosed by the four coordinates 701-704, and the video area on the lines (boundaries) connecting coordinates 701 and 702, 702 and 704, 701 and 703, and 703 and 704 are transmitted to the image processing circuit 302 along with the in-camera VFX video. The image processing circuit 302 then overwrites the video in these areas with green screen video.

[0055] Figure 7(b) shows the in-camera VFX footage after it has been overwritten with green screen footage, and the shaded area 710 indicates the area of ​​the footage that has been overwritten with green screen footage.

[0056] Then, in S406, the in-camera VFX image, which has been overwritten with green screen footage, is recorded in the recording unit 311.

[0057] As described above, according to the first embodiment, if the imaging range of the camera 200 exceeds the display area of ​​the display device 160, the video area outside the display area of ​​the display device 160 is overwritten with a green screen image. This makes it possible to record the video in a state that is easy to edit later. Specifically, it is expected to reduce the workload during editing and the likelihood of errors in cutting out the background area.

[0058] Furthermore, if the original footage is needed for later editing (for example, if an overwrite error occurs with the green screen footage), the in-camera VFX footage recorded in the recording unit 250 of camera 200 before it was overwritten with the green screen footage can be used.

[0059] In the first embodiment, the markers 191 to 194 for acquiring the display device position were arranged around the outer perimeter of the display device 160. However, the markers for acquiring the display device position may also be arranged at the four endpoints of the display area of ​​the display device 160. In this case, however, if any of the endpoints are not included in the video, the outer perimeter of the display device 160 cannot be detected by image processing. Therefore, it is necessary to register the marker positions in advance as absolute coordinates and then use the position and orientation information of the camera 200 to convert them to the camera viewpoint coordinate system.

[0060] Furthermore, in the first embodiment, the markers 191 to 194 for acquiring the position of the display device were placed on the outer perimeter of the display area of ​​the display device 160 and the position information was detected from the captured video. However, markers (transmitters) that have a wireless communication function and can transmit position information may also be used. By transmitting the position information of the display device 160 to the shooting system control device 300 via wireless communication, the display area of ​​the display device 160 within the imaging range can be calculated by combining it with the viewpoint (position and orientation) information of the camera 200 received from the viewpoint detection device 120. In this case, it is preferable to place the markers inside or on the back of the display device 160 so that they do not appear in the video.

[0061] Furthermore, although the first embodiment described the markers 191 to 194 for acquiring the position of the display device as being installed on the outer periphery of the display area of ​​the display device 160, a frame with predetermined chromaticity and saturation may be displayed around the display area, and this frame may be detected from the captured video.

[0062] Furthermore, in the first embodiment, the area outside the display device 160 was overwritten with a green screen image, but it may also be overwritten with a chroma key image of a color other than green.

[0063] Furthermore, in this embodiment, the entire area outside the display device 160 is overwritten with a green screen image. However, if a shooting set is located in the ground area at the bottom of the screen, the area at the bottom, even if it is outside the display device 160, may not be overwritten with a green screen image. The determination of whether or not a shooting set is located may be made, for example, by image analysis, or the user may provide information indicating the presence of a shooting set to the imaging system control device 300. The determination method is not particularly limited.

[0064] Furthermore, if the real subject 180 is included in the video area outside the display area of ​​the display device 160, the area containing the real subject may be separated using known techniques, so that the area where the real subject 180 exists is not overwritten with the green screen image.

[0065] Alternatively, the video area outside the display area of ​​the display device 160 may be calculated by image matching the in-camera VFX video received from the camera 200 with the background video data transmitted to the display control device 110. Specifically, a projection transformation is performed on the background video data transmitted to the display control device 110 using camera attitude information, and the background video data after the projection transformation is aligned (image matching) with the in-camera VFX video received from the camera 200. This makes it possible to determine that the video area that matches the background video data is the display area of ​​the display device 160, and the other areas are video areas outside the display area of ​​the display device 160.

[0066] <Second Embodiment> A second embodiment of the present invention will be described below. In the first embodiment, if the imaging range of the camera 200 exceeds the range of the display device 160, the area outside the display area of ​​the display device 160 in the video is overwritten with a green screen image. In contrast, the second embodiment is characterized by overwriting areas other than the actual subject with a green screen image.

[0067] In the second embodiment, the configuration of the virtual studio 100, camera 200, and shooting system control device 300 is the same as that described in the first embodiment with reference to Figures 1 to 3, so a detailed explanation is omitted here.

[0068] However, in this embodiment, the first imaging unit 220 of the camera 200 has a configuration in which two photodiodes are provided for one microlens, making it possible to acquire pupil-splitting images (left viewpoint image and right viewpoint image).

[0069] The operation of the virtual studio 100 during image capture, which is characteristic of this embodiment, will be explained using the flowchart shown in Figure 8. Note that the following operations are performed while the in-camera VFX image is being captured.

[0070] In S801, the imaging range detection unit 303 of the shooting system control device 300 acquires the in-camera VFX image and the pupil-splitting images (left viewpoint image, right viewpoint image) in their RAW state before development from the camera 200.

[0071] In S802, the imaging range detection unit 303 of the shooting system control device 300 detects the display position acquisition markers 191 to 194 included in the in-camera VFX image acquired in S801.

[0072] In S803, it is determined whether the imaging range of the camera 200 exceeds the display area of ​​the display device 160. If it does not exceed, proceed to S804; if it does exceed, proceed to S805.

[0073] In S804, the shooting system control device 300 records the in-camera VFX video received from the camera 200 directly to the recording unit 311.

[0074] Meanwhile, in S805, the imaging range detection unit 303 of the shooting system control device 300 calculates distance information for each pixel from the pupil-splitting images (left viewpoint image, right viewpoint image) in their RAW state before development. Then, it extracts the subject area in the in-camera VFX video where the real subject 180 exists from the calculated distance information. The information of the subject area extracted here is transmitted to the image processing circuit 302.

[0075] In S806, the image processing circuit 302 overwrites the areas of the in-camera VFX video other than the subject area of ​​the real subject 180 with a green screen image.

[0076] In S807, the in-camera VFX video, which was overwritten with green screen footage in S806, is recorded to the recording unit 311.

[0077] In S808, the imaging system control device 300 determines whether or not to terminate imaging. The imaging system control device 300 can determine to terminate imaging, for example, when imaging is completed according to a predetermined timeline, or when the user instructs to terminate imaging through the operation unit 313. If the imaging system control device 300 determines to terminate imaging, it terminates the imaging system control operation; otherwise, it repeats the operations from S801.

[0078] The following provides a detailed explanation of the operation of S805, S806, and S807. In S805, the phase difference between pixels of the two images is calculated using stereo matching processing, with the pupil-splitting images (left viewpoint image, right viewpoint image) in RAW state acquired by the imaging range detection unit 303 in S801 being used. Note that any known method can be used for calculating the phase difference using stereo matching processing; for example, a method described in Japanese Patent Application Publication No. 2009-165115 can be used. Then, distance information for each pixel is calculated using the calculated phase difference.

[0079] Furthermore, the imaging range detection unit 303 detects the real subject 180 from the in-camera VFX video acquired by the unit. For example, if the real subject 180 is a person, the location of the real subject 180 in the in-camera VFX video can be calculated using known face detection or human body detection techniques.

[0080] Then, by combining the calculated distance information for each pixel with the detection results from face detection, human body detection, etc., of 180 real-world subjects, a more accurate image area of ​​180 real-world subjects is determined. Specifically, pixels that are near the video region of the real subject 180 detected by human body detection, and that show distance information close to the distance information of the pixels in the center of the real subject 180 detected by human body detection, should be determined to be the video region where the real subject 180 exists. The imaging range detection unit 303 then transmits information about the video region in which the real subject 180 exists to the image processing circuit 302.

[0081] In S806, information about the video area in which the real subject 180 exists, received from the imaging range detection unit 303, is used to overwrite the video area other than the video area in which the real subject 180 exists with a green screen image.

[0082] Figure 9 shows the in-camera VFX footage after it has been overwritten with green screen footage. 900 represents the entire image area of ​​the in-camera VFX footage, and 910 represents the area of ​​the real subject 180 in the in-camera VFX footage. The shaded area 920, excluding the area of ​​the real subject 180, represents the image area that has been overwritten with green screen footage.

[0083] In S807, the in-camera VFX video, which was overwritten with green screen footage in S806, is recorded to the recording unit 311.

[0084] As described above, according to the second embodiment, if the imaging range of the camera 200 exceeds the display area of ​​the display device 160, the area of ​​the video other than the real subject 180 is overwritten with a green screen image. This makes it possible to record the video in a state where the real subject 180 can be easily cut out in later editing.

[0085] The control methods used in the first embodiment and the control methods used in the second embodiment may be used interchangeably depending on the application and the functions installed in the camera 200 and the shooting system control device 300.

[0086] For example, if you want to later replace only the area outside the display device 160 with CG or the like during editing, or if you want to crop the area outside the display device 160 (and resize the remaining video area), you can use the method of the first embodiment.

[0087] Furthermore, if you want to replace everything except the real subject 180 with CG or other means during post-production editing, or if you can accurately extract the background area using information (such as distance information) that can only be used when the camera 200 is capturing an image, as mentioned above, then you can use the method of this embodiment.

[0088] <Third Embodiment> A third embodiment of the present invention will be described below. In the first and second embodiments, when the imaging range of the camera 200 exceeded the display area of ​​the display device 160, the area outside the display area of ​​the display device 160 or the real subject 180 was overwritten with a green screen image by image processing and recorded. In contrast, in this embodiment, when the imaging range of the camera 200 exceeds the display area of ​​the display device 160, the display device 160 displays a green screen image.

[0089] In the third embodiment, the configuration of the virtual studio 100, camera 200, and shooting system control device 300 is the same as that described in the first embodiment with reference to Figures 1 to 3, so a detailed explanation is omitted here.

[0090] The operation during image capture in the virtual studio 100 in this embodiment will be explained using the flowchart shown in Figure 10. Note that the following operations are performed while the in-camera VFX image is being captured.

[0091] In S1001, the imaging range detection unit 303 of the shooting system control device 300 acquires in-camera VFX video from the camera 200.

[0092] In S1002, the imaging range detection unit 303 of the shooting system control device 300 detects the display position acquisition markers 191 to 194 included in the in-camera VFX image acquired in S401.

[0093] In S1003, it is determined whether the imaging range of the camera 200 exceeds the display area of ​​the display device 160. If it does not exceed, proceed to S1004; if it does exceed, proceed to S1005.

[0094] In S1004, the shooting system control device 300 instructs the display control device 110 to display the background image on the display device 160.

[0095] Meanwhile, in S1005, the shooting system control device 300 instructs the display control device 110 to display a green screen image on the display device 160. In this case, the captured in-camera VFX image will look like Figure 11(a).

[0096] In Figure 11(a), 1100 represents the entire image area of ​​the in-camera VFX video, 1110 represents the real subject 180 in the in-camera VFX video, and the vertical line area 1120 represents the area in the in-camera VFX video where the display device 160 is captured. Also, 1131 to 1134 represent the markers 191 to 194 used to acquire the position of the display device in the in-camera VFX video, respectively. The horizontal line area 1140 represents the area outside the display device 160 in the in-camera VFX video.

[0097] In S1006, the shooting system control device 300 acquires in-camera VFX video again from camera 200. At this time, if background video is displayed on the display device 160 in S1004, the real subject 180 and the background video displayed on the display device 160 will be included in the in-camera VFX video. In other words, normal in-camera VFX is captured during this time. On the other hand, if green screen video is displayed on the display device 160 in S1005, the real subject 180 and the green screen video displayed on the display device 160 will be included in the in-camera VFX video.

[0098] Then, in S1007, the shooting system control device 300 records the in-camera VFX video received from the camera 200 to the recording unit 311.

[0099] In S1008, the imaging system control device 300 determines whether or not to terminate imaging. The imaging system control device 300 can determine to terminate imaging, for example, when imaging according to a predetermined timeline is completed, or when the user instructs to terminate imaging through the operation unit 313. If the imaging system control device 300 determines that imaging should be terminated, it terminates the imaging system control operation; otherwise, it repeats the operations from S1001.

[0100] As explained above, if the imaging range of camera 200 exceeds the range of display device 160, displaying a green screen image on display device 160 allows the video to be shot and recorded in a way that makes it easy to cut out the real subject 180 in later editing.

[0101] Furthermore, the second embodiment and this embodiment can be used interchangeably depending on the functions installed in the camera 200 and the shooting system control device 300. In other words, if you want to replace the area around the real subject 180 with CG or the like in post-processing, but it is not possible to accurately cut out the real subject 180 during imaging as in the second embodiment, then you can simply display a green screen image on the display device 160 as in this embodiment.

[0102] Furthermore, if you want to overwrite and record the area 1140 outside the display device 160 and the markers 1131-1134 for acquiring the display device position in the in-camera VFX video of Figure 11(a) with a green screen video, you can do so using the same method as in the first embodiment.

[0103] As a result, the in-camera VFX image looks like Figure 11(b). In Figure 11(b), the shaded area 1150 is where a green screen image is recorded through image processing.

[0104] <Variation> In the first to third embodiments described above, when shooting was to continue, the process was repeated by returning to S401 in Figure 4, S801 in Figure 8, and S1001 in Figure 10, respectively, each time an in-camera VFX image was received.

[0105] In contrast, if the camera 200's viewpoint has not changed, the detection of the display device position acquisition markers 191-194 in S402, S802, and S1002, and the determination in S403, S803, and S1003 may be omitted, and the most recent detection result and determination result may be used. This reduces the processing load on the imaging range detection unit 303 of the imaging system control device 300.

[0106] <Other Embodiments> Furthermore, the present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0107] <Summary> This embodiment includes the following configuration.

[0108] (Item 1) Display means and An imaging means for capturing and outputting an image that includes at least a portion of the display area of ​​the aforementioned display means within its field of view, A determination means for determining whether the video includes video of an area outside the display area, If it is determined that the aforementioned video contains video of the aforementioned external region, the processing means replaces the video of the said external region with video of a predetermined color. An imaging system characterized by having the following features. (Item 2) The system further includes detection means for detecting subjects included in the aforementioned video, The imaging system according to item 1, characterized in that, when the processing means determines that the image includes an image of the external region, it replaces the image of the region excluding the subject with an image of a predetermined color, and the region excluding the subject includes the external region. (Item 3) The system further includes a display control means for controlling the image displayed on the display means, The display control means is If it is determined that the video does not contain video of the external area, the background video is displayed on the display means. If it is determined that the aforementioned video contains video from the external region, the display means will display video of a predetermined color. The imaging system according to item 1 or 2, characterized by being controlled in such a manner. (Item 4) The imaging system according to any one of items 1 to 3, further comprising a storage means for storing the image processed by the processing means when it is determined that the image includes the image of the external region, and for storing the image output from the imaging means when it is determined that the image does not include the image of the external region. (Item 5) Display means and A display control means for controlling the image displayed on the display means, An imaging means for capturing and outputting an image that includes at least a portion of the display area of ​​the aforementioned display means within its field of view, The system includes a determination means for determining whether the video includes video of an area outside the display area, The display control means is If it is determined that the video does not contain video of the external area, the background video is displayed on the display means. If it is determined that the aforementioned video contains video from the external region, the display means will display video of a predetermined color. An imaging system characterized by controlling it in such a way. (Item 6) The determination means includes a detection means for detecting a predetermined portion of the display means from the video, The imaging system according to any one of items 1 to 5, characterized in that the determination means determines that the image of the external region is included when the predetermined portion is detected. (Item 7) The imaging system according to item 6, characterized in that the predetermined portion is a marker or light source of a predetermined hue, saturation, and brightness, installed on the outer periphery of the display area. (Item 8) The imaging system according to item 6, characterized in that the predetermined portion is a color marker or light source having predetermined hue, saturation, and brightness, positioned at the corner of the display area. (Item 9) The determination means includes a detection means for detecting a predetermined portion of the display means from the video, The predetermined portion is a frame having predetermined hue, saturation, and brightness displayed around the display area, The imaging system according to any one of items 1 to 4, characterized in that, when the processing means determines that the image includes an image of the external region, it further replaces the surrounding image with an image of a predetermined color. (Item 10) An acquisition means for acquiring the viewpoint of the imaging means, The system further includes a transmitter that emits a signal indicating the position of the outer periphery of the display area, The imaging system according to any one of items 1 to 5, characterized in that the determination means receives a signal from the transmitter to detect the position of the outer perimeter of the display area, and determines whether the image includes an image of the external area based on the detected position of the outer perimeter of the display area and the viewpoint of the imaging means. (Item 11) The imaging system according to any one of items 1 to 5, characterized in that the determination means matches the video data displayed on the display means with the video data captured by the imaging means, and determines that the video area that does not match is an image of an area outside the display area. (Item 12) The imaging system according to any one of items 1 to 11, characterized in that the display means is composed of a plurality of display panels. (Item 13) If it is determined that the video includes video of the external area, the system further has a determination means for determining whether or not there is a shooting set below the video. The imaging system according to any one of items 1 to 4, characterized in that, when the processing means determines that there is a shooting set below the image, it does not replace the image below the image, even if it is in an area outside the display area. (Item 14) The system further includes an acquisition means for acquiring the viewpoint of the imaging means, The imaging system according to any one of items 1 to 13, characterized in that the determination means performs the determination when there is a change in the viewpoint, and uses the most recent determination result when there is no change in the viewpoint. (Item 15) The imaging system according to any one of items 1 to 4, characterized in that it stores the image output from the imaging means and the image processed by the processing means in a storage means. (Item 16) The acquisition means includes an acquisition step of acquiring an image whose field of view includes at least a portion of the display area of ​​the display means, The determination means includes a determination step of determining whether the image includes an image of an area outside the display area, If the processing means determines that the image contains images of the external region, it performs a processing step of replacing the images of the external region with images of a predetermined color. An image processing method characterized by having the following features. (Item 17) The acquisition means includes an acquisition step of acquiring an image whose field of view includes at least a portion of the display area of ​​the display means, The determination means includes a determination step of determining whether the image includes an image of an area outside the display area, A display control step in which the display control means controls the display to display a background image on the display means when it is determined that the image does not include an image of the external area, and to display an image of a predetermined color on the display means when it is determined that the image includes an image of the external area. An imaging method characterized by having the following features. (Item 18) A program that causes a computer to perform each step of the image processing method described in item 16. (Item 19) A program that causes a computer to perform each step of the imaging method described in item 17. (Item 20) A computer-readable storage medium containing the program described in item 18 or 19.

[0109] The present invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0110] 100…Virtual studio, 110…Display control device, 120…Viewpoint detection device, 130…Marker for camera viewpoint detection, 140…Lighting control device, 150…Lighting, 160…Display device, 170…Imaging range, 180…Real subject, 191~194…Marker for acquiring display device position, 200…Camera, 210…First optical system, 220…First imaging unit, 230…Image processing unit, 240…Control unit, 250 ...Recording unit, 260...Second optical system, 270...Second imaging unit, 300...Shooting system control device, 301...Control unit, 302...Image processing circuit, 303...Imaging range detection unit, 304...First I / F, 305...Second I / F, 306...Third I / F, 307...Fourth I / F, 308...ROM, 309...RAM, 310...Storage unit, 311...Recording unit, 312...Display unit, 313...Operation unit, 314...Clock control circuit

Claims

1. Display means and An imaging means for capturing and outputting an image that includes at least a portion of the display area of ​​the aforementioned display means within its field of view, A determination means for determining whether the video includes video of an area outside the display area, If it is determined that the aforementioned video contains video of the aforementioned external region, the processing means replaces the video of the said external region with video of a predetermined color. An imaging system characterized by having the following features.

2. The system further includes detection means for detecting subjects included in the aforementioned video, The imaging system according to claim 1, characterized in that, when the processing means determines that the image includes an image of the external region, it replaces the image of the region excluding the subject with an image of the predetermined color, and the region excluding the subject includes the external region.

3. The system further includes a display control means for controlling the image displayed on the display means, The display control means is If it is determined that the video does not contain video of the external area, the background video is displayed on the display means. If it is determined that the aforementioned video contains video from the external region, the display means will display video of a predetermined color. The imaging system according to claim 1, characterized in that it is controlled in such a manner.

4. The imaging system according to claim 1, further comprising a storage means for storing the image processed by the processing means when it is determined that the image includes the image of the external region, and for storing the image output from the imaging means when it is determined that the image does not include the image of the external region.

5. Display means and A display control means for controlling the image displayed on the display means, An imaging means for capturing and outputting an image that includes at least a portion of the display area of ​​the aforementioned display means within its field of view, The system includes a determination means for determining whether the video includes video of an area outside the display area, The display control means is If it is determined that the video does not contain video of the external area, the background video is displayed on the display means. If it is determined that the aforementioned video contains video from the external region, the display means will display video of a predetermined color. An imaging system characterized by controlling it in such a way.

6. The determination means includes a detection means for detecting a predetermined portion of the display means from the video, The imaging system according to any one of claims 1 to 5, characterized in that the determination means determines that the image of the external region is included when the predetermined portion is detected.

7. The imaging system according to claim 6, characterized in that the predetermined portion is a color marker or light source having predetermined hue, saturation, and brightness, installed on the outer periphery of the display area.

8. The imaging system according to claim 6, characterized in that the predetermined portion is a color marker or light source having predetermined hue, saturation, and brightness, positioned at the corner of the display area.

9. The determination means includes a detection means for detecting a predetermined portion of the display means from the video, The predetermined portion is a frame having predetermined hue, saturation, and brightness displayed around the display area, The imaging system according to any one of claims 1 to 4, characterized in that, when the processing means determines that the image includes an image of the external region, it further replaces the surrounding image with an image of a predetermined color.

10. An acquisition means for acquiring the viewpoint of the imaging means, The system further includes a transmitter that emits a signal indicating the position of the outer periphery of the display area, The imaging system according to any one of claims 1 to 5, characterized in that the determination means receives a signal from the transmitter to detect the position of the outer perimeter of the display area, and determines whether the image includes an image of the external area based on the detected position of the outer perimeter of the display area and the viewpoint of the imaging means.

11. The imaging system according to any one of claims 1 to 5, characterized in that the determination means matches the video data displayed on the display means with the video data captured by the imaging means, and determines that the video area that does not match is an image of an area outside the display area.

12. The imaging system according to any one of claims 1 to 5, characterized in that the display means is composed of a plurality of display panels.

13. If it is determined that the video includes video of the external area, the system further has a determination means for determining whether or not there is a shooting set below the video. The imaging system according to any one of claims 1 to 4, characterized in that, when the processing means determines that there is a shooting set below the image, it does not replace the image below the image, even if it is in an area outside the display area.

14. The system further includes an acquisition means for acquiring the viewpoint of the imaging means, The imaging system according to any one of claims 1 to 5, characterized in that the determination means performs the determination when there is a change in the viewpoint, and uses the most recent determination result when there is no change in the viewpoint.

15. The imaging system according to any one of claims 1 to 4, characterized in that the image output from the imaging means and the image processed by the processing means are stored in a storage means.

16. The acquisition means includes an acquisition step of acquiring an image whose field of view includes at least a portion of the display area of ​​the display means, The determination means includes a determination step of determining whether the image includes an image of an area outside the display area, If the processing means determines that the image contains images of the external region, it performs a processing step of replacing the images of the external region with images of a predetermined color. An image processing method characterized by having the following features.

17. The acquisition means includes an acquisition step of acquiring an image whose field of view includes at least a portion of the display area of ​​the display means, The determination means includes a determination step of determining whether the image includes an image of an area outside the display area, A display control step in which the display control means controls the display to display a background image on the display means when it is determined that the image does not include an image of the external area, and to display an image of a predetermined color on the display means when it is determined that the image includes an image of the external area. An imaging method characterized by having the following features.

18. A program for causing a computer to perform each step of the image processing method described in claim 16.

19. A program for causing a computer to perform each step of the imaging method described in claim 17.

20. A computer-readable storage medium storing the program described in claim 18 or 19.

Images