Virtual Studio System

Abstract

To provide a virtual studio system configured to more easily create a video that a creator has in mind, in creating a new video content by synthesizing a real subject with a background in a three-dimensional virtual space.SOLUTION: A virtual studio system 1 includes: a camera 100 for imaging a real subject P; a camera tracker 110 which is fixed to the camera; a rendering unit which draws an image of a three-dimensional virtual space; a virtual camera 300 which is arranged in the three-dimensional virtual space, has parameters of a position, an attitude and an angle of view, the parameters of the position and the attitude being operated in accordance with the position and the attitude of the camera, on the basis of camera move signals output from the camera tracker, and configured to specify an imaging range on the basis of the parameters of the position, the attitude and the angle of view of the virtual camera; and a processing apparatus 200 including a synthesizing unit which generates a synthetic video by synthesizing an image of the subject captured by the camera with an image of the imaging range in the three-dimensional virtual space.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a virtual studio system for creating an image by synthesizing an actual subject and the background of a three-dimensional virtual space.

Background Art

[0002] In recent years, with the development of computer and video technologies, when creating video contents such as movies, TV programs, or commercials, it has become widespread to use images created by computer graphics (hereinafter referred to as CG).

[0003] However, when creating video contents by CG, costs corresponding to its quality are required. Especially when creating a person or the like, a great deal of economic and time costs are required for model creation and motion creation. Therefore, it has been difficult to say that it is an easy thing that anyone can handle.

[0004] Therefore, it is known to combine and photograph an actual subject and a CG background. The actual subject is photographed by a camera under a single-color background such as green, and the photographed real image is subjected to a transparency process by specifying the color of the background. A so-called chroma key synthesis method of combining the obtained processed real image and the CG background image to obtain a composite image can create a composite image at a relatively low cost, and thus is often used when creating a TV program or the like.

[0005] However, in the chroma key synthesis method, when the camera is moved, the CG background image also has to be moved along with the movement of the camera, otherwise the composite image will be incongruous. It takes a great deal of effort to move (shift the viewpoint) the CG background image by post-processing. Therefore, when performing such shooting by chroma key synthesis, the camera is usually installed at a fixed position.

[0006] Therefore, a photographing system that generates a composite image by synthesizing an image of an actual subject and an image in a three-dimensional virtual space is known.

[0007] The photographing system is also called a virtual studio. It generates an image in a three-dimensional virtual space in synchronization with the operation of a camera that photographs an actual subject, and synthesizes the real image photographed by the camera and the image of the three-dimensional virtual space to obtain a composite video. This eliminates the need for post-processing to synthesize the real image and the CG background image as in the above-described chroma key synthesis, and it is possible to obtain a composite image with less discomfort by changing the image of the three-dimensional virtual space as the camera moves. Therefore, it has been particularly popular in recent years.

[0008] However, in the conventional photographing system (virtual studio), when changing the image of the three-dimensional virtual space as the camera moves, it is necessary to digitize the movement of the camera, and a camera or a crane / dolly equipped with a special function for detecting the movement of the camera is required. Therefore, it was necessary to assemble the system using expensive equipment.

[0009] On the other hand, as shown in, for example, Japanese Patent Application Laid-Open No. 2015 / 098807 (Patent Document 1), it is also known that in the photographing system (virtual studio), it is possible to identify the position of the camera relatively inexpensively by using a tracker (tracking device) and a marker (tracking sign) for identifying the position of the camera.

[0010] However, the invention described in Patent Document 1 aims to generate a composite image with high reality and immersion. It synthesizes an image of a subject and an image of a three-dimensional virtual space already stored in the spatial image storage unit. Furthermore, it is possible to detect the movement of the subject (the person being photographed) by a motion sensor, add the content stored in the content storage unit based on the detected specific movement, and display the obtained composite image on a monitor visible to the subject (the person being photographed) being photographed, so as to obtain a composite image with a high sense of immersion as if one has entered the three-dimensional virtual space oneself.

[0011] That is, the invention described in Patent Document 1 mainly focuses on relatively small-scale amusement use for the subject (the person being photographed) to enjoy, and repeatedly uses the images of the three-dimensional virtual space stored in the spatial image memory unit in advance and the content stored in the content memory unit for a large number of subjects (the persons being photographed). It was difficult to say that it was suitable for creating new video content such as recorded content like movies, TV programs, commercials, or live broadcast content such as streaming distribution or live distribution.

Prior Art Documents

Patent Documents

[0012]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0013] Therefore, an object of the present invention is to provide a virtual studio system that makes it easier to create the video envisioned by the video creator when creating new video content including recorded content such as movies, TV programs, commercials, or live broadcast content such as streaming distribution or live distribution by using a video that synthesizes a real subject and the background of a three-dimensional virtual space.

Means for Solving the Problems

[0014] The present invention made to solve the above problems is a virtual studio system that creates a video by synthesizing a real subject and the background of a three-dimensional virtual space, a camera that photographs a real subject, a camera tracker fixed to the camera, which detects the position and orientation of the camera and outputs a camera movement signal, a rendering unit that depicts an image of a three-dimensional virtual space, A virtual camera that is arranged in the three-dimensional virtual space, has parameters of position, orientation, and field of view angle, and whose parameters of its own position and orientation are manipulated according to the position and orientation of the camera based on the camera movement signal output by the camera tracker, and that specifies a projection range based on its own parameters of position, orientation, and field of view angle. A synthesizing unit that generates a synthesized video by synthesizing an image of a subject photographed by the camera and an image of a projection range in the three-dimensional virtual space. It is characterized by this.

[0015] Another aspect of the present invention made to solve the above problems is a virtual production system for creating a video by synthesizing a real subject and the background of a three-dimensional virtual space, A camera that photographs a real subject, A camera tracker that is fixed to the camera, detects the position and orientation of the camera, and outputs a camera movement signal, A rendering unit that depicts an image of a three-dimensional virtual space, A virtual camera that is arranged in the three-dimensional virtual space, has parameters of position, orientation, and field of view angle, and whose parameters of its own position and orientation are manipulated according to the position and orientation of the camera based on the camera movement signal output by the camera tracker, and that specifies a projection range based on its own parameters of position, orientation, and field of view angle. A panel that is installed around the subject and outputs an image of a projection range in the three-dimensional virtual space specified by the virtual camera. By photographing the subject with the camera in a state where an image of a projection range in the three-dimensional virtual space is output to the panel, a synthesized video obtained by synthesizing an image of the subject and an image of a projection range in the three-dimensional virtual space is photographed. It is characterized by this.

[0016] Further, the camera has a focus function and the virtual camera has focus parameters. When performing a focus operation by operating the focus parameters of the virtual camera according to the focus operation amount detected by the focus detection means attached to the camera, the image of the projection range in the three-dimensional virtual space also changes due to the focus operation of the virtual camera, and it is possible to create the video envisioned by the video creator. In addition, since the focus operation of the virtual camera is performed in conjunction with simply performing the focus operation of the camera, it is very convenient.

[0017] In addition, the focus function of the camera performs focus by rotating the focus ring, the focus detection means includes a rotation detection unit attached to the focus ring, and a signal output unit that outputs a focus operation signal according to the focus rotation amount detected by the rotation detection unit. When performing the focus operation of the virtual camera according to the focus operation signal, it becomes easy to convert the rotation amount of the rotation operation of the focus ring of the camera into a focus operation signal.

[0018] Furthermore, the rotation detection unit includes a gear member fixed to the focus ring, and an encoder that meshes with the gear member to digitize the focus rotation amount. The signal output unit includes a motor member that rotates in conjunction with the focus rotation amount digitized by the encoder, and a focus tracker that is attached to the rotation axis of the motor member and outputs at least one of its position and orientation as a focus operation signal. When performing a focus operation by operating the parameters of the virtual camera according to the focus operation signal output by the focus tracker rotating in conjunction with the rotation operation of the focus ring, by outputting the rotation of the focus tracker as a focus operation signal, it becomes possible to align the output format with the camera operation signal of the camera tracker, and it becomes easy to operate the virtual camera by inputting it into the parameters of the virtual camera.

[0019] In addition, the camera has a zoom function and the virtual camera has zoom parameters, when performing a zoom operation by operating the zoom parameters of the virtual camera according to the zoom operation amount detected by the zoom detection means mounted on the camera, the image of the projection range in the three-dimensional virtual space also changes due to the zoom operation of the virtual camera, and it is possible to create the video envisioned by the video creator. Also, since the focus operation of the virtual camera is performed in conjunction with just performing the zoom operation of the camera, it is very convenient.

[0020] In addition, the zoom function of the camera is such that zooming is performed by rotating the zoom ring, the zoom detection means includes a rotation detection unit mounted on the zoom ring and a signal output unit that outputs a zoom operation signal according to the zoom rotation operation amount detected by the rotation detection unit. When performing a zoom operation of the virtual camera according to the zoom operation signal, it becomes easy to convert the rotation amount of the rotation operation of the zoom ring of the camera into a zoom operation signal.

[0021] Furthermore, the rotation detection unit includes a gear member fixed to the zoom ring and an encoder that meshes with the gear member to digitize the zoom rotation operation amount, the signal output unit includes a motor member that rotates in conjunction with the zoom rotation operation amount digitized by the encoder, and a zoom tracker that is attached to the rotation axis of the motor member and outputs at least one of its position and orientation as a zoom operation signal, when performing a zoom operation of the virtual camera based on the zoom operation signal output by the zoom tracker rotating in conjunction with the rotation operation of the zoom ring, by outputting the rotation of the zoom tracker as a zoom operation signal, it becomes possible to align the output format with the camera operation signal of the camera tracker, and it becomes easy to operate the virtual camera by inputting it into the parameters of the virtual camera.

[0022] In addition, when the encoder of the rotation detector and the motor member of the signal output unit are connected by wire or wirelessly and are arranged separately, by configuring the signal output unit including the motor member as a device separate from the camera, the components attached to the camera are reduced. In addition to having little impact on the handling of the camera, the focus tracker and zoom tracker can output the focus operation signal and zoom operation signal in a stable state.

[0023] Furthermore, it is provided with an infrared projection device that projects infrared rays. The infrared projection device consists of a set of two units installed facing the outer edge of a predetermined section, or two sets of four units installed facing the outer edge of the predetermined section one set at a time. When the tracker is an infrared tracker that outputs at least one of its own position and orientation within the section as a signal by receiving the infrared rays projected by the infrared projection device, it is particularly desirable because inexpensive and high-performance tracking by the infrared method becomes possible.

Effects of the Invention

[0024] According to the virtual studio system of the present invention, it is possible to create a composite video by post-processing an image obtained by photographing a subject using a well-known conventional camera for video shooting with a screen such as a green screen and the background of a three-dimensional virtual space. The camera that the cameraman is familiar with can be used as it is, and there is no need to introduce new shooting equipment. Therefore, it has a great advantage in terms of cost, and the time required for skillful operation can also be omitted. In addition, by operating the virtual camera in the three-dimensional virtual space according to the position and orientation of the camera that photographs the real subject, and synthesizing the image of the projection range specified by the virtual camera to generate a composite video, the scale of the three-dimensional virtual space is made wider than the real space, the movement amount of the virtual camera is increased more than the movement amount of the camera, and the angle of view of the virtual camera can be set to telephoto or wide angle according to the type of the camera lens. Combined with the focus and zoom operations, it becomes easier to create the video that the video creator imagines.

[0025] Further, according to another virtual studio system of the present invention, an image of a three-dimensional virtual space as a background projected in real time on panels arranged around a subject is changed in conjunction with the movement of a camera, and the camera captures a composite image while the image is changing. As a result, the subject to be photographed can view the panel, and the sense of presence during the performance can be significantly improved. In addition, since a screen for chroma key processing (such as a green screen) is not required, the colors of clothing and hair can be freely selected regardless of the background color, and defects such as omissions in fine parts such as hair tips can be avoided.

Brief Description of the Drawings

[0026]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Embodiments for Carrying Out the Invention

[0027] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In this specification, the term "image" refers to both still images and moving images in which still images are continuous.

[0028] FIG. 1 is an explanatory diagram showing the usage state of the virtual studio system 1 of the present invention. As shown in this figure, the virtual studio system 1 is a photographing system for using a predetermined section 10 such as indoors as a photographing area and creating a video by synthesizing an image of an actual subject P photographed by a camera 100 and an image of a three-dimensional virtual space.

[0029] The subject P may be of any kind as long as it has an entity such as a person, an animal, or an object.

[0030] The screen 20 located behind the subject P in the section 10 is unified in a single color such as green, for example, and a process (chroma key) of specifying and transmitting the color is enabled. Note that one side or the entire surface of the wall may be unified in a single color such as green by pasting or painting wallpaper.

[0031] In addition, on the outer edge 11 in the section 10, two sets of four infrared projection devices 30 facing each other in pairs are installed above the wall surface. In the present embodiment, the two sets of four infrared projection devices 30 are installed along the two diagonals of the section 10 by installing them at the four corners of the outer edge 11 of the section 10 so as to face the center of the section 10. However, for example, the two sets of four infrared projection devices 30 may be installed by installing them at the midpoints of each of the four sides of the outer edge 11 of the section 10 so as to face the center of the section 10.

[0032] FIG. 2 is a block diagram showing the overall configuration of the virtual studio system 1 according to the present invention, and FIG. 3 is a configuration diagram showing the detailed configuration of the devices constituting the virtual studio system 1 according to the present invention. As shown in these figures, the virtual studio system 1 is roughly composed of a camera 100 for photographing an actual subject P and a processing device 200 for depicting a three-dimensional virtual space and synthesizing it with the image photographed by the camera 100.

[0033] The camera 100 can be a camera for video shooting (digital cinema camera), and does not require a special function such as outputting numerical values such as position coordinates. In the present embodiment, the camera 100 uses a conventionally well-known camera for video shooting equipped with a lens having a focus function and a zoom function, and is fixed to a camera rig 101 for attaching peripheral devices.

[0034] By using a conventionally well-known camera for video shooting as it is in this way, the camera operator can use the camera he or she is familiar with as it is, and there is no need to introduce new equipment. Therefore, it has a great advantage in terms of cost, and the time required for skillful operation can also be omitted.

[0035] Note that the present invention can also be used with a camera having only one of the focus function and the zoom function.

[0036] A camera tracker 110 for detecting the position and orientation of the camera 100 is fixed to the camera 100, and in the present embodiment, it is fixed by being attached to the camera rig 101.

[0037] The camera tracker 110 is a device having a function of detecting its own position and orientation and outputting a camera movement signal S10, and includes a sensor unit 111 that receives the infrared rays projected from the infrared ray projection device 30 and detects its own position and orientation within the section 10, and a transmission unit 112 that transmits the camera movement signal S10 obtained by digitizing its own position and orientation detected by the sensor unit 111 by wireless communication.

[0038] The focus function of the camera 100 performs focus adjustment according to the operation of a focus ring 102 that can be manually rotated.

[0039] The focus ring 102 is attached with a focus detection means 120 capable of detecting the amount of focus operation. The focus detection means 120 comprises a rotation detection unit 130 consisting of a gear member 131 and an encoder 132, and a signal output unit 140 consisting of a motor member 141 and a focus tracker 150.

[0040] The gear member 131 is a gear ring with an adjustable inner diameter and gear-shaped protrusions formed on its outer circumference. It is attached so that its inner diameter substantially coincides with the outer diameter of the focus ring 102 and rotates integrally.

[0041] The encoder 132 is a rotary encoder for detecting the amount of rotation as a physical change amount by a sensor element and transmitting a focus rotation amount signal S21. It comprises a rotation part 133 for receiving the operation of the detection target as the amount of rotation, and a conversion part 134 for converting the rotation amount of the rotation part 133 into a digitized focus rotation amount signal S21. The rotation part 133 is meshed with the gear-shaped protrusions provided on the outer circumference of the gear member 131 and is attached and fixed to the camera rig 101. Note that the configuration of the conversion part 134 can be an optical type, a mechanical type, or the configuration of other conventionally well-known rotary encoders.

[0042] An operation knob 135 pivotally supported by the camera rig 101 and rotating in conjunction with the rotation part 133 is attached to the rotation part 133 (see FIG. 4). As the operation knob 135, a conventionally well-known operation knob for follow focus can be used. Alternatively, an operation bar may be fixed to and attached to the gear member 131 (not shown). In this way, by attaching the operation knob 135 or the operation bar that operates in conjunction with the gear member 131, it becomes easy for the cameraman to indirectly operate the focus ring 102.

[0043] The motor member 141 is a stepping motor that rotates based on the digitized focus rotation amount signal S21. In addition, a motor capable of controlling the exact rotation position and rotation speed, such as a servo motor, can be used. A controller 142 and a power supply 143 for driving and controlling the motor member 141 are separately provided.

[0044] The focus tracker 150 is a device having a function of detecting its own position and orientation and outputting a focus operation signal S20. It includes a sensor unit 151 that receives the infrared rays projected from the infrared ray projection device 30 and detects its own position and orientation within the section 10, and a transmission unit 152 that transmits the focus operation signal S20 obtained by digitizing its own position and orientation detected by the sensor unit 151 by wireless communication.

[0045] That is, when the focus tracker 150 rotates by the motor member 141, the rotation direction and rotation amount of the focus tracker 150 are output as the focus operation signal S20, and the focus operation signal S20 is reflected in the focus parameters of the virtual camera 300, enabling the focus operation of the virtual camera 300.

[0046] In this embodiment, the focus tracker 150 is assumed to perform a rotational operation. However, for example, as a configuration in which the focus tracker 150 is slid, the moving direction and moving amount of the focus tracker 150 may be output as the focus operation signal S20 (not shown).

[0047] FIG. 5 is a configuration diagram showing the signal output unit 140 in this embodiment. As shown in this figure, the signal output unit 140 includes the motor member 141, the controller 142, and the power supply 143 within the case body 144, and is an independent device with the focus tracker 150 exposed from the upper surface of the case body 144. It is connected via a cable for signal transmission from the rotation detection unit 130. Note that the connection is not limited to wired connection using a cable, and may also be wireless connection using wireless communication means (not shown).

[0048] The controller 142 drives the motor 141 according to the focus rotation amount signal S21 transmitted from the encoder 132 of the rotation detection unit 130. For example, a conventionally well-known stepping motor controller can be used. The controller 142 is provided with input / output terminals such as USB terminals, etc., and by connecting to a personal computer, it is possible to set the rotation rate of the motor member 141 and monitor input values, etc.

[0049] Furthermore, within the case body 144 of the signal output unit 140, a fan 145 and a power supply 146 for cooling the motor member 141 are provided. The power supplies 143, 146 are not limited to rechargeable batteries such as lithium ion polymer secondary batteries, and commercial power supplies such as household outlets may also be used.

[0050] In this embodiment, a battery which is a lithium ion polymer secondary battery and a commercial power supply which is a household outlet are used in combination. When connected to the commercial power supply, the current supplied from the commercial power supply is supplied to the battery, the controller, and the fan via an AC adapter. When not connected to the commercial power supply, the current supplied from the battery is supplied to the controller and the fan, enabling independent operation.

[0051] Note that the signal output unit 140 may be placed directly on the floor surface, but it is particularly desirable to lift it to a predetermined height from the floor surface and fix it using, for example, a stand, etc., to make it easier to handle.

[0052] The zoom function of the camera 100 is an optical zoom type that performs zooming by operating the internal lens position in response to the operation of a zoom ring 103 that can be manually rotated.

[0053] A zoom detection means 160 capable of detecting the zoom operation amount is attached to the zoom ring 103. The zoom detection means 160 includes a rotation detection unit 170 composed of a gear member 171 and an encoder 172, and a signal output unit 180 composed of a motor member 181 and a zoom tracker 190.

[0054] The gear member 171 is a gear ring having an adjustable inner diameter and formed with gear-shaped protrusions on its outer circumference, and is attached so as to rotate integrally with the inner diameter substantially matching the outer diameter of the zoom ring 103.

[0055] The encoder 172 is a rotary encoder for detecting the rotation amount as a physical change amount by a sensor element and transmitting a zoom rotation amount signal S31, and includes a rotation unit 173 for receiving the operation of the detection target as a rotation amount, and a conversion unit 174 for converting the rotation amount of the rotation unit 173 into a digitized zoom rotation amount signal S31. The rotation unit 173 is engaged with the gear-shaped protrusions provided on the outer circumference of the gear member 171 and is attached and fixed to the camera rig 101. Note that the configuration of the conversion unit can be an optical type, a mechanical type, or the configuration of other conventionally known rotary encoders.

[0056] An operation knob that is pivotally supported by the camera rig 101 and rotates in conjunction with the rotation unit 173 is attached to the rotation unit 173, and since it has the same configuration as the rotation unit 133 and the operation knob 135 of the focus detection means 120 shown in FIG. 4, a detailed description thereof is omitted.

[0057] The motor member 181 is a stepping motor that rotates based on the digitized zoom rotation amount signal S31. In addition, other motors such as servo motors that can control accurate rotation positions and rotation speeds can also be used. A controller and a power source for driving and controlling the motor member are separately provided.

[0058] The zoom tracker 190 is a device having a function of detecting its own position and orientation and outputting a zoom operation signal S30, and includes a sensor unit 191 that receives infrared rays projected from the infrared projection device 30 and detects its own position and orientation within the section 10, and a transmission unit 192 that transmits, by wireless communication, the zoom operation signal S30 obtained by digitizing the position and orientation of itself detected by the sensor unit 191.

[0059] That is, when the zoom tracker 190 rotates by the motor member 181, the rotation direction and rotation amount of the zoom tracker 190 are output as the zoom operation signal S30, and the zoom operation signal S30 is reflected in the parameters of the zoom of the virtual camera 300, enabling the zoom operation of the virtual camera 300.

[0060] In this embodiment, the zoom tracker 190 is configured to rotate. However, for example, the zoom tracker 190 may be configured to slide, and the moving direction and moving amount of the zoom tracker 190 may be output as the zoom operation signal S30 (not shown).

[0061] The signal output unit 180 has the same configuration as the signal output unit 140 (see FIG. 4), and thus a detailed description thereof is omitted. The signal output unit 180 is an independent device and is connected via a signal transmission cable from the rotation detection unit 170. Note that the connection is not limited to a wired connection using a cable, and may be a wireless connection using wireless communication means (not shown).

[0062] Each of the transmission units 152 and 192 in the present embodiment performs wireless communication using Bluetooth (registered trademark) and can perform wireless communication by pairing with the processing device 200.

[0063] The processing device 200 is for rendering a three-dimensional virtual space and synthesizing it with the image captured by the camera 100, and includes a rendering unit 210, a synthesizing unit 220, an arithmetic unit 230, and a storage unit 240.

[0064] The rendering unit 210 is software that can render a three-dimensional virtual space and operate a virtual camera 300 arranged in the three-dimensional virtual space. For example, software such as a game engine can be used.

[0065] The virtual camera 300 is arranged in the three-dimensional virtual space to specify a projection range, has parameters of position, orientation, and field angle, and can change the projection range specified by the virtual camera 300 by operating the numerical values of the respective parameters.

[0066] In addition, the virtual camera 300 further has parameters of focus and zoom. By operating the parameters of focus and zoom, focus operation and zoom operation can be performed like a real camera 100 having focus function and zoom function, and focusing can be performed only on the part of the image in the three-dimensional virtual space that the photographer wants to focus on, or zooming in or out to obtain an image that the photographer imagines.

[0067] In the present embodiment, the position and orientation of the virtual camera 300 are configured to move in synchronization with the position and orientation of the camera 100 that photographs a real subject P. For example, the amount of movement of the position and orientation of the virtual camera 300 may be made larger than the amount of movement of the position and orientation of the camera 100, so that shooting can be performed in a three-dimensional virtual space with a larger scale than the real space.

[0068] The composition unit 220 is software for generating a composite video by compositing a real image of the subject P photographed by the camera 100 and a CG image of the projection range specified by the virtual camera 300 in the three-dimensional virtual space. For example, software such as video editing software can be used.

[0069] The arithmetic unit 230 provides the arithmetic processing capabilities required in the rendering unit 210, the composition unit 220, and the other parts of the processing device 200, and uses a CPU and a memory.

[0070] The storage unit 240 is an auxiliary storage device for storing data, and uses a storage medium such as an HDD or an SSD. Also, the software constituting the rendering unit 210 and the composition unit 220 is stored in the storage unit 240.

[0071] The real-shot monitor 40 is a monitor for displaying an image of the real subject P photographed by the camera 100.

[0072] The virtual space monitor 50 is a monitor for displaying an image of the virtual space depicted by the rendering unit 210. Note that it may be configured to display the projection range projected by the virtual camera 300.

[0073] The composite video monitor 60 is a monitor for displaying the composite video composite by the composition unit 220.

[0074] The infrared projection device 30 is a device for projecting infrared rays. Two sets of four units are installed in pairs above the wall surfaces at the four corners of the outer edge 11 in the section 10. The infrared rays projected from the infrared projection device 30 are received by the respective sensor units in each of the trackers (camera tracker 110, focus tracker 150, zoom tracker 190) and are used to detect the positions and postures of the respective trackers within the section 10.

[0075] Note that the infrared projection device 30 may perform wired communication via a cable or wireless communication via Bluetooth (registered trademark). By connecting (pairing) with the processing device 200, it is possible to synchronize the timing of infrared projection and the like.

[0076] Hereinafter, based on an embodiment, the usage method of the virtual studio system 1 according to the present invention will be described.

[0077] <Equipment Installation> First, in the room where shooting is to be performed, two sets of four infrared projection devices 30 facing each other in pairs are installed above the wall surface of the outer edge 11 in the shooting section 10. At the same time, the camera 100, the processing device 200, the signal output units 140 and 180, and each monitor (actual shooting monitor 40, virtual space monitor 50, composite video monitor 60) are installed and connected respectively. Also, each tracker (camera tracker 110, focus tracker 150, zoom tracker 190) is paired with the processing device 200. Furthermore, a screen 20 for transmission processing unified in a single color such as green is installed at a position behind the subject P. At this time, a plurality of screens 20 may be installed. For example, by using two screens 20 and arranging them in an L shape, or using three screens 20 and arranging them in a U shape, the corners and ends of the section 10 can be widely used for shooting, and the degree of freedom of movement of the subject P and the camera 100 can be increased.

[0078] <Virtual Camera Setting> Once the machine setup is completed, the rendering unit 210 of the processing device 200 performs the rendering of the three-dimensional virtual space and activates the virtual camera 300 within the three-dimensional virtual space. Thereafter, if necessary, the parameters of the virtual camera 300 are manipulated to adjust the initial position of the virtual camera 300 in the three-dimensional virtual space, and the movement, focus operation, and zoom operation of the camera 100 are performed. It is confirmed that the camera movement signal S10 transmitted by the camera tracker 110 is reflected in the position and orientation parameters of the virtual camera 300, the focus operation signal S20 transmitted by the focus tracker 150 is reflected in the focus parameter of the virtual camera 300, and the zoom operation signal S30 transmitted by the zoom tracker 190 is reflected in the zoom parameter of the virtual camera 300.

[0079] <Shooting in the real space> Once the equipment installation and virtual camera settings are completed, the preparations for shooting are complete. Place the subject P at a predetermined position in front of the screen 20 and shoot the subject P with the camera 100. At this time, the movement, focus operation, and zoom operation of the camera 100 can be performed. The image captured by the camera 100 is displayed on the actual shooting monitor 40, allowing the editor to check it in real time. Note that multiple actual shooting monitors may be prepared so that multiple people can check simultaneously.

[0080] <Shooting in the three-dimensional virtual space> Simultaneously with shooting the subject P in the real space with the camera 100, an image of the projection range projected by the virtual camera 300 within the three-dimensional virtual space is captured. The image captured by the virtual camera 300 is displayed on the virtual space monitor 50, allowing the editor to check it in real time.

[0081] When the camera 100 is moved, the camera tracker 110 attached to the camera rig 101 detects its own position and orientation and outputs a camera movement signal S10, and the camera movement signal S10 is reflected in the position and orientation parameters of the virtual camera 300 to enable the movement of the virtual camera 300.

[0082] When the focus operation of the camera 100 is performed, a focus rotation amount signal S21 is output by the rotation detection unit 130 attached to the focus ring 102, and the focus tracker 150 of the signal output unit 140 that has received the focus rotation amount signal S21 rotates.

[0083] When the focus tracker 150 rotates, the rotation direction and rotation amount of the focus tracker 150 are output as a focus operation signal S20, and the focus operation signal S20 is reflected in the focus parameters of the virtual camera 300, enabling the focus operation of the virtual camera 300.

[0084] Furthermore, when the zoom operation of the camera 100 is performed, a zoom rotation amount signal S31 is output by the rotation detection unit 170 attached to the zoom ring 103, and the zoom tracker 190 of the signal output unit 180 that has received the zoom rotation amount signal S31 rotates.

[0085] When the zoom tracker 190 rotates, the rotation direction and rotation amount of the zoom tracker 190 are output as a zoom operation signal S30, and the zoom operation signal S30 is reflected in the zoom parameters of the virtual camera 300, enabling the zoom operation of the virtual camera 300.

[0086] In this way, by uniformly using each tracker (camera tracker 110, focus tracker 150, zoom tracker 190) that can output its own position and orientation as signals, there is no need for the camera 100 itself to have special functions, and the parameters of the virtual camera 300 can be simultaneously operated by a unified-form signal without passing through unnecessary hardware or software such as a controller between each tracker and the processing device 200. Therefore, the linkage between the real camera 100 and the virtual camera 300 can be performed simply, reliably, and inexpensively.

[0087] <Transparency processing> The image of the subject P captured by the camera 100 is first subjected to a transparency process by the synthesizing unit 220 in the processing device 200 to make the color of the screen 20 used as the background a specified color and transmit it.

[0088] <Synthesis processing> After the transparency process is completed, the synthesizing unit 220 in the processing device 200 synthesizes the image of the subject P after the transparency process and the image of the projection range captured by the virtual camera 300 in the three-dimensional virtual space to generate a synthesized video. The synthesized video is displayed on the monitor 60 for the synthesized video, and can be confirmed by the editor in real time. In addition, it can be stored in the storage unit 240 in the processing device 200 or output externally.

[0089] FIG. 4 is a block diagram showing the overall configuration in different embodiments of the virtual studio system 2 of the present invention. As shown in this figure, the virtual studio system 2 is different from the virtual studio system 1 in that instead of the screen 20 with a single color background installed around the subject P, a panel 70 is used, and the processing device 200 does not have a synthesizing unit 220.

[0090] The panel 70 is a device capable of displaying an image such as a liquid crystal panel (LCD) or an organic EL panel (OLED), and is required to have a height and width greater than that of the subject P, and is preferably a high-resolution device capable of displaying an image with as high a definition as possible. Note that a plurality of panels 70 may be installed at different angles or depths.

[0091] The image of the projection range in the three-dimensional virtual space specified by the virtual camera 300 is displayed on the panel 70.

[0092] The virtual studio system 2 of this embodiment captures a composite video by capturing a subject P with the camera 100 in a state where an image of a projection range in the three-dimensional virtual space is displayed on the panel 70. That is, since the image captured by the camera 100 directly becomes the composite video, a monitor 60 for the composite video is not required.

[0093] According to the virtual studio system 2 of this embodiment, the subject P, i.e., the person to be photographed, can view the panel 70, and the sense of presence can be significantly improved when performing acting. In addition, since a screen (such as a green screen) for chroma key processing is not required, the color of clothing and hair can be freely selected without depending on the background color, and defects such as omission errors in fine parts such as hair tips can be avoided.

[0094] In each embodiment of the present invention, each tracker (camera tracker 110, zoom tracker 150, focus tracker 190) uses infrared detection means in which its respective sensor units 111, 151, 191 receive infrared light projected from the infrared projection device 30 to detect its own position and orientation within the section 10. However, other detection means for detecting its own position and orientation, such as an optical method or a magnetic method, which are conventionally well-known, may also be used.

[0095] As described above, according to the virtual studio system of the present invention, it is possible to create a composite video by post-processing and synthesizing an image of a subject photographed using a conventionally well-known camera for video shooting with a screen such as a green screen and the background of a three-dimensional virtual space. Since the camera that the cameraman is familiar with can be used as it is without the need to introduce new shooting equipment, it has a great advantage in terms of cost, and the time required for skillful operation can also be omitted. Further, by operating a virtual camera in a three-dimensional virtual space according to the position and orientation of a camera that photographs an actual subject, and generating a composite video by synthesizing an image of a projection range specified by the virtual camera, the scale of the three-dimensional virtual space can be made wider than the real space, the movement amount of the virtual camera can be increased more than the movement amount of the camera, and the angle of view of the virtual camera can be set to telephoto or wide angle according to the type of the camera lens. Together with the operations of focus and zoom, it becomes easier to create the video that the video creator imagines.

[0096] Further, according to another virtual studio system of the present invention, a composite video is captured by capturing with a camera in a state where an image of a three-dimensional virtual space as a background projected in real time onto a panel arranged around a subject changes in conjunction with the movement of the camera. Since the subject to be photographed can see the panel, the sense of presence can be significantly improved when performing acting. In addition, since a screen (such as a green screen) for chroma key processing is not required, the color of clothing and hair can be freely selected without depending on the background color, and it has the advantage of being able to avoid problems such as omission of fine parts such as hair tips.

Explanation of Signs

[0097] 1, 2 virtual studio system, 10 compartments, 11 outer edges, 20 screens, 30 infrared projection devices, 40 live-action monitors, 50 virtual space monitors, 60 composite video monitors, 70 panels, 100 cameras, 101 camera rigs, 102 focus rings, 103 zoom rings, 110 camera trackers, 111 sensor units, 112 transmission units, 120 focus detection means, 130 rotation detection units, 131 gear members, 132 encoders, 133 rotating parts, 134 conversion parts, 135 operation knobs, 140 signal output units, 141 motor members, 142 controllers, 143 power supplies, 144 case bodies, 145 fans, 146 power supplies, 150 focus trackers, 151 sensor units, 152 transmission units, 160 zoom detection means, 170 rotation detection units, 171 gear members, 172 encoders, 173 rotating parts, 174 conversion parts, 180 signal output units, 181 motor members, 190 zoom trackers, 191 sensor units, 192 transmission units, 200 processing devices, 210 rendering units, 220 composite units, 230 arithmetic units, 240 storage units, P subject, S10 camera movement signal, S20 focus operation signal, S21 focus rotation amount signal, S30 zoom operation signal, S31 zoom rotation amount signal

Claims

1. A virtual studio system for creating an image by synthesizing a real subject and the background of a three-dimensional virtual space, a camera having a focus function or a zoom function by a rotation operation of a focus ring or a zoom ring, and photographing a real subject disposed within a predetermined section serving as a photographing area; a camera tracker fixed to the camera, detecting the position and orientation of the camera by detecting its own position and orientation with respect to the predetermined section, and outputting a camera movement signal; a rotation detection unit provided on the focus ring or the zoom ring, and including an encoder that quantifies the rotation operation of the focus ring or the zoom ring as a rotation amount; a signal output unit disposed within the predetermined section away from the camera, connected to the rotation detection unit by wire or wirelessly, and having a motor member and a tracker, wherein the motor member rotates according to the rotation amount quantified by the rotation detection unit, and the tracker changes its own position or orientation within the predetermined section by rotating or sliding in conjunction with the rotation of the motor member, and outputs at least one of its own position and orientation with respect to the predetermined section as an operation signal; a rendering unit that depicts an image of a three-dimensional virtual space; a virtual camera disposed within the three-dimensional virtual space, having parameters of position, orientation, and field angle, and parameters of focus or zoom, wherein the parameters of position and orientation are operated based on the camera movement signal output by the camera tracker, the parameters of focus or zoom are operated based on the operation signal output by the tracker, and a projection range is specified based on its own position, orientation, field angle, and parameters of focus or zoom; a synthesizing unit that generates a synthesized image by synthesizing the image of the subject photographed by the camera and the image of the projection range within the three-dimensional virtual space. A virtual studio system characterized by the above.

2. A virtual studio system for creating an image by synthesizing a real subject and the background of a three-dimensional virtual space, a camera having a focus function or a zoom function by a rotation operation of a focus ring or a zoom ring, and photographing a real subject disposed within a predetermined section serving as a photographing area; A camera tracker that is fixed to the camera, detects its own position and orientation with respect to the predetermined section, detects the position and orientation of the camera, and outputs a camera movement signal; A rotation detection unit including an encoder provided on the focus ring or the zoom ring, which quantifies the rotation operation of the focus ring or the zoom ring as a rotation amount; A signal output unit that is arranged within the predetermined section away from the camera, is connected to the rotation detection unit by wire or wirelessly, and has a motor member and a tracker. The motor member rotates according to the rotation amount quantified by the rotation detection unit, and the tracker changes its own position or orientation within the predetermined section by rotating or sliding in conjunction with the rotation of the motor member, and outputs at least one of its own position and orientation with respect to the predetermined section as an operation signal; A rendering unit that depicts an image of a three-dimensional virtual space; A virtual camera that is arranged within the three-dimensional virtual space, has parameters of position, orientation, and field angle, and parameters of focus or zoom, the parameters of the position and orientation are operated based on the camera movement signal output by the camera tracker, the parameters of the focus or zoom are operated based on the operation signal output by the tracker, and the projection range is specified based on its own position, orientation, field angle, and the parameters of focus or zoom; A panel that is installed around the subject and outputs an image of the projection range within the three-dimensional virtual space specified by the virtual camera; By photographing the subject with the camera in a state where an image of the projection range within the three-dimensional virtual space is output to the panel, a composite video obtained by synthesizing the image of the subject and the image of the projection range within the three-dimensional virtual space is photographed; A virtual studio system characterized by the above.

3. The camera tracker and the tracker are detection means for detecting their own position and orientation in a unified manner selected from an infrared method, an optical method, and a magnetic method; The virtual studio system according to claim 1 or 2, characterized by the above.

4. An infrared projection device that projects infrared rays is provided; The infrared projection device consists of two units arranged in a pair and facing the outer edge of the predetermined section, or four units arranged in two pairs with each pair facing the outer edge of the predetermined section. Each of the camera tracker and the tracker has a sensor unit, and is an infrared tracker that outputs, as a signal, at least one of its own position and orientation with respect to the predetermined section by receiving the infrared rays projected from the infrared projection device with the sensor unit. The virtual studio system according to claim 1, 2, or 3, characterized by the above.

Images