Three-dimensional image generation system, camera arrangement structure, and composite image generation system

Abstract

This provides a means to create a three-dimensional image of the entire filming location, including the surrounding space. [Solution] A three-dimensional video system comprising at least a plurality of cameras 10 placed at the shooting site and a three-dimensional video generation unit 20 that generates a three-dimensional video based on a plurality of two-dimensional videos each captured by the plurality of cameras 10. The three-dimensional video exhibits a form that includes at least a video that has been made three-dimensional using the two-dimensional videos of all cameras 10 and a video that has been made three-dimensional without using the two-dimensional videos of all cameras 10. Furthermore, when one of the plurality of cameras 10 is designated as the main camera and the other cameras 10 as sub-cameras, the main camera is arranged to satisfy both of the following conditions: (Condition 1) At least a portion of the shooting range of the main camera overlaps with at least a portion of the shooting range of at least one sub-camera. (Condition 2) The shooting range of the main camera is not entirely occupied by background material D for chroma key processing.

Description

Technical Field

[0001] The present invention relates to a three-dimensional video generation system for generating a three-dimensional video based on a plurality of two-dimensional videos respectively captured by a plurality of cameras, an arrangement structure of cameras at a shooting site, and a composite video generation system.

Background Art

[0002] As shown in the following prior art documents, a three-dimensional video generated based on a plurality of two-dimensional videos respectively captured by a plurality of cameras is also called a "volumetric video".

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Conventional 3D video production methods have only proposed a usage approach that assumes the object being surrounded by cameras is photographed from different angles using multiple cameras, and then the space (region) other than the object is removed from these images to create 3D data of only the object, which is then combined with separately prepared 3D background data to create the final output. The idea of ​​creating 3D video of the entire shooting environment has not been proposed. This is presumably because there was a preconceived notion that generating 3D video requires work in a dedicated shooting studio with many cameras already installed, and since the background surrounding the shooting studio is unnecessary for the final output, it is natural to cover it with chroma key background material.

[0006] Therefore, one of the objectives of the present invention is to provide a means that makes it possible to create a three-dimensional image of the entire space of the filming location. [Means for solving the problem]

[0007] The present invention, made to solve the above problems, comprises at least a plurality of cameras arranged at the shooting site, and a three-dimensional image generation unit that generates a three-dimensional image based on a plurality of two-dimensional images each captured by the plurality of cameras, wherein the three-dimensional image exhibits an appearance that includes at least an image that has been made three-dimensional using the two-dimensional images of all of the plurality of cameras, and an image that has been made three-dimensional using the two-dimensional images of some of the plurality of cameras. Furthermore, the present invention relates to a camera arrangement structure for generating three-dimensional images of a shooting location, comprising a plurality of cameras, each capable of capturing two-dimensional images, wherein when one of the plurality of cameras is designated as the main camera and the others as sub-cameras, the main camera is arranged to satisfy both of the following conditions: (Condition 1) at a position where at least a portion of the shooting range of the main camera overlaps with at least a portion of the shooting range of at least one of the sub-cameras, and (Condition 2) at a position where the shooting range of the main camera is not entirely occupied by background material for chroma key processing. Furthermore, the present invention relates to a composite image generation system for displaying a composite image formed by combining multiple images to a user, comprising at least: a real image acquisition unit that acquires real images, which are images taken in approximately the user's line of sight, in a manner that includes depth information; a measurement unit that acquires measurement information including the user's position and orientation; a three-dimensional image management unit that manages three-dimensional images generated from multiple two-dimensional images taken by multiple cameras arranged at the shooting site; a synthesis unit that generates a composite image by synthesizing a processed image, which is obtained by removing arbitrary parts based on the depth information from the real image, and the three-dimensional image based on the measurement information; and a display unit that can be attached to the user's head and displays the composite image to the user, wherein the three-dimensional image exhibits a manner that includes at least an image that has been made three-dimensional using the two-dimensional images of all cameras and an image that has been made three-dimensional without using the two-dimensional images of all cameras. [Effects of the Invention]

[0008] According to the present invention, by creating a three-dimensional image of the entire shooting location, it becomes possible to open up avenues for utilizing three-dimensional images in a wider range of applications. [Brief explanation of the drawing]

[0009] [Figure 1] A schematic plan view showing an example of the construction of a three-dimensional imaging system according to Example 1. [Figure 2] A screenshot from a 3D video of a construction site. [Figure 3] A schematic plan view showing an example of the construction of a three-dimensional imaging system according to Example 2. [Figure 4] An illustrative diagram showing the configuration of the composite video generation system according to Example 3. [Figure 5] A schematic diagram showing the image of the processed video generation process. [Figure 6] A schematic diagram illustrating the process of generating a composite image. [Modes for carrying out the invention]

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

Embodiment

[0011] [Three-dimensional video generation system (configured with an enclosed area by cameras)]

[0012] <1>Overall configuration (Fig. 1) The three-dimensional video generation system is configured to include at least a plurality of cameras 10 and a three-dimensional video generation unit 20. Each component can be realized by arbitrarily combining hardware and software. In addition, a configuration in which each part is an individual device, a configuration in which a plurality of each part is incorporated into one device and integrated, etc. can be adopted. In the present invention, the number of each component, the location of each component, the number of videos used for generating the three-dimensional video, etc. are not limited unless otherwise specified below. Hereinafter, details of each component will be described.

[0013] <2>Cameras (Fig. 1) The camera 10 is a device installed at the shooting site for acquiring two-dimensional videos. In the present invention, the type of the camera 10 is not particularly limited, and it is not limited to general digital cameras or video cameras capable of shooting still images or moving images. A depth camera capable of giving depth information to two-dimensional videos may also be used.

[0014] <2.1>Regarding the number of cameras (Fig. 1) In the present invention, the number of cameras 10 used at the shooting site is not particularly limited. However, for example, if the number of cameras 10 is too large, it is conceivable that many cameras will be three-dimensionally reflected in the generated three-dimensional video, which may not be preferable from the perspective of immersion for the user experiencing the three-dimensional video. On the contrary, if the number of cameras is too small, it is conceivable that it will be difficult to obtain a finely three-dimensionalized three-dimensional video. Therefore, in the present invention, unlike the specialized photography studios described in the prior art, it is not assumed to use dozens of cameras 10. Considering not choosing the shooting location and portability, the number of cameras used is set to be 2 or more and 10 or less, more preferably 3 or more and 5 or less. In the present embodiment shown in FIG. 1, four cameras 10 (10a, 10b, 10c, 10d) are used. By changing the line-of-sight direction of each camera 10 inward by 90 degrees each, they are arranged so as to surround and view any area of the shooting scene in plan view.

[0015] <3>Three-dimensional video generation unit (FIG. 1) The three-dimensional video generation unit 20 has a function of generating a three-dimensional video based on the two-dimensional videos captured by each camera 10. The three-dimensional video generated by the three-dimensional video generation unit 20 includes at least a video three-dimensionalized using the two-dimensional videos of all the cameras 10 and a video three-dimensionalized without using the two-dimensional videos of all the cameras 10. Note that for the information processing used in generating the three-dimensional video in the present invention, since known technologies can be used, detailed descriptions are omitted.

[0016] <4>Arrangement conditions of cameras (FIG. 1) In the system according to the present embodiment among the three-dimensional video generation systems according to the present invention, when any one of the plurality of cameras 10 is used as the main camera and the other cameras 10 are used as sub-cameras, it is configured to satisfy all of the following conditions 1 to 3. (Condition 1) At least a part of the shooting range of the main camera and at least a part of the shooting range of at least one sub-camera overlap. (Condition 2) All of the shooting range of the main camera is not occupied by the background material for chroma key processing. This condition includes cases where there is no background material at all in the shooting range of the main camera or where a part of the shooting range of the main camera is covered by the background material. For example, in the case of the conventional photography studio shown in the prior art, when all of the studio walls, floors, ceilings, etc. are covered with the background material for chroma key, this condition is not satisfied. (Condition 3) At least one sub-camera must be included in the shooting range of the main camera.

[0017] The above conditions may be applied to all of the cameras 10, or to only some of the cameras 10.

[0018] Figure 1 shows the arrangement of each camera 10 and the overlap of the shooting ranges of each camera 10 when viewed from above. Furthermore, the information processing device that constitutes the three-dimensional image generation unit 20 is covered with background material D so that it does not appear in the three-dimensional image. The following provides details on how each camera 10 satisfies the above-mentioned placement conditions.

[0019] <4.1>When camera 10a is used as the main camera (Figure 1) (1) Regarding Condition 1 Condition 1 is satisfied because at least a portion of the shooting range of the main camera 10a overlaps with at least a portion of the shooting ranges of the sub-cameras 10b, 10c, and 10d. (2) Regarding condition 2 Condition 2 is satisfied because the entire shooting range of the main camera 10a is not occupied by the background material D for chroma keying. (3) Regarding condition 3 Since the shooting range of the main camera 10a includes the sub-camera 10b, condition 3 is satisfied.

[0020] <4.2>When camera 10b is used as the main camera (Figure 1) (1) Regarding Condition 1 Condition 1 is satisfied because at least a portion of the shooting range of the main camera 10b overlaps with at least a portion of the shooting ranges of the sub-cameras 10a, 10c, and 10d. (2) Regarding condition 2 Although a portion of the main camera 10b's shooting range is occupied by the background material D used for chroma keying, the entire shooting range is not occupied by the background material D used for chroma keying, thus satisfying condition 2. (3) Regarding condition 3 Since the shooting range of the main camera 10b includes the sub-camera 10a, condition 3 is satisfied.

[0021] <4.3>When cameras 10c and 10d are used as the main cameras (Figure 1) Regarding cameras 10c and 10d, it is clear from Figure 1 that they satisfy conditions 1 to 3 described above, so a detailed explanation will be omitted.

[0022] <4.4> Overlap of shooting range In the camera arrangement described in this embodiment (Figure 1), an area (overlapping area 30) is formed within the shooting site A where the shooting ranges of each camera 10 overlap. Object C within this overlapping region 30 will have a three-dimensional image generated based on the two-dimensional images obtained from each of the four cameras 10 (10a to 10d). Furthermore, for areas other than the aforementioned overlapping area 30, such as the walls B, floor, and ceiling of the filming location A, and each object C present in the filming location A, images of an appropriate nature will be generated using two-dimensional images obtained from one to three cameras 10.

[0023] <4.5> Regarding the manner in which the image is incorporated into a three-dimensional image. An example of how the object C, camera 10, and surrounding wall B are incorporated into the three-dimensional image shown in Figure 1 will be explained below. (1) Object C2 is made three-dimensional based on the two-dimensional images from cameras 10a, 10b, and 10d. (2) Object C3 is made three-dimensional based on the two-dimensional images from cameras 10a, 10b, and 10c. (3) Object C4 is made three-dimensional based on the two-dimensional images from camera 10a and camera 10d. (4) Camera 10b is displayed two-dimensionally (planarly) within the three-dimensional image based on the two-dimensional image from camera 10a. (Note that if depth information is linked to the image captured by camera 10a, it can also be incorporated into the three-dimensional image in a three-dimensional state based on the depth information.) (5) Of the wall B on the back side of camera 10b, the part where the shooting range of camera 10a and the shooting range of camera 10c overlap (upper right corner in Figure 1) will be made three-dimensional based on the two-dimensional images of camera 10a and camera 10d respectively, and the part where the shooting range of camera 10a and the shooting range of camera 10d overlap (upper left corner in Figure 1) will be made three-dimensional based on the two-dimensional images of camera 10a and camera 10d respectively. Furthermore, the portion of wall B that is only within the shooting range of camera 10a (the middle portion in Figure 1) will be displayed two-dimensionally (planarly) within the three-dimensional image based on the two-dimensional image from camera 10a. (Note that if depth information is linked to the image captured by camera 10a, it can also be incorporated into the three-dimensional image in a three-dimensional state based on the depth information.)

[0024] <5> Image of 3D image generation (Figure 2) Figure 2 shows a screenshot of a three-dimensional image generated by the three-dimensional image generation system according to the present invention. In the three-dimensional image generated by the three-dimensional image generation system according to the present invention, the entire work site, which is the shooting location, and the workers, which are enclosed in a thick frame (shown for ease of explanation) in Figure 2, are rendered in three dimensions, and the user can view this three-dimensional image from any viewpoint. In this invention, the display destination of the generated three-dimensional image is not particularly limited and can be any known display device, such as a display built into or attached to an information processing device, or an HMD (head-mounted display) worn by the user.

[0025] <6> summary According to the three-dimensional image generation system of the present invention, at least one of the effects described below can be obtained. (1) In addition to creating a three-dimensional image of the object defined as the subject in conventional three-dimensional video (volumetric video) as described in the Prior Art section, by processing the image to include the images of objects (people, objects, structures, etc.) surrounding the subject in the three-dimensional image, it is possible to create a three-dimensional image of the entire space of the shooting location. (2) Because the system can be built with fewer cameras than those used in conventional shooting studios as described in the prior art, it can also be applied to recording locations other than shooting studios (such as construction sites) as three-dimensional images. [Examples]

[0026] [Three-dimensional image generation system (configuration without a camera-enclosed area)]

[0027] <1> Overall structure (Figure 3) In the three-dimensional image generation system according to the present invention, multiple cameras 10 may be arranged so as not to surround any area of ​​the shooting site A in a planar view. For example, in the plan view shown in Figure 3, three cameras 10 (10a, 10b, 10c) are used, and the lines of sight of the left and right cameras 10b and 10c are rotated inward by approximately 60 degrees relative to the line of sight of the central camera 10a, resulting in a configuration in which the wall B on the back side of camera 10a is not included in the shooting range of the two-dimensional image used for the three-dimensional image.

[0028] <2> Camera placement conditions (Figure 3) Each of the cameras 10 (10a, 10b, 10c) is positioned such that it satisfies only conditions 1 and 2 of the positioning conditions described in Example 1. Object C1 is then rendered in 3D based on the 2D images from cameras 10a, 10b, and 10c, respectively. Object C2 and the approximate right half of wall B1 are rendered in 3D based on the 2D images from cameras 10a and 10b, respectively. Object C3 and the approximate left half of wall B1 are rendered in 3D based on the 2D images from cameras 10a and 10c, respectively.

[0029] <3> summary The three-dimensional image generation system according to this embodiment is suitable for use in shooting locations where the generation of 3D images covering the entire circumference is not required. [Examples]

[0030] [Composite Video Generation System]

[0031] The three-dimensional images generated by the three-dimensional image generation system according to the present invention can also be used as source images in a system (synthetic image generation system) that displays a composite image to the user, which is created by combining real-world images of the user's surroundings with source images, as described below.

[0032] <1> Overall structure (Figure 4) The composite image generation system shown in Figure 4 comprises at least a real-world image acquisition unit 100, a measurement unit 200, a three-dimensional image management unit 300, a synthesis unit 400, and a display unit 500. Each component can be implemented by arbitrarily combining hardware and software, and various configurations can be adopted, such as each component being a separate device, or multiple components being integrated into a single device. Furthermore, in this invention, there are no particular limitations on the number of parts, the location of each part, or the number of images used for synthesis. The details of each part are explained below.

[0033] <2> Real-world image acquisition unit (Figure 4) The real-world image acquisition unit 100 has the function of acquiring real-world images 110 in the approximate direction of the user's line of sight, in a manner that includes depth information. This "Real-World Image 110" is a video recording of the real world, including the approximate direction of the user's line of sight; in other words, it is a real-time video showing the space around the user. The real-world image acquisition unit 100 can use a camera built into a head-mounted display (hereinafter also referred to as "HMD") worn by the user, a camera attached externally to the HMD, or any other camera that can be worn by the user. Cameras that can be used as the real-world image acquisition unit 100 include so-called depth cameras or depth sensors, which have a built-in depth sensor. In this invention, the real-world image acquisition unit 100 only needs to be a camera capable of capturing images in at least the approximate direction of the user's line of sight, and does not exclude cameras capable of capturing images around the user's entire surroundings.

[0034] <3> Measurement unit (Figure 4) The measurement unit 200 has the function of acquiring user measurement information 210. This "measurement information 210" includes at least information that allows for the recognition of the user's orientation and position. The synthesis unit 400, described later, generates a synthesized image 420 based on this measurement information 210. The measurement unit 200 can use a group of sensors such as an angle sensor, an acceleration sensor, and a gyroscope, as well as a motion capture device, a GPS device, and the like. Furthermore, the measurement unit 200 includes all types, such as those installed in the living space where the user resides, those built into the HMD worn by the user, those separately attached to the user's body, and those held by the user.

[0035] <4> Three-dimensional image management unit (Figure 4) The three-dimensional image management unit 300 has the function of managing at least one three-dimensional image 310. This "three-dimensional image 310" includes a stereoscopic image consisting of still images or videos (including slow-motion videos and time-lapse videos) that include a 3D model generated based on multiple two-dimensional images obtained by capturing a certain subject (including anything such as a person, object, or structure) with multiple cameras. For example, a three-dimensional image generated by the three-dimensional image generation system according to the present invention, as described in Examples 1 and 2, can be used.

[0036] In the present invention, "management" of the three-dimensional video 310 by the three-dimensional video management unit 300 includes the act of generating the three-dimensional video 310 based on a plurality of two-dimensional videos received by the three-dimensional video management unit 300, as well as the act of receiving and saving three-dimensional videos 31 that are generated in real time (streaming video) or have already been generated by other functional units. Therefore, the three-dimensional video management unit 300 can be configured to have the functions of the three-dimensional video generation system according to the present invention, or to be configured to handle three-dimensional videos generated by the three-dimensional video generation system according to the present invention. In other words, the three-dimensional video 310 managed by the three-dimensional video management unit 300 may be a video generated by the three-dimensional video management unit 300 itself, which is equipped with the functions of the three-dimensional video generation system according to the present invention, or it may be a video generated by a three-dimensional video generation system according to the present invention, which is provided separately from the three-dimensional video management unit 300.

[0037] <5> Composite section (Figure 4) The synthesis unit 400 has the function of generating an image (processed image) in which at least an arbitrary portion based on depth information has been removed from the real image 110 (depth-based removal processing), and the function of synthesizing the processed image and the three-dimensional image 410 acquired and / or stored by the three-dimensional image management unit 300 based on the measurement information 210 acquired by the measurement unit 200. The synthesis unit 400 can be configured with hardware such as a CPU, memory, and GPU controlled by software. In the present invention, the synthesis unit 400 may further combine and synthesize other source images (such as VR images, CG images, and 360-degree images) in addition to the three-dimensional image 310 and the processed image.

[0038] <5.1> Example of removal process (Figure 4) In the present invention, the removal process performed on the real image 110 includes at least a depth-based removal process, and also includes a chroma-based removal process as necessary. "Depth-based removal processing" is a process that removes elements from a real-world image 110 containing depth information based on the depth information values. For example, it involves removing parts of the image that have depth information exceeding a predetermined distance. "Chroma removal processing" is a process that performs removal based on the values ​​of color space information on the image after depth removal processing has been performed on the real image 110. More specifically, it is a process that removes the parts corresponding to the homogeneous color of background materials placed in the user's living space that are visible within the field of view of the real image 110, and makes it possible to composite other images onto the removed parts. These processes can be found in detail in the patent documents conceived by the present applicant (Japanese Patent Publication No. 6717486, Japanese Patent Publication No. 6991494, Japanese Patent Publication No. 7157271, etc.).

[0039] <5.2> Image of processed video generation (Figure 5) Figure 5 shows an image illustrating the results of applying each removal process to the real-world image 110. The real-world image 110 shown in Figure 5(a) is an image captured by a real-world image acquisition unit 100 installed on an HMD worn by the user, and shows the user's hands and the background in the direction of the user's line of sight. By applying a depth-based removal process to this real-world image 110, for example, which removes areas with depth information of 50 cm or more, a processed image 410a in which only the user's hands are extracted can be generated, as shown in Figure 5(b). If the user's hands are sufficiently extracted in the processed image 410a after this depth-based decompression process, this processed image 410a can be used as the final processed image 410 for synthesis. For example, if you want to extract them in more detail, you can further apply chroma-based decompression to the processed image 410a to remove the background color area remaining around the contour lines of the user's hands, generating the processed image 410b shown in Figure 5(c), and this processed image 410b can be used as the final processed image 410.

[0040] <6> Display unit (Figure 4) The display unit 500 is a device equipped with the function of displaying the composite image generated by the synthesis unit 400 to the user. The display unit 500 can use a user-worn HMD, goggles, or other portable devices such as smartphones that can be housed in a housing to function as goggles.

[0041] <7> Image of the generation of a composite image (Figure 6) Figure 6 shows an example of how the composite image is generated. Figure 6(a) shows a three-dimensional image 310 from a three-dimensional image management unit 300 equipped with the functions of the three-dimensional image generation system according to the present invention, in which the entire shooting site and workers (objects) are rendered in three dimensions. Figure 6(b) shows the processed image 410 after the real image 110 acquired by the real image acquisition unit 100 has been processed by the synthesis unit 400 to remove at least arbitrary parts based on depth information (depth-based removal processing). Only the hands of the user wearing the HMD and the rod held by both hands are visible. Figure 6(c) is a composite image 420 created by combining the images shown in Figures 6(a) and (b) using the compositing unit 400 based on the user's measurement information. In this image, the user's hands and a stick are visible in front of the workers and the work site.

[0042] <8> summary The composite image generation system according to the present invention can achieve, for example, the following effects. (1) Because the composite image includes a three-dimensional image generated from multiple two-dimensional images taken by multiple cameras, it is possible to provide users with a more realistic composite image experience. (2) In a composite image, even if there are areas in a particular field of view where the user cannot see them because they are obstructed by 3D models of people or objects included in the three-dimensional image, the user can see those areas by moving or changing the direction of their head to avoid the obstructing 3D models. (3) By moving so that the user overlaps with the 3D model in the three-dimensional image, the user can experience a view as if they were one with the 3D model. (4) Synthesized video can be used for on-site instruction, training materials for workers (distributing and saving footage of skilled workers at work), recording and managing work details, maintenance work such as visual inspections of equipment at work sites, and as reference material for tracing the cause of malfunctions by looking back at past incidents. (5) By using real-time generated streaming video for the three-dimensional image, the user can see real-time video of the shooting site, which is the source of the three-dimensional image. For example, when a construction site is used as the shooting site, and a skilled technician who is a user of the system according to this embodiment is giving instructions to assembly workers at the construction site, the skilled technician can display a composite image, which is generated from streaming video of the construction site captured in real time, on a display unit worn by the skilled technician. This allows the skilled technician, even from a distance, to perceive the construction site in three dimensions as if they were actually there, and to give instructions to the assembly workers. [Explanation of Symbols]

[0043] 10: Camera 20: 3D image generation section 21: Three-dimensional images A: Filming location B:Wall C: Object D: Background material 100: Real-world image acquisition unit 110: Real footage 200: Measurement Unit 210: Measurement Information 300: Three-Dimensional Video Management Department 310: Three-dimensional images 400: Synthesis part 410: Processed video 420: Composite video 500: Display section

Claims

1. Multiple cameras are positioned at the filming location, A three-dimensional image generation unit generates a three-dimensional image based on multiple two-dimensional images captured by multiple cameras, It must have at least the following: The three-dimensional image is characterized by having an appearance that includes at least an image that has been converted into a three-dimensional image using two-dimensional images from all of the multiple cameras, and an image that has been converted into a three-dimensional image using two-dimensional images from some of the multiple cameras. Three-dimensional image generation system.

2. When one of the aforementioned multiple cameras is designated as the primary camera and the others as secondary cameras, The main camera, (Condition 1) A position where at least a portion of the shooting range of the main camera and at least a portion of the shooting range of at least one sub-camera overlap, And, (Condition 2) A position in which the shooting range of the main camera is not entirely occupied by background material for chroma keying, It is characterized by being arranged to satisfy both conditions, The three-dimensional image generation system according to claim 1.

3. The aforementioned main camera further, (Condition 3) A position in which at least one sub-camera is included in the shooting range of the main camera, A feature characterized by being arranged to satisfy the following conditions: The three-dimensional image generation system according to claim 2.

4. The number of the aforementioned cameras is characterized by being between two and ten. A three-dimensional image generation system according to any one of claims 1 to 3.

5. The number of cameras is characterized by being 3 or more and 5 or less. A three-dimensional image generation system according to any one of claims 1 to 3.

6. A camera arrangement structure for generating three-dimensional images of a filming location, Each unit has multiple cameras capable of capturing two-dimensional images. When one of the aforementioned multiple cameras is designated as the primary camera and the others as secondary cameras, The main camera mentioned above, (Condition 1) A position where at least a portion of the shooting range of the main camera and at least a portion of the shooting range of at least one sub-camera overlap, And, (Condition 2) A position in which the shooting range of the main camera is not entirely occupied by background material for chroma keying, It is characterized by being arranged in such a way that it satisfies both of the following conditions: Camera placement structure in a three-dimensional video shooting location.

7. The aforementioned main camera further, (Condition 3) A position in which at least one sub-camera is included in the shooting range of the main camera, A characteristic feature is that it is arranged to satisfy the following conditions: The camera arrangement structure in a three-dimensional image shooting site according to claim 5.

8. A composite video generation system for displaying a composite video, which is made by combining multiple videos, to a user, A real-world image acquisition unit acquires real-world images, which are images taken in approximately the direction of the user's line of sight, in a manner that incorporates depth information. A measurement unit that acquires measurement information including the user's position and orientation, The 3D video management unit manages the 3D video generated from multiple 2D images captured by multiple cameras positioned at the filming location, A synthesis unit generates a composite image by combining the processed image, which is obtained by removing arbitrary parts based on the depth information from the real image, with the three-dimensional image, based on the measurement information. A display unit that can be attached to the user's head and displays the synthesized image to the user, It must have at least the following: The aforementioned three-dimensional image is The present invention is characterized by exhibiting an embodiment that includes at least an image that has been converted to three dimensions using two-dimensional images from all cameras, and an image that has been converted to three dimensions without using two-dimensional images from all cameras. A system for generating composite images.

Images