Method for calculating camera coordinates and orientation at a filming location, camera calibration system for a three-dimensional image generation system, and composite image generation system.

Abstract

This provides a means to perform camera calibration work with high accuracy. [Solution] A method for determining the coordinates and orientations of each camera B in the coordinate system of a three-dimensional marker A, using a calculation unit C consisting of an information processing device, in a shooting location where a three-dimensional marker A and multiple cameras B are arranged, wherein the three-dimensional marker A has four or more detection units 20 that are installed so as to fit within the field of view of each camera B and are arranged so that not all detection units 20 are located on the same plane, and any three detection units form a triangular plane E. The calculation unit C has at least the following functions: an extraction process for at least one of the multiple cameras B, which extracts one or more triangles D with any three detection units 20 as vertices from the four or more detection units 20 included in the image captured by camera B; and a determination process for determining whether or not to use the triangular plane E corresponding to the triangle D extracted in the extraction process as the target of calculation for determining the coordinates and orientations of camera B.

Description

Technical Field

[0001] The present invention relates to a method for calculating the coordinates and orientation of a camera at a shooting site, a camera calibration device for a three-dimensional video generation system, and a composite video generation system.

Background Art

[0002] When performing motion capture, generation of xR video, etc. using at least one installed camera, it is necessary to calibrate (calibrate) the coordinates and orientation of each camera in advance. Conventional camera calibration methods include a method (Patent Document 1) of photographing a detection object whose coordinates and orientation are specified by surveying, etc. with a camera and using the position of the detection object in the photographed image as reference information, and a jig provided with a detection object, an acceleration sensor, etc. A method (Non-Patent Document 1) is known in which the jig is appropriately moved within the viewing angle of the camera while being held in the hand, and the movement of the detection object on the photographed video of the camera and the acceleration obtained from the jig are used as reference information.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the case of the above method, there are at least the following improvement points. (1) In the case of the method described in Patent Document 1, the coordinates and orientation of each detection object must be determined in advance, which requires separate surveying work. (2) In order to perform the calibration work, it is necessary to move the jig.

[0006] Therefore, one of the objectives of the present invention is to provide a means that enables the calibration of cameras to be performed with high accuracy. [Means for solving the problem]

[0007] The present invention, made to solve the above problems, is a method for determining the coordinates and orientations of each camera in the coordinate system of a three-dimensional marker, using a calculation unit consisting of an information processing device, in a shooting site where a three-dimensional marker and multiple cameras are arranged, wherein the three-dimensional marker has four or more detection units that are installed so as to fit within the field of view of each camera and are arranged so that not all detection units are located on the same plane, and any three detection units form a triangular surface, and the calculation unit has at least the functions of: extracting one or more triangles with arbitrary three detection units as vertices from four or more detection units included in the captured image of at least one of the multiple cameras, and determining whether or not to use the triangular surface corresponding to the triangle extracted in the extraction process as the target of calculation for determining the coordinates and orientations of the camera. Furthermore, the present invention relates to a camera calibration system used in a three-dimensional image generation system that generates a three-dimensional image based on multiple two-dimensional images captured by multiple cameras, and comprises at least three: a three-dimensional marker having four or more detection units that are positioned within the field of view of each camera and that are not all located on the same plane, and which forms a triangular surface with any three detection units; and a calculation unit that determines the coordinates and orientation of each camera in the coordinate system of the three-dimensional marker based on the relative coordinates of each detection unit obtained based on the captured image of each camera, wherein the calculation unit has at least the function of extracting one or more triangles with any three detection units as vertices from four or more detection units included in the captured image of at least one of the multiple cameras, and determining whether or not to use the triangular surface corresponding to the triangle extracted in the extraction process as the target of calculation for determining the coordinates and orientation of the camera. Furthermore, the present invention relates to a composite image generation system for displaying a composite image to a user, which is composed of multiple images, comprising: multiple cameras; a three-dimensional marker having four or more detection units positioned so as to fit within the field of view of each camera and such that not all detection units are located on the same plane, and which forms a triangular plane with any three detection units; a calculation unit that determines the coordinates and orientation of the cameras in the coordinate system of the three-dimensional marker based on the relative coordinates of each detection unit obtained based on the images captured by each camera; a real-world image acquisition unit that acquires real-world images, which are images captured in approximately the direction of the user's line of sight, in a manner that includes depth information; a measurement unit that acquires measurement information including the user's position and orientation; and an arbitrary part based on the depth information from the real-world image. The system comprises at least a synthesis unit that generates a composite image by combining a processed image from which a fractional part has been removed and a three-dimensional image based on a plurality of two-dimensional images captured by each of the plurality of cameras, based on the measurement information, and a display unit that can be attached to the user's head and displays the composite image to the user, wherein the calculation unit has at least the function of extracting one or more triangles whose vertices are any three detection parts from four or more detection parts included in the image captured by at least one of the plurality of cameras, and determining whether or not to use the triangular plane corresponding to the triangle extracted in the extraction process as the target of calculation for determining the coordinates and orientation of the camera. [Effects of the Invention]

[0008] According to the present invention, camera calibration can be performed with high accuracy. [Brief explanation of the drawing]

[0009] [Figure 1] A schematic diagram showing the usage of the present invention in a filming location. [Figure 2] A schematic perspective view showing the overall structure of the 3D marker. [Figure 3] A diagram showing an example of an identifier. [Figure 4] A reference diagram to explain the judgment method [1]. [Figure 5] Reference diagram for explaining determination method [2]. [Figure 6] Reference diagram for explaining determination method [3]. [Figure 7] Schematic plan view showing the positional relationship between the stereo marker and each camera. [Figure 8] Schematic right side view showing the positional relationship between the stereo marker and one of the cameras. [Figure 9] Image diagram of a photographed image by one of the cameras. [Figure 10] Reference diagram showing the determination results of a plurality of triangles extracted from the photographed image. [Figure 11] Schematic diagram showing the configuration of the composite video generation system according to Example 2. [Figure 12] Schematic diagram showing the generation image of the processed video. [Figure 13] Schematic diagram showing the generation image of the composite video.

Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

Embodiment

[0011] [Calculation Method for Coordinates and Directions of Cameras at the Shooting Site]

[0012] <1>Overall Configuration (Fig. 1) The calculation method for the coordinates and directions of the cameras at the shooting site according to the present invention is a method that is pre-implemented to calibrate the cameras when generating a three-dimensional video by appropriately synthesizing two-dimensional videos of subjects such as people, objects, and backgrounds in the shooting site taken by each camera based on the coordinates and directions of each camera. In the present invention, the display destination of the generated three-dimensional video is not particularly limited, and known display devices such as a display built in or externally attached to an information processing device, and an HMD (head-mounted display) worn by a user can be targeted. When implementing the method according to the present invention, as shown in Figure 1, a three-dimensional marker A is set up so as to surround it with multiple cameras B (B1 to B3) installed at the shooting site, and the coordinates and orientation of each camera B are calibrated by a calculation unit C consisting of an information processing device from the image information captured by each camera B on the three-dimensional marker A. After each camera B has been calibrated, the 3D marker A can be removed. The details of each component are described below.

[0013] <2> Camera (Figure 1) Camera B is a device installed at the shooting location to photograph subjects such as people and objects within the shooting location. Before photographing subjects, Camera B is also used to photograph 3D markers installed at the shooting location in order to calibrate its own coordinates and orientation. In this invention, there are no particular limitations on the type or number of cameras B used. General digital cameras and video cameras capable of capturing still images and videos, or depth cameras capable of adding depth information to two-dimensional images, can be used. In this embodiment (Figure 1), three cameras B (B1 to B3) are installed around the three-dimensional marker A.

[0014] <3> Three-dimensional markers (Figures 1-3) 3D marker A is an object that is photographed by camera B in order to calibrate the coordinates and orientation (sometimes called posture, rotational posture, or angle) of camera B installed at the site. Camera B calibrated using the 3D marker A according to this embodiment can be used for purposes such as recording the movements of people and objects in a motion capture system, or capturing images to be used as composite material for various types of images in xR systems including VR, MR, and AR. The three-dimensional marker A according to this embodiment is configured to include at least a frame 10 and four or more detection units 20. The details of each component are described below.

[0015] (1) Frame (Figure 1) The frame 10 is a component that holds the position of each detection unit 20 as a three-dimensional marker A. In this embodiment, the frame 10 is configured to have four virtual triangular surfaces E.

[0016] (1.1) Frame shape (Figures 1 and 2) In this invention, the shape of the frame 10 is not particularly limited, but a configuration that is convenient for on-site assembly and installation of the three-dimensional marker A is desirable. For example, if it is necessary to assemble the three-dimensional marker A on-site, it is desirable to have a configuration in which the number of components constituting the frame 10 is small, or a configuration that makes it easy to hold the four detection units 20 (described later) so that they are not located on the same plane. Furthermore, from the standpoint of ensuring the installation stability of the three-dimensional marker A, a configuration that allows for a wide base surface of the frame 10 and a configuration that makes it difficult for the three-dimensional marker A to tip over are preferable. In this embodiment (Figure 2), the frame 10 has a tetrahedron shape, which is formed by assembling a three-dimensional frame using multiple wires 11 of equal length as a framework.

[0017] (1.2) Frame material In this invention, the material of the frame 10 is not particularly limited, but from the viewpoint of ensuring portability when disassembling and assembling the frame 10, it is desirable to use a material that is as lightweight as possible.

[0018] (1.3) Presence or absence of facing material (Figure 2) In the present invention, there is no particular limitation on whether or not a physical surface material is provided on the frame 10. It can be designed arbitrarily as long as no adverse effects occur, such as the detection unit 20 being obscured by the surface material when the camera photographs the three-dimensional marker A. For example, the frame 10 may be constructed by arranging triangular surface materials made of transparent material in a tetrahedron shape. For example, even if a surface material is used for the bottom portion of frame 10 in Figure 1, it will not negatively affect the camera calibration process.

[0019] (2) Detection unit (Figure 2) The detection unit 20 is a component that obtains positional information of the detection unit 20 by image processing of the captured image, which is captured by the camera B. The detection unit 20 can be configured with at least one or more identifiers 21, as shown in Figure 2.

[0020] (2.1) Identifier (Figure 3) The identifier 21 can be composed of characters, figures, or light-emitting elements, or a combination thereof, which are known as recognition codes. For example, AR markers or other two-dimensional codes can be used. Figure 3 shows an example of how the identifier 21 can be constructed using a geometric shape. This identifier 21 divides the inside of a circle into four sections, leaving one diagonal section blank, and provides corners that radiate outwards at 120° intervals from the outside of the circle. During the camera calibration process, the center point 22 of the identifyer 21 is identified by image processing of the image of the identifyer 21, and the position of the center point 22 is treated as the position of the detection unit 20.

[0021] (2.2) Number and location of detection units (Figure 2) In the three-dimensional marker A according to this embodiment, four or more detection units 20 are provided, and the frame 10 is configured to hold the position of all detection units 20 so that they are not located on the same plane in the spatial coordinate system. This is to ensure that, when photographing the 3D marker A with a camera, a depth difference is as likely as possible between at least one detection unit 20 and the other detection units 20, regardless of the direction from which the image is taken.

[0022] (2.3) Installation location of the detection unit relative to the frame In the three-dimensional marker A according to this embodiment, the location on the frame 10 for the detection unit 20 is not particularly limited and can be set as appropriate, such as the ends or middle parts of the wires 11 that make up the frame 10, the corners formed between the wires 11, or the surface of the facing material that makes up the frame 10. For example, in the frame 10 shown in Figure 2, the detection unit 20 is installed at the top of a tetrahedron-shaped three-dimensional frame formed by connecting the ends of the wires 11.

[0023] (2.4) Number and orientation of the identifiers (Figure 2) In the three-dimensional marker A according to this embodiment, each detection unit 20 may be configured such that multiple identification bodies 21 are provided and each identification body 21 faces a different direction. This is because, for example, when there are multiple cameras to be calibrated, if one of the identified objects 21 is difficult to capture from the shooting direction of one camera, it increases the possibility of capturing the other identified object 21 which is facing a different direction.

[0024] (2.5) Size of the identifier (not shown) In the present invention, the sizes of each detection unit 20 and each identifying element 21 do not need to be the same. For example, for the identifier 21c placed inside the frame 10 of the detection unit 20 located furthest from the camera (for example, the detection unit 20d as viewed from a camera positioned towards the front of the page in Figure 1), the size of the figure may be made slightly larger than that of the other identifiers 21 in order to suppress a decrease in detection accuracy.

[0025] (2.6) Relative coordinates and separation distance between detection units (Figure 2) With the above configuration, when viewed as a coordinate system of its own, the three-dimensional marker A has a predetermined relative coordinate system (X1~X4, Y1~Y4, Z1~Z4) between each detection unit 20 (identifier 21), with a certain point within the three-dimensional marker as the origin.

[0026] <4> Calculation unit (Figure 1) The calculation unit C is a functional unit that uses the captured video from each camera B and the relative coordinates of each detection unit 20 provided on the 3D marker A to determine the coordinates and orientation of each camera B. In the present invention, the calculation unit C can be realized by any combination of hardware and software, and can also be configured with each part as a separate device, or with multiple parts integrated into a single device. In this embodiment (Figure 1), the calculation unit C is configured by a program installed in the information processing device. In the present invention, the calculation unit C performs the following processing mainly using the video footage captured by at least one of the multiple cameras B. Each process may be rearranged or executed in parallel, as long as it does not contradict the procedure. Furthermore, in this invention, the following processes may be performed for all cameras B.

[0027] (1) Extraction process The extraction process involves extracting triangles whose vertices are any three detection units 20 from four or more detection units 20 contained in the video footage captured by camera B. Details of the extraction process will be described later. <5> This will be explained in the section on implementation procedures. Furthermore, the user may manually set which detection unit 20 corresponds to each vertex of the triangle in the 3D marker A, or it may be automatically recognized using an image recognition function or the like separately provided in the calculation unit C.

[0028] (2) Judgment process (Figures 4-6) The determination process determines whether or not to use the triangular face corresponding to the triangle extracted in the extraction process as the target of calculations to determine the camera's coordinates and orientation. For example, if the triangular plane formed by the three detection points that make up the 3D marker A is an equilateral triangle, and the triangle extracted from the captured image is also close to an equilateral triangle, then camera B is almost directly facing this triangular plane, and the difference in the depth coordinates of each vertex relative to camera B is small, which can be a factor in reducing the accuracy when calibrating the coordinates and orientation of camera B. Therefore, in the present invention, any method that can be used as the determination process is one that can determine the degree of alignment between camera B and triangular surface E, or one that can determine the magnitude of the difference in the depth direction coordinates of any three detection parts constituting the three-dimensional marker A relative to camera B, and any method within that range can be used. The following is an example of a determination method.

[0029] (2.1) Decision method [1] (Figure 4) This method involves selecting a triangle extracted during the extraction process and determining its camera coordinates and orientation if the difference between the length of its longest side and the sum of the lengths of its other two sides is less than or equal to a predetermined value. For example, if triangle D extracted from the image captured by camera B has the shape shown in Figure 4(a), the sum of the lengths of the other two sides (L2, L3) (L2+L3) is only slightly larger than the length of the longest side (L1). Therefore, it is determined that triangle D can be used as a target for calculation. On the other hand, if the extracted triangle D is a triangle that is close to an equilateral triangle as shown in Figure 4(b), the sum of the lengths of the other two sides (L2, L3) (L2+L3) is approximately twice the length of the longest side (L1), so this triangle D is determined to be unusable as a target for calculation.

[0030] (2.2) Decision method [2] (Figure 5) This method uses a triangle as the target for calculations to determine the camera's coordinates and orientation if the smallest of the three interior angles of the extracted triangle is less than or equal to a predetermined angle. The predetermined angle in this method can be set as appropriate, for example, to about 30°. For example, if triangle D extracted from the image captured by camera B has the shape shown in Figure 5, the smallest interior angle is approximately 20°, so this triangle is deemed suitable for use in calculations. On the other hand, although not shown in the diagram, if the triangulation D is a triangle that is close to an equilateral triangle, all of its interior angles are approximately 60°, so it is determined that it cannot be used as a target for calculation.

[0031] (2.3) Decision method [3] (Figure 6) This method excludes the triangular surface E from calculations to determine the coordinates and orientation of camera B if the absolute value (θ) of the angle between the normal vector of the triangular surface E corresponding to the extracted triangle D and the optical axis vector of camera B (0 ≤ θ ≤ 90°) is less than or equal to a predetermined angle. The predetermined angle in this method can be set as appropriate, for example, to 45°. For example, as shown in Figure 6, if the absolute value (θ) of the angle between the normal vector F of the triangular face E of the 3D marker A and the optical axis vector G of the camera B is approximately 50°, then the triangles in the captured image extracted with the detection unit of this triangular face as the vertex can be included as the target of the calculation.

[0032] (2.4) Combinations of decision methods Furthermore, the above-described determination methods may be combined using logical OR or logical AND as appropriate. For example, the system may be configured to execute multiple determination methods and then make a final determination to use the triangular faces corresponding to triangles determined to be calculation targets by all determination methods as calculation targets for determining the camera's coordinates and orientation, or to make a final determination to use the triangular faces corresponding to triangles determined to be calculation targets by at least one determination method as calculation targets for determining the camera's coordinates and orientation. This configuration is effective when a single determination method alone cannot sufficiently exclude calculation targets.

[0033] (3) Calculation of camera coordinates and orientation Finally, the calculation unit C, after the determination process, uses the relative coordinates of each detection point on the triangular surface E corresponding to the triangle D that was the target of the calculation to determine the coordinates and orientation of the camera in the coordinate system of the 3D marker in the manner of triangulation. If multiple triangles D are used in the calculation, the final coordinates and orientation of camera B may be obtained by taking the average of the coordinates and orientation of camera B obtained for each triangle D. Furthermore, if there is a significant discrepancy in the coordinates and orientation of camera B obtained from each triangle D, error handling may be performed.

[0034] <5> Implementation Procedure The following describes an example of the procedure for implementing the method according to the present invention.

[0035] (1) Installation status (Figures 7 and 8) Figure 7 shows the positional relationship between the 3D marker A and each camera B when the 3D marker A is viewed from above, and Figure 8 shows the positional relationship between the 3D marker A and camera B1 when the 3D marker A is viewed from the right side.

[0036] In the method according to the present invention, it is sufficient to install the three-dimensional marker A so that it fits within the field of view of each camera B, and there is no need to pre-determine the installation location and orientation of the three-dimensional marker A by surveying the site or the like. For example, as shown in Figure 7, in the case of camera B1, the three detection units 20 (20a, 20b, 20c) located at the vertices of the virtual triangular plane E1, and the one detection unit 20 (20d) located at the back are all visible in the image. Similarly, in the case of camera B2, the three detection units 20 (20a, 20c, 20d) located at the vertices of the virtual triangular surface E2, and the one detection unit 20 (20b) located on the far side of the triangular surface E2 are all captured in the image. Similarly, camera B3 also shows three detection units 20 (20a, 20b, 20d) located at the vertices of the virtual triangular surface E3, as well as one detection unit 20 (20c) located at the back of the triangular surface E3.

[0037] (2) Detection by the detection unit and extraction of triangles (Figure 9) Figure 9 is an illustrative diagram of the image 30 captured by camera B1. The calculation unit C detects the four detection units 20 (20a, 20b, 20c, 20d) included in the captured image 30 shown in Figure 9. Then, a triangle D is extracted, with any three points from the four detection points 20 of the captured image 30 as its vertices. In this embodiment, four triangles D (D1 to D4) are extracted from the captured image.

[0038] (3) Judgment process (Figure 10) Next, the calculation unit C uses one or a combination of the aforementioned determination methods to determine whether or not to use the four triangles D1 to D4 extracted in the extraction process, specifically the triangular faces E1 to E4 corresponding to each triangle D1 to D4, as targets for calculations to determine the camera's coordinates and orientation. As shown in Figure 10, in this embodiment, of the four triangles D1 to D4 extracted from the image 30 captured by camera B1, it was determined that the triangular face E1 corresponding to triangle D1 would not be used as a target for calculation, while the remaining three triangular faces E2, E3, and E4 corresponding to triangles D2, D3, and D4 were determined to be used as targets for calculation.

[0039] (4) Calculation process (Figure 10) Finally, the calculation unit C uses the relative coordinates of the three detection units 20 that make up each triangular face E2, E3, and E4 corresponding to the triangles D2, D3, and D4 that were the subject of the calculation, to determine the coordinates (Xc1-2~4, Yc1-2~4, Zc1-2~4) and orientation (αc1-2~4, βc1-2~4, γc1-2~4) of camera B1 in the manner of triangular surveying, and then takes the average of these to determine the final coordinates (Xc1, Yc1, Zc1) and orientation (αc1, βc1, γc1) of camera B1.

[0040] (5) Other cameras The calculation unit C can perform the same processing as described above for the other cameras B2 and B3 to determine the final coordinates (Xc2, Yc2, Zc2) and orientation (αc2, βc2, γc2) of camera B2, and the final coordinates (Xc3, Yc3, Zc3) and orientation (αc3, βc3, γc3) of camera B3.

[0041] <6> summary According to the method of the present invention, by excluding the triangular faces E that constitute the three-dimensional marker A, which are factors that reduce accuracy, from the calculations when determining the coordinates and orientation of the camera B, the calibration work for each camera B can be performed with greater accuracy. Furthermore, by adjusting the predetermined conditions of each judgment method, or by configuring multiple judgment methods to be arbitrarily combined using logical OR or logical AND as appropriate, the judgment conditions can be adjusted, thus enabling the adjustment of the accuracy of the camera B calibration process. [Examples]

[0042] [Composite Video Generation System]

[0043] In a shooting location where the coordinates and orientations of each camera have been determined using the camera coordinate and orientation calculation method according to the present invention, a system (synthetic image generation system) can be implemented that provides a user with a composite image, which is created by combining a three-dimensional image formed by combining two-dimensional images captured by each camera at the shooting location as source image, and then combining the source image with real-world images of the user's surroundings.

[0044] <1> Overall structure (Figure 11) The composite image generation system shown in Figure 11 comprises at least a real-world image acquisition unit 100, a measurement unit 200, a three-dimensional image management unit 300, a synthesis unit 400, and a display unit 500. Each component can be implemented by arbitrarily combining hardware and software, and various configurations can be adopted, such as each component being a separate device, or multiple components being integrated into a single device. Furthermore, in this invention, there are no particular limitations on the number of parts, the location of each part, or the number of images used for synthesis. The details of each part are explained below.

[0045] <2> Real-world image acquisition unit (Figure 11) The real-world image acquisition unit 100 has the function of acquiring real-world images 110 in the approximate direction of the user's line of sight, in a manner that includes depth information. This "Real-World Image 110" is a video recording of the real world, including the approximate direction of the user's line of sight; in other words, it is a real-time video showing the space around the user. The real-world image acquisition unit 100 can use a camera built into a head-mounted display (hereinafter also referred to as "HMD") worn by the user, a camera attached externally to the HMD, or any other camera that can be worn by the user. Cameras that can be used as the real-world image acquisition unit 100 include so-called depth cameras or depth sensors, which have a built-in depth sensor. In this invention, the real-world image acquisition unit 100 only needs to be a camera capable of capturing images in at least the approximate direction of the user's line of sight, and does not exclude cameras capable of capturing images around the user's entire surroundings.

[0046] <3> Measurement unit (Figure 11) The measurement unit 200 has the function of acquiring user measurement information 210. This "measurement information 210" includes at least information that allows for the recognition of the user's orientation and position. The synthesis unit 400, described later, generates a synthesized image 420 based on this measurement information 210. The measurement unit 200 can use a group of sensors such as an angle sensor, an acceleration sensor, and a gyroscope, as well as a motion capture device, a GPS device, and the like. Furthermore, the measurement unit 200 includes all types, such as those installed in the living space where the user resides, those built into the HMD worn by the user, those separately attached to the user's body, and those held by the user.

[0047] <4> Three-dimensional video management department (Figure 11) The three-dimensional image management unit 300 has the function of managing at least one three-dimensional image 310. This "3D image 310" includes stereoscopic images consisting of still images or videos (including slow-motion videos and time-lapse videos) that contain a 3D model generated based on multiple 2D images obtained by capturing a subject (including anything such as a person, object, or structure) with multiple cameras.

[0048] In the present invention, "management" of the three-dimensional video 310 by the three-dimensional video management unit 300 includes the act of generating the three-dimensional video 310 based on a plurality of two-dimensional videos received by the three-dimensional video management unit 300, as well as the act of receiving and saving three-dimensional videos 31 that are generated in real time (streaming video) or have already been generated by other functional units. Therefore, the three-dimensional video management unit 300 can be configured to have the functions of the three-dimensional video generation system according to the present invention, or to be configured to handle three-dimensional videos generated by the three-dimensional video generation system according to the present invention. In other words, the three-dimensional video 310 managed by the three-dimensional video management unit 300 may be a video generated by the three-dimensional video management unit 300 itself, which is equipped with the functions of the three-dimensional video generation system according to the present invention, or it may be a video generated by a three-dimensional video generation system according to the present invention, which is provided separately from the three-dimensional video management unit 300.

[0049] <5> Composite section (Figure 11) The synthesis unit 400 has the function of generating an image (processed image) in which at least an arbitrary portion based on depth information has been removed from the real image 110 (depth-based removal processing), and the function of synthesizing the processed image and the three-dimensional image 410 acquired and / or stored by the three-dimensional image management unit 300 based on the measurement information 210 acquired by the measurement unit 200. The synthesis unit 400 can be configured with hardware such as a CPU, memory, and GPU controlled by software. In the present invention, the synthesis unit 400 may further combine and synthesize other source images (such as VR images, CG images, and 360-degree images) in addition to the three-dimensional image 310 and the processed image.

[0050] <5.1> Example of removal process In the present invention, the removal process performed on the real image 110 includes at least a depth-based removal process, and also includes a chroma-based removal process as necessary. "Depth-based removal processing" is a process that removes elements from a real-world image 110 containing depth information based on the depth information values. For example, it involves removing parts of the image that have depth information exceeding a predetermined distance. "Chroma removal processing" is a process that performs removal based on the values ​​of color space information on the image after depth removal processing has been performed on the real image 110. More specifically, it is a process that removes the parts corresponding to the homogeneous color of background materials placed in the user's living space that are visible within the field of view of the real image 110, and makes it possible to composite other images onto the removed parts. These processes can be found in detail in the patent documents conceived by the present applicant (Japanese Patent Publication No. 6717486, Japanese Patent Publication No. 6991494, Japanese Patent Publication No. 7157271, etc.).

[0051] <5.2> Image of processed video generation (Figure 12) Figure 12 shows an image illustrating the results of applying each removal process to the real-world image 110. The real-world image 110 shown in Figure 12(a) is an image captured by a real-world image acquisition unit 100 installed on an HMD worn by the user, and shows the user's hands and the background in the direction of the user's line of sight. By applying a depth-based removal process to this real-world image 110, for example, which removes areas with depth information of 50 cm or more, a processed image 410a in which only the user's hands are extracted can be generated, as shown in Figure 12(b). If the user's hands are sufficiently extracted in the processed image 410a after this depth-based decompression process, this processed image 410a can be used as the final processed image 410 for synthesis. For example, if you want to extract them in more detail, you can further apply chroma-based decompression to the processed image 410a to remove the background color area remaining around the contour lines of the user's hands, generating the processed image 410b shown in Figure 12(c), and use this processed image 410b as the final processed image 410.

[0052] <6> Display unit (Figure 11) The display unit 500 is a device equipped with the function of displaying the composite image generated by the synthesis unit 400 to the user. The display unit 500 can use a user-worn HMD, goggles, or other portable devices such as smartphones that can be housed in a housing to function as goggles.

[0053] <7> Image of the generation of a composite image (Figure 13) Figure 13 shows an example of how the composite image is generated. Figure 13(a) shows a three-dimensional image 310 from a three-dimensional image management unit 300 equipped with the functions of the three-dimensional image generation system according to the present invention, in which the entire shooting site and workers are rendered in three dimensions. Figure 13(b) shows the processed image 410 after the real image 110 acquired by the real image acquisition unit 100 has been processed by the synthesis unit 400 to remove at least arbitrary parts based on depth information (depth-based removal processing). Only the hands of the user wearing the HMD and the rod held by both hands are visible. Figure 13(c) is a composite image 420 created by combining the images shown in Figures 13(a) and (b) using the composite unit 400 based on user D's measurement information. In this image, the user's hands and the stick are visible in front of the workers and the work site.

[0054] <8> summary The composite image generation system according to the present invention can achieve, for example, the following effects. (1) Because the composite image includes a three-dimensional image generated from multiple two-dimensional images taken by multiple cameras, it is possible to provide users with a more realistic composite image experience. (2) In a composite image, even if there are areas in a particular field of view where the user cannot see them because they are obstructed by 3D models of people or objects included in the three-dimensional image, the user can see those areas by moving or changing the direction of their head to avoid the obstructing 3D models. (3) By moving so that the user overlaps with the 3D model in the three-dimensional image, the user can experience a view as if they were one with the 3D model. (4) Synthesized video can be used for on-site instruction, training materials for workers (distributing and saving footage of skilled workers), recording and managing work details, maintenance work such as visual inspections of equipment at work sites, and as reference material for tracing the cause of malfunctions by looking back at past incidents. (5) By using real-time generated streaming video for the three-dimensional image, the user can see real-time video of the shooting site, which is the source of the three-dimensional image. For example, when a construction site is used as the shooting site, and a skilled technician who is a user of the system according to this embodiment is giving instructions to assembly workers at the construction site, the skilled technician can display a composite image, which is generated from streaming video of the construction site captured in real time, on a display unit worn by the skilled technician. This allows the skilled technician, even from a distance, to perceive the construction site in three dimensions as if they were actually there, and to give instructions to the assembly workers. [Explanation of Symbols]

[0055] A: 3D marker 10: Frame 11:Wire rod 20: Detection unit 21: Identifier 22: Center point B: Camera 30: Captured image C: Calculation part D: Triangle E: triangular surface F: Normal vector G: Optical axis vector 100: Real-world image acquisition unit 110: Real footage 200: Measurement Unit 210: Measurement Information 300: Three-Dimensional Video Management Department 310: Three-dimensional images 400: Synthesis part 410: Processed video 420: Composite video 500: Representation section

Claims

1. A method for determining the coordinates and orientations of each camera in the coordinate system of a three-dimensional marker, using a calculation unit consisting of an information processing device, in a shooting location where a three-dimensional marker and multiple cameras are arranged, The aforementioned three-dimensional marker has four or more detection units that are positioned so as to fit within the field of view of each camera and such that not all detection units are located on the same plane, and any three detection units form a triangular surface. The calculation unit described above, An extraction process is performed in which, in at least one of multiple cameras, one or more triangles are extracted from four or more detection points included in the video footage captured by that camera, with any three detection points as vertices. A determination process to determine whether or not to use the triangular plane corresponding to the triangle extracted in the extraction process as the target of calculations for determining the coordinates and orientation of the camera, It is characterized by having at least the function of performing the following: Method for calculating the camera's coordinates and orientation at a filming location.

2. The aforementioned determination process, The length of the line segment obtained from the aforementioned triangle, The interior angles of the aforementioned triangle, and The normal vector of the triangular surface corresponding to the aforementioned triangle, A method characterized by using at least one of the following as an element of the determination condition, A method for calculating the coordinates and orientation of a camera at a shooting site according to claim 1.

3. The aforementioned determination process, The method is characterized in that, if the difference between the length of the longest side of the aforementioned triangle and the sum of the lengths of the other two sides is less than or equal to a predetermined value, the triangle is used as the object of calculation for determining the coordinates and orientation of the camera. A method for calculating the coordinates and orientation of a camera at a shooting site according to claim 1.

4. The aforementioned determination process, The method is characterized in that, if the smallest of the three interior angles of the aforementioned triangle is less than or equal to a predetermined angle, the triangle is used as the object of calculation for determining the coordinates and orientation of the camera. A method for calculating the coordinates and orientation of a camera at a shooting site according to claim 1.

5. The aforementioned determination process, The method is characterized in that if the absolute value (θ) of the angle between the normal vector of the triangular surface corresponding to the triangle and the optical axis vector of the camera (0 ≤ θ ≤ 90°) is less than or equal to a predetermined angle, the triangle is excluded from the calculations for determining the coordinates and orientation of the camera. A method for calculating the coordinates and orientation of a camera at a shooting site according to claim 1.

6. A camera calibration system used in a three-dimensional image generation system that generates a three-dimensional image based on multiple two-dimensional images captured by multiple cameras, A three-dimensional marker having four or more detection units positioned so as to fit within the field of view of each camera, and such that not all detection units are located on the same plane, with any three detection units forming a triangular surface, A calculation unit determines the coordinates and orientation of each camera in the coordinate system of the three-dimensional marker based on the relative coordinates of each detection unit obtained from the captured images of each camera. It must have at least the following: The calculation unit described above, An extraction process is performed in which, in at least one of multiple cameras, one or more triangles are extracted from four or more detection points included in the video footage captured by that camera, with any three detection points as vertices. A determination process to determine whether or not to use the triangular plane corresponding to the triangle extracted in the extraction process as the target of calculations for determining the coordinates and orientation of the camera, It is characterized by having at least the function of performing the following: Camera calibration system for 3D image generation systems.

7. A composite video generation system for displaying a composite video, which is made by combining multiple videos, to a user, Multiple cameras, A three-dimensional marker having four or more detection units positioned so as to fit within the field of view of each camera, and such that not all detection units are located on the same plane, with any three detection units forming a triangular surface, A calculation unit determines the coordinates and orientation of the cameras in the coordinate system of the three-dimensional marker based on the relative coordinates of each detection unit obtained from the images captured by each camera. A real-world image acquisition unit acquires real-world images, which are images taken in approximately the direction of the user's line of sight, in a manner that incorporates depth information. A measurement unit that acquires measurement information including the user's position and orientation, A synthesis unit generates a composite image by combining a processed image obtained by removing arbitrary portions based on depth information from the actual image, and a three-dimensional image obtained by combining multiple two-dimensional images taken by multiple cameras, based on the measurement information. A display unit that can be attached to the user's head and displays the synthesized image to the user, It must have at least the following: The calculation unit described above, An extraction process is performed in which, in at least one of multiple cameras, one or more triangles are extracted from four or more detection points included in the video footage captured by that camera, with any three detection points as vertices. A determination process to determine whether or not to use the triangular plane corresponding to the triangle extracted in the extraction process as the target of calculations for determining the coordinates and orientation of the camera, It is characterized by having at least the function of performing the following: A system for generating composite images.

Images