Three-dimensional marker and camera calibration method

Abstract

To perform calibration work of cameras only by work for installing a three-dimensional marker at a free position in a site.SOLUTION: A three-dimensional marker A to be photographed by each camera for calibrating relative coordinates and relative attitudes among a plurality of cameras, includes at least a frame 10 and four or more detection parts 20 (20a to 20d) held in position by the frame 10, where the four or more detection parts 20 (20a to 20d) are configured not to be positioned on the same plane. The three-dimensional marker A is so installed that it fits within the angle of view of each camera, and triangulation calculation is performed using the relative coordinates of each detection part in the captured image obtained by photographing the three-dimensional marker A with each camera, thereby acquiring the coordinates and the attitudes of the cameras in the coordinate system of the three-dimensional marker A.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a three-dimensional marker and a camera calibration method used when calibrating the coordinates and orientation of a camera used in a motion capture system or an xR (cross reality) system including VR, MR, AR, etc. [Background technology]

[0002] When using at least one camera to perform motion capture or generate xR video, the coordinates and orientation of each camera must be calibrated in advance. Conventional camera calibration methods include a method in which a camera is used to photograph a detected object whose coordinates and orientation have been identified through surveying or other means, and the position of the detected object in the captured image is used as reference information (Patent Document 1), and a method in which a jig equipped with a detected object and an acceleration sensor is held in the hand and moved appropriately within the camera's field of view, and the movement of the detected object in the camera's captured image and the acceleration obtained from the jig are used as reference information (Non-Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2021-182259 [Non-patent literature]

[0004] [Non-Patent Document 1] "3. Calibration | Tutorial | Support | Acuity Inc. | Acuity Measurement and Inspection DX with Image Processing and Machine Learning" https: / / www.acuity-inc.co.jp / support / tutorial / 576 / Summary of the Invention [Problem to be solved by the invention]

[0005] The above method has at least the following improvements. (1) In the case of the method described in Patent Document 1, it is necessary to specify the coordinates and orientation of each detection object in advance, which requires a separate surveying operation. (2) In order to perform the calibration work, it is necessary to move the jig.

[0006] Therefore, an object of the present invention is to provide a three-dimensional marker that can be used to perform camera calibration simply by placing it in a desired position within the site. [Means for solving the problem]

[0007] The present invention, which has been made to solve the above problems, provides a three-dimensional marker that comprises at least a frame and four or more detection units that are held in position by the frame, and is configured so that the four or more detection units are not located on the same plane.

[0008] With this configuration, it is possible to determine the coordinates and orientation of the camera in the coordinate system of the three-dimensional marker based on the relative coordinates of each detection unit detected from an image of the three-dimensional marker captured by the camera. [Effects of the Invention]

[0009] According to the present invention, the camera calibration can be performed simply by placing the three-dimensional marker at any desired position within the site. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a schematic perspective view showing the overall configuration of a three-dimensional marker according to the present invention; [Figure 2] FIG. 10 is a diagram showing an example of an identifier. [Figure 3] FIG. 2 is a schematic plan view showing the positional relationship between the three-dimensional marker and each camera. [Figure 4] FIG. 10 is a schematic right side view showing the positional relationship between the three-dimensional marker and one of the cameras. [Figure 5] An illustration of an image captured by one of the cameras. [Figure 6] FIG. 10 is a schematic perspective view showing another example of the configuration of the three-dimensional marker according to the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. [Example]

[0012] <1> Overall configuration (Fig. 1) The three-dimensional marker A according to the present invention is an object that is to be photographed by a camera installed at the site in order to calibrate the coordinates and attitude (also referred to as the direction or angle) of the camera. A camera calibrated using the three-dimensional marker A of the present invention can be used to record the movements of people and objects in a motion capture system, or to capture images to be used as material for compositing various types of images in xR systems including VR, MR, and AR. The three-dimensional marker A according to the present invention is configured to include at least a frame 10 and detection units 20 at four or more locations. Each component will be described in detail below.

[0013] <2> Frame (Figure 1) The frame 10 is a member that serves as the three-dimensional marker A and maintains the position of each detection unit 20.

[0014] <2.1> Frame shape (Fig. 1) In the present invention, the shape of the frame 10 is not particularly limited, but a highly convenient configuration is desirable for assembling and installing the three-dimensional marker A on-site. For example, if it is necessary to assemble the three-dimensional marker A on-site, it is desirable to have a configuration in which the number of components making up the frame 10 is small, and a configuration in which it is easy to maintain the positions of the four detection bodies described below so that they are not positioned on the same plane. Furthermore, from the viewpoint of ensuring the stability of the installation of the three-dimensional marker A, it is preferable to have a configuration that allows the bottom surface of the frame 10 to be wide, and a configuration that makes it difficult for the three-dimensional marker A to tip over. As one preferred example of a configuration based on the above viewpoint, in FIG. 1, the frame 10 is assembled using a plurality of wires 11 as a framework, and is configured to have a three-dimensional frame shape (tetrahedron shape) without any surface materials.

[0015] <2.2> Frame material In the present invention, the material of the frame 10 is not particularly limited, but it is desirable to use a material that is as light as possible in order to ensure portability when disassembling and assembling the frame 10.

[0016] <2.3> Presence or absence of surface material (Fig. 1) In the present invention, there is no particular restriction on whether or not a physical surface material is provided on the frame 10, and it can be designed arbitrarily as long as it does not cause any adverse effects, such as the detection unit 20 being hidden by the surface material when photographing the three-dimensional marker A with a camera. For example, even if a surface material is used for the portion that constitutes the bottom surface of the frame 10 in FIG. 1, this does not adversely affect the calibration work of the camera.

[0017] <3> Detection unit (Fig. 1, Fig. 2) The detection unit 20 is a member that is photographed by a camera, and position information of the detection unit 20 is obtained by image processing of the photographed image. The detection unit 20 can be configured with at least one or more identifiers 21.

[0018] <3.1> Identification body (Fig. 2) The identifier 21 can be configured with characters, figures, or illuminants, or a combination of these, which are known as recognition codes, and for example, an AR marker or other two-dimensional codes can be used. 2 shows an example of an identifier 21 configured as a graphic. This identifier 21 is a circle divided into four parts, with one diagonal corner left blank, and corners extending radially outward at 120° intervals from the circle. During the camera calibration work, the image of the identifying object 21 is processed to identify the center point 22 of the identifying object 21, and the position of the center point 22 is treated as the position of the detection unit 20.

[0019] <3.2> Number and location of detectors (Fig. 1) In the three-dimensional marker A according to the present invention, the detection units 20 are provided in four or more locations, and are configured so that all of the detection units 20 are held in position by the frame 10 so that they are not located on the same plane in the spatial coordinate system. This is to ensure that when the three-dimensional marker A is photographed with a camera, a depth difference occurs between at least one detection unit 20 and the other detection units 20 regardless of the direction from which the photograph is taken.

[0020] <3.3> Location of the detector relative to the frame In the three-dimensional marker A of the present invention, the location of installation of the detection unit 20 on the frame 10 is not particularly limited, and can be set appropriately at the end or middle of the wire 11 that constitutes the frame 10, the corner formed between the wires 11, the surface of the panel 12 that constitutes the frame 10, etc. For example, in the frame 10 shown in FIG. 1, the detection unit 20 is installed at the top of a tetrahedron-shaped three-dimensional frame formed by connecting the ends of wires 11 together.

[0021] <3.4> Number and orientation of identifiers (Figure 1) In the three-dimensional marker A according to the present invention, each detection unit 20 may be configured so that a plurality of the above-described identifiers 21 are prepared and each identifier 21 faces in a different direction. This is to increase the possibility of capturing the other identifying object 21 with a different orientation, for example, when there are multiple cameras to be calibrated and one identifying object 21 is difficult to capture in the shooting direction from one camera.

[0022] <3.5> Size of identifier (not shown) In the present invention, the sizes of the detection units 20 and the identification objects 21 do not need to be the same. For example, for the identifying object 21c placed inside the frame 10 of the detecting unit 20 located furthest from the camera (for example, the detecting unit 20d when viewed from the camera placed on the front side of the paper in Figure 1), the size of the figure may be made slightly larger than that of the other identifying objects 21 in order to prevent a decrease in detection accuracy.

[0023] <3.6> Relative coordinates and distance between detectors With the above configuration, when the three-dimensional marker A is viewed as its own coordinate system, the relative coordinates (X1 to X4, Y1 to Y4, Z1 to Z4) between the respective detection units 20 (identifiers 21) are predetermined.

[0024] <4> Calibration method (Figures 3 to 5) An example of a method for calibrating a camera using the three-dimensional marker according to FIG. 1 will be described with reference to FIGS.

[0025] (1) Installation status (Fig. 3, Fig. 4) Figure 3 is a diagram showing the positional relationship between three-dimensional marker A and each camera B (B1 to B3) when the three-dimensional marker is viewed from above, and Figure 4 is a diagram showing the positional relationship between three-dimensional marker A and camera B1 when the three-dimensional marker A is viewed from the right side. The coordinates and orientations of each camera are shown in Table 1 below.

[0026] [Table 1] TIFF2025140996000002.tif33150

[0027] In the calibration method of the present invention, the three-dimensional marker A only needs to be installed so that it fits within the field of view of each camera; there is no need to survey the site or determine in advance the installation location and orientation of the three-dimensional marker A. For example, in the case of camera B1, as shown in Figures 3 and 4, the installation orientation of the three-dimensional marker A is adjusted so that the three detection units 20 (20a, 20b, 20c) located at the vertices of an imaginary triangle and the one detection unit 20 (20d) located at the back of the imaginary plane are each reflected. In addition, in the case of camera B2, the installation orientation of the three-dimensional marker A is adjusted so that the three detection units 20 (20a, 20c, 20d) located at the vertices of an imaginary triangle and the one detection unit 20 (20b) located at the back of the imaginary plane are each reflected, as shown in Figure 3. Similarly, for camera B3, the installation orientation of the three-dimensional marker A can be adjusted so that the three detection units 20 (20a, 20b, 20d) and the one detection unit 20 (20c) located at the back of the virtual plane are each reflected, as shown in Figure 3.

[0028] (2) Camera footage (Figure 5) FIG. 5 is a conceptual diagram of an image C captured by the camera B1. In this embodiment, of the four detection units 20 included in the captured image C shown in Figure 5, one detection unit 20d located in the triangle formed by connecting the three outer detection units 20 (20a, 20b, 20c) is recognized as the back detection unit, and triangulation calculations are performed using the relative coordinates of each detection unit 20 found in three triangles D (D1, D2, D3) that include this back detection unit 20d and two of the other three detection units 20, thereby making it possible to determine the coordinates (Xc1, Yc1, Zc1) and orientation (αc1, βc1, γc1) of camera B1. Similarly, for cameras B2 and B3, the coordinates and orientation of each camera can be determined by performing triangulation calculations using the relative coordinates of each detection unit 20 found in three triangles D obtained from the four detection units detected in the captured image C.

[0029] <5> summary In this way, with the three-dimensional marker of the present invention, the camera coordinates and orientation can be calibrated simply by placing it stationary in any position within the site, eliminating the need to determine the coordinates of the installation location in advance through surveying, etc. Furthermore, higher calibration accuracy can be achieved with the same amount of effort as the conventional simple calibration method in which calibration is performed by arranging multiple detection markers on the same plane. [Example]

[0030] FIG. 5 shows a modified example of the three-dimensional marker A according to the present invention.

[0031] <1> Modification 1 (Figure 6(a)) In Figure 6(a), the frame 10 is not formed into a three-dimensional frame shape, but is composed of wires 11 that extend radially in three directions from a certain point when viewed in a plane, and wires 11 that extend upward from the same point, and a detection unit 20 is provided at the open end of each wire 11.

[0032] <2> Modification 2 (Figure 6(b)) In Figure 6(b), a frame 10 is formed by a rectangular surface material 12 in a plan view and a single wire material 11 extending upward from the surface material 12, and detection units 20 made of light-emitting bodies are provided at the three corners of the surface material 12 and at the open end of the wire material 11.

[0033] <3> summary In any of the above-described modified examples, as long as the wire 11 does not interfere with the photographing of the detecting section 20, there is no problem in calibrating the camera according to the present invention. [Explanation of symbols]

[0034] A: 3D marker B: Camera C: Photographed image D: Triangle 10: Frame 11: Wire rod 12: Surface material 20:Detection unit 21: Identification body 22: Center point

Claims

1. The frame and four or more detection units whose positions are held by the frame; and The four or more detection units are not located on the same plane. 3D marker.

2. The frame is a three-dimensional frame formed by assembling a plurality of wire rods, The detection unit is disposed at the top of the three-dimensional frame. The three-dimensional marker according to claim 1 .

3. The three-dimensional frame is tetrahedral. The three-dimensional marker according to claim 2 .

4. The detection unit is characterized in that it comprises one or more identifiers consisting of characters, figures, illuminants, or a combination thereof. The three-dimensional marker according to claim 1 .

5. The detection unit is characterized in that it comprises a plurality of identifiers having different orientations. The three-dimensional marker according to claim 4.

6. The three-dimensional marker according to claim 1 is installed so as to be within the angle of view of the camera, The coordinates and the orientation of the camera in the coordinate system of the three-dimensional marker are calculated based on the relative coordinates of each detection unit detected from the image captured by the camera. How to calibrate your camera.

Images