CALIBRATION METHOD AND CALIBRATION PROGRAM

Abstract

To provide a calibration method or the like that can improve the accuracy of identifying a relation among multiple state images based on the multiple state images acquired by a stereo camera at each of multiple different positional attitudes.SOLUTION: A calibration sheet 20 (object) having, on a flat designated surface, designated areas 21, 22, and 23 in which local areas 210, 220, and 230 having three different attributes (RGB values) are arranged so as to be distributed. Three-dimensional positions of respective representative points (centers of gravity) of the designated areas 21, 22, and 23 reflected in each of multiple state images acquired by stereo cameras at different positional attitudes are identified. An affine transformation matrix is obtained so that the three-dimensional position of each representative point in a first state image coordinate system coincides with the three-dimensional position of each representative point in a second state image coordinate system.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a technique for calibrating a stereo camera in a plurality of positions and orientations. [Background technology]

[0002] A technology has been proposed in which the relationship between a reference image, which is one of multiple calibration pattern images, and the other calibration pattern images is set using affine transformation parameters based on the reference image, and the parameters are optimized (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2011-118553 A Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention aims to provide a calibration method, etc. that can improve the accuracy of identifying the relationship between multiple state images based on multiple state images acquired by a stereo camera in each of multiple different position and postures. [Means for solving the problem]

[0005] The calibration method of the present invention comprises the steps of: A calibration method for determining a relationship between a first state image coordinate system, which is a coordinate system of the first state image, and a second state image coordinate system, which is a coordinate system of the second state image, based on a first state image and a second state image configured by pixels having pixel values ​​each of which is a distance to an object and a first attribute variable, a second attribute variable, and a third attribute variable representing a first attribute, a second attribute, and a third attribute of the object, respectively, acquired by a stereo camera in a first position and posture and a second position and posture, a step of arranging an object having a first designated region in which first local regions having the first attribute are distributed, a second designated region in which second local regions having the second attribute are distributed, and a third designated region in which third local regions having the third attribute are distributed, on a flat designated surface, such that the first designated region, the second designated region, and the third designated region are reflected in the first state image and the second state image, respectively; specifying three-dimensional positions of a first representative point, a second representative point and a third representative point which are representative points of the first designated area, the second designated area and the third designated area reflected in the first state image and the second state image, respectively; and calculating an affine transformation matrix based on the three-dimensional positions of the first representative point, the second representative point, and the third representative point identified based on the first state image and the three-dimensional positions of the first representative point, the second representative point, and the third representative point identified based on the second state image, so that the positions of the first representative point, the second representative point, and the third representative point in the first state image coincide with the positions of the first representative point, the second representative point, and the third representative point in the second state image.

[0006] According to the calibration method of the present invention, it is possible to improve the accuracy of identifying the relationship between a plurality of state images based on a plurality of state images captured by a stereo camera in each of a plurality of different position and orientations. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating the configuration of a calibration system. [Diagram 2] FIG. 4 is a diagram illustrating the configuration of a calibration sheet. [Diagram 3] 1 is a flowchart showing the procedure of a calibration method. [Figure 4] 1 is a flowchart showing steps of a method for integrating multiple status images. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] (Calibration system configuration) A calibration method according to one embodiment of the present invention is implemented using a calibration system including a calibration device 10, a first stereo camera 11, a second stereo camera 12, and a calibration sheet 20, as shown in Fig. 1. In this embodiment, the number of stereo cameras is "2", but it may be any number equal to or greater than "3".

[0009] The calibration device 10 is configured to include an information processing terminal or a computer, and is configured such that an arithmetic processing device (such as a CPU) reads out data and a calibration program stored in a storage device (memory), and performs calibration processing on the data in accordance with the program.

[0010] The first stereo camera 11 is composed of a stereo camera that captures a first state image in a first position and orientation. The second stereo camera 12 is composed of a stereo camera that captures a second state image in a second position and orientation. The "position and orientation" is defined by, for example, the position of a representative point in real space (for example, the center point of the image sensor of one of the cameras that constitute the stereo camera), and the azimuth and elevation angles of the optical axis. Each pixel that constitutes each of the first and second state images has a distance (D) to the object and RGB values ​​(first attribute variable, second attribute variable, and third attribute variable) as pixel values.

[0011] The first stereo camera 11 and the second stereo camera 12 may be separate stereo cameras, or may be the same stereo camera in different positions and orientations at different times. Each of the first stereo camera 11 and the second stereo camera 12 may be installed at a fixed location, mounted on an unmanned aerial vehicle, or mounted on a camera crane.

[0012] (Calibration sheet configuration) 2, the calibration sheet 20 (object) is a substantially rectangular sheet (e.g., made of rubber, resin, and / or metal) with a pattern on its surface (designated surface) as described below. The pattern includes a first designated area 21, a second designated area 22, and a third designated area 23, all of which are substantially rectangular.

[0013] The first specified region 21 is composed of a plurality of substantially square first local regions 210 colored in red (first attribute: RGB value (255,0,0)) that are distributed in a checkerboard pattern. Substantially square intermediate regions between the first local regions 210 are colored in red that is brighter (or has a lower saturation) than the first local regions 210. The first specified region 21 is surrounded by a first frame region 212 colored in red (with the same brightness and saturation as the first local regions 210) that extends along the periphery of the first specified region 211.

[0014] The second specified region 22 is composed of a plurality of substantially square second local regions 220 colored in green (second attribute: RGB value (0, 255, 0)) that are distributed in a checkerboard pattern. Substantially square intermediate regions between the second local regions 220 are colored in green that is brighter (or has a lower saturation) than the second local regions 220. The second specified region 22 is surrounded by a second frame region 222 colored in green (with the same brightness and saturation as the second local regions 220) that extends along the periphery of the second specified region 22.

[0015] The third specified region 23 is composed of a plurality of substantially square third local regions 230 colored in red (third attribute: RGB value (0,0,255)) that are distributed in a checkerboard pattern. Substantially square intermediate regions between the third local regions 230 are colored in blue that is brighter (or has a lower saturation) than the third local regions 230. The third specified region 23 is surrounded by a third frame region 232 that extends along its periphery and is colored in blue (with the same brightness and saturation as the third local regions 230).

[0016] The other area of ​​the calibration sheet 20 is made up of a plurality of approximately square local areas 200 colored in black (fourth attribute: RGB value (0,0,0)), which are dispersed and arranged in a checkerboard pattern. Approximately square intermediate areas between the local areas 200 are colored in black with a higher brightness (or lower saturation) than the local areas 200. A black frame area 202 (with the same brightness and saturation as the local areas 200) is colored on the periphery of the calibration sheet 20.

[0017] The first designated region 21, the second designated region 22, and the third designated region 23 may have various shapes such as a substantially rectangular shape, a substantially circular shape, a substantially elliptical shape, a substantially triangular shape, a substantially parallelogram shape, a substantially trapezoid shape, and a substantially N-sided shape (N is an integer of 5 or more). The first local region 210, the second local region 220, and the third local region 230 may have various shapes such as a substantially rectangular shape (substantially square stripes), a substantially circular shape, a substantially elliptical shape, a substantially triangular shape, a substantially parallelogram shape, a substantially trapezoid shape, and a substantially N-sided shape (N is an integer of 5 or more). The local regions 200, the first local region 210, the second local region 220, and the third local region 230 may be regularly distributed and arranged to form a checkered pattern or a lattice pattern, or may be irregularly arranged such that the vertical and horizontal intervals between the regions are not constant.

[0018] The shapes and / or sizes of the first specified region 21, the second specified region 22, and the third specified region 23 may be the same as or different from each other. The shapes and / or sizes of the first local region 210, the second local region 220, the third local region 230, and the local region 200 may be the same as or different from each other. The widths of the frame region 202, the first frame region 212, the second frame region 222, and the third frame region 232 may be the same as or different from each other. At least a part of the frame region 202, the first frame region 212, the second frame region 222, and the third frame region 232 may be omitted.

[0019] The first frame region 212 may have an attribute (hue: for example, green, blue, purple, yellow, or black) different from the first attribute (hue: red). Similarly, the second frame region 222 may have an attribute (hue: for example, red, blue, purple, yellow, or black) different from the second attribute (hue: green). Also, the third frame region 232 may have an attribute (hue: for example, red, green, purple, yellow, or black) different from the third attribute (hue: blue). The frame region 202 may have an attribute (hue: for example, red, green, blue, purple, yellow, or black) different from the fourth attribute (hue: black).

[0020] (Calibration method) A calibration method for identifying the relationship between a first state image coordinate system, which is the coordinate system of a first state image captured by the first stereo camera 11 in a first position and orientation, and a second state image coordinate system, which is the coordinate system of a second state image captured by the second stereo camera 12 in a second position and orientation, will be described with reference to the flowchart of FIG. 3.

[0021] First, the calibration sheet 20 is placed so as to be included in the imaging ranges of the first stereo camera 11 and the second stereo camera 12 (FIG. 3 / STEP 10).

[0022] Next, the first stereo camera 11 acquires a first state image in which the first designated area 21, the second designated area 22, and the third designated area 23 of the calibration sheet 20 are entirely reflected, and the second stereo camera 12 acquires a second state image in which the first designated area 21, the second designated area 22, and the third designated area 23 of the calibration sheet 20 are entirely reflected (FIG. 3 / STEP 12). It is not necessary for the entire calibration sheet 20 to be reflected in each of the first state image and the second state image. As described above, each pixel constituting each of the first state image and the second state image has a distance (D) to the object and an RGB value (a first attribute variable, a second attribute variable, and a third attribute variable) as a pixel value. The first state image and the second state image (or data representing them) are transmitted to the calibration device 10 from each of the first stereo camera 11 and the second stereo camera 12.

[0023] The calibration device 10 specifies the centers of gravity of the first designated area 21, the second designated area 22, and the third designated area 23 based on the RGB values ​​of the pixel values ​​of each pixel in each of the first and second state images (FIG. 3 / STEP 14). Specifically, in each of the first and second state images, rectangular (or polygonal) areas that have RGB values ​​of (255,0,0), (0,255,0), and (0,0,255), respectively, and that are a group of at least the designated number of pixels and are closest to a convex hull are recognized as the first designated area 21, the second designated area 22, and the third designated area 23. For example, the square (integer) of the larger pixel width of the vertical or horizontal width of the state image divided by 40 is defined as the designated number of pixels. At this time, areas in which R, G, and B polygonal areas are adjacent to each other are excluded.

[0024] Then, in each of the first state image and the second state image, one or more pixels corresponding to the center of gravity of each of the first designated region 21, the second designated region 22, and the third designated region 23 are identified.

[0025] As described above, the first designated region 21 is made up of a plurality of first local regions 210 arranged in a checkerboard pattern and is surrounded by the first frame region 212, so that the accuracy of identifying the pixel group corresponding to the first designated region 21 in each state image, and further the pixel (or pixel group) corresponding to the center of gravity of the first designated region 21 is improved. The same applies to the second designated region 22 and the third designated region 23.

[0026] The calibration device 10 specifies the three-dimensional positions of the centers of gravity based on each of the first stereo camera 11 and the second stereo camera 12 based on the position of the pixel in the image coordinate system, the distance (D) of the pixel value of the pixel, and the internal parameters of the first stereo camera 11 and the second stereo camera 12 (matrix that converts a camera coordinate system based on one of the stereo cameras into a coordinate system of an image captured by the one camera) (FIG. 3 / STEP 16). That is, the three-dimensional positions of the centers of gravity in the first camera coordinate system based on the first stereo camera 11 (the imaging element of one of the cameras constituting the first stereo camera) and the three-dimensional positions of the centers of gravity in the second camera coordinate system based on the second stereo camera 12 (the imaging element of one of the cameras constituting the first stereo camera) are specified.

[0027] Then, an affine transformation matrix representing the relationship between the first state image coordinate system (u1, v1) and the second state image coordinate system (u2, v2) is calculated based on the three-dimensional positions of the centers of gravity of the first designated area 21, the second designated area 22, and the third designated area 23 relative to the first stereo camera 11, and the three-dimensional positions of the centers of gravity of the first designated area 21, the second designated area 22, and the third designated area 23 relative to the second stereo camera 12 (and internal camera parameters if necessary) (Figure 3 / STEP 18).

[0028] Specifically, a matrix representing the translation and rotation (as well as scaling) that matches the center of gravity positions of the first designated area 21, the second designated area 22, and the third designated area 23 in the first state image coordinate system (u1, v1) with the second state image coordinate system (u2, v2) is calculated as an affine transformation matrix.

[0029] Note that, when the internal parameters of the first stereo camera 11 and the second stereo camera 12 are the same, the affine transformation matrix may be defined by a translation matrix and a rotation matrix. Also, when the relationship between the scale of each of the first state image coordinate system (u1, v1) and the second state image coordinate system (u2, v2) and the real space scale is matched in advance by the internal parameters of the first stereo camera 11 and the second stereo camera 12, the affine transformation matrix may be defined by a translation matrix and a rotation matrix.

[0030] (How to integrate status images) A method for integrating a plurality of state images captured by a plurality of stereo cameras after the calibration has been performed as described above will be described with reference to the flowchart of FIG.

[0031] First, a first state image and a second state image in which a common object is reflected are captured by the first stereo camera 11 and the second stereo camera 12, respectively (FIG. 4 / STEP 22).

[0032] A spatial coordinate system transformation device (or an information processing terminal that is the same as or different from the information processing terminal that constitutes the calibration device 10) defines point cloud data by defining each pixel of the first state image and the second state image as a point having an RGB value as a pixel value and existing at a three-dimensional position corresponding to the distance (D) (or depth) to the object, also as a pixel value (FIG. 4 / STEP 24).

[0033] The spatial coordinate system transformation device transforms the point cloud data derived from the second state image into point cloud data in the first state image coordinate system using the affine transformation matrix calculated as described above (FIG. 4 / STEP 26).

[0034] Then, the spatial coordinate system transformation device integrates the transformed point cloud data and the point cloud data derived from the first state image to generate an integrated integrated point cloud in which the integrated point cloud data is reflected in the first state image coordinate system (Figure 4 / STEP 28).

[0035] (Another embodiment of the present invention) In the above embodiment, the first attribute, the second attribute, and the third attribute are different hues, and the first attribute variable, the second attribute variable, and the third attribute variable are defined by different RGB values, but in another embodiment, the first attribute variable, the second attribute variable, and the third attribute variable may be defined by HSV values ​​or MKY values. The first attribute, the second attribute, and the third attribute may be saturation and / or brightness in addition to hue, and the first attribute variable, the second attribute variable, and the third attribute variable may be defined by saturation and / or brightness in addition to different RGB values, HSV values, or MKY values. The first attribute, the second attribute, and the third attribute may be saturation and / or brightness that are different from each other instead of hue, and the first attribute variable, the second attribute variable, and the third attribute variable may be defined by saturation and / or brightness. [Explanation of symbols]

[0036] 10. Calibration device 11. First stereo camera 12. Second stereo camera 20. Calibration sheet 200. Local area 202. Frame area 21‥1st designated area 210. First local area 212. First frame area 22‥Second designated area 220. Second local area 222. Second frame area 23‥Third designated area 230. Third local area 232...Third frame area.

Claims

1. A calibration method for determining a relationship between a first state image coordinate system, which is a coordinate system of the first state image, and a second state image coordinate system, which is a coordinate system of the second state image, based on a first state image and a second state image configured by pixels having pixel values ​​each representing a distance to an object and a first attribute variable, a second attribute variable, and a third attribute variable representing a first attribute, a second attribute, and a third attribute variable of the object, each acquired by a stereo camera in a first position and posture and a second position and posture, a step of arranging an object having a first designated area in which first local areas having the first attribute are distributed, a second designated area in which second local areas having the second attribute are distributed, and a third designated area in which third local areas having the third attribute are distributed, on a flat designated surface, such that the first designated area, the second designated area, and the third designated area are reflected in the first state image and the second state image, respectively; specifying three-dimensional positions of a first representative point, a second representative point and a third representative point which are representative points of the first designated area, the second designated area and the third designated area reflected in the first state image and the second state image, respectively; and determining an affine transformation matrix based on the three-dimensional positions of the first representative point, the second representative point, and the third representative point identified based on the first state image and the three-dimensional positions of the first representative point, the second representative point, and the third representative point identified based on the second state image, such that the positions of the first representative point, the second representative point, and the third representative point in the first state image coincide with the positions of the first representative point, the second representative point, and the third representative point in the second state image. Calibration method.

2. 2. The calibration method according to claim 1, the first local regions are distributed in a checkerboard or lattice pattern in the first specified region, the second local regions are distributed in a checkerboard or lattice pattern in the second specified region, The third local regions are distributed in a checkerboard or lattice pattern in the third specified region. Calibration method.

3. 3. The calibration method according to claim 2, a periphery of the first specified area is surrounded by a first frame area having the first attribute, a periphery of the second specified area is surrounded by a second frame area having the second attribute, The periphery of the third specified area is surrounded by a third frame area having the third attribute. Calibration method.

4. 3. The calibration method according to claim 2, On the designated surface, fourth local regions having a fourth attribute different from any of the first attribute, the second attribute, and the third attribute are dispersed and arranged in a checkerboard or lattice pattern in regions surrounding the first designated region, the second designated region, and the third designated region. Calibration method.

5. 5. The calibration method according to claim 4, The periphery of the designated surface is surrounded by a fourth frame region having the fourth attribute. Calibration method.

6. The calibration method according to any one of claims 1 to 5, The first attribute, the second attribute, and the third attribute are hues, and the first attribute variables, the second attribute variables, and the third attribute variables are RGB values, HSV values, or CMY values. Calibration method.

7. The calibration method according to any one of claims 1 to 5, As the object, a sheet-like object is arranged, at least one of the front surface and the back surface of which is the designated surface. Calibration method.

8. A calibration program that provides an information processing terminal with a function for carrying out the calibration method according to any one of claims 1 to 5.

Images