Calibration method and calibration device for camera

Abstract

To provide a calibration method and a calibration device for a camera, capable of improving accuracy of calibration of a camera.SOLUTION: A calibration method includes: an image acquisition step of acquiring a plurality of calibration images captured by a camera, the calibration images each being a dot pattern with a plurality of dots periodically arranged; a pattern information setting step of setting pattern information including information regarding an arrangement of the dots; a feature point detection step of detecting a feature point showing a location of a dot from a calibration image; and a parameter calculation step of calculating a parameter of the camera according to the feature point and the pattern information. The feature point detection step detects a dot centroid position as the feature point, according to a projection image projecting a contour of a dot image DI on an image plane IP of the calibration image against a virtual plane VP set within a three-dimensional space.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a camera calibration method and calibration device. [Background technology]

[0002] Generally, when a camera is used for object detection, the camera is calibrated. In camera calibration, a planar calibration pattern having multiple feature points aligned two-dimensionally is prepared, and the calibration pattern is photographed from multiple directions by the camera. Then, feature points are detected from the multiple images photographed by the camera, and the camera parameters are estimated using the coordinates of the detected feature points. The camera parameters include the camera's internal parameters (focal length, optical center) or distortion parameters (distortion coefficients).

[0003] For example, Patent Document 1 discloses a method for calibrating a camera using a dot pattern in which a plurality of dots (black circles) are arranged in a lattice as a calibration pattern. In this method, the positions of the centers of gravity of dots included in the dot pattern are detected as feature points on an image (calibration image) captured by the camera, and camera parameters are obtained.

[0004] Patent Document 1 also discloses a grid pattern and a checkered pattern as calibration patterns other than the dot pattern. When these calibration patterns are used, intersections (corners) between straight lines are detected as feature points. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2022-30807 Summary of the Invention

Problems to be Solved by the Invention

[0006] Among the calibration patterns disclosed in Patent Document 1, for example, when a grid pattern shown in IX-A of FIG. 9 or a checker pattern shown in IX-B of FIG. 9 is used, intersections (corners) H1 or H2 of straight lines of each pattern on the calibration image are detected as feature points (that is, edge detection). In such edge detection, it is necessary to use the luminance gradient (differential value) of the edge. Therefore, the detection positions (coordinates) of the intersections H1 or H2 are easily affected by luminance noise, and there is a problem that the detection accuracy of the feature points is not good as compared with the dot pattern.

[0007] On the other hand, when the dot pattern shown in IX-C of FIG. 9 is used, since the barycentric position of the dot D of the dot pattern is detected as a feature point on the calibration image, it is less affected by luminance noise as compared with the grid pattern or the checker pattern, but there are problems as described below.

[0008] FIGS. 10 and 11 show examples of photographing the dot D on the dot pattern by the camera CAM. In FIG. 10, O c indicates the origin (camera origin) of the camera coordinate system with reference to the camera CAM. Also, in FIGS. 10 and 11, the U-axis and the V-axis are coordinate axes that constitute an image coordinate system (two-dimensional orthogonal coordinate system) defined on the image plane M of the camera CAM.

[0009] In an image captured using a camera having a normal lens (non - telecentric optical system), there is a characteristic that the magnification changes depending on the distance from the camera. That is, the longer the distance from the camera, the smaller the resulting image, and the shorter the distance, the larger the resulting image. Therefore, as shown in FIG. 10, when the dot D on the dot pattern is photographed by the camera CAM from an oblique direction, as shown in FIG. 11, the image of the dot D (hereinafter referred to as the "dot image") DI on the image captured by the camera CAM becomes an oval shape distorted in a specific direction according to the positional relationship with the camera CAM. That is, in the dot image DI on the image captured by the camera CAM, the portion on the front side (the side with a shorter distance from the camera CAM, the +V side) is relatively smaller than the portion on the back side (the side with a longer distance from the camera CAM, the -V side). Therefore, when trying to obtain the dot centroid position two - dimensionally from the shape of the dot image DI on the image, due to the influence of the non - telecentric optical system, the dot centroid position may be obtained at a position deviated from the original position.

[0010] FIGS. 12 and 13 are diagrams for explaining the influence of the inclination of the camera CAM with respect to the dot pattern (dot D) on the detection result of the dot centroid position.

[0011] FIG. 12 shows a state where the camera CAM is facing the dot pattern (dot D) directly (a state where the optical axis AX of the camera CAM is orthogonal to the dot D, that is, a state where the camera CAM is not inclined with respect to the dot pattern). In this case, since the dot image DI on the image plane M of the camera CAM and the dot D before projection are in a similar shape, the dot centroid position G1 on the image plane M (the centroid position obtained two - dimensionally from the shape of the dot image DI on the image plane M) coincides with the actual dot centroid position G0 projected on the image plane M (the projection point obtained by projecting the centroid position g of the dot D before projection onto the image plane M), and the dot centroid position can be detected without error.

[0012] FIG. 13 shows a state in which the camera CAM is tilted with respect to the dot pattern (dots D). In this case, due to the influence of the non-telecentric optical system, the dot image DI on the image plane M has an asymmetric shape with one side (the left side in FIG. 13) and the other side (the right side in FIG. 13) sandwiching the actual dot centroid position G0 projected onto the image plane M (the projection point of the centroid position g of the dot D before projection onto the image plane M). Therefore, the dot centroid position G1 on the image plane M does not coincide with the actual dot centroid position G0 projected onto the image plane M, resulting in a detection error of the dot centroid position.

[0013] The occurrence of such a detection error in the dot centroid position causes a factor that reduces the accuracy of camera calibration.

[0014] The present invention has been made in view of such circumstances, and an object thereof is to provide a camera calibration method and a calibration apparatus capable of improving the accuracy of camera calibration.

Means for Solving the Problems

[0015] The present invention comprises the following aspects in order to achieve the above object.

[0016] A camera calibration method according to a first aspect includes an image acquisition step of acquiring a plurality of calibration images obtained by photographing a dot pattern in which a plurality of dots are periodically arranged with a camera, a pattern information setting step of setting pattern information including information regarding the arrangement of the dots, a feature point detection step of detecting feature points indicating the positions of the dots from the calibration images, and a parameter calculation step of calculating camera parameters based on the feature points and the pattern information. The feature point detection step detects the position of the centroid of the dot as a feature point based on a projection image obtained by projecting the contour of the dot on the image plane of the calibration image onto a virtual plane set in a three-dimensional space.

[0017] In the calibration method of the camera according to the second aspect, in the first aspect, in the feature point detection step, a virtual plane optimized so that a circle fits the projection image is obtained, and based on the projection image projected onto the obtained virtual plane, the position of the center of gravity of the dot is detected as a feature point.

[0018] In the calibration method of the camera according to the third aspect, in the first aspect, in the feature point detection step, when an approximate circle is an approximate circle obtained by approximating a point group composed of a plurality of projection points obtained by projecting a plurality of contour points indicating the contour of a dot on the image plane onto a virtual plane, an optimized virtual plane is obtained such that an evaluation value indicating the fitting error of the approximate circle with respect to the point group is minimized or equal to or less than a threshold value, and the position of the point obtained by projecting the center of the approximate circle on the optimized virtual plane onto the image plane is set as the position of the center of gravity of the dot.

[0019] In the calibration method of the camera according to the fourth aspect, in the third aspect, the feature point detection step includes a search step of searching for an optimized virtual plane from a plurality of virtual planes having different postures in the three-dimensional space based on an evaluation value indicating the fitting error of the approximate circle with respect to the point group.

[0020] In the calibration method of the camera according to the fifth aspect, in the fourth aspect, in the search step, while changing the virtual plane to a plurality of postures by rotating the virtual plane around an axis parallel to the major axis direction of the dot on the image plane, an evaluation value is calculated for each posture, and an optimized virtual plane is determined based on the evaluation value calculated for each posture.

[0021] In the calibration method of the camera according to the sixth aspect, in the fourth aspect, the feature point detection step includes a contour point extraction step of extracting a plurality of contour points of the dot from the image plane, a half-line group generation step of generating a half-line group composed of a plurality of half-lines passing through the plurality of contour points of the dot on the image plane with the optical center of the camera as a starting point, and an approximate circle calculation step of calculating an approximate circle based on the intersection group of the half-line group and the virtual plane.

[0022] The calibration apparatus for a camera according to the seventh aspect includes an image acquisition unit that acquires a plurality of calibration images obtained by photographing a dot pattern in which a plurality of dots are periodically arranged with the camera, a pattern information setting unit that sets pattern information including information regarding the arrangement of dots in the dot pattern, a feature point detection unit that detects feature points from the calibration images, and a parameter calculation unit that calculates parameters of the camera based on the feature points and the pattern information. The feature point detection unit detects the position of the center of gravity of the dots as feature points based on a projection image obtained by projecting the contour of the dots on the image plane of the calibration image onto a virtual plane set in a three-dimensional space.

Advantages of the Invention

[0023] According to the present invention, the accuracy of camera calibration can be improved.

Brief Description of the Drawings

[0024]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Embodiments for Carrying Out the Invention

[0025] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0026] 〔Projection Model of Camera〕 First, the projection model of the camera in this embodiment will be described. FIG. 1 is an explanatory diagram for explaining the projection model of the camera in this embodiment.

[0027] As shown in FIG. 1, the world coordinate system (world coordinate system) is a coordinate system representing positions in a three-dimensional space (real space), with the origin at O w and a three-dimensional orthogonal coordinate system with the X w axis, Y w axis, and Z w axis as coordinate axes. Note that any coordinate system may be used for the world coordinate system as long as it can specify positions in a three-dimensional space (three-dimensional positions). The camera coordinate system has the optical axis center O c of the camera as the origin, the right direction from the origin O c as the X c axis, the downward direction as the Y c axis, and the optical axis direction as the Z c axis, which is a three-dimensional orthogonal coordinate system. The image coordinate system has the upper left of the image plane IP, which is at a focal distance f from the origin O c of the camera coordinate system in the Z c direction, as the origin, and a two-dimensional orthogonal coordinate system (pixel coordinate system) having a U axis and a V axis in directions parallel to the X c axis and the Y c axis, respectively.

[0028] First, the coordinates (x w , y w , z w ) of a point P (object point) in a three-dimensional space in the world coordinate system can be converted into the coordinates (x, y, z) in the camera coordinate system according to the following equation (1) using the rotation matrix R and the translation vector t of the camera.

[0029]

Equation

[0030] Here, [R|t] is a matrix (external parameter matrix) for converting from the world coordinate system to the camera coordinate system, representing the pose and position of the camera in the world coordinate system. Each component r 11 , r 12 , ···, r 33 , t x , t y , t z of [R|t] is called an external parameter of the camera.

[0031] Next, when the coordinates (pixel coordinates) of the projection point Q obtained by projecting the point P at (x, y, z) onto the image plane IP as seen from the camera coordinate system are (u, v), the relational expressions shown in the following equations (2) to (7) hold.

[0032]

Equation

[0033]

Equation

[0034]

Equation

[0035]

Equation

[0036]

Number

[0037]

Number

[0038] Here, (x′, y′) represents the coordinates of the projection point obtained by projecting a point P at (x, y, z) onto the normalized image plane (z = 1) as seen from the camera coordinate system. Also, (x″, y″) represents the coordinates of the projection point (distorted point) obtained by projecting the point P onto the normalized image plane when considering the lens distortion of the camera.

[0039] Also, f x and f y represent the focal lengths in the x - direction and y - direction expressed in pixel units. Also, c x and c y represent the optical center in the image coordinate system (the position where the optical axis of the camera intersects the image plane IP, the optical center in pixel units). Also, k1, k2, k3 are the radial distortion coefficients, and p1, p2 are the tangential distortion coefficients. In this specification, the focal lengths f x , f y , the optical centers c x , c y are referred to as the internal parameters of the camera, and the distortion coefficients k1, k2, k3, p1, p2 are referred to as the distortion parameters of the camera.

[0040] 〔Calibration Device〕 FIG. 2 is a block diagram showing an example of the schematic configuration of the calibration device 10 of the present embodiment. As shown in FIG. 2, the calibration device 10 includes an arithmetic control unit 20 and a storage unit 22. Also, a camera 12, an operation unit 14, and an output unit 16 are connected to the calibration device 10.

[0041] The camera 12 generates images IM (calibration images) obtained by photographing a calibration pattern from different directions, and outputs them to the calibration device 10. The camera 12 used in the present embodiment has a non - telecentric optical system (an optical system having non - telecentricity on the subject side), and photographing is performed by an image sensor via a lens group as the non - telecentric optical system.

[0042] The operation unit 14 includes operation members such as a keyboard and a mouse, and accepts input of various operations by an operator.

[0043] The output unit 16 is a device for outputting the calculation result etc. by the arithmetic control unit 20. The output unit 16 includes, for example, an operation UI (User Interface) and a monitor (for example, a liquid crystal display etc.) for displaying the calculation result. Further, in addition to or instead of the monitor, the output unit 16 may include a printer or a speaker etc.

[0044] The arithmetic control unit 20 controls the operation of the calibration device 10. The arithmetic control unit 20 is constituted by an arithmetic device such as a personal computer, and includes an arithmetic circuit constituted by various processors (Processor) and memories etc. The various processors include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), and programmable logic devices [for example, SPLD (Simple Programmable Logic Devices), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Arrays)] etc. Note that the various functions of the calibration device 10 may be realized by one processor, or may be realized by a plurality of processors of the same type or different types.

[0045] The storage unit 22 stores a control program and various types of data. The storage unit 22 is constituted by, for example, a hard disk drive (HDD) or a solid state drive (SSD). The storage unit 22 may include a temporary storage element constituted by a random access memory (RAM) such as a dynamic random access memory (DRAM) or a static random access memory (SRAM), and may function as a work area of the arithmetic control unit 20.

[0046] A plurality of calibration images IM captured by the camera 12 are temporarily stored in the storage unit 22. Further, the camera parameters CP obtained by the arithmetic control unit 20 are stored in the storage unit 22. The camera parameters CP include internal parameters (focal length, optical center) of the camera 12 and distortion parameters (radial and tangential distortion coefficients).

[0047] The arithmetic control unit 20 functions as an image acquisition unit 30, a pattern information setting unit 32, a feature point detection unit 34, and a parameter calculation unit 36 by reading and executing the control program stored in the storage unit 22. The functions of each part constituting the arithmetic control unit 20 will be described later.

[0048] Hereinafter, a processing procedure (an example of a calibration method) of the calibration process executed by the calibration device 10 of the present embodiment will be described. FIG. 3 is a flowchart showing the overall process of the calibration process executed by the calibration device 10 of the present embodiment. It is assumed that various initial setting processes such as operation confirmation of each part of the calibration device 10 are performed at the start of the flowchart shown in FIG. 3.

[0049] (Step S10: Image acquisition step) First, the image acquisition unit 30 acquires a plurality of calibration images IM captured by the camera 12 and stores them in the storage unit 22. In the present embodiment, as an example of the calibration pattern, as shown in FIG. 4, a dot pattern DP (also referred to as a "circle grid") in which a plurality of dots D (black circles) are arranged vertically and horizontally is used. Note that if feature points are distributed in a shape and arrangement that are easily detected by an existing search method, the shape or arrangement of each dot D in the dot pattern DP is not limited to the example shown in FIG. 4.

[0050] The plurality of calibration images IM are images obtained by photographing the dot pattern DP as a calibration pattern from different directions with the camera 12. The number of calibration images IM is at least two or more, preferably five or more, more preferably ten or more. Note that the photographing of the calibration pattern (dot pattern DP) by the camera 12 may be performed as long as it is performed at least before the image acquisition step.

[0051] The image acquisition unit 30 may acquire each calibration image IM from the camera 12 through a cable or a recording medium. Further, each calibration image IM may be stored in an external storage device such as an external server installed outside the calibration device 10, and the image acquisition unit 30 may acquire each calibration image IM from the external storage device via a wired or wireless network.

[0052] (Step S12: Pattern Information Setting Step) Next, the pattern information setting unit 32 sets pattern information including information regarding the arrangement of dots D in the dot pattern DP. For example, the pattern information includes the number of dots in the vertical direction (column direction) and the horizontal direction (row direction) of the dot pattern DP. In the example shown in FIG. 4, the number of dots in the vertical direction of the dot pattern DP is 5, and the number of dots in the horizontal direction is 8. Further, the pattern information includes the total lengths S1 and S2 in the vertical and horizontal directions of the dot pattern DP in addition to the number of dots in the vertical and horizontal directions of the dot pattern DP. Note that instead of the total lengths S1 and S2 in the vertical and horizontal directions of the dot pattern DP, pitches P1 and P2 in the vertical and horizontal directions of the dot pattern DP may be included. Thereby, it becomes possible to specify the relative positional relationship of each dot D on the dot pattern DP. Note that the pattern information is information used to specify a point (object point) in the three-dimensional space corresponding to a feature point (image point) in the image coordinate system in the feature point detection unit 34 described later.

[0053] The pattern information setting unit 32 may, for example, acquire the pattern information input by the operator via the operation unit 14. Alternatively, the pattern information may be stored in advance in the storage unit 22, and the pattern information setting unit 32 may acquire it from the storage unit 22. Note that the pattern information setting step only needs to be performed at least before the parameter calculation step described later. For example, it may be performed before the image acquisition step is performed, or may be performed after the feature point detection step is performed.

[0054] (Step S14: Feature Point Detection Step) Next, the feature point detection unit 34 executes a feature point detection process for detecting a plurality of feature points for each of the plurality of calibration images IM captured by the camera 12. Specifically, the feature point detection unit 34 sequentially reads the plurality of calibration images IM stored in the storage unit 22. Then, for each of the read calibration images IM, after performing predetermined image processing (such as grayscale conversion) on the calibration image IM, the feature point detection unit 34 detects each feature point (image point) from the calibration image IM and obtains the coordinates (pixel coordinates) of each feature point in the image coordinate system. In the present embodiment, the position of the center of gravity (dot center of gravity position) of each dot D (dot image DI) is detected as a feature point on the calibration image IM. The coordinates of the feature points detected by the feature point detection unit 34 are temporarily stored in the storage unit 22. Note that the details of the feature point detection process executed by the feature point detection unit 34 will be described later.

[0055] If there is a calibration image IM for which the detection of feature points has failed among the plurality of calibration images IM, the feature point detection unit 34 performs an exclusion process of excluding the calibration image IM for which the detection of feature points has failed from the target of the parameter calculation process described later. As a result, in the parameter calculation process, the camera parameters CP can be calculated based on the calibration images IM after the exclusion process (that is, the calibration images IM for which the detection of feature points has been successful) among the plurality of calibration images IM.

[0056] (Step S16: Parameter calculation step) Next, the parameter calculation unit 36 executes the calculation process of the camera parameters CP. Specifically, the parameter calculation unit 36 calculates the camera parameters CP based on the positions (dot centroid positions) of the feature points on each calibration image IM detected by the feature point detection unit 34 and the pattern information set by the pattern information setting unit 32. The camera parameters CP can be calculated using a known method (for example, Zhang's method). Zhang's method is a method of optimizing parameters so that the positions of the feature points (image points) on the captured image (calibration image IM) and the positions of the points (object points, known) in the three-dimensional space corresponding to the feature points have a correct correspondence relationship (Z. Zhang, "A flexible new technique for camera calibration", IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.22, No.11, pp.1330-1334, 2000.).

[0057] The camera parameters CP calculated in the parameter calculation unit 36 are the internal parameters (focal lengths f x 、f y 、optical centers c x 、c y ) and distortion parameters (distortion coefficients k1, k2, k3, p1, p2) of the camera 12. The parameter calculation unit 36 stores the calculated camera parameters CP in the storage unit 22.

[0058] (Step S18: Output step) Next, the parameter calculation unit 36 outputs the calculation result of the camera parameters CP to the output unit 16. As a result, the operator can obtain the camera parameters CP, and thus can perform image correction and the like on the images captured by the camera 12 based on the camera parameters CP.

[0059] Thus, the flowchart of the entire calibration process executed by the calibration device 10 ends.

[0060] [Feature Point Detection Process] Next, the details of the feature point detection process executed in the feature point detection step will be described. The feature point detection process in this embodiment is a process of detecting the dot centroid position as a feature point on the calibration image IM. Note that the feature point detection process is sequentially performed for each calibration image IM.

[0061] First, the detection principle of the feature point detection process will be described with reference to FIG. 5. FIG. 5 is an explanatory diagram for explaining the detection principle of the feature point detection process.

[0062] When the dot pattern DP is photographed in a state of being inclined with respect to the camera 12, due to the influence of the non - telecentric optical system of the camera 12, the dots D on the dot pattern DP are distorted and projected onto the image plane IP of the camera 12. Therefore, when trying to directly detect the dot centroid position in the dot image DI in the calibration image IM two - dimensionally, there is a problem that the detected dot centroid position is shifted from the original position.

[0063] Therefore, in this embodiment, the feature point detection process is performed three - dimensionally using a virtual plane VP as described below. That is, in this feature point detection process, as shown in FIG. 5, when a virtual plane set in the three - dimensional space is used as the virtual plane VP, the virtual plane VP optimized by a fitting method such as the least - squares method is obtained so that the projection image of the contour of the dot image DI on the image plane IP of the calibration image IM approaches a circle on the virtual plane VP.

[0064] Specifically, starting from the origin O of the camera coordinate system c (the optical center of the camera 12), a group of half - lines B composed of a plurality of half - lines passing through a plurality of contour points of the dot image DI on the image plane IP is generated, and the intersection group T between the group of half - lines B and the virtual plane VP is obtained. Then, when the circle obtained by approximating the intersection group T by the least - squares method is used as the approximate circle, the virtual plane VP optimized so that the evaluation value indicating the fitting error of the approximate circle with respect to the intersection group T becomes small is obtained.

[0065] Here, as the virtual plane VP approaches a state parallel to the dot pattern DP (not shown in FIG. 5) in the three-dimensional space, the evaluation value indicating the fitting error of the approximate circle with respect to the intersection group T on the virtual plane VP becomes smaller. Therefore, the virtual plane VP optimized so that the evaluation value indicating the fitting error of the approximate circle with respect to the intersection group T on the virtual plane VP becomes smaller can be regarded as being in a posture substantially parallel to the dot pattern DP in the three-dimensional space. Note that substantially parallel includes not only the case of being exactly parallel but also the case of being approximately parallel.

[0066] And when the center VC of the approximate circle on the virtual plane VP optimized in this way is projected onto the image plane IP, the position of the projection point IC projected onto the image plane IP indicates the dot centroid position where the distortion error due to projection is minimized on the image plane IP.

[0067] In the present embodiment, among a plurality of virtual planes VP with different postures in the three-dimensional space, a virtual plane VP optimized so that the evaluation value indicating the fitting error as described above becomes smaller is searched for. And the position of the projection point IC obtained by projecting the center VC of the approximate circle on the optimized virtual plane VP onto the image plane IP is set as the dot centroid position. As described above, since the optimized virtual plane VP can be regarded as being substantially parallel to the dot pattern DP in the three-dimensional space, by setting the projection point IC obtained by projecting the center VC of the approximate circle on the optimized virtual plane VP onto the image plane IP as the dot centroid position, the coordinates (pixel coordinates) of the dot centroid position where the distortion error due to projection is minimized can be obtained.

[0068] FIGS. 6 and 7 are flowcharts showing an example of the procedure of the feature point detection process. Hereinafter, an example of the procedure of the feature point detection process will be described according to the flowcharts shown in FIGS. 6 and 7.

[0069] (Step S20: Contour point extraction step) First, the feature point detection unit 34 reads the calibration image IM, performs a predetermined image process (such as grayscale conversion process) on the calibration image IM, and then extracts a plurality of contour points indicating the contour of the dot image DI on the image plane IP of the calibration image IM. The number of contour points extracted at this time is at least three or more. The extraction of the contour (a plurality of contour points) of the dot image DI uses a known method such as edge detection or binarization processing. Thereby, the coordinates of a plurality of contour points (contour pixels) indicating the contour of the dot image DI on the image plane IP are obtained.

[0070] (Step S22: Half-line group generation step) Next, as shown in FIG. 5, the feature point detection unit 34 c generates a half-line group B composed of a plurality of half-lines passing through each of the contour points obtained in the contour point extraction step, starting from the origin O of the camera coordinate system.

[0071] (Step S24: Dot major axis direction calculation step) Next, the feature point detection unit 34 calculates the major axis direction of the dot image DI on the image plane IP (hereinafter referred to as the "dot major axis direction").

[0072] (Step S26: Angle initialization step) Next, the feature point detection unit 34 initializes the angle θ of the virtual plane VP to 0°. Here, the angle θ of the virtual plane VP is an angle indicating the rotation position when the virtual plane VP is rotated around an axis parallel to the dot major axis direction in the loop process described later. In the present embodiment, when the virtual plane VP is in the initial rotation position (θ = 0°), it is assumed that the virtual plane VP is arranged at a position parallel to the image plane IP in the three-dimensional space, but it is not limited thereto, and the virtual plane VP may be arranged at an arbitrary position.

[0073] (Step S28: Loop start step) The loop start step is the start of a loop process (Loop1) that repeats the processes from step S30 to step S56 described later N times. Note that this loop process is an example of the search step of the present invention.

[0074] (Step S30: Positive side rotation angle calculation step) Next, the feature point detection unit 34 obtains, as the first angle θ1, a value obtained by adding a preset step angle δ to the angle θ of the virtual plane VP as shown in the following formula (8) (however, δ θ > 0. The same applies hereinafter). θ

Equation

[0075] (Step S32: Virtual plane rotation step) Next, the feature point detection unit 34 performs a process of rotating the virtual plane VP around an axis parallel to the dot major axis direction so that the virtual plane VP becomes the first angle θ1. Hereinafter, the virtual plane VP that has become the first angle θ1 is referred to as the "first virtual plane VP1".

[0076] (Step S34: Intersection point group calculation step) Next, the feature point detection unit 34 calculates a first intersection point group T1 between the half-line group B and the first virtual plane VP1.

[0077] (Step S36: Circle approximation step) Next, the feature point detection unit 34 performs a circle approximation process on the first intersection point group T1 by the least squares method or the like, and calculates the position and shape (center position and radius) of the approximate circle on the first virtual plane VP1.

[0078] (Step S38: First evaluation value calculation step) Next, the feature point detection unit 34 calculates, as a first evaluation value ε1, an evaluation value indicating the fitting error of the approximate circle with respect to the first intersection point group T1 on the first virtual plane VP1. As the evaluation value calculated as the first evaluation value ε1, for example, the sum of squared residuals or the sum of absolute residuals is applied. The sum of squared residuals is the sum of the squares of the residuals, which are the differences between the distances from the center position of the approximate circle to each point of the first intersection point group T1 and the radius of the approximate circle, and the sum of absolute residuals is the sum of the absolute values of those residuals.

[0079] (Step S40: Negative side angle calculation step) Next, as shown in the following formula (9), the feature point detection unit 34 subtracts a preset step angle δ θ from the angle θ of the virtual plane VP by the value thus obtained as the second angle θ2.

Equation

[0080] (Step S42: Virtual plane rotation step) Next, the feature point detection unit 34 performs a process of rotating the virtual plane VP around an axis parallel to the dot major axis direction so that the virtual plane VP becomes the second angle θ2. Hereinafter, the virtual plane VP that has become the second angle θ2 is referred to as the "second virtual plane VP2".

[0081] (Step S44: Intersection point group calculation step) Next, the feature point detection unit 34 calculates a second intersection point group T2 between the half-line group B and the second virtual plane VP2.

[0082] (Step S46: Circle approximation step) Next, the feature point detection unit 34 performs a circle approximation process on the second intersection point group T2 by the least squares method or the like, and calculates the position and shape (center position and radius) of the approximate circle on the second virtual plane VP2.

[0083] (Step S48: Second evaluation value calculation step) Next, the feature point detection unit 34 calculates, as the second evaluation value ε2, an evaluation value indicating the fitting error of the approximate circle with respect to the second intersection group T2 on the second virtual plane VP2. The evaluation value calculated as the second evaluation value ε2 is the same as the first evaluation value ε1 described above, and the sum of squared residuals or the sum of absolute residuals is applied.

[0084] (Step S50: Evaluation value comparison step) Next, the feature point detection unit 34 compares the magnitudes of the first evaluation value ε1 and the second evaluation value ε2. When the first evaluation value ε1 is smaller than the second evaluation value ε2 (Yes), the process proceeds to step S52. When the first evaluation value ε1 is greater than or equal to the second evaluation value ε2 (No), the process proceeds to step S54. Note that when the first evaluation value ε1 and the second evaluation value ε2 are equal, the process may proceed to step S52 instead of step S54.

[0085] (Steps S52 and S54: Angle update step) Next, when the first evaluation value ε1 is smaller than the second evaluation value ε2, the feature point detection unit 34 updates the angle θ of the virtual plane VP to the first angle θ1 (step S52). When the first evaluation value ε1 is greater than or equal to the second evaluation value ε2, the feature point detection unit 34 updates the angle θ of the virtual plane VP to the second angle θ2 (step S54).

[0086] (Step S56: Step angle update step) Next, the feature point detection unit 34 θ updates the step angle δ θ to δ θ ×α. However, the coefficient α is a real number such that 0 < α ≤ 1. For example, the coefficient α is empirically set based on the required calculation accuracy and the like. Thereby, when the coefficient α is set to less than 1, the step angle δ θSince it gradually (step by step) becomes smaller, it is possible to accurately obtain a virtual plane VP optimized so that the evaluation value indicating the fitting error of the circle with respect to the intersection point group T becomes smaller. Further, the step angle update step may be performed only when the smaller evaluation value among the first evaluation value ε1 and the second evaluation value ε2 is smaller than a preset threshold value (evaluation value threshold value). As a result, as the evaluation value becomes smaller, the step angle δ θ can be gradually reduced to perform the pursuit calculation, and the processing speed of the feature point detection process can be improved.

[0087] (Step S58: Loop end step) The loop end step is the end point of the above-described loop process (Loop1). In the present embodiment, when the processes from step S30 to step S56 are repeated N times, the loop process ends. Note that, as the end condition of the loop process, when the number of times of the loop process reaches the upper limit value N (where N is an integer of 2 or more) (first end condition), and the updated step angle δ θ is smaller than a preset threshold value (angle threshold value) (second end condition), the loop process may be ended when either one of the conditions is satisfied. Further, the loop process may be ended when the smaller evaluation value among the first evaluation value ε1 and the second evaluation value ε2 satisfies the condition that it is smaller than the threshold value (evaluation value threshold value) (third end condition).

[0088] By performing the above loop process, it is possible to search for a virtual plane VP optimized to an angle θ such that the evaluation value indicating the fitting error of the approximate circle with respect to the intersection point group T on the virtual plane VP becomes the minimum or less than the threshold value among a plurality of virtual planes VP having different postures in the three-dimensional space.

[0089] (Step S60: Intersection point group calculation step) Next, the feature point detection unit 34 calculates an intersection point group T between the virtual plane VP optimized to the angle θ by the above loop process and the half line group B.

[0090] (Step S62: Circle approximation step) Next, the feature point detection unit 34 performs circle approximation processing on the intersection group T obtained in the intersection group calculation step (step S60) by the least squares method or the like, and calculates the position and shape (center position and radius) of the approximate circle on the virtual plane VP.

[0091] (Step S64: Half-line generation step) Next, the feature point detection unit 34 uses the origin O of the camera coordinate system c as a starting point to generate a half-line CL passing through the center VC of the approximate circle obtained in the circle approximation step (step S62).

[0092] (Step S66: Image plane intersection calculation step) Next, the feature point detection unit 34 calculates the pixel position (pixel coordinates) of the intersection of the half-line CL and the image plane IP. This intersection becomes the projection point IC obtained by projecting the center VC of the approximate circle on the virtual plane VP onto the image plane IP.

[0093] (Step S68: Feature point determination step) Next, the feature point detection unit 34 determines the position (dot centroid position) of the feature points of the calibration image IM to the position (pixel coordinates) of the intersection on the image plane IP obtained in step S66 (that is, the projection point IC).

[0094] Thus, the flowchart of the dot detection process ends.

[0095] 〔Effect〕 Next, the effects of this embodiment will be described.

[0096] Figure 8 is a graph for explaining the effects of the present embodiment. In Figure 8, the horizontal axis represents the inclination angle (unit: degree (°)) of the dot pattern DP with respect to the camera 12, and the vertical axis represents the detection error (unit: pixel) of the dot centroid position. The graph shown by the solid line in Figure 8 indicates the detection error when the dot centroid position is detected by the feature point detection process in the present embodiment. Also, the graph shown by the broken line in Figure 8 indicates the detection error when the dot centroid position is directly detected two-dimensionally from the calibration image as a comparative example.

[0097] As can be seen from Figure 8, in the comparative example, as the inclination angle (absolute value) of the dot pattern DP with respect to the camera 12 increases, the detection error (absolute value) of the dot centroid position increases.

[0098] On the other hand, in the feature point detection process in the present embodiment, an optimized virtual plane VP is obtained by fitting a circle to the projection image obtained by projecting the contour of the dot image DI on the image plane IP. Then, since the position of the projection point obtained by projecting the center of the circle (approximate circle) on the optimized virtual plane VP onto the image plane IP is defined as the dot centroid position, it is possible to obtain the coordinates (pixel coordinates) of the dot centroid position with the minimum distortion error due to projection.

[0099] Therefore, in the present embodiment, compared with the comparative example, the detection error of the dot centroid position is overall smaller. In particular, in the comparative example, as the inclination angle (absolute value) of the camera 12 with respect to the dot pattern DP increases, the detection error (absolute value) of the dot centroid position tends to increase. However, in the present embodiment, even when the inclination angle (absolute value) increases, the detection error (absolute value) of the dot centroid position can be suppressed within a certain range (in the example shown in Figure 8, the detection error (absolute value) is within 0.1 pixel).

[0100] Therefore, according to this embodiment, since the dot centroid position can be detected without being affected by the non-telecentric optical system, it has excellent tilt resistance characteristics, and the position (dot centroid position) of the feature points on the calibration image IM can be accurately obtained. As a result, it is possible to improve the calibration accuracy of the camera 12.

[0101] Also, according to this embodiment, since it has excellent tilt resistance characteristics, the degrees of freedom of the posture and position of the camera 12 with respect to the dot pattern DP are high, and it is excellent in convenience for the user.

[0102] Also, according to this embodiment, when searching for the optimized virtual plane VP in the three-dimensional space, it is performed by rotating around an axis parallel to the direction orthogonal to the direction in which the dot image DI is likely to be distorted (dot major axis direction) on the image plane IP of the calibration image IM. Thereby, by rotating the virtual plane VP around one axis, it becomes possible to efficiently search for the optimized virtual plane VP. Note that the search for the optimized virtual plane VP is not limited to one axis, and may be searched by rotating the virtual plane VP around any two axes or three axes.

[0103] As described above, the embodiments of the present invention have been described. However, the present invention is not limited to the above examples, and of course, various improvements and modifications may be made without departing from the gist of the present invention.

Explanation of Reference Numerals

[0104] 10... Calibration device, 12... Camera, 14... Operation unit, 16... Output unit, 20... Arithmetic control unit, 22... Storage unit, 30... Image acquisition unit, 32... Pattern information setting unit, 34... Feature point detection unit, 36... Parameter calculation unit, IM... Calibration image, IP... Image plane, VP... Virtual plane, DI... Dot image, VC... Center, IC... Projection point, B... Half-line group, T... Intersection group, CL... Straight line, DP... Dot pattern, D... Dot

Claims

1. An image acquisition step of acquiring a plurality of calibration images obtained by photographing a dot pattern in which a plurality of dots are periodically arranged with a camera; A pattern information setting step of setting pattern information including information regarding the arrangement of the dots; A feature point detection step of detecting feature points indicating the positions of the dots from the calibration images; A parameter calculation step of calculating parameters of the camera based on the feature points and the pattern information; comprising: In the feature point detection step, based on a projection image obtained by projecting the contour of the dot on the image plane of the calibration image onto a virtual plane set in a three-dimensional space, the position of the center of gravity of the dot is detected as the feature point. A calibration method for a camera.

2. In the feature point detection step, an optimized virtual plane is obtained such that a circle fits the projection image, and based on the projection image projected onto the obtained virtual plane, the position of the center of gravity of the dot is detected as the feature point. The calibration method for a camera according to Claim 1.

3. In the feature point detection step, when a circle obtained by approximating a point group composed of a plurality of projection points obtained by projecting a plurality of contour points indicating the contour of the dot on the image plane onto the virtual plane is used as an approximate circle, an optimized virtual plane is obtained such that an evaluation value indicating the fitting error of the approximate circle with respect to the point group is minimized or equal to or less than a threshold value, and the position of the point obtained by projecting the center of the approximate circle on the optimized virtual plane onto the image plane is set as the position of the center of gravity of the dot. The calibration method for a camera according to Claim 1.

4. The feature point detection step includes a search step of searching for the optimized virtual plane from among a plurality of virtual planes having different postures in a three-dimensional space based on an evaluation value indicating the fitting error of the approximate circle with respect to the point group. The calibration method for a camera according to Claim 3.

5. In the search step, while changing the virtual plane to a plurality of postures by rotating the virtual plane around an axis parallel to the major axis direction of the dot on the image plane, the evaluation value is calculated for each posture, and based on the evaluation value calculated for each posture, the optimized virtual plane is determined. The calibration method for a camera according to Claim 4.

6. The feature point detection step includes: a contour point extraction step of extracting the plurality of contour points of the dot from the image plane; a half-line group generation step of generating a group of half-lines composed of a plurality of half-lines each passing through the plurality of contour points of the dot on the image plane with the optical center of the camera as a starting point; an approximate circle calculation step of calculating the approximate circle based on the intersection group of the half-line group and the virtual plane; The camera calibration method according to claim 4, comprising the above steps.

7. an image acquisition unit that acquires a plurality of calibration images obtained by photographing a dot pattern in which a plurality of dots are periodically arranged with a camera; a pattern information setting unit that sets pattern information including information regarding the arrangement of the dots in the dot pattern; a feature point detection unit that detects feature points from the calibration images; a parameter calculation unit that calculates parameters of the camera based on the feature points and the pattern information; comprising: The feature point detection unit detects the position of the center of gravity of the dot as the feature point based on a projection image obtained by projecting the contour of the dot on the image plane of the calibration image onto a virtual plane set in a three-dimensional space; A camera calibration device.

Images