Calibration method and calibration device for camera

Abstract

This calibration method for a camera includes: an image acquisition step for acquiring a plurality of calibration images obtained by imaging, with a camera, a dot pattern in which a plurality of dots are disposed periodically; a pattern information setting step for setting pattern information including information on the disposition of the dots; a feature point detection step for detecting feature points indicating the positions of the dots by using the calibration images; and a parameter calculation step for calculating parameters for the camera on the basis of the feature points and the pattern information. The feature point detection step involves detecting, as the feature points, gravity center positions of the dots on the basis of projection images obtained by projecting, onto virtual planes VP set within a three-dimensional space, the outlines of dot images DI on image planes IP of the calibration images.

Description

Camera calibration method and calibration device

[0001] The present invention relates to a camera calibration method and calibration device.

[0002] Generally, when a camera is used for object detection, the camera is calibrated. In camera calibration, a planar calibration pattern having a plurality of feature points aligned two-dimensionally is prepared, and the calibration pattern is photographed from multiple directions with the camera. Then, the feature points are detected from the multiple images photographed by the camera, and the camera parameters are estimated using the coordinates of the detected feature points. The camera parameters include the camera's internal parameters (focal length, optical center) or distortion parameters (distortion coefficients).

[0003] For example, Patent Document 1 discloses a method for calibrating a camera using a dot pattern in which a plurality of dots (black circles) are arranged in a grid pattern as a calibration pattern. In this method, camera parameters are obtained by detecting, as feature points, the positions of the centers of gravity of dots included in the dot pattern on an image captured by the camera (calibration image).

[0004] Furthermore, Patent Document 1 also discloses a grid pattern and a checkered pattern as calibration patterns other than dot patterns. When these calibration patterns are used, intersections (corners) of straight lines are detected as feature points.

[0005] Japanese Patent Application Laid-Open No. 2022-30807

[0006] Among the calibration patterns disclosed in Patent Document 1, for example, when the grid pattern shown in 1025A of Fig. 25 or the checkered pattern shown in 1025B of Fig. 25 is used, intersections (corners) H1 or H2 between straight lines of each pattern are detected as feature points (i.e., edge detection) on the calibration image. Such edge detection requires the use of the luminance gradient (differential value) of the edge. Therefore, the detected position (coordinates) of the intersection H1 or H2 is easily affected by luminance noise, resulting in a problem that the detection accuracy of feature points is poor compared to dot patterns.

[0007] On the other hand, when the dot pattern shown in 1025C in Figure 25 is used, the center of gravity positions of the dots D of the dot pattern are detected as feature points on the calibration image, so it is less susceptible to the effects of luminance noise compared to grid patterns or checker patterns, but there are issues as described below.

[0008] 26 and 27 show examples of dots D on a dot pattern photographed by a camera CAM. ｃ indicates the origin (camera origin) of the camera coordinate system based on the camera CAM. In addition, in Figures 26 and 27, the U axis and V axis are coordinate axes that constitute the image coordinate system (two-dimensional Cartesian coordinate system) defined on the image plane M of the camera CAM.

[0009] Images captured using a camera with a normal lens (non-telecentric optical system) have the characteristic that the magnification changes depending on the distance from the camera. That is, the longer the distance from the camera, the smaller the resulting image, and the shorter the distance, the larger the resulting image. Therefore, when dots D on a dot pattern are captured by camera CAM from an oblique angle as shown in FIG. 26 , the image of the dot D on the image captured by camera CAM (hereinafter referred to as the "dot image") DI, as shown in FIG. 27 , becomes an oval shape distorted in a specific direction depending on the positional relationship with camera CAM. That is, in the dot image DI on the image captured by camera CAM, the front side (the side closer to camera CAM, the +V side) becomes relatively smaller than the back side (the side farther from camera CAM, the -V side). Therefore, when attempting to determine the dot centroid position two-dimensionally from the shape of the dot image DI on the image (image captured by camera CAM), the non-telecentric optical system may cause the dot centroid position to be determined at a position shifted from its original position.

[0010] 28 and 29 are diagrams for explaining the influence that the inclination of the camera CAM with respect to the dot pattern (dots D) has on the detection result of the dot centroid position.

[0011] 28 shows a state in which the camera CAM faces the dot pattern (dots D) (a state in which the optical axis AX of the camera CAM is perpendicular to the dots D, i.e., a state in which the camera CAM is not tilted with respect to the dot pattern). In this case, the dot images DI on the image plane M of the camera CAM and the dots D before projection have similar shapes, so the dot centroid position G1 on the image plane M (the centroid position determined two-dimensionally from the shape of the dot images DI on the image plane M) coincides with the actual dot centroid position G0 projected onto the image plane M (the projected point obtained by projecting the centroid position g of the dots D before projection onto the image plane M), and the dot centroid position can be detected without error.

[0012] Fig. 29 shows a state in which the camera CAM is tilted toward the dot pattern (dots D). In this case, due to the influence of the non-telecentric optical system, the dot image DI on the image plane M has an asymmetric shape on one side (the left side in Fig. 29) and the other side (the right side in Fig. 29) of the actual dot centroid position G0 projected onto the image plane M (the projection point obtained by projecting the centroid position g of the dot D before projection onto the image plane M). As a result, the dot centroid position G1 on the image plane M does not coincide with the actual dot centroid position G0 projected onto the image plane M, resulting in a detection error in the dot centroid position.

[0013] If such an error occurs in detecting the dot centroid position, it will lead to a decrease in the accuracy of camera calibration.

[0014] The present invention has been made in view of the above circumstances, and has as its object to provide a camera calibration method and calibration device that can improve the accuracy of camera calibration.

[0015] In order to achieve the above object, the present invention comprises the following aspects.

[0016] The camera calibration method according to the first aspect includes an image acquisition step of acquiring a plurality of calibration images obtained by photographing with a camera a dot pattern in which a plurality of dots are periodically arranged; a pattern information setting step of setting pattern information including information regarding the arrangement of the dots; a feature point detection step of detecting feature points indicating the positions of the dots from the calibration images; and a parameter calculation step of calculating camera parameters based on the feature points and the pattern information, wherein the feature point detection step detects the positions of the centers of gravity of the dots as feature points based on a projection image in which the outlines of the dots on the image plane of the calibration image are projected onto a virtual plane set in three-dimensional space.

[0017] In the camera calibration method according to the second aspect, in the first aspect, the feature point detection step obtains a virtual plane optimized so that a circle fits onto the projected image, and detects the position of the center of gravity of the dot as a feature point based on the projected image projected onto the obtained virtual plane.

[0018] In the camera calibration method according to the third aspect, in the first aspect, the feature point detection step determines an optimized virtual plane such that, when an approximate circle is defined as a circle obtained by approximating a point cloud consisting of a plurality of projection points obtained by projecting a plurality of contour points indicating the contours of a dot on an image plane onto a virtual plane, an evaluation value indicating the fitting error of the approximated circle to the point cloud is minimized or is equal to or less than a threshold value, and the position of the point obtained by projecting the center of the approximated circle on the optimized virtual plane onto the image plane is defined as the position of the center of gravity of the dot.

[0019] In a camera calibration method according to a fourth aspect, in the third aspect, the feature point detection step includes a search step of searching for an optimized virtual plane from among a plurality of virtual planes with different orientations in three-dimensional space, based on an evaluation value indicating a fitting error of an approximation circle to a point cloud.

[0020] A camera calibration method according to a fifth aspect is the same as that of the fourth aspect, in which the search step changes the virtual plane into a plurality of postures by rotating the virtual plane around an axis parallel to the long axis direction of the dots on the image plane, while calculating an evaluation value for each posture, and determining an optimized virtual plane based on the evaluation value calculated for each posture.

[0021] A camera calibration method according to a sixth aspect is the fourth or fifth aspect, wherein the feature point detection step includes a contour point extraction step of extracting a plurality of contour points of a dot from an image plane, a ray group generation step of generating a group of half rays consisting of a plurality of half rays that start at the optical center of the camera and pass through each of the plurality of contour points of the dot on the image plane, and an approximation circle calculation step of calculating an approximation circle based on a group of intersections between the group of half rays and a virtual plane.

[0022] A camera calibration device according to a seventh aspect includes an image acquisition unit that acquires a plurality of calibration images obtained by capturing with a camera a dot pattern in which a plurality of dots are periodically arranged; a pattern information setting unit that sets pattern information including information regarding the arrangement of dots in the dot pattern; a feature point detection unit that detects feature points from the calibration images; and a parameter calculation unit that calculates camera parameters based on the feature points and the pattern information, wherein the feature point detection unit detects the positions of the centers of gravity of the dots as feature points based on a projected image in which the outlines of the dots on the image plane of the calibration image are projected onto a virtual plane set in three-dimensional space.

[0023] A camera calibration method according to an eighth aspect of the present invention includes an image acquisition step of acquiring a plurality of calibration images obtained by photographing a calibration pattern including a very small target in a defocused state; a pattern information setting step of setting pattern information related to the calibration; a feature point detection step of detecting, from the calibration images, the positions of the centers of gravity of the defocused images of the targets as feature points; and a parameter calculation step of calculating camera parameters based on the pattern information and the positions of the feature points.

[0024] A ninth aspect of the present invention relates to a camera calibration method according to the eighth aspect, wherein the size of the target is determined according to at least one of the focal length of a camera that photographs the target, the distance between the camera and the target, and the pixel size of the camera's imaging element.

[0025] A tenth aspect of the present invention relates to a camera calibration method according to the eighth aspect, wherein the target has a size such that the size of the image on the imaging element of the camera obtained when photographed in an in-focus state is one pixel or less. However, depending on the required tolerance, a target may be used whose size exceeds one pixel when photographed in an in-focus state on the imaging element of the camera.

[0026] An eleventh aspect of the present invention provides a camera calibration method according to any one of the eighth to tenth aspects, wherein the pattern information includes information relating to the arrangement of targets in the calibration pattern.

[0027] A twelfth aspect of the present invention provides a camera calibration method according to any one of the eighth to eleventh aspects, wherein the target has a shape that is line-symmetric or point-symmetric in two directions that are orthogonal to each other.

[0028] A thirteenth aspect of the present invention provides the camera calibration method of the twelfth aspect, wherein the targets are dot-shaped.

[0029] A fourteenth aspect of the present invention provides the camera calibration method of the twelfth aspect, wherein the target is a point light source.

[0030] A camera calibration method according to a fifteenth aspect of the present invention includes an image acquisition step of acquiring a calibration image by using a camera positioned optically conjugate to the light source to capture reflected light of illumination light irradiated from a light source onto a calibration object including a reflector having the property of reflecting the illumination light back in the incident direction; a pattern information setting step of setting pattern information related to the calibration object; a feature point detection step of detecting the focusing positions of the reflected light from the calibration image as feature points; and a parameter calculation step of calculating camera parameters based on the pattern information and the positions of the feature points.

[0031] A sixteenth aspect of the present invention provides a camera calibration method according to the fifteenth aspect, wherein in the image acquisition step, images of reflected light from at least two reflectors of the calibration object are captured by a camera.

[0032] A seventeenth aspect of the present invention relates to the camera calibration method of the fifteenth or sixteenth aspect, wherein the pattern information includes information relating to the arrangement and spacing of reflectors on the calibration object.

[0033] An eighteenth aspect of the present invention provides a camera calibration method according to any one of the fifteenth to seventeenth aspects, wherein the reflectors are spherical and arranged in an array on the calibration object.

[0034] A nineteenth aspect of the present invention provides a camera calibration method according to the eighteenth aspect, wherein the surface of the reflector is a mirror surface or a rough surface that provides specularly reflected light that can be distinguished from light that is diffusely reflected when the light is incident non-perpendicularly on the surface of the reflector.

[0035] A twentieth aspect of the present invention provides the camera calibration method of the eighteenth aspect, wherein the reflector is made of glass or sapphire.

[0036] According to the present invention, the accuracy of camera calibration can be improved.

[0037] 17 is a diagram illustrating a projection model of a camera. FIG. 18 is a block diagram illustrating an example of a schematic configuration of a calibration device. FIG. 19 is a flowchart illustrating the overall process of calibration processing performed by the calibration device of the first embodiment. FIG. 19 is a diagram illustrating an example of a calibration pattern (dot pattern) used in the first embodiment. FIG. 19 is an explanatory diagram illustrating the detection principle of feature point detection processing. FIG. 19 is a flowchart illustrating an example of the procedure for feature point detection processing. FIG. 19 is a graph illustrating the effect of the first embodiment. FIG. 19 is a diagram illustrating an example of a target photographed by a camera. FIG. 20 is a plan view illustrating an example of a calibration pattern. FIG. 21 is an image of a target photographed (Example and Comparative Example). FIG. 21 is a flowchart illustrating the overall process of calibration processing performed by the calibration device of the second embodiment. FIG. 22 is a graph illustrating the tilt resistance of the calibration method. FIG. 23 is a graph illustrating the luminance noise resistance of the calibration method (Example). FIG. 24 is a graph illustrating the luminance noise resistance of the calibration method (Comparative Example 1). FIG. 25 is a graph illustrating the luminance noise resistance of the calibration method (Comparative Example 2). 1 is a graph showing the resistance to tilt of a calibration method; FIG. 2 is a graph showing the resistance to luminance noise of a calibration method (Example); FIG. 3 is a graph showing the resistance to luminance noise of a calibration method (Comparative Example 1); FIG. 4 is a graph showing the resistance to luminance noise of a calibration method (Comparative Example 2); FIG. 5 is a diagram showing an example of a calibration pattern; FIG. 6 is a diagram showing an example of dots photographed by a camera; FIG. 7 is a diagram for explaining the influence of the tilt of a camera relative to a dot pattern on detection results; FIG. 8 is a diagram for explaining the influence of the tilt of a camera relative to a dot pattern on detection results.

[0038] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

[0039] [Camera Projection Model] First, the camera projection model will be described. Fig. 1 is an explanatory diagram for explaining the camera projection model. Note that the camera projection model shown in Fig. 1 is common to each embodiment described below.

[0040] As shown in FIG. 1, the world coordinate system is a coordinate system that represents positions in a three-dimensional space (real space), and has an origin O ｗ and X ｗ Axis, Y ｗ axis, Z ｗ The world coordinate system is a three-dimensional Cartesian coordinate system with the axes O and O being the coordinate axes. Note that any coordinate system may be used as the world coordinate system as long as it can identify a position in three-dimensional space (three-dimensional position). The camera coordinate system is ｃ is the origin, and the origin O ｃ From there, turn right and press X. ｃ axis, downward direction is Y ｃ axis, the optical axis direction is Z ｃ The image coordinate system is a three-dimensional Cartesian coordinate system with the origin O of the camera coordinate system. ｃ From Z ｃ The upper left corner of the image plane IP, which is a focal distance f away in the X direction, is set as the origin. ｃ axis and Y ｃ It is a two-dimensional orthogonal coordinate system (pixel coordinate system) having a U axis and a V axis in directions parallel to the axes.

[0041] First, the coordinates (x) of point P (object point) in the world coordinate system in the three-dimensional space are ｗ , y ｗ , z ｗ ) can be converted into coordinates (x, y, z) in the camera coordinate system using the camera rotation matrix R and translation vector t, as shown in the following equation (1).

[0042]

[0043] Here, [R|t] is a matrix (external parameter matrix) that transforms from the world coordinate system to the camera coordinate system, and represents the orientation and position of the camera in the world coordinate system. 11 , r 12 , ..., r 33 , t x, t y , t z are called the extrinsic parameters of the camera.

[0044] Next, if the coordinates (pixel coordinates) of a projection point Q obtained by projecting a point P located at (x, y, z) in the camera coordinate system onto an image plane IP are (u, v), the following relationship shown in equations (2) to (7) holds.

[0045]

[0046]

[0047]

[0048]

[0049]

[0050]

[0051] Here, (x', y') represent the coordinates of the projected point obtained by projecting point P, which is located at (x, y, z) when viewed from the camera coordinate system, onto the normalized image plane (z = 1). Also, (x'', y'') represent the coordinates of the projected point (distorted point) obtained by projecting point P onto the normalized image plane when taking into account the lens distortion of the camera.

[0052] Also, f ｘ , f ｙ denotes the focal length in the x and y directions expressed in pixels, and c ｘ , c ｙ denotes the optical center in the image coordinate system (the position where the optical axis of the camera intersects with the image plane IP, the optical center in pixel units). １ , k ２ , k ３ is the radial distortion coefficient, and p １ , p ２ is the tangential distortion coefficient. In this specification, the focal length f ｘ , f ｙ , optical center c ｘ , c ｙ are called the intrinsic parameters of the camera, and the distortion coefficient k １ , k ２ , k ３, p １ , p ２ are called the distortion parameters of the camera.

[0053] <First embodiment> [Calibration device] Fig. 2 is a block diagram showing an example of a schematic configuration of a calibration device 10 according to the first embodiment. As shown in Fig. 2, the calibration device 10 includes an arithmetic control unit 20 and a storage unit 22. In addition, a camera 12, an operation unit 14, and an output unit 16 are connected to the calibration device 10.

[0054] The camera 12 generates images IM (calibration images) by capturing images of the calibration pattern from different directions, and outputs the images to the calibration device 10. The camera 12 has a non-telecentric optical system (an optical system that is non-telecentric on the subject side), and captures images using an image sensor via a lens group that serves as the non-telecentric optical system.

[0055] The operation unit 14 includes operation members such as a keyboard and a mouse, and receives inputs of various operations from an operator.

[0056] The output unit 16 is a device for outputting the calculation results and the like obtained by the calculation control unit 20. The output unit 16 includes, for example, an operation UI (User Interface) and a monitor (for example, a liquid crystal display) for displaying the calculation results. The output unit 16 may also include a printer, a speaker, or the like in addition to or instead of the monitor.

[0057] The arithmetic control unit 20 controls the operation of the calibration device 10. The arithmetic control unit 20 is configured by an arithmetic device such as a personal computer, and includes an arithmetic circuit configured by various processors, memories, etc. The various processors include a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), and a programmable logic device (e.g., simple programmable logic device (SPLD), complex programmable logic device (CPLD), and field programmable gate array (FPGA)). Note that the various functions of the calibration device 10 may be realized by a single processor, or by multiple processors of the same or different types.

[0058] The storage unit 22 stores control programs and various data. The storage unit 22 is configured, for example, by a hard disk drive (HDD) or a semiconductor storage device (SSD: Solid State Drive). The storage unit 22 may include a temporary storage element configured, for example, by a random access memory (RAM) such as a dynamic random access memory (DRAM) or a static random access memory (SRAM), and may function as a work area for the arithmetic control unit 20.

[0059] The storage unit 22 temporarily stores a plurality of calibration images IM captured by the camera 12. The storage unit 22 also stores camera parameters CP calculated by the calculation and control unit 20. The camera parameters CP include internal parameters of the camera 12 (focal length, optical center) and distortion parameters (radial and tangential distortion coefficients).

[0060] The calculation control unit 20 reads and executes the control program stored in the storage unit 22, thereby functioning as an image acquisition unit 30, a pattern information setting unit 32, a feature point detection unit 34, and a parameter calculation unit 36. The functions of each unit constituting the calculation control unit 20 will be described later.

[0061] The following describes the procedure of the calibration process (an example of a calibration method) executed by the calibration device 10 of the first embodiment. Fig. 3 is a flowchart showing the overall process of the calibration process executed by the calibration device 10 of the first embodiment. It should be noted that at the start of the flowchart shown in Fig. 3, various initial setting processes such as checking the operation of each part of the calibration device 10 have been performed.

[0062] (Step S10: Image Acquisition Step) First, the image acquisition unit 30 acquires a plurality of calibration images IM captured by the camera 12 and stores them in the storage unit 22. In this embodiment, as the calibration pattern, a dot pattern DP (also called a "circle grid") in which a plurality of dots D (black circles) are arranged vertically and horizontally is used, as an example shown in Fig. 4. Note that the shape or arrangement of each dot D in the dot pattern DP is not limited to the example shown in Fig. 4, as long as the feature points are distributed in a shape or arrangement that is easy to detect using an existing search method.

[0063] The multiple calibration images IM are images of a dot pattern DP serving as a calibration pattern photographed from different directions by the camera 12. The number of calibration images IM is at least two, preferably five, and more preferably ten. Note that the photographing of the calibration pattern (dot pattern DP) by the camera 12 should be performed at least before the image acquisition step is performed.

[0064] The image acquisition unit 30 may acquire each calibration image IM from the camera 12 via a cable or a recording medium. Alternatively, each calibration image IM may be stored in an external storage device such as an external server installed outside the calibration device 10, and the image acquisition unit 30 may acquire each calibration image IM from the external storage device via a wired or wireless network.

[0065] (Step S12: Pattern Information Setting Step) Next, the pattern information setting unit 32 sets pattern information including information regarding the arrangement of dots D in the dot pattern DP. For example, the pattern information includes the number of dots in the vertical direction (column direction) and horizontal direction (row direction) of the dot pattern DP. In the example shown in FIG. 4, the number of dots in the vertical direction of the dot pattern DP is 5, and the number of dots in the horizontal direction is 8. In addition to the number of dots in the vertical and horizontal directions of the dot pattern DP, the pattern information also includes the total length S1, S2 of the dot pattern DP in the vertical and horizontal directions. Note that instead of the total length S1, S2 of the dot pattern DP in the vertical and horizontal directions, the pitch P1, P2 of the dot pattern DP in the vertical and horizontal directions may also be included. This makes it possible to identify the relative positional relationship of each dot D in the dot pattern DP. Note that the pattern information is information used by the feature point detection unit 34 (described later) to identify points (object points) in three-dimensional space corresponding to feature points (image points) in the image coordinate system.

[0066] The pattern information setting unit 32 may acquire, for example, pattern information input by an operator via the operation unit 14. Alternatively, the pattern information may be stored in advance in the storage unit 22, and the pattern information setting unit 32 may acquire the pattern information from the storage unit 22. Note that the pattern information setting step only needs to be performed before at least the parameter calculation step described below, and may be performed, for example, before the image acquisition step or after the feature point detection step.

[0067] (Step S14: Feature Point Detection Step) Next, the feature point detection unit 34 executes a feature point detection process to detect multiple feature points for each of the multiple calibration images IM captured by the camera 12. Specifically, the feature point detection unit 34 sequentially reads the multiple calibration images IM stored in the storage unit 22. Then, for each of the read calibration images IM, the feature point detection unit 34 performs predetermined image processing (such as grayscale conversion) on the calibration image IM, detects each feature point (image point) from the calibration image IM, and determines the coordinates (pixel coordinates) of each feature point in the image coordinate system. In this embodiment, the position of the center of gravity (dot center of gravity position) of each dot D (dot image DI) is detected as a feature point on the calibration image IM. The coordinates of the feature points detected by the feature point detection unit 34 are temporarily stored in the storage unit 22. Details of the feature point detection process executed by the feature point detection unit 34 will be described later.

[0068] If there is a calibration image IM in which feature points have not been detected successfully among the plurality of calibration images IM, the feature point detection unit 34 performs an exclusion process to exclude the calibration image IM in which feature points have not been detected successfully from the target of the parameter calculation process described below. As a result, in the parameter calculation process, the camera parameters CP can be calculated based on the calibration image IM after the exclusion process (i.e., the calibration image IM in which feature points have been successfully detected) among the plurality of calibration images IM.

[0069] (Step S16: Parameter Calculation Step) Next, the parameter calculation unit 36 ​​executes a calculation process of the camera parameters CP. Specifically, the parameter calculation unit 36 ​​calculates the camera parameters CP based on the positions (dot centroid positions) of the feature points on each calibration image IM detected by the feature point detection unit 34 and the pattern information set by the pattern information setting unit 32. The camera parameters CP can be calculated using a known method (e.g., Zhang's method). The Zhang's method is a method of optimizing parameters so that the positions of feature points (image points) on a captured image (calibration image IM) correspond correctly to the positions of points (known object points) in three-dimensional space corresponding to those feature points (Z. Zhang, "A flexible new technique for camera calibration," IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 22, No. 11, pp. 1330-1334, 2000).

[0070] The camera parameters CP calculated by the parameter calculation unit 36 ​​are the internal parameters of the camera 12 (focal length f ｘ , f ｙ , optical center c ｘ , c ｙ ) and distortion parameters (distortion coefficient k １ , k ２ , k ３ , p １ , p ２ The parameter calculation unit 36 ​​stores the calculated camera parameters CP in the storage unit 22.

[0071] (Step S18: Output Step) Next, the parameter calculation unit 36 ​​outputs the calculation results of the camera parameters CP to the output unit 16. This allows the operator to acquire the camera parameters CP, and therefore enables image correction and the like to be performed on the image captured by the camera 12 based on the camera parameters CP.

[0072] This completes the flowchart of the overall calibration process executed by the calibration device 10.

[0073] [Feature Point Detection Processing] Next, the feature point detection processing executed in the feature point detection step will be described in detail. The feature point detection processing in the first embodiment is processing for detecting the dot centroid positions as feature points on the calibration image IM. Note that the feature point detection processing is performed sequentially for each calibration image IM.

[0074] First, the detection principle of the feature point detection process will be described with reference to Fig. 5. Fig. 5 is an explanatory diagram for explaining the detection principle of the feature point detection process.

[0075] When an image is captured with the dot pattern DP tilted relative to the camera 12, the dots D on the dot pattern DP are distorted and projected onto the image plane IP of the camera 12 due to the influence of the non-telecentric optical system of the camera 12. Therefore, when an attempt is made to directly detect the positions of the dot centroids of the dot images DI in the calibration image IM in two dimensions, the detected positions of the dot centroids will be displaced from their original positions.

[0076] Therefore, in the first embodiment, the feature point detection process is performed three-dimensionally using a virtual plane VP as described below. That is, in this feature point detection process, assuming that the virtual plane VP is a virtual plane set in three-dimensional space as shown in Fig. 5, the virtual plane VP is optimized by a fitting method such as the least squares method so that the projected image obtained by projecting the contours of the dot images DI on the image plane IP of the calibration image IM onto the virtual plane VP approximates a circle.

[0077] Specifically, the origin O of the camera coordinate system ｃA group of ray lines B consisting of a plurality of ray lines each passing through a plurality of contour points of the dot image DI on the image plane IP is generated with the optical center of the camera 12 as the starting point, and a group of intersections T between the group of ray lines B and a virtual plane VP is obtained. Then, when the circle obtained by approximating the group of intersections T using the least squares method is taken as the approximate circle, an optimized virtual plane VP is obtained so as to reduce an evaluation value indicating the fitting error of the approximate circle to the group of intersections T.

[0078] Here, the closer the virtual plane VP is to being parallel to the dot pattern DP (not shown in FIG. 5 ) in three-dimensional space, the smaller the evaluation value indicating the fitting error of the approximation circle to the group of intersections T on the virtual plane VP becomes. Therefore, the virtual plane VP optimized so as to reduce the evaluation value indicating the fitting error of the approximation circle to the group of intersections T on the virtual plane VP can be considered to be substantially parallel to the dot pattern DP in three-dimensional space. Note that "substantially parallel" includes not only the case where the plane is strictly parallel, but also the case where the plane is approximately parallel.

[0079] When the center VC of the approximate circle on the thus optimized virtual plane VP is projected onto the image plane IP, the position of the projection point IC projected onto the image plane IP indicates the dot center of gravity position on the image plane IP where the distortion error due to projection is minimized.

[0080] In the first embodiment, a virtual plane VP optimized to minimize the evaluation value indicating the fitting error described above is searched for among a plurality of virtual planes VP with different orientations in three-dimensional space. The position of a projection point IC obtained by projecting the center VC of the approximation circle on the optimized virtual plane VP onto the image plane IP is determined as the dot centroid position. As described above, since the optimized virtual plane VP can be considered substantially parallel to the dot pattern DP in three-dimensional space, the coordinates (pixel coordinates) of the dot centroid position with the minimum distortion error due to projection can be obtained by projecting the center VC of the approximation circle on the optimized virtual plane VP onto the image plane IP and determining the projection point IC as the dot centroid position.

[0081] 6 and 7 are flowcharts showing an example of the procedure of the feature point detection process, which will be described below with reference to the flowcharts shown in FIGS.

[0082] (Step S20: Contour Point Extraction Step) First, the feature point detection unit 34 reads the calibration image IM, performs predetermined image processing (grayscale conversion processing, etc.) on the calibration image IM, and then extracts a plurality of contour points that indicate the contours of the dot images DI on the image plane IP of the calibration image IM. At this time, the number of contour points extracted is at least three. The contours (plurality of contour points) of the dot images DI are extracted using known techniques such as edge detection and binarization processing. As a result, the coordinates of a plurality of contour points (contour pixels) that indicate the contours of the dot images DI on the image plane IP are determined.

[0083] (Step S22: Half-line group generation step) Next, the feature point detection unit 34 detects the origin O of the camera coordinate system as shown in FIG. ｃ A group B of half-lines is generated, which is made up of a plurality of half-lines passing through each of the contour points found in the contour point extraction step, with the start point being .

[0084] (Step S24: Dot Long-Axis Direction Calculation Step) Next, the feature point detection unit 34 calculates the long-axis direction of the dot image DI on the image plane IP (hereinafter referred to as "dot long-axis direction").

[0085] (Step S26: Angle Initialization Step) Next, the feature point detection unit 34 initializes the angle θ of the virtual plane VP to 0°. Here, the angle θ of the virtual plane VP is an angle that indicates the rotation position when the virtual plane VP is rotated around an axis parallel to the dot major axis direction in a loop process described later. In this embodiment, when the virtual plane VP is at the initial rotation position (θ = 0°), the virtual plane VP is positioned in a position in three-dimensional space that is parallel to the image plane IP. However, the present invention is not limited to this, and the virtual plane VP may be positioned at any position.

[0086] (Step S28: Loop Start Step) The loop start step is the start point of a loop process (Loop 1) that repeats the processes from step S30 to step S56 (described later) N times. This loop process is an example of a search step of the present invention.

[0087] (Step S30: Positive Side Rotation Angle Calculation Step) Next, the feature point detection unit 34 calculates a preset step angle δ with respect to the angle θ of the virtual plane VP, as shown in the following equation (8). θ The value obtained by adding the above angle is calculated as the first angle θ1 (where δ θ >0. And so on.

[0088] (Step S32: Virtual Plane Rotation Step) Next, the feature point detection unit 34 performs a process of rotating the virtual plane VP around an axis parallel to the dot major axis direction so that the virtual plane VP is at a first angle θ1. Hereinafter, the virtual plane VP at the first angle θ1 will be referred to as a "first virtual plane VP1."

[0089] (Step S34: Intersection Group Calculation Step) Next, the feature point detection unit 34 calculates a first intersection group T1 between the half-ray group B and the first virtual plane VP1.

[0090] (Step S36: Circle Approximation Step) Next, the feature point detection unit 34 performs circle approximation processing using the least squares method or the like on the first intersection group T1 to calculate the position and shape (center position and radius) of an approximate circle on the first virtual plane VP1.

[0091] (Step S38: First Evaluation Value Calculation Step) Next, the feature point detection unit 34 calculates, as the first evaluation value ε1, an evaluation value indicating the fitting error of the approximation circle with respect to the first group of intersections T1 on the first virtual plane VP1. The evaluation value calculated as the first evaluation value ε1 may be, for example, the sum of squared residuals or the sum of absolute residuals. The sum of squared residuals is the sum of squared residuals, which are the differences between the radius of the approximation circle and the distance from the center position of the approximation circle to each point of the first group of intersections T1, and the sum of absolute residuals is the sum of the absolute values ​​of the residuals.

[0092] (Step S40: Minus Side Angle Calculation Step) Next, the feature point detection unit 34 calculates a predetermined step angle δ with respect to the angle θ of the virtual plane VP, as shown in the following equation (9). θ The value obtained by subtracting this is determined as the second angle θ2.

[0093] (Step S42: Virtual Plane Rotation Step) Next, the feature point detection unit 34 performs a process of rotating the virtual plane VP around an axis parallel to the dot major axis direction so that the virtual plane VP is at a second angle θ2. Hereinafter, the virtual plane VP at the second angle θ2 will be referred to as a "second virtual plane VP2."

[0094] (Step S44: Intersection Group Calculation Step) Next, the feature point detection unit 34 calculates a second intersection group T2 between the half-line group B and the second virtual plane VP2.

[0095] (Step S46: Circle Approximation Step) Next, the feature point detection unit 34 performs circle approximation processing using the least squares method or the like on the second intersection group T2 to calculate the position and shape (center position and radius) of an approximate circle on the second virtual plane VP2.

[0096] (Step S48: Second evaluation value calculation step) Next, the feature point detection unit 34 calculates, as the second evaluation value ε2, an evaluation value that indicates the fitting error of the approximation circle with respect to the second group of intersections T2 on the second virtual plane VP2. The evaluation value calculated as the second evaluation value ε2 is the same as the first evaluation value ε1 described above, and the residual sum of squares or the residual sum of absolute values ​​is applied.

[0097] (Step S50: Evaluation Value Comparison Step) Next, the feature point detection unit 34 compares the magnitude of the first evaluation value ε1 and the second evaluation value ε2. If the first evaluation value ε1 is smaller than the second evaluation value ε2 (Yes), the process proceeds to step S52, and if the first evaluation value ε1 is greater than or equal to the second evaluation value ε2 (No), the process proceeds to step S54. Note that if the first evaluation value ε1 and the second evaluation value ε2 are equal, the process may proceed to step S52 instead of step S54.

[0098] (Steps S52 and S54: Angle update step) Next, the feature point detection unit 34 updates the angle θ of the virtual plane VP to the first angle θ1 if the first evaluation value ε1 is smaller than the second evaluation value ε2 (step S52), and updates the angle θ of the virtual plane VP to the second angle θ2 if the first evaluation value ε1 is greater than or equal to the second evaluation value ε2 (step S54).

[0099] (Step S56: Step Angle Update Step) Next, the feature point detection unit 34 updates the step angle δ θ δ θ ×α, where the coefficient α is a real number in the range of 0<α≦1. For example, the coefficient α is set empirically based on the required calculation accuracy, etc. As a result, when the coefficient α is set to less than 1, the step angle δ is updated every time the number of loop processes increases. θ becomes smaller gradually (in stages), it is possible to accurately obtain an optimized virtual plane VP so that the evaluation value indicating the fitting error of the circle to the group of intersections T becomes smaller. Furthermore, the step angle update step may be performed only when the smaller evaluation value of the first evaluation value ε1 and the second evaluation value ε2 is smaller than a preset threshold value (evaluation value threshold value). In this way, as the evaluation value becomes smaller, the step angle δ θ It is possible to gradually decrease the value of the vector to perform the calculations, thereby improving the processing speed of the feature point detection process.

[0100] (Step S58: Loop End Step) The loop end step is the end point of the above-mentioned loop process (Loop 1). In this embodiment, the loop process ends when the processes from step S30 to step S56 have been repeated N times. The conditions for ending the loop process are that the number of times the loop process has been performed reaches an upper limit N (where N is an integer of 2 or more) (first end condition) and that the updated step angle δ θThe loop processing may be terminated when either one of the following conditions is satisfied: when the first evaluation value ε1 or the second evaluation value ε2 becomes smaller than a predetermined threshold value (angle threshold value) (second termination condition); or when the smaller of the first evaluation value ε1 and the second evaluation value ε2 becomes smaller than a threshold value (evaluation value threshold value) (third termination condition).

[0101] By performing the above loop processing, it is possible to search for a virtual plane VP that is optimized to an angle θ such that the evaluation value indicating the fitting error of the approximation circle to the group of intersections T on the virtual plane VP is minimum or below a threshold value, from among multiple virtual planes VP with different orientations in three-dimensional space.

[0102] (Step S60: Intersection Group Calculation Step) Next, the feature point detection unit 34 calculates the intersection group T between the virtual plane VP optimized to the angle θ by the above-described loop processing and the group of half lines B.

[0103] (Step S62: Circle approximation processing step) Next, the feature point detection unit 34 performs circle approximation processing using the least squares method or the like on the intersection group T obtained in the intersection group calculation step (step S60), and calculates the position and shape (center position and radius) of the approximate circle on the virtual plane VP.

[0104] (Step S64: Half-line Generation Step) Next, the feature point detection unit 34 detects the origin O of the camera coordinate system. ｃ A half line CL passing through the center VC of the approximate circle obtained in the circle approximation processing step (step S62) is generated with the start point at .

[0105] (Step S66: Image Plane Intersection Calculation Step) Next, the feature point detection unit 34 calculates the pixel position (pixel coordinates) of the intersection between the half line CL and the image plane IP. This intersection is the projection point IC obtained by projecting the center VC of the approximation circle on the virtual plane VP onto the image plane IP.

[0106] (Step S68: Feature point determination step) Next, the feature point detection unit 34 determines the position (dot center of gravity position) of the feature point of the calibration image IM to be the position (pixel coordinates) of the intersection (i.e., the projection point IC) on the image plane IP obtained in step S66.

[0107] This completes the flowchart of the dot detection process.

[0108] [Advantages of First Embodiment] Next, advantages of the first embodiment will be described.

[0109] Fig. 8 is a graph for explaining the effects of the first embodiment. In Fig. 8, the horizontal axis represents the tilt angle (unit: degrees (°)) of the dot pattern DP relative to the camera 12, and the vertical axis represents the detection error (unit: pixels) of the dot centroid position. The graph shown by the solid line in Fig. 8 represents the detection error when the dot centroid position is detected by the feature point detection process in the first embodiment. Furthermore, the graph shown by the dashed line in Fig. 8 represents, as a comparative example, the detection error when the dot centroid position is directly detected two-dimensionally from the calibration image.

[0110] As can be seen from FIG. 8, in the comparative example, the larger the inclination angle (absolute value) of the dot pattern DP relative to the camera 12, the larger the detection error (absolute value) of the dot centroid position.

[0111] In contrast, in the feature point detection process in the first embodiment, an optimized virtual plane VP is obtained by fitting a circle to a projected image obtained by projecting the contours of the dot images DI onto the image plane IP. Then, the position of the projection point obtained by projecting the center of the circle (approximate circle) on the optimized virtual plane VP onto the image plane IP is set as the dot centroid position, so it is possible to obtain the coordinates (pixel coordinates) of the dot centroid position with the minimum distortion error due to projection.

[0112] Therefore, in the first embodiment, the detection error of the dot centroid position is smaller overall than in the comparative example. In particular, in the comparative example, the detection error (absolute value) of the dot centroid position tends to increase as the tilt angle (absolute value) of the camera 12 with respect to the dot pattern DP increases, but in the first embodiment, even if the tilt angle (absolute value) increases, it is possible to keep the detection error (absolute value) of the dot centroid position within a certain range (in the example shown in Figure 8, the detection error (absolute value) is within 0.1 pixels).

[0113] Therefore, according to the first embodiment, the dot centroid positions can be detected without being affected by a non-telecentric optical system, and therefore the tilt tolerance is excellent, and the positions of the feature points (dot centroid positions) on the calibration image IM can be determined with high accuracy, thereby improving the accuracy of the calibration of the camera 12.

[0114] Furthermore, according to the first embodiment, the tilt resistance is excellent, so there is a high degree of freedom in the attitude and position of the camera 12 relative to the dot pattern DP, which is very convenient for the user.

[0115] Furthermore, according to the first embodiment, when searching for an optimized virtual plane VP in three-dimensional space, the search is performed by rotating the image plane IP of the calibration image IM around an axis parallel to the direction perpendicular to the direction in which the dot image DI is likely to distort (the dot major axis direction). This allows for efficient searching for the optimized virtual plane VP by rotating the virtual plane VP around one axis. The search for the optimized virtual plane VP is not limited to one axis, and the virtual plane VP may be rotated around any two or three axes.

[0116] Second Embodiment Next, a second embodiment will be described. Note that the configuration of the calibration device 10 is the same as that of the first embodiment (see FIG. 2), and therefore a description thereof will be omitted.

[0117] In the second embodiment, calibration processing of the camera 12 is performed based on a plurality of calibration images IM obtained by photographing a very small target in a defocused state with the camera 12. Fig. 9 is a diagram showing an example of photographing a calibration pattern CA with the camera 12, and Fig. 10 is a plan view of the calibration pattern CA.

[0118] As shown in FIG. 9 , the camera 12 includes a lens (objective lens) 120 and an image sensor (e.g., including a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor)) 122. For simplicity, the example shown in FIG. 9 illustrates only two targets T1 and T2 as representatives of the multiple targets T on the calibration pattern CA. Reference symbols PX1 and PX2 in FIG. 9 denote the optical axes of light traveling from the targets T1 and T2 toward the image sensor 122, respectively. In the example shown in FIG. 9 , the optical axis PX1 coincides with the optical axis AX of the lens 120. Furthermore, in FIG. 9 , the in-focus position at which the images of the targets T1 and T2 are focused via the lens 120 is designated F1. Among the defocus positions at which the images of the targets T1 and T2 are out of focus, the defocus position further behind the in-focus position F1 (on the opposite side from the lens 120) is designated F2, and the defocus position further in front of the in-focus position F1 (toward the lens 120) is designated F3. 9 shows an example in which the image sensor 122 is positioned at a defocus position F2. The symbols f′ and f in the figure indicate the front and rear focal positions of the lens 120, respectively.

[0119] The targets T on the calibration pattern CA are tiny dots (circle markers). The shape of these dots in a planar view is approximately circular. Here, the size (diameter) of the targets T is determined by the relationship with at least one of, for example, the focal length of the camera 12 (the focal length of the lens 120), the distance between the camera 12 and the targets T (the distance along the optical axis between the lens 120 and the targets T), the pixel size of the image sensor 122 of the camera 12, and the calibration tolerance (pixel error). Note that the targets T are not limited to tiny dots, and may also be tiny point light sources.

[0120] It is preferable that the size (diameter) of the target T on the calibration pattern CA is such that when the target T is photographed by the image sensor 122 at the in-focus position F1, the size of the area occupied by the image of the target T on the image sensor 122 is one pixel or less. For example, when the focal length (rear focal length) of the lens 120 is 8 mm, the center-to-center distance between the camera 12 and the target T is 300 mm, and the pixel size of the image sensor 122 is 2.74 μm, it is preferable that the diameter φ of the target T is 102.75 μm or less. However, depending on the tolerance for calibration, the size of the target T may be larger than the above example.

[0121] When the lens 120 is an appropriately designed and manufactured lens (specifically, when the optical axis AX of the lens 120 and the center of the aperture stop AP of the lens 120 are aligned), when the target T is photographed, the height of the incident light at the position of the lens 120 is Δ1 = Δ2. In this case, the image height at the defocus position F2 (the dimension of the defocused image of the target T in the YC direction) is also δ1 = δ2. Although not shown in the figure, the same is true at the defocus position F3.

[0122] 1011A in Fig. 11 shows an image (example) of the target T captured by the image sensor 122 positioned at the defocus position F2. In 1011A in Fig. 11, as described in Fig. 9, the center line of the image of the target T overlaps and substantially coincides with the optical axis PX of the light from the target T. The same is true when the target T is captured by the image sensor 122 positioned at the defocus position F3.

[0123] 1011B in Fig. 11 shows an image (comparison example) of a dot (a non-minimal target) whose image captured in an in-focus state is about the same size as 1011A in Fig. 11. In this case, as shown in the enlarged view shown in 1011C in Fig. 11, the center line L1 of the dot image is shifted from the line L2 passing through the center of gravity of the dot image.

[0124] In contrast, when an extremely small target T is photographed in a defocused state, the center line of the image of the target T coincides with the optical axis (AX or AX1), so the center (feature point) of the target T can be detected with high accuracy.

[0125] In the second embodiment, the target T on the calibration pattern CA is minimized, so that the position of the feature point (the position of the center of gravity of the target T) can be detected with high accuracy without being affected by a non-telecentric optical system, even if the calibration pattern CA is photographed at an angle with respect to the camera 12. On the other hand, minimizing the target T makes it more susceptible to the influence of luminance noise, but in the second embodiment, the target T is photographed in a defocused state, so that the position of the feature point can be detected with the influence of luminance noise effectively suppressed.

[0126] In the second embodiment, the very small target T has a substantially circular shape (dot shape), but is not limited to this. It may have any shape as long as the center of gravity of the target T coincides with the center position in the defocused image, and may have, for example, a line-symmetric shape or a point-symmetric shape in two directions orthogonal to each other, such as an ellipse.

[0127] The following describes the procedure of the calibration process (an example of a calibration method) executed by the calibration device 10 of the second embodiment. Fig. 12 is a flowchart showing the overall process of the calibration process executed by the calibration device 10 of the second embodiment. It should be noted that at the start of the flowchart shown in Fig. 12, various initial setting processes such as checking the operation of each part of the calibration device 10 have been performed.

[0128] (Step S110: Image Acquisition Step) First, the image acquisition unit 30 acquires a plurality of calibration images IM captured by the camera 12 and stores them in the storage unit 22. In the second embodiment, a calibration pattern in which extremely small targets T (dots or point light sources) that satisfy the above size conditions are arranged vertically and horizontally is used as the calibration pattern CA.

[0129] The image acquisition unit 30 may acquire each calibration image IM from the camera 12 via a cable or a recording medium. Alternatively, each calibration image IM may be stored in an external storage device such as an external server installed outside the calibration device 10, and the image acquisition unit 30 may acquire each calibration image IM from the external storage device via a wired or wireless network.

[0130] The multiple calibration images IM are images of the calibration pattern captured from different directions by the camera 12. The number of calibration images IM is at least two, preferably five, and more preferably ten. Note that it is sufficient that the calibration pattern is captured by the camera 12 at least before the image acquisition step is performed.

[0131] (Step S112: Pattern Information Setting Step) Next, the pattern information setting unit 32 sets pattern information including information regarding the arrangement of the targets T on the calibration pattern CA. For example, the pattern information includes the number of targets in the vertical direction (column direction) and horizontal direction (row direction) of the calibration pattern CA. In the example shown in FIG. 10 , the number of targets in the vertical direction of the calibration pattern CA is 5, and the number of targets in the horizontal direction of the calibration pattern CA is 8. In addition to the number of targets in the vertical and horizontal directions of the calibration pattern CA, the pattern information also includes the total lengths S1, S2 of the calibration pattern CA in the vertical and horizontal directions (see FIG. 10 ). Note that instead of the total lengths S1, S2 of the calibration pattern CA in the vertical and horizontal directions, pitches (intervals) P1, P2 of the calibration pattern CA in the vertical and horizontal directions may be included. This makes it possible to identify the relative positional relationship of the targets T on the calibration pattern CA. Note that the pattern information is information used by the feature point detection unit 34, which will be described later, to identify points (object points) in three-dimensional space corresponding to feature points (image points) in the image coordinate system.

[0132] The pattern information setting unit 32 may acquire, for example, pattern information input by an operator via the operation unit 14. Alternatively, the pattern information may be stored in advance in the storage unit 22, and the pattern information setting unit 32 may acquire the pattern information from the storage unit 22. Note that the pattern information setting step only needs to be performed before at least the parameter calculation step described below, and may be performed, for example, before the image acquisition step or after the feature point detection step.

[0133] (Step S114: Feature Point Detection Step) Next, the feature point detection unit 34 executes a feature point detection process to detect multiple feature points for each of the multiple calibration images IM captured by the camera 12. Specifically, the feature point detection unit 34 sequentially reads the multiple calibration images IM stored in the storage unit 22. Then, for each of the read calibration images IM, the feature point detection unit 34 performs predetermined image processing (e.g., binarization processing or grayscale conversion) on the calibration image IM, and then detects each feature point (image point) from the calibration image IM and calculates the coordinates (pixel coordinates) of each feature point in the image coordinate system. In this embodiment, the feature point is detected as the center of gravity of the image of the target T from the calibration image IM captured by the camera 12 in a defocused state. The coordinates of the feature points detected by the feature point detection unit 34 are temporarily stored in the storage unit 22.

[0134] If there is a calibration image IM in which feature points have not been detected successfully among the plurality of calibration images IM, the feature point detection unit 34 performs an exclusion process to exclude the calibration image IM in which feature points have not been detected successfully from the target of the parameter calculation process described below. As a result, in the parameter calculation process, the camera parameters CP can be calculated based on the calibration image IM after the exclusion process (i.e., the calibration image IM in which feature points have been successfully detected) among the plurality of calibration images IM.

[0135] (Step S116: Parameter Calculation Step) Next, the parameter calculation unit 36 ​​executes a calculation process of the camera parameters CP. Specifically, the parameter calculation unit 36 ​​calculates the camera parameters CP based on the positions of the feature points on each calibration pattern image IM detected by the feature point detection unit 34 and the pattern information set by the pattern information setting unit 32. As in the first embodiment, the camera parameters CP can be calculated using a known method (for example, the Zhang method).

[0136] The camera parameters CP calculated by the parameter calculation unit 36 ​​are the internal parameters of the camera 12 (focal length f ｘ , f ｙ , optical center c ｘ , c ｙ ) and distortion parameters (distortion coefficient k １ , k ２ , k ３ , p １ , p ２ The parameter calculation unit 36 ​​stores the calculated camera parameters CP in the storage unit 22.

[0137] (Step S118: Output Step) Next, the parameter calculation unit 36 ​​outputs the calculation results of the camera parameters CP to the output unit 16. This allows the operator to acquire the camera parameters CP, and therefore enables image correction and the like to be performed on the image captured by the camera 12 based on the camera parameters CP.

[0138] This completes the flowchart of the overall calibration process executed by the calibration device 10.

[0139] Example of the Second Embodiment Fig. 13 is a graph showing the tilt tolerance of the calibration method, where the horizontal axis represents the tilt angle (degrees) of the camera relative to the calibration pattern, and the vertical axis represents pixel error (pixels).

[0140] The example (defocused) of the second embodiment shows an example in which the target according to the second embodiment is photographed in a defocused state and calibration is performed. Comparative Example 1 shows an example in which calibration is performed using the dot pattern exemplified by 1025C in Fig. 25, and Comparative Example 2 shows an example in which calibration is performed using the checker pattern exemplified by 1025B in Fig. 25.

[0141] In the case of Comparative Example 1, as shown in Figure 27, objects closer to the camera are photographed larger, so the pixel error (absolute value) increases as the tilt angle (absolute value) relative to the dot pattern increases, resulting in poor tilt tolerance.

[0142] In the example, the pixel error (absolute value) is small regardless of the tilt angle (absolute value) of the camera 12 relative to the target. Also in the comparative example 2, the pixel error (absolute value) is small regardless of the tilt angle (absolute value) of the camera relative to the checkered pattern.

[0143] 14 to 16 are graphs showing the luminance noise resistance of the calibration method, and correspond to the example (defocus), comparative example 1 (dot pattern), and comparative example 2 (checkered pattern), respectively. The horizontal axis of each of Figs. 14 to 16 indicates the sample number, and the vertical axis indicates the pixel error (pixel).

[0144] 14 to 16, the luminance noise added when capturing the calibration patterns (CA in FIG. 10, 1025C and 1025B in FIG. 25) increases in the order of Noise=0.0, 4.0, and 8.0.

[0145] In Comparative Example 2 ( FIG. 16 ), the luminance gradient (differential value) is used when detecting the edges of the checkered pattern, and therefore the pixel error increases significantly depending on the luminance noise. This shows that, among the examples shown in FIGS. 14 to 16 , this example is the most sensitive to luminance noise.

[0146] On the other hand, the dot pattern according to Comparative Example 1 (FIG. 15) has higher resistance to luminance noise than the checkered pattern.

[0147] Furthermore, in the example (FIG. 14), the luminance noise resistance is higher than in either the first or second comparative example.

[0148] The above results are summarized in the table below. The examples are excellent in both tilt resistance and luminance noise resistance.

[0149]

[0150] Effect of Second Embodiment According to the second embodiment, the calibration process is performed based on a plurality of calibration images IM obtained by capturing a calibration pattern CA including a very small target T with the camera 12 in a defocused state. Therefore, even if the calibration pattern CA is captured at an angle with respect to the camera 12, the positions of the feature points (the positions of the center of gravity of the target T) can be detected with high accuracy without being affected by a non-telecentric optical system. Furthermore, by capturing the target T in a defocused state, the positions of the feature points can be detected while effectively suppressing the influence of luminance noise associated with the very small size of the target T. Therefore, it is possible to accurately detect the feature points of the calibration images IM, and the accuracy of the calibration of the camera 12 can be improved.

[0151] Third Embodiment In addition to the above-described problems of the present invention, there is another point to consider. That is, when a planar pattern such as a dot pattern, a grid pattern, or a checkered pattern is used as a calibration object, it may not be possible to accurately grasp position errors and the like of feature points on each pattern. This makes it difficult to perform calibration on the calibration object, which can lead to a decrease in the calibration accuracy of the camera.

[0152] Therefore, an object of the third embodiment is to provide a camera calibration method that can improve the accuracy of camera calibration.

[0153] The third embodiment will be described below. Note that the calibration device 10 is similar to that of the first embodiment (see FIG. 2 ), and therefore description thereof will be omitted. In the third embodiment, a calibration object MR (see FIGS. 17 and 18 ), which will be described later, is used instead of the calibration pattern described above.

[0154] FIG. 17 is a perspective view showing the calibration object MR, and FIG. 18 is a cross-sectional view taken along line AA-AA in FIG.

[0155] The calibration object MR is a spherical target in which a plurality of balls B are arranged in a housing F. The intervals and diameters of the balls B are known. Information about the calibration object MR (e.g., information about the type of calibration object MR, and the arrangement, intervals, and diameters of the balls B; hereinafter, referred to as pattern information) is stored in the memory unit 22 of the calibration device 10.

[0156] The housing F is a flat (plate-shaped) member made of a material that is highly rigid and stable over time (e.g., metal or ceramics). The housing F has a plurality of recesses H formed in an array.

[0157] The ball B is, for example, a spherical member formed from a material (e.g., metal) that has high rigidity and stability over time. The ball B is fitted into the recess H and fixed to the housing F. The ball B is an example of a reflector, and has the ability to reflect (specularly reflect) illumination light that is incident perpendicularly to its surface back in the direction of incidence. The ball B has high sphericity (see, for example, Japanese Industrial Standard JIS B 1501: 2009), and its surface is mirror-finished. Here, the surface roughness (arithmetic mean roughness) Ra (Japanese Industrial Standard JIS B0601: 2001) of the mirror surface is, for example, 0.2 μm or less.

[0158] The surface of ball B may be rough rather than mirror-finished, as long as it can detect reflected light, as described below. Here, a rough surface is a surface that is rougher than a mirror-finish (for example, Ra > 0.2 μm) but satisfies at least one of the following conditions: (A) A surface that can reflect specularly reflected light (L21 and L22) with an intensity equal to or greater than a predetermined value; (B) A surface that can obtain specularly reflected light (L21 and L22) that is distinguishable from (e.g., stronger than) diffusely reflected light that is incident non-perpendicularly on the surface of ball B from light source 50 and diffusely reflected (a surface where the difference between the intensity of reflected light of illumination light that is incident perpendicularly on the surface of ball B and the intensity of diffusely reflected light that is diffusely reflected around the perpendicular incidence position is equal to or greater than a threshold value).

[0159] The ball B may also be made of glass or sapphire and have retroreflective properties (see Japanese Industrial Standards JIS Z8713: 1995).

[0160] 17 and 18, the balls B are arranged at equal intervals, but this is not limiting. For example, the intervals d1 and d2 between the balls B only need to be known, and the intervals d1 and d2 between the balls B may be different or may be unequal. The diameters of the balls B may also be different from one another. The calibrator object MR may be, for example, a one-dimensional calibrator (ball-bar calibrator) in which a plurality of balls B are arranged in a linear housing (bar), or may be an L-shaped housing (bar) in which a plurality of balls B are arranged.

[0161] With the calibration object MR configured as described above, it is possible to measure the position (three-dimensional coordinates) of each ball B arranged in the housing F using a high-precision measuring device (traceable three-dimensional coordinate measuring device). This makes it possible to perform traceable calibration on the calibration object MR that is actually used. This makes it possible to perform highly accurate calibration of the camera 12 based on high-precision pattern information (including information on the arrangement, spacing, and diameters of the balls B).

[0162] In the third embodiment, the ball B is used as an example of a reflector, but the present invention is not limited to this. For example, a calibration object having a retroreflector (retroreflector, see Japanese Industrial Standard JIS Z8713: 1995, for example, a cube corner) may be used. In the case of a retroreflector, the range of tilt angles at which retroreflector performance can be obtained for incident light may be narrower than that of the ball B, but a retroreflector can be used when an imaging system that can tolerate the tilt angle conditions is used.

[0163] For the sake of simplicity, the following description will be given using an example in which the balls B are arranged at equal intervals along two mutually perpendicular directions, and the distances d1 and d2 between adjacent balls B along these two directions are equal.

[0164] 19 is a diagram for explaining an example of calibration using a calibrator object MR. In the example shown in FIG. 19, for the sake of simplicity, the light source 50 is placed on the perpendicular bisector of the line segment connecting the centers of the balls B1 and B2, but this is not limiting.

[0165] 19 , when photographing the calibration object MR, illumination light is irradiated onto the calibration object MR coaxially with the lens 124 of the camera 12 (coaxial illumination). The imaging system of the calibration object MR includes a light source 50, a half mirror 52, and the camera 12. The camera 12 includes a lens (objective lens) 124 and an imaging element (e.g., a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor)) 126. The calculation and control unit 20 (see FIG. 2 ) of the calibration device 10 controls the light emission of the light source 50 and the imaging of the camera 12.

[0166] The light source 50 is a device that irradiates the calibration object MR with visible light, and includes, for example, a light-emitting diode, etc. The light source 50 is disposed at a position optically conjugate with the camera origin (object-side principal point) Oc via a half mirror 52. Here, the camera origin Oc is located, for example, at the center (optical axis center) of the lens 124 of the camera 12.

[0167] Components L11 and L12 of the illumination light from the light source 50 that are incident perpendicularly on the surfaces of the balls B1 and B2 are reflected (retroreflected) along their respective incident light paths toward the light source 50. Then, reflected light L21 and L22 reflected from the surfaces of the balls B1 and B2 are reflected by the half mirror 52, pass through the lens 124, and form images on the image sensor 126. As a result, images of the reflected light L21 and L22 retroreflected by the balls B1 and B2 are obtained.

[0168] 19 indicate positions (light-collecting positions) where the reflected light L21 and L22 from the balls B1 and B2, respectively, are focused on the imaging element 126, and symbol P0 indicates a conjugate image of the light source 50.

[0169] The image acquisition unit 30 acquires a calibration image IM including images P1 and P2 of the reflected light L21 and L22 captured by the image sensor 126.

[0170] The pattern information setting unit 32 acquires pattern information relating to the calibration object MR from the storage unit 22 .

[0171] The feature point detection unit 34 detects the images P1 and P2 (bright field observation images) of the reflected light L21 and L22, and detects the positions of the centers of gravity of the images P1 and P2.

[0172] The parameter calculation unit 36 ​​calculates the camera parameters CP based on the positions of the feature points (the centers of gravity of the images P1 and P2 of the reflected light L21 and L22) detected by the feature point detection unit 34 and the pattern information set by the pattern information setting unit 32 (for example, information regarding the arrangement and spacing of the balls B1 and B2 on the calibration object MR).

[0173] As described above, in the third embodiment, the light source 50 is installed at a position conjugate with the camera origin Oc (object-side principal point) via the half mirror 52. Therefore, the imaging center (center of gravity) of the light source 50 reflected by the calibration object MR is on the line connecting the viewpoint (camera origin Oc) and the center of the calibration object MR.

[0174] In the third embodiment, reflected light L21 and L22 reflected perpendicularly to the surfaces of balls B1 and B2 are observed. Therefore, according to the third embodiment, distortion and pixel error due to the tilt of the pattern do not occur, and the accuracy of the detected pixel positions of the calibration object MR is not affected by the posture of the calibration object MR. This allows the calibration of the camera 12 to be performed with high accuracy. In addition, a traceable calibration object can be provided.

[0175] In the example shown in Figure 19, reflected light L21 and L22 from two balls B1 and B2 were photographed, but the number of balls used for calibration is not limited to two, and three or more balls may be used.

[0176] 19, the illumination light transmitted through the half mirror 52 reaches the balls B1 and B2, and the reflected light reflected by the half mirror 52 reaches the camera 12. However, the arrangement of the optical system in the imaging system is not limited to this. For example, the illumination light reflected by the half mirror 52 may reach the balls B1 and B2, and the reflected light transmitted through the half mirror 52 may reach the camera 12.

[0177] The following describes the procedure of the calibration process (an example of a calibration method) executed by the calibration device 10 of the third embodiment. Fig. 20 is a flowchart showing the overall process of the calibration process executed by the calibration device 10 of the third embodiment. It should be noted that at the start of the flowchart shown in Fig. 20, various initial setting processes such as checking the operation of each part of the calibration device 10 have been performed.

[0178] (Step S210: Image Acquisition Step) First, the image acquisition unit 30 acquires a plurality of calibration images IM captured by the camera 12 and stores them in the storage unit 22.

[0179] The image acquisition unit 30 may acquire each calibration image IM from the camera 12 via a cable or a recording medium. Alternatively, each calibration image IM may be stored in an external storage device such as an external server installed outside the calibration device 10, and the image acquisition unit 30 may acquire each calibration image IM from the external storage device via a wired or wireless network.

[0180] (Step S212: Pattern Information Setting Step) Next, the pattern information setting unit 32 sets pattern information related to the calibration object MR. The pattern information includes, for example, information related to the arrangement and spacing of the balls B1 and B2 on the calibration object MR. The pattern information is information used to define points (object points) in three-dimensional space that correspond to feature points (image points) in the image coordinate system.

[0181] The pattern information setting unit 32 may acquire, for example, pattern information input by an operator via the operation unit 14. Alternatively, the pattern information may be stored in advance in the storage unit 22, and the pattern information setting unit 32 may acquire the pattern information from the storage unit 22. Note that the pattern information setting step only needs to be performed before at least the parameter calculation step described below, and may be performed, for example, before the image acquisition step or after the feature point detection step.

[0182] (Step S214: Feature Point Detection Step) Next, the feature point detection unit 34 executes a feature point detection process to detect multiple feature points for each of the multiple calibration images IM captured by the camera 12. Specifically, the feature point detection unit 34 sequentially loads the multiple calibration images IM stored in the storage unit 22. Then, for each loaded calibration image IM, the feature point detection unit 34 performs predetermined image processing (e.g., binarization or grayscale conversion) on the calibration image IM, detects each feature point (image point) from the calibration image IM, and calculates the coordinates (pixel coordinates) of each feature point in the image coordinate system. In this embodiment, for example, images P1 and P2 of reflected light L21 and L22 are detected from the bright-field observation images of balls B1 and B2 shown in FIG. 19 , and the center of gravity positions of each image P1 and P2 are detected as feature points. The coordinates of the feature points detected by the feature point detection unit 34 are temporarily stored in the storage unit 22.

[0183] If there is a calibration image IM in which feature points have not been detected successfully among the plurality of calibration images IM, the feature point detection unit 34 performs an exclusion process to exclude the calibration image IM in which feature points have not been detected successfully from the target of the parameter calculation process described below. As a result, in the parameter calculation process, the camera parameters CP can be calculated based on the calibration image IM after the exclusion process (i.e., the calibration image IM in which feature points have been successfully detected) among the plurality of calibration images IM.

[0184] (Step S216: Parameter Calculation Step) Next, the parameter calculation unit 36 ​​executes a calculation process of the camera parameters CP. Specifically, the parameter calculation unit 36 ​​calculates the camera parameters CP based on the positions of the feature points on each calibration image IM detected by the feature point detection unit 34 and the pattern information set by the pattern information setting unit 32. As in the first embodiment, the camera parameters CP can be calculated using a known method (for example, the Zhang method).

[0185] The camera parameters CP calculated by the parameter calculation unit 36 ​​are the internal parameters of the camera 12 (focal length f ｘ , f ｙ , optical center c ｘ , c ｙ ) and distortion parameters (distortion coefficient k １ , k ２ , k ３ , p １ , p ２ The parameter calculation unit 36 ​​stores the calculated camera parameters CP in the storage unit 22.

[0186] (Step S218: Output Step) Next, the parameter calculation unit 36 ​​outputs the calculation results of the camera parameters CP to the output unit 16. This allows the operator to acquire the camera parameters CP, and therefore enables image correction and the like to be performed on the image captured by the camera 12 based on the camera parameters CP.

[0187] This completes the flowchart of the overall calibration process executed by the calibration device 10.

[0188] [Example of the Third Embodiment] Fig. 21 is a graph showing the tilt tolerance of the calibration method, where the horizontal axis represents the tilt angle (degrees) of the camera relative to the calibration object MR or the calibration pattern, and the vertical axis represents the pixel error (pixel).

[0189] The examples of the third embodiment show examples in which calibration was performed using a calibration object MR (spherical target). Comparative Example 1 shows an example in which calibration was performed using the dot pattern exemplified by 1025C in Fig. 25, and Comparative Example 2 shows an example in which calibration was performed using the checker pattern exemplified by 1025B in Fig. 25.

[0190] In the case of Comparative Example 1, as shown in Figure 27, objects closer to the camera are photographed larger, so the pixel error (absolute value) increases as the tilt angle (absolute value) relative to the dot pattern increases, resulting in poor tilt tolerance.

[0191] In the example, the pixel error (absolute value) is small regardless of the tilt angle (absolute value) of the camera 12 with respect to the calibration object MR. Also in the comparative example 2, the pixel error (absolute value) is small regardless of the tilt angle (absolute value) of the camera with respect to the checker pattern.

[0192] 22 to 24 are graphs showing the luminance noise resistance of the calibration methods, corresponding to the working example (spherical target), comparative example 1 (dot pattern), and comparative example 2 (checkered pattern), respectively. The horizontal axis of each of Figs. 22 to 24 indicates the sample number, and the vertical axis indicates the pixel error (pixel). Note that the range of only the vertical axis of Fig. 22 has been narrowed to prevent the graph from collapsing.

[0193] 22 to 24, the luminance noise added when photographing the calibration object MR or the calibration pattern increases in the order of Noise=0.0, 4.0, and 8.0.

[0194] In Comparative Example 2 (FIG. 24), the luminance gradient (differential value) is used when detecting the edges of the checkered pattern, and therefore the pixel error increases significantly depending on the luminance noise. It can be seen that, among the examples shown in FIGS. 22 to 24, this is the most sensitive to luminance noise.

[0195] On the other hand, the dot pattern according to Comparative Example 1 (FIG. 23) has higher resistance to luminance noise than the checkered pattern.

[0196] Furthermore, in the example (FIG. 22), as is clear from the range of the vertical axis, the luminance noise resistance is higher than in either of the comparative examples 1 and 2.

[0197] The above results are summarized in the table below. The examples are excellent in both tilt resistance and luminance noise resistance.

[0198]

[0199] Effect of Third Embodiment According to the third embodiment, a calibration object MR including a reflector (ball B or retroreflector) having a reflective property that reflects illumination light back in the incident direction is irradiated with illumination light (coaxial illumination light) from a light source 50 positioned conjugate with the camera origin Oc, and the reflected light from the reflector is captured by the camera 12 to obtain a calibration image IM. Then, by detecting the position of the reflected light as a feature point from this calibration image IM, it is possible to achieve highly accurate calibration with excellent resistance to tilt and brightness noise.

[0200] Furthermore, according to the third embodiment, by using the above-described calibration object MR, it becomes possible to measure the position (three-dimensional coordinates) of the reflector (ball B or retroreflector) using a high-precision measuring device (traceable three-dimensional coordinate measuring device). This makes it possible to perform traceable calibration of the calibration object MR, thereby improving calibration precision.

[0201] Although the embodiments of the present invention have been described above, the present invention is not limited to the above examples, and various improvements and modifications may be made without departing from the spirit of the present invention.

[0202] 10...calibration device, 12...camera, 14...operation unit, 16...output unit, 20...arithmetic and control unit, 22...storage unit, 30...image acquisition unit, 32...pattern information setting unit, 34...feature point detection unit, 36...parameter calculation unit, 50...light source, 52...half mirror, 120...lens, 122...imaging element, 124...lens, 126...imaging element, IM...calibration image, IP...image plane, VP...virtual plane, DI...dot image, VC...center, IC...projection point, B...group of ray lines, T...group of intersection points, CL...line, DP...dot pattern, D...dot, T, T1, T2...target, F1...in-focus position, F2...defocus position, F3...defocus position, MR...calibration object, B...ball, F...housing

Claims

1. An image acquisition step of acquiring a plurality of calibration images obtained by photographing a dot pattern in which a plurality of dots are periodically arranged with a camera; a pattern information setting step of setting pattern information including information regarding the arrangement of the dots; a feature point detection step of detecting feature points indicating the positions of the dots from the calibration images; and a parameter calculation step of calculating parameters of the camera based on the feature points and the pattern information, wherein the feature point detection step detects the position of the center of gravity of the dots as the feature points based on a projection image obtained by projecting the contours of the dots on the image plane of the calibration images onto a virtual plane set in a three-dimensional space. A calibration method for a camera.

2. The feature point detection step obtains the virtual plane optimized so that a circle fits the projection image, and detects the position of the center of gravity of the dots as the feature points based on the projection image projected onto the obtained virtual plane. The calibration method for a camera according to claim 1.

3. When the feature point detection step uses, as an approximate circle, a circle obtained by approximating a point group composed of a plurality of projection points obtained by projecting a plurality of contour points indicating the contours of the dots on the image plane onto the virtual plane, the feature point detection step obtains the virtual plane optimized so that an evaluation value indicating the fitting error of the approximate circle with respect to the point group is minimized or equal to or less than a threshold value, and sets the position of the point obtained by projecting the center of the approximate circle on the optimized virtual plane onto the image plane as the position of the center of gravity of the dots. The calibration method for a camera according to claim 1.

4. The feature point detection step includes a search step of searching for the optimized virtual plane from among a plurality of virtual planes having different postures in a three-dimensional space based on an evaluation value indicating the fitting error of the approximate circle with respect to the point group. The calibration method for a camera according to claim 3.

5. The search step rotates the virtual plane around an axis parallel to the major axis direction of the dots on the image plane to change the virtual plane to a plurality of postures, calculates the evaluation value for each posture, and determines the optimized virtual plane based on the evaluation values calculated for each posture. The calibration method for a camera according to claim 4.

6. The feature point detection step includes: a contour point extraction step of extracting the plurality of contour points of the dot from the image plane; a half-line group generation step of generating a group of half-lines composed of a plurality of half-lines each passing through the plurality of contour points of the dot on the image plane with the optical center of the camera as a starting point; and an approximate circle calculation step of calculating the approximate circle based on the intersection group of the group of half-lines and the virtual plane. The calibration method of the camera according to claim 4.

7. An image acquisition unit that acquires a plurality of calibration images obtained by photographing a dot pattern in which a plurality of dots are periodically arranged with a camera; a pattern information setting unit that sets pattern information including information regarding the arrangement of the dots in the dot pattern; a feature point detection unit that detects feature points from the calibration images; and a parameter calculation unit that calculates parameters of the camera based on the feature points and the pattern information. The feature point detection unit detects the position of the center of gravity of the dot as the feature point based on a projection image obtained by projecting the contour of the dot on the image plane of the calibration image onto a virtual plane set in a three-dimensional space. A camera calibration apparatus.

8. An image acquisition step of acquiring a plurality of calibration images obtained by photographing a calibration pattern including a minimum target in a defocused state; a pattern information setting step of setting pattern information regarding calibration; a feature point detection step of detecting, as a feature point, the position of the center of gravity of the defocused image of the target from the calibration images; and a parameter calculation step of calculating camera parameters based on the pattern information and the position of the feature point. A camera calibration method including the above steps.

9. The size of the target is determined according to at least one of the focal length of the camera that photographs the target, the distance between the camera and the target, and the pixel size of the imaging element of the camera. The camera calibration method according to claim 8.

10. The target has a size such that the size of the image on the imaging element of the camera obtained when photographed in an in-focus state is 1 pixel or less. The camera calibration method according to claim 8.

11. The calibration method of a camera according to any one of claims 8 to 10, wherein the pattern information includes information regarding the arrangement of the targets in the calibration pattern.

12. The calibration method of a camera according to any one of claims 8 to 10, wherein the targets have a shape that is line-symmetric or point-symmetric in two directions orthogonal to each other.

13. The calibration method of a camera according to claim 12, wherein the targets are dot-shaped.

14. The calibration method of a camera according to claim 12, wherein the targets are point light sources.

15. An image acquisition step of irradiating reflected light of illumination light irradiated from a light source onto a calibration device including a reflector having a reflection performance of reflecting the illumination light back in the incident direction, with a camera disposed at a position optically conjugate with the light source to obtain a calibration image; a pattern information setting step of setting pattern information regarding the calibration device; a feature point detection step of detecting a feature point with the condensing position of the reflected light as the feature point from the calibration image; and a parameter calculation step of calculating camera parameters based on the pattern information and the position of the feature point.

16. The calibration method of a camera according to claim 15, wherein in the image acquisition step, the camera captures reflected light from at least two reflectors of the calibration device.

17. The calibration method of a camera according to claim 15, wherein the pattern information includes information regarding the arrangement and interval of the reflectors in the calibration device.

18. The calibration method of a camera according to any one of claims 15 to 17, wherein the reflectors are spherical and arranged in an array on the calibration device.

19. The calibration method of a camera according to claim 18, wherein the surface of the reflector is a mirror surface, or a rough surface from which specular reflection light distinguishable from the reflected light diffusely reflected when incident non-perpendicularly to the surface of the reflector can be obtained.

20. The calibration method of a camera according to claim 18, wherein the reflector is made of glass or sapphire.

Images