Camera calibration device and camera calibration method

Abstract

This camera calibration device comprises: a deviation amount calculation unit that calculates a first deviation amount between a detection position of an object for calibration detected for each division region of an image obtained by imaging the object by a camera through a refraction layer and a first calculation position of the object for each region of an image that can be imaged by the camera without going through the refraction layer, and a second deviation amount between a second calculation position of the object for each re-division region calculated using a calculation formula of a distortion model obtained by modeling the camera and the refraction layer and a detection position of the object; an assessment unit that assesses the second deviation amount and changes the calculation formula of the distortion model; and a correction amount calculation unit that calculates a correction amount for correcting the detection position of the object so as to be the first calculation position for each of the re-division regions on the basis of the changed calculation formula, and calibrates the camera on the basis of the correction amount.

Description

Camera calibration device and camera calibration method

[0001] The present invention relates to a camera calibration device and a camera calibration method.

[0002] There is a growing need to enhance safe driving at intersections, with the New Car Assessment Program (NCAP), a representative automobile assessment program, introducing Autonomous Emergency Braking (AEB) at intersections from 2020. For example, when a car turns right or left at an intersection, it needs to be able to detect objects with a wide horizontal angle of view in order to identify pedestrians and other objects around the intersection.

[0003] Generally, cameras are installed in the center of the vehicle interior. In this case, the camera recognizes the outside world through the windshield in front of the camera (hereinafter referred to as the "refractive layer"). In the wide-angle portion of the refractive layer, the angle of incidence of light rays onto the refractive layer becomes large. When the effect of refraction is large, it becomes difficult for an environment recognition device, which recognizes the environment using images captured by the camera, to accurately detect objects.

[0004] One known method for solving this problem is to divide the image on the imaging surface into small regions, approximate the image distortion for each small region using a function model, and then perform highly accurate distortion correction using this approximated function model. However, if this method does not take into account the boundaries of the function models for each small region, the distortion correction amount in the boundary region becomes discontinuous, resulting in locally reduced correction accuracy.

[0005] For example, Patent Document 1 discloses a calibration device including a feature point coordinate acquisition unit, an ideal coordinate acquisition unit, an area division processing unit, a correction formula calculation unit, and an image correction unit. The feature point coordinate acquisition unit described in Patent Document 1 acquires feature point coordinates that become feature points during calibration from a calibration target image captured by a camera to be calibrated. The ideal coordinate acquisition unit calculates ideal coordinates as true values ​​during calibration. The area division processing unit divides the calibration target image into N small areas. The correction formula calculation unit calculates coefficients of a distortion correction formula by least-squares approximation using coordinate data obtained by the feature point coordinate acquisition unit and the ideal coordinate acquisition unit. Furthermore, the image correction unit corrects the captured image using a distortion model formula that approximates the distortion characteristics. The technology described in Patent Document 1 enables high-precision calibration without local degradation of correction accuracy at the boundaries of small areas, even when calibrating a camera with extreme distortion characteristics.

[0006] Patent No. 5998532

[0007] However, while the technology described in Patent Document 1 corrects camera distortion, it does not take into account the influence of distortion of light rays incident through a refractive layer, and does not describe a method for eliminating this influence.

[0008] Furthermore, due to manufacturing variations that occur during the manufacturing process of the refractive layer, a situation may arise in which the thickness of some regions of the refractive layer differs from the thickness of other regions, or the refractive characteristics of a region with a crack caused by hitting it with a stone or the like differ from the refractive characteristics of other regions. However, in the technology described in Patent Document 1, although a model formula is selected for each region, it is optimized so that the detected feature points are arranged in a known order, and therefore it is not possible to deal with the above-mentioned manufacturing variations and local changes in the refractive layer such as cracks.

[0009] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a camera calibration method that can also accommodate local changes in the refractive layer.

[0010] The camera calibration device of the present invention includes a deviation amount calculation unit that calculates a first deviation amount between the detected position of the object detected for each divided area of ​​a predetermined size in an image in which the camera captures the calibration object through a refractive layer and a first calculated position of the object for each area of ​​the image that the camera can capture without passing through the refractive layer, as well as a second calculated position of the object for each sub-divided area obtained by subdividing the divided area, calculated using a calculation formula of a distortion model that models the camera and the refractive layer, and the detected position of the object; an evaluation unit that evaluates the second deviation amount for each sub-divided area and changes the calculation formula of the one distortion model in accordance with the evaluation result; and a correction amount calculation unit that calculates a correction amount for each sub-divided area based on the changed calculation formula to correct the detected position of the object to the first calculated position, and calibrates the camera based on the correction amount.

[0011] The present invention provides a camera calibration method that can accommodate local variations in refractive layers.

[0012] 1 is a block diagram showing a schematic configuration example of a camera calibration device according to a first embodiment of the present invention. FIG. 2 is a perspective view showing an example of a two-dimensional calibration chart on which quadrangular markers are arranged according to the first embodiment of the present invention. FIG. 3 is a perspective view showing an example of a three-dimensional calibration chart on which circular markers are arranged according to the first embodiment of the present invention. FIG. 4 is a horizontal cross-sectional view of a refractive layer in a state where a camera is mounted on a vehicle according to the first embodiment of the present invention. FIG. 5 is a vertical cross-sectional view of a refractive layer in a state where a camera is mounted on a vehicle according to the first embodiment of the present invention. FIG. 6 is a diagram showing an example of an image of a calibration chart captured through a windshield. FIG. 7 is a diagram showing an example of a sub-pixel according to the first embodiment of the present invention. FIG. 8 is a diagram showing how ray displacement is calculated from an approximated curved surface of a refractive layer modeled by a model generation unit according to the first embodiment of the present invention. FIG. 9 is a diagram showing an example of a refractive layer whose curved surface is approximated by a quadratic polynomial according to the first embodiment of the present invention. FIG. 10 is a diagram showing an example of markers that appear in an image captured of a calibration chart on which black dots are arranged in a grid pattern according to the first embodiment of the present invention. FIG. 11 is a diagram showing a state in which a calibration chart according to the first embodiment of the present invention is captured by a camera. FIG. 12 is a diagram showing the relationship between the detected position and the calculated position of markers on the calibration chart that appear in an image according to the first embodiment of the present invention. FIG. 13 is a diagram showing the amount of deviation for each divided region according to the first embodiment of the present invention. FIG. 1 is a diagram showing an example of correspondence between divided regions and re-divided regions, and distortion model calculation formulas applied to the divided regions according to the first embodiment of the present invention. FIG. 2 is a graph plotting a first deviation amount calculated in a region without a local change and a first deviation amount calculated in a region with a local change according to the first embodiment of the present invention. FIG. 3 is a graph plotting an actual first deviation amount and a deviation amount indicated by an approximate value when the same distortion model calculation formula as other regions is applied to a region with a local change according to the first embodiment of the present invention. FIG. 4 is a graph plotting an actual first deviation amount and a deviation amount indicated by an approximate value when a distortion model calculation formula revised by a result evaluation unit according to the first embodiment of the present invention is applied. FIG. 5 is a diagram showing an example of region division based on an evaluation result by a result evaluation unit according to the first embodiment of the present invention. FIG. 6 is a flowchart showing an example of the procedure of a camera calibration process by the camera calibration device according to the first embodiment of the present invention.FIG. 1 is a flowchart showing an example of the procedure for camera calibration processing by the camera calibration device according to the first embodiment of the present invention. FIG. 2 is a block diagram showing an example of the hardware configuration of a computer according to the first embodiment of the present invention. FIG. 3 is a diagram showing an example of setting the number of input images according to a second amount of deviation according to the second embodiment of the present invention. FIG. 4 is a diagram showing an example of the installation position of a calibration chart with respect to a camera according to the second embodiment of the present invention. FIG. 5 is a diagram showing the position of a calibration chart 20(1) that appears in the entire image according to the second embodiment of the present invention. FIG. 6 is a diagram showing the position of a calibration chart 20(2) that appears closer to the left side of the image according to the second embodiment of the present invention. FIG. 7 is a diagram showing the position of a calibration chart 20(3) that appears closer to the upper side of the image according to the second embodiment of the present invention. FIG. 8 is a flowchart showing an example of the procedure for camera calibration processing according to the second embodiment of the present invention. FIG. 9 is a diagram showing an example of setting a distortion model calculation formula according to a first amount of deviation according to the third embodiment of the present invention. FIG. 10 is a flowchart showing an example of the procedure for camera calibration processing according to the third embodiment of the present invention.

[0013] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functions or configurations are designated by the same reference numerals, and redundant description will be omitted. The present invention is applicable to, for example, a computing device for vehicle control capable of communicating with an on-board ECU (Electronic Control Unit) for an Advanced Driver Assistance System (ADAS) or Autonomous Driving (AD).

[0014] 1 is a block diagram showing an example of the schematic configuration of a camera calibration device 100 according to a first embodiment of the present invention. The camera calibration device 100 calibrates a camera 10 by executing the camera calibration method according to the first embodiment.

[0015] The camera calibration device 100 includes, for example, a marker position detection unit 101, a model generation unit 102, an image position calculation unit 103, a parameter estimation unit 104, a region deviation amount calculation unit 105, a result evaluation unit 106, and a correction amount calculation unit 107.

[0016] The camera 10 includes a lens and an image sensor (not shown). The camera 10 may be either a monocular camera or a stereo camera. The camera 10 captures an image of an object via the lens with the image sensor to obtain an image P1. The image P1 is input to the camera calibration device 100 and is used as an input image in the camera calibration device 100.

[0017] <Example of Calibration Chart> Here, we will explain a calibration chart, which is an example of an object captured by the camera 10. The object for calibration is at least one of a two-dimensional calibration chart and a three-dimensional calibration chart. Furthermore, the calibration chart is configured by equally spaced markers, such as squares or circles, arranged in two or three dimensions.

[0018] Fig. 2A is a perspective view showing an example of a two-dimensional calibration chart 21 on which square markers are arranged, and Fig. 2B is a perspective view showing an example of a three-dimensional calibration chart 22 on which circular markers are arranged.

[0019] In the following description, the calibration charts 21 and 22 will be collectively referred to as the calibration chart 20. The markers arranged on the calibration chart 20 are often arranged at equal intervals on a flat surface without flexure in order to simplify the calculations used for calibration in the camera calibration device 100 and to simplify the manufacture of the calibration chart 20. However, as long as the camera calibration device 100 can accurately grasp the positional relationship of each marker in advance, there is no need to place any restrictions on the spacing or shape of the markers. The worker moves while holding the calibration chart 20. Then, an image of the calibration chart 20 is captured according to the position to which the calibration chart 20 has moved.

[0020] Next, the positional relationship between the refractive layer 1 and the camera 10 will be described with reference to Figures 3A and 3B. The calibration method according to this embodiment includes a process in which the camera 10 captures an image of the calibration chart 20 through the refractive layer 1 to obtain an image P1. The refractive layer 1 is assumed to be, for example, the windshield of an automobile, but the refractive layer 1 may also be the rear window of an automobile, a transparent resin part, or the like.

[0021] 3A and 3B are schematic diagrams showing the relationship between the refractive layer 1 and light rays. Fig. 3A is a horizontal cross-sectional view of the refractive layer 1 when the camera 10 is mounted on a car. Fig. 3B is a vertical cross-sectional view of the refractive layer 1 when the camera 10 is mounted on a car.

[0022] 3A , when the refractive layer 1 is located in front of the camera 10, has a convex shape, and the convex portion is formed in a direction away from the camera 10, a light ray R11 passing near the front surface of the refractive layer 1 is not significantly affected by the refractive layer 1. On the other hand, a light ray R12 passing through the horizontal wide-angle portion of the refractive layer 1 and entering the camera 10 is significantly affected by the refractive layer 1.

[0023] 3B , when the refractive layer 1 is located in front of the camera 10, the camera 10 is disposed near the left end of the refractive layer 1, and the optical axis of the camera 10 passes near the center of the refractive layer 1, a light ray R11 passing near the center of the refractive layer 1 is not significantly affected by the refractive layer 1. On the other hand, a light ray R22 incident on the camera 10 from a direction below the optical axis of the camera 10, i.e., a light ray R22 incident on the camera 10 from the wide-angle portion of the refractive layer 1 in the horizontal direction, is significantly affected by the refractive layer 1.

[0024] <Image of Calibration Chart Captured Through Windshield> Next, an image captured by a camera through the windshield and output will be described with reference to FIG.

[0025] FIG. 4 shows an example of an image of a calibration chart captured through a windshield. Conventional cameras can capture images of the calibration chart at horizontal angles of view ranging from −20 degrees to +20 degrees and vertical angles of view ranging from −20 degrees to +20 degrees. Here, the markers (one example of feature points) on a calibration chart captured on a vehicle without a windshield are represented by multiple white dots 200 spaced at approximately equal intervals in the vertical and horizontal directions. On the other hand, the markers on a calibration chart captured on a vehicle with a windshield are represented by multiple black dots 210 that are shifted relative to the white dots 200. When the horizontal and vertical angles of view are near 0 degrees (the center of FIG. 1 ), the black dots 210 are positioned approximately the same relative to the white dots 200, with almost no shift. On the other hand, as shown in the lower right of FIG. 1 , as the horizontal and vertical angles of view increase, the black dots 210 shift, resulting in a difference d1 in the vertical direction. Although not shown, a difference between the black dots 210 and the white dots 200 also occurs in the vertical direction. That is, the image P1 obtained by the camera 10 capturing an image of the calibration chart 20 through the refractive layer 1 is an image in which the influence of the refractive layer 1 is reflected.

[0026] Returning to the explanation of FIG. 1 , the marker position detection unit 101 shown in FIG. 1 detects markers on the calibration chart 20 that appear in the image P1. More specifically, the marker position detection unit 101 detects markers on the calibration chart 20 that appear in the image P1 using methods such as the Hough transform, corner detection, and luminance centroid calculation. The marker position detection unit 101 also has a function of detecting in subpixel units at which pixel on the image P1 the detected marker is located. A subpixel is a pixel that is a unit of less than one pixel, and while the pixels of the image P1 are represented by integer values, the subpixel unit is represented by a decimal number.

[0027] <Examples of Pixels and Sub-Pixels> Here, sub-pixels will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of a sub-pixel. Fig. 5 shows how a plurality of pixels are arranged.

[0028] 5 are arranged in the X-axis and Y-axis directions. Each pixel is, for example, a plurality of pixels 31 whose position coordinates are (2, 1). The length of this pixel 31 in the X-axis direction is px [mm], and the length in the Y-axis direction is py [mm].

[0029] Here, an example will be described in which the attention point 32 in the figure is expressed in pixel units and sub-pixel units. In pixel units, the position coordinates of the attention point 32 are expressed as (4, 4). On the other hand, in sub-pixel units 32s, the position coordinates of the attention point 32 are expressed as (3.2, 3.8). In this way, in sub-pixel units, the position coordinates of the attention point 32 can be specified in more detail than in pixel units.

[0030] Returning to the description of FIG. 1 , the model generation unit (model generation unit 102) shown in FIG. 1 generates a distortion model by modeling the camera (camera 10), the calibration chart (calibration chart 20), and the refractive layer (refractive layer 1) based on multiple images captured by the camera (camera 10) and input to the deviation amount calculation unit (region deviation amount calculation unit 105). Specifically, the model generation unit 102 has a function of modeling the internal parameters of the camera 10, lens distortion, translational and rotational components of the calibration chart 20, and the approximate curved surface or translational and rotational components of the refractive layer 1 to generate a distortion model. The model generation unit 102's modeling makes it possible to calculate where a three-dimensional point is theoretically located on the image. Thereafter, the result evaluation unit 106, which will be described later, evaluates the deviation between the calculated three-dimensional point and a point on the image P1 captured by the camera 10, thereby confirming the validity of the model, parameter settings, and estimation (model). Here, the modeling of the refractive layer 1 performed by the model generating unit 102 will be described.

[0031] <Example of Approximate Curved Surface of Refractive Layer and Light Ray Displacement> FIG. 6 is a diagram showing how the model generating unit 102 obtains the light ray displacement from the approximate curved surface of the refractive layer 1 modeled.

[0032] 6 shows how the camera 10 captures an image of an object including the calibration chart 20 via the refractive layer 1. Here, the curved surface on the front side of the refractive layer 1 as viewed from the camera 10 is called the front-side approximated curved surface 41, and the curved surface on the back side of the refractive layer 1 is called the back-side approximated curved surface 42. The refractive layer 1 is configured to be sandwiched between the front-side approximated curved surface 41 and the back-side approximated curved surface 42 with respect to the camera 10, and has a parameter of refractive index n.

[0033] As an example of modeling an approximated curved surface, the model generation unit 102 approximates and models the curved surface of the refractive layer 1 using a quadratic polynomial (hereinafter also referred to as a "distortion model calculation formula") shown in the following formula (1): In the following formula (1), Z in the height direction relative to the xy plane is expressed as a function of x and y.

[0034]

[0035] p in formula (1) 00 , p 10 , p 01 , p 20 , p 11 , p 02 are the coefficients of each term, and x ２ , y ２ is a square term. In the subscripts of each P in formula (1) (for example, 10, 20), the first part represents the degree of x and the second part represents the degree of y. ２０ represents the coefficient of the term where x is the square and y is the zeroth power. The near-side approximated curved surface 41 and the far-side approximated curved surface 42 expressed by equation (1) have a translation component T and a rotation component R, and each surface is positioned with respect to the camera 10. However, the method of selecting variables in modeling is not limited to that shown in equation (1), and a higher-order polynomial (for example, a fifth-order polynomial shown in equation (6) described later) may also be used.

[0036] The model generation unit 102 identifies, with respect to the camera 10, the point at which a ray of light emitted in the direction of the refractive layer 1 enters the front-side approximated surface 41 as the intersection point between the ray of light and the approximated surface. The model generation unit 102 then calculates a normal (near-front side) at the identified intersection point based on equation (1) of the modeled approximated surface, a translation component T, and a rotation component R. Because the model generation unit 102 can grasp the incident ray, intersection point, normal, and refractive index with respect to the front-side approximated surface 41 through modeling, it can calculate the exit ray of light from the front-side approximated surface 41 by substituting this information into an equation such as Snell's law.

[0037] The model generation unit 102 also regards the emergent ray from the front-side approximated curved surface 41 as an incident ray for the back-side approximated curved surface 42. Then, similar to the calculation for the front-side approximated curved surface 41, the model generation unit 102 performs processing to determine the emergent ray from the back-side approximated curved surface 42 using Snell's law, thereby determining the emergent ray from the back-side approximated curved surface 42. This emergent ray is the ray emerging from the refractive layer 1. Then, the model generation unit 102 determines the normal (back surface) at the intersection point based on equation (1), the translation component T, and the rotation component R of the modeled approximated surface.

[0038] The image position calculation unit 103 calculates the position of the marker from the difference between the position of the marker in the image P1 captured without passing through the refractive layer 1 (the exiting light ray shown by the dotted line in the figure) and the position of the marker in the image P1 captured through the refractive layer 1 (the exiting light ray shown by the solid line in the figure).

[0039] <Example of Approximated Curved Surface of Refractive Layer> The refractive layer 1 whose curved surface is approximated by Equation (1) is expressed in Fig. 7 shown below. Fig. 7 is a diagram showing an example of the refractive layer 1 whose curved surface is approximated by a quadratic polynomial.

[0040] 7 , even if the refractive layer 1 has a complex curved surface, the model generation unit 102 can approximate the curved surface of the refractive layer 1 by using formula (1). Furthermore, the model generation unit 102 can also represent the refractive layer 1 as a plane by substituting a specific numerical value for the coefficient p in formula (1). The following formula (1)′ is obtained by substituting a numerical value for the coefficient p in formula (1).

[0041]

[0042] <Example of Approximate Plane of Refractive Layer> Fig. 8 is a diagram showing an example of markers that appear in an image P1 obtained by capturing an image of a calibration chart 20 on which black dots 51 are arranged in a grid pattern. The center of Fig. 8 is set to position coordinates of x = 0, y = 0. When the black dot 51 at position coordinates (4, -2) is captured by the camera 10, it is represented at the position of a white dot 52 due to distortion of the refractive layer 1. Here, the position coordinates of the white dot 52 are set to (x distortion present, y distortion present). In this way, by representing the refractive layer 1 as a plane, the influence of the distortion of the refractive layer 1 becomes clear.

[0043] Returning to the explanation of Fig. 1, the image position calculation unit 103 shown in Fig. 1 converts the marker positions on the three-dimensional calibration chart 20 into positions on the image. Here, the process of converting the marker positions will be explained with reference to Figs. 9 to 11.

[0044] 9 is a diagram showing the state in which the calibration chart 20 is imaged by the camera 10. Here, the Zhang method will be described, in which the parameters of the camera 10 are modeled and the camera 10 is considered as a pinhole camera. By the Zhang method, an object in the world coordinate system is converted into an image in the camera coordinate system.

[0045] An example of the calibration chart 20 expressed in world coordinates is shown in the upper part of Fig. 9. World coordinates are a coordinate system used in three-dimensional space in the real world, and in world coordinates, the positions of markers are expressed in millimeters [mm].

[0046] A plurality of markers are arranged at equal intervals on the calibration chart 20. In world coordinates, the position coordinates of each marker, which is represented by a black dot on the calibration chart 20, are expressed using coordinates of row number Mx and column number My, with the upper left corner of the calibration chart 20 as the origin. For example, since the marker 25 on the calibration chart 20 is located in the third row and sixth column of the world coordinates, the coordinates of the marker 25 are expressed as position coordinates (Mx ３６ , My ３６ ) is shown by

[0047] The lower part of Fig. 9 shows how the calibration chart 20 is imaged by the camera 10. This shows a method for converting the three-dimensional marker positions on the calibration chart into image coordinate positions on the image. Image coordinates are a coordinate system used in images, and in image coordinates, image positions are expressed in pixel units [px].

[0048] The calibration chart 20 is set to rotate or translate relative to the camera 10. Here, the rotation of the calibration chart 20 is represented by "R", and the translation of the calibration chart 20 is represented by "t". The position coordinates of the marker 25 on the calibration chart 20 are expressed as P ｉｊ (X ｉｊ , Y ｉｊ , Z ｉｊ The position of the camera 10 in the camera coordinate system is expressed as (X camera, Y camera, Z camera). The unit of the camera coordinate system is mm.

[0049] The position of any point in the image P1 captured by the camera 10 is specified by the image coordinates. Here, the position of the origin (0, 0) of the image coordinates is defined as the optical center through which the Z axis (Z camera) indicating the optical axis of the camera 10 passes, and is called the principal point (Cx, Cy). The marker 25 of the calibration chart 20 captured by the camera 10 is on a straight line 26 that passes through the marker 28 of the calibration chart 27 shown in the image P1. Furthermore, the coordinates of any position on the image coordinates can be expressed as (x ｉｊ , y ｉｊ On the other hand, the coordinates of any position on the calibration chart 27 imaged through the refractive layer 1 are expressed as (u ｉｊ , v ｉｊ )

[0050] The model generation unit 102 models the distortion of the lens of the camera 10 and sets parameters used in the following equations (2) and (3). The equation (2) is a set of coordinates P ｉｊ (X ｉｊ , Y ｉｊ , Z ｉｊ ) is rotated (R) and translated (t) to determine the three-dimensional position of the marker.

[0051] The parameters of the calibration chart include, for example, the following: Two-dimensional chart plane coordinates: Mx ｉｊ , My ｉｊ ・Rotational component: R ・Translational component: T

[0052]

[0053] For example, the term marked "external parameters" in the above formula (2) is a term that represents the influence of rotation (R) and translation (t) on the calibration chart 20. Also, the term marked "two-dimensional chart plane coordinates" in formula (2) represents the position of the marker in the world coordinate system shown in the upper part of FIG.

[0054] The following equation (3) is the three-dimensional point (X ｉｊ , Y ｉｊ ) is an equation used to determine the position in image P1 where the object appears. Here, the coordinate system of image P1 is also called image coordinates.

[0055]

[0056] For example, the term marked "image coordinates" on the left side of the above formula (3) is a term that represents the position in image coordinates of the marker 28 of the calibration chart 27 that appears in the image P shown in FIG. 9. The term marked "internal parameters" on the right side of the formula (3) is a term that represents camera parameters. Examples of camera parameters include the following: Focal length: f Focal length in the x direction: f ｘ Focal length in the y direction: f ｙ Coordinates of the optical center (principal point): c ｘ , c ｙ ・Pixel size: p ｘ , p ｙ Shear coefficient: s

[0057] f in the above camera parameters ｘ is "f / p ｘ " and f ｙ is "f / p ｙ The pixel size p ｘ , p ｙEach of these represents the size of a pixel. The shear coefficient s in the camera parameters is calculated from the information on the pixel tilt α of the CMOS sensor (not shown) which is the image receiving unit of the camera using a predetermined formula (f ｘ The value is calculated by using the tangent function α.

[0058] By using the above formula (3) and the like, the model generation unit 102 can determine the positions of markers that can be captured by the camera 10 without passing through the refractive layer 1, from the image captured by the camera 10 through the refractive layer 1. The following formula (4) is an example of a calculation formula for the camera lens distortion model generated by the model generation unit 102.

[0059]

[0060] In equation (4), p1 and p2 respectively represent coefficients in the distortion model. Furthermore, x and y in equation (4) respectively represent the coordinates of the marker on the image when a distortion-free camera (pinhole camera) is used. In reality, deviations occur in the x and y coordinates due to factors such as distortion of the lens of the camera 10. The s coordinate of the marker when deviations occur is expressed as (x distortion, y distortion).

[0061] The image position calculation unit 103 converts the position of each marker on the calibration chart 20 into a position on the image by setting and calculating each parameter for the refractive layer 1 and the distortion model generated by the model generation unit 102.

[0062] <Relationship Between Detected and Calculated Positions of Markers> Fig. 10 is a diagram showing the relationship between the detected and calculated positions of markers on the calibration chart 20 shown in the image. In Fig. 10, the positions of the markers on the calibration chart 20 are represented by "X".

[0063] The position of the marker on the calibration chart 20 captured by the camera 10 is the detected position Q in the image P1 shown in FIG. ｉｊ (u ｉｊ , v ｉｊ On the other hand, the image position calculation unit 103 shown in FIG. 1 calculates the position of the marker using the above-mentioned equations (2) and (3), thereby obtaining the calculated position P' ｉｊ (x ｉｊ , yｉｊ Then, the image position calculation unit 103 calculates the calculated positions P′ for all the markers on the calibration chart 20. ｉｊ (x ｉｊ , y ｉｊ ) is calculated.

[0064] The parameter estimation unit 104 compares the detection result of the marker detected from the image P1 by the marker position detection unit 101 with the calculation result of the marker calculated by the image position calculation unit 103 to calculate the amount of deviation between the feature amount detected at the detection position and the feature amount calculated at the calculation position.The parameter estimation unit 104 then estimates parameters that minimize the amount of deviation.The parameter estimation unit 104 can estimate the parameters using, for example, the following equation (5).

[0065]

[0066] Q in formula (5) ｉｊ and P' ｉｊ represent the following position coordinates: Position coordinates of the marker detected by the marker position detection unit 101: Q ｉｊ (u ｉｊ , v ｉｊ Position coordinates of the marker calculated by the image position calculation unit 103: P' ｉｊ (x ｉｊ , y ｉｊ S(Rt) in equation (5) is the sum of the distances between all detected points and calculated points. The parameter estimation unit 104 then estimates parameters that minimize the sum of the distances between each detected position and calculated position.

[0067] The correction amount calculation unit 107 shown in FIG. 1 has a function of creating a correction table (not shown) based on each parameter value calculated by the model generation unit 102 and the parameter estimation unit 104. The correction table indicates the positions of points on the image P1 when there is no influence from the refractive layer 1 or lens distortion. This correction table is used to simultaneously perform geometric correction, which corrects lens distortion using only the image P1 captured through the refractive layer 1, and correction for the influence of the refractive layer 1 on the image. Since such correction is performed by the correction amount calculation unit 107, it is possible to grasp the influence of the refractive layer 1. Therefore, according to this embodiment, it is possible to simulate a state in which the refractive layer 1 is absent even in a situation where it is difficult to separate light rays incident on the camera 10 from light rays incident on the refractive layer 1, such as when the camera is mounted on a vehicle.

[0068] The area deviation amount calculation unit 105 has a function of dividing the image P1 into several areas and calculating the amount of deviation for each of the divided areas based on the presence or absence of distortion of the refractive layer 1 or the lens. It is desirable that all of the areas divided by the area deviation amount calculation unit 105 be the same size. The deviation amount calculation unit (area deviation amount calculation unit 105) calculates a first deviation amount, which is the amount of deviation between the detected position of the object detected for each area of ​​a predetermined size in an image in which the camera (camera 10) captures the calibration object through the refractive layer (refractive layer 1), and a first calculated position, which is the calculated position of the object calculated for each area of ​​the image that the camera (camera 10) can capture without passing through the refractive layer (refractive layer 1).

[0069] <Example of Calculating a Displacement Amount> Here, the manner in which the region displacement amount calculation unit 105 divides the image P1 and calculates the first displacement amount will be described with reference to FIG. 11 . FIG. 11 is a diagram showing the displacement amount for each divided region of the image P1. The image P1 is divided at equal intervals in both the vertical and horizontal directions. In FIG. 11 , the image P1 is divided into three regions vertically and three regions horizontally. Note that the number of regions divided into the image P1 by the region displacement amount calculation unit 105 is not limited to three, and may be two or another number. The displacement amount calculation unit (region displacement amount calculation unit 105) calculates the first calculated position of the object using a calculation formula for the modeled refraction layer (refraction layer 1).

[0070] Here, a surface located an appropriate distance from the front of the camera 10 and directly facing the lens surface of the camera 10 is defined as an infinite plane 61. Rectangular regions are displayed on this infinite plane 61 according to the size of each region of the divided image P1. As an example, the region deviation amount calculation unit 105 sets a white point 63 passing through the center of the divided image P1 for each divided region. Then, the region deviation amount calculation unit 105 calculates a black point 64 affected by the distortion of the refractive layer 1 and the lens, which corresponds to the white point 63 set for each divided region.

[0071] Next, the area deviation amount calculation unit 105 calculates where the white points 63 and black points 64 on the infinite plane 61 are located on the image coordinates 62. The image coordinates 62 shown in Fig. 11 show white points 65 and black points 66 that correspond to the white points 63 and black points 64 on the infinite plane 61, respectively. The area deviation amount calculation unit 105 then calculates a "deviation amount 67" that is the distance between the white points 65 and black points 66 for each divided area.

[0072] Furthermore, the region deviation amount calculation unit 105 according to this embodiment further divides each divided region obtained by dividing the image P1 into, for example, 3 vertically and 3 horizontally to generate sub-divided regions. Next, the region deviation amount calculation unit 105 calculates a second calculated position in each sub-divided region. The second calculated position is a calculated position of the object for each sub-divided region, calculated using a distortion model calculation formula such as the above-described formula (1) or formula (4) that models the camera (camera 10) and the refractive layer (refractive layer 1). Next, the region deviation amount calculation unit 105 calculates a second deviation amount, which is the amount of deviation between the calculated second calculated position and the actual detected position of the marker affected by distortion of the refractive layer 1 and the lens.

[0073] Next, the evaluation unit (result evaluation unit 106) evaluates the second deviation amount for each region calculated by the region deviation amount calculation unit 105 and changes the calculation formula for the distortion model according to the evaluation result. The result evaluation unit 106 then outputs the evaluation result to the correction amount calculation unit 107. More specifically, the result evaluation unit 106 determines whether the second deviation amount calculated for each re-divided region is equal to or less than a predetermined first threshold, and outputs the evaluation result to the correction amount calculation unit 107. If the evaluation result by the result evaluation unit 106 indicates that the second deviation amount is equal to or less than the first threshold, the correction amount calculation unit 107 calculates the correction amount using the distortion model calculation formula applied to the initially divided regions, such as 3 × 3.

[0074] On the other hand, if the evaluation result by the result evaluation unit 106 indicates that the second deviation amount is greater than the first threshold, it is assumed that the validity of the distortion model formula used to calculate the second deviation amount with respect to the entity shown in the re-divided region is low. Therefore, the result evaluation unit 106 reviews the distortion model formula to be applied to the re-divided region. As the review of the distortion model formula, the result evaluation unit 106 may increase the order of the distortion model formula or apply a different distortion model formula that enables the second deviation amount to be smaller.

[0075] For example, the result evaluation unit 106 can change the distortion model calculation formula to the following formula (6) in which the order is increased to 5. Note that the order of the changed distortion model calculation formula is not limited to 5 and may be another order.

[0076]

[0077] Furthermore, the result evaluation unit 106 may change the distortion model calculation formula to a model that is combined with another approximated surface, such as the following formula (7).

[0078]

[0079] <Example of how the amount of deviation appears> Next, how the amount of deviation between the detected position and the calculated position appears when the first amount of deviation calculated by the area deviation amount calculation unit 105 becomes large will be described. Fig. 12 is a diagram showing an example of the correspondence between the divided area and the re-divided area and the distortion model calculation formula applied to the divided area. Fig. 12 shows an example of the correspondence between the divided area and the re-divided area and the distortion model calculation formula applied to the divided area. １ 12 shows how the distortion model calculation formula indicated by (x)" is applied. Furthermore, of the sub-division areas Ar11 to Ar14 obtained by further dividing the sub-division area Ar1 shown in FIG. 12, the upper left sub-division area Ar11 is assumed to be an area in which local changes such as manufacturing variations or cracks have occurred.

[0080] 13 is a graph plotting the first deviation amount calculated in a region without a local change and the first deviation amount calculated in a region with a local change, where the vertical axis of the graph shown in FIG. 13 represents the magnitude of the deviation amount and the horizontal axis represents the position of the divided region in the x direction.

[0081] In Fig. 13, the first deviation amount in the region without local change is indicated by a bold solid line. １ The deviation amount in the feature amount approximated using "f(x)" is shown by a thick dashed line. The first deviation amount in the area where there is a local change is shown by a solid line, and the "f(x)" in the distortion model calculation formula optimized for the entire re-divided area is shown by a solid line. ２ The deviation amount in the feature amount approximated using (x)" is indicated by a thin dashed line. As illustrated in FIG. 13 , the distortion model calculation formula that can derive a deviation amount that approximates the actual first deviation amount differs depending on whether the target is a region with no local change or a region with local change.

[0082] FIG. 14 shows that the same distortion model calculation formula (f １ 14 is a graph plotting the actual first deviation amount and the deviation amount indicated by the approximate value when the distortion model calculation formula (x) is applied. １The deviation amount at the feature point approximated using (x)" is indicated by a thick dashed line. In the example shown in FIG. 14, in the area Ari whose range is indicated by the double arrow, a difference occurs between the deviation amount approximated by the distortion model calculation formula and the first deviation amount. That is, the distortion model calculation formula (f １ In (x), it is clear that it is difficult to approximate the first deviation amount in the area Ari.

[0083] Fig. 15 is a graph plotting the actual first deviation amount and the deviation amount indicated by the approximate value when the distortion model formula revised by the result evaluation unit 106 is applied. In Fig. 15, the deviation amount indicated by the approximate value using the revised distortion model formula is indicated by a thin dashed line. By reviewing the distortion model formula by the result evaluation unit 106, it becomes possible to appropriately approximate feature points of the object in areas where local changes have occurred.

[0084] FIG. 16 is a diagram showing an example of region division based on the evaluation results by the result evaluation unit 106. FIG. 16 shows a 16×16 region obtained by further dividing a 4×4 region. Divided region Ar14, located in the second column from the left and the second row from the top in FIG. 16, is further divided into four sub-division regions Ar141 to Ar144. The second position indicated by a checkered circle located within sub-division region Ar144 is the position of a feature point approximated by the distortion model calculation formula applied to sub-division region Ar144. However, the actual detection position of the detected object is detection position 68, which is diagonally downward and to the left of the second position.

[0085] In other words, this indicates that the distortion model calculation formula applied to the sub-division area Ar144 is unable to approximate the feature point at the detection position 68. In this case, in this embodiment, the area deviation amount calculation unit 105 re-divides the area, the result evaluation unit 106 reviews the distortion model calculation formula, and the result evaluation unit 106 evaluates the calculation results using the revised distortion model calculation formula. The sub-division area Ar233, which is adjacent to the right of the sub-division area Ar144, is also further divided into four sub-division areas because the second deviation amount is equal to or greater than the predetermined first threshold.

[0086] If the evaluation unit (result evaluation unit 106) determines that the second deviation amount in the re-divided region is less than the first threshold value due to application of the revised distortion model calculation formula, the correction amount calculation unit 107 calculates the correction amount using the distortion model calculation formula set for the divided region. On the other hand, if the second deviation amount in the re-divided region is still equal to or greater than the first threshold value, the result evaluation unit 106 determines whether the size of the re-divided region is equal to or greater than a predetermined second threshold value, and outputs the determination result to the region deviation amount calculation unit 105.

[0087] The second threshold is a threshold set to define the minimum size of the re-divided regions, and is set to a value such as 10 x 10 (pixels) or 10 x 10 (mm). The threshold value is an example, and may be set to a value other than these. If the second threshold is too large, it may be impossible to properly measure the distance to a vehicle traveling in front of the windshield, which is the refractive layer 1, and if it is too small, the calculation time and processing load will increase. The second threshold is set to an optimal value taking these factors into consideration.

[0088] If the evaluation unit (result evaluation unit 106) determines that the second deviation amount is equal to or greater than the first threshold and that the size of the re-divided region is equal to or greater than the second threshold, the deviation amount calculation unit (region deviation amount calculation unit 105) further divides the re-divided region to generate sub-sub-divided regions, calculates the second deviation amount for each sub-sub-divided region, and changes the distortion model calculation formula (distortion model calculation formula) based on the second deviation amount.The region deviation amount calculation unit 105 then outputs the deviation amount calculated using the distortion model calculation formula to the correction amount calculation unit 107.

[0089] In other words, the result evaluation unit 106 continues the evaluation process of the second deviation amount in the sub-divided area until the result evaluation unit 106 determines that the second deviation amount is less than the first threshold value, or until it determines that the size of the sub-divided area is less than the second threshold value.

[0090] The correction amount calculation unit 107 calculates a correction amount for correcting the detected position of the object to the first calculated position for each sub-divided area based on a calculation formula changed in accordance with the evaluation result by the evaluation unit (result evaluation unit 106), and calibrates the camera (camera 10) based on the correction amount. By performing such calibration by the correction amount calculation unit 107, the image output from the camera calibration device 100 is one in which the influence of the refractive layer 1 has been removed. The image output from the camera calibration device 100 is then input to a downstream environment recognition device (not shown) or the like, and used for recognizing the environment around the automobile, such as recognizing vehicles or people traveling ahead.

[0091] Furthermore, if the evaluation unit (result evaluation unit 106) determines that the second deviation amount is equal to or greater than the first threshold and that the size of the re-divided area is less than the second threshold, the correction amount calculation unit (correction amount calculation unit 107) calculates the correction amount for each re-divided area using the distortion model calculation formula (distortion model calculation formula) set for the divided area. In other words, if the target re-divided area has a negligible size, its size will be less than the second threshold, and no further distortion model calculation will be performed. Therefore, according to this embodiment, the calculation load can be reduced.

[0092] <Example of Camera Calibration Method According to First Embodiment> Next, a camera calibration method according to the first embodiment performed by the camera calibration device 100 will be described with reference to Fig. 17 and Fig. 18. Fig. 17 and Fig. 18 are flowcharts showing an example of the procedure of camera calibration processing by the camera calibration device 100.

[0093] First, the camera 10 (see FIG. 1) of the camera calibration device 100 captures an image of the calibration chart 20 through the refractive layer 1 (step S1). Next, the marker position detection unit 101 detects the positions of the markers from the image P1 output from the camera 10 (step S2). Next, the model generation unit 102 generates the distortion model described above (step S3).

[0094] Next, the image position calculation unit 103 calculates the image position of the marker in the camera coordinate system from the position of each marker on the calibration chart represented by the world coordinate system (step S4). Next, the parameter estimation unit 104 estimates parameters that minimize the sum of the distances between the detected positions of the markers detected in step S2 and the calculated positions of the markers calculated in step S4 (step S5).

[0095] Next, the region deviation amount calculation unit 105 calculates a first calculation position for each divided region obtained by dividing the image P1 (step S6). The first calculation position is the position of a marker that can be captured by the camera 10 without passing through the refractive layer 1. Next, the region deviation amount calculation unit 105 calculates a first deviation amount, which is the deviation amount between the detected position of the object and the first calculation position, for each divided region (step S7). Next, the result evaluation unit 106 determines whether the first deviation amount calculated in step S7 is less than a predetermined third threshold (step S8). The third threshold is a value for determining the validity of the distortion model calculation formula used to calculate the first calculation position, and an appropriate value can be set based on experiments, etc.

[0096] If it is determined in step S8 that the first deviation is equal to or greater than the third threshold (NO in step S8), the result evaluation unit 106 changes the calculation formula for the distortion model based on the first deviation. More specifically, the result evaluation unit 106 instructs the model generation unit 102 to regenerate the distortion model. On the other hand, if it is determined that the first deviation is less than the third threshold (YES in step S8), the result evaluation unit 106 instructs the region deviation calculation unit 105 to re-divide the divided region. Then, based on the instruction, the region deviation calculation unit 105 further divides the divided region to create re-divided regions (step S9 in FIG. 18 ).

[0097] Next, the result evaluation unit 106 changes the distortion model calculation formula to be applied to the re-divided regions generated in step S9 (step S10). In step S10, the result evaluation unit 106, for example, increases the order of the distortion model calculation formula or changes it to another polynomial that is expected to reduce the second deviation amount. By changing the distortion model calculation formula, it becomes possible to approximate feature points that could not be represented by the distortion model calculation formula applied to the divided regions, such as those shown in region Ari in Fig. 14.

[0098] Next, the area displacement amount calculation unit 105 calculates a second calculation position using the changed distortion model calculation formula (step S11). The second calculation position is the calculation position of the object calculated using the distortion model calculation formula such as the above-mentioned formula (1) or (4).

[0099] Next, the region deviation amount calculation unit 105 calculates a second deviation amount, which is the deviation amount between the detected position of the object and the second calculated position (step S12). Next, the result evaluation unit 106 determines whether the second deviation amount is less than a first threshold (step S13). If it is determined in step S13 that the second deviation amount is equal to or greater than the first threshold (NO in step S13), the result evaluation unit 106 determines whether the size of the re-divided region is equal to or greater than a second threshold (step S14).

[0100] If it is determined in step S14 that the size of the re-divided region is equal to or larger than the second threshold value (YES in step S14), the result evaluation unit 106 instructs the region displacement amount calculation unit 105 to re-divide the divided region. Then, the processing of step S9 is performed. That is, the region displacement amount calculation unit 105 further divides the divided region to create re-divided regions.

[0101] On the other hand, if it is determined in step S13 that the second deviation amount is less than the first threshold value (YES in step S13), or if it is determined in step S14 that the re-divided area is less than the second threshold value (NO in step S14), the correction amount calculation unit 107 calculates a correction amount (step S15). Specifically, the correction amount calculation unit 107 calculates the difference value between the first calculation position and the second calculation position as the correction amount.

[0102] Next, the correction amount calculation unit 107 calibrates the camera using the calculated correction amount (step S16). Specifically, the correction amount calculation unit 107 corrects the detected position of the object to the first calculated position using the correction amount. After the processing of step S16, the camera calibration process by the camera calibration device 100 ends.

[0103] <Example of Hardware Configuration of Computer> Next, the hardware configuration of the computer 70 constituting the camera calibration device 100 will be described with reference to Fig. 19. Fig. 19 is a block diagram showing an example of the hardware configuration of the computer 70. The computer 70 is an example of hardware used as a computer that can operate as the camera calibration device 100 according to this embodiment. The camera calibration device 100 according to this embodiment realizes a camera calibration method in which the functional blocks of the camera calibration device 100 shown in Fig. 1 cooperate with each other by causing the computer 70 to execute a program.

[0104] The computer 70 includes a CPU (Central Processing Unit) 71, a ROM (Read Only Memory) 72, and a RAM (Random Access Memory) 73, each connected to a bus 74. The computer 70 further includes a non-volatile storage 75 and a network interface 76.

[0105] The CPU 71 reads out program code of software that realizes each function according to this embodiment from the ROM 72, loads it into the RAM 73, and executes it. Variables, parameters, etc. generated during the calculation processing of the CPU 71 are temporarily written to the RAM 73, and these variables, parameters, etc. are read out by the CPU 71 as appropriate. Note that an MPU (Micro Processing Unit) may be used instead of the CPU 71, or the CPU 71 may be used in combination with a GPU (Graphics Processing Unit). The functions of each part of the camera calibration device 100 shown in FIG. 1 are realized by the CPU 71.

[0106] Examples of the non-volatile storage 75 include a hard disk drive (HDD), a solid state drive (SSD), a flexible disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, and non-volatile memory. The non-volatile storage 75 stores an operating system (OS), various parameters, and programs for operating the computer 70. The ROM 72 and the non-volatile storage 75 store programs and data necessary for the CPU 71 to operate, and are used as examples of computer-readable, non-transitory storage media that store programs executed by the computer 70. The non-volatile storage 75 also stores, for example, an image P1. The non-volatile storage 75 also stores the distortion model generated by the model generation unit 102, the parameters estimated by the parameter estimation unit 104, the first and second displacement amounts calculated by the region displacement amount calculation unit 105, the evaluation results of the displacement amounts evaluated by the result evaluation unit 106, and the correction amount calculated by the correction amount calculation unit 107.

[0107] The network interface 76 may be, for example, a network interface card (NIC), and various data may be transmitted and received between devices via an in-vehicle local area network (LAN) or dedicated line connected to the terminal of the NIC.

[0108] The camera calibration device 100 according to the first embodiment described above generates a distortion model that includes not only the lens of the camera 10 but also the refractive layer 1. The camera calibration device 100 then calibrates the camera 10 based on the amount of deviation between calculated positions (first calculated position, second calculated position) that are theoretical detection positions of the object obtained by the calculation formula of the distortion model and the actual detection position of the object. Therefore, according to this embodiment, the camera 10 can be calibrated with high accuracy even when light rays from the object are incident not only through the lens of the camera 10 but also through the refractive layer 1.

[0109] In the above-described embodiment, a distortion model is set for each divided region obtained by dividing the image captured by the camera 10. Then, the amount of deviation between the calculated position calculated using the distortion model formula and the actual detected position of the object is evaluated. In other words, it is evaluated whether each distortion model formula set for the divided region is a distortion model formula that can reduce the amount of deviation. Then, the distortion model is changed as necessary. Therefore, according to this embodiment, an appropriate distortion model is set for each region, such as the central portion or wide-angle portion, of the refractive layer 1, thereby improving the calculation accuracy of the correction amount obtained based on the distortion model formula.

[0110] Furthermore, in the above-described embodiment, the detection position of the object is converted by calibration of the camera 10 to a position (first position) that is not affected by distortion of the refractive layer 1 or the lens of the camera 10. Therefore, according to this embodiment, an environment recognition device or the like that recognizes the environment using an image captured by the camera 10 can accurately grasp the position of the object.

[0111] For example, in an application for detecting pedestrians when a car turns right or left at an intersection, an image P captured by a camera 10 using a lens capable of capturing a wide angle of view is input to an environment recognition device, etc. Even in such an application, the camera calibration process according to the first embodiment can be performed to reduce the influence of image deviation incident from the wide-angle portion of the refractive layer. Therefore, according to this embodiment, the environment recognition device, etc., can reliably detect pedestrians.

[0112] Therefore, even when distance measurement is performed using a monocular camera or a stereo camera, there is no need to provide dedicated equipment for geometric correction because the camera 10 can be calibrated simply by using at least one calibration chart 20. Furthermore, according to this embodiment, the correction amount for the image P1 is calculated, so it is also possible to provide an imaging device and an image correction device that can simulate a situation where the refractive layer 1 is not present, from the image P1 captured through the refractive layer 1.

[0113] Furthermore, in the above-described embodiment, the correction amount is calculated based on the amount of deviation between the calculated position of the object obtained by the calculation formula of the distortion model and the actual detected position of the object, so the camera 10 can be appropriately calibrated based on the correction amount calculated with high accuracy regardless of the mounting position of the camera 10 relative to the refractive layer 1. Therefore, according to this embodiment, the required accuracy for the mounting position of the camera 10 can be relaxed.

[0114] Furthermore, in the above-described embodiment, the validity of the distortion model is examined for the sub-divided regions obtained by further dividing the divided region, and if the validity is not found, i.e., if the second deviation amount is equal to or greater than the first threshold, the distortion model is revised. Therefore, according to this embodiment, even if a local change occurs in the refractive layer 1 during the manufacturing process, such as when a region of a different thickness is created in a part of the refractive layer 1 from other regions, or when a crack occurs in a region due to a stone or the like being hit, an optimal distortion model corresponding to the local change is set for that region (sub-divided region). Therefore, according to this embodiment, a camera calibration method that can also accommodate local changes in the refractive layer is provided.

[0115] Furthermore, in the above-described embodiment, if the validity of the distortion model is confirmed in the re-divided region, that is, if the second deviation amount is less than the first threshold, the distortion model is not re-examined and the deviation amount is not re-calculated. Therefore, according to this embodiment, it is possible to reduce the calculation load on the camera calibration device 100 when calculating the correction amount for calibrating the camera 10.

[0116] Second Embodiment Next, a camera calibration method according to a second embodiment of the present invention, which is performed by the camera calibration device 100, will be described.

[0117] The configuration of the camera calibration device 100 according to the second embodiment can be the same as that of the camera calibration device 100 according to the first embodiment. The camera calibration device 100 according to the second embodiment calibrates the camera 10 by changing the number of images P1 input from the camera 10 in accordance with the magnitude of the second deviation amount in the re-divided region calculated by the region deviation amount calculation unit 105 according to the first embodiment described above.

[0118] More specifically, when the evaluation unit (result evaluation unit 106) of the camera calibration device 100 according to the second embodiment determines that the second deviation is equal to or greater than the first threshold, it determines whether the number of input images (images P) corresponds to the second deviation. If it determines that the number does not correspond to the second deviation, it outputs an instruction to input a number of images corresponding to the second deviation. Specifically, the result evaluation unit 106 instructs the camera 10 to increase the number of images P1 to be input from the camera 10 for regions where the second deviation is large. Then, the camera 10 captures the images P based on this instruction. Increasing the number of input images P1 from the camera 10 improves the accuracy of object position estimation, thereby also improving the accuracy of calculation of the correction amount for each sub-division region. Therefore, according to this embodiment, it is possible to further improve the camera calibration accuracy while responding to local changes occurring in the refractive layer 1.

[0119] Fig. 20 is a diagram showing an example of setting the number of input images according to the second shift amount, in which the second shift amount [px] for each sub-division area is shown in the upper part of Fig. 20, and the number of input images (images P1) required for each sub-division area is shown in the lower part of Fig. 20.

[0120] The evaluation unit (result evaluation unit 106) pre-sets the number of images (images P) that need to be input for each divided region based on the second deviation amount calculated during initial setting. Assume that the minimum number of input images for image P1 from camera 10 is one. For example, if the second deviation amount is less than 1.0, the result evaluation unit 106 sets the number of input images to one. Furthermore, if the second deviation amount is 1.0 or greater but less than 3.0, the result evaluation unit 106 sets the number of input images to two. Furthermore, if the second deviation amount is 3.0 or greater but less than 4.0, the result evaluation unit 106 sets the number of input images to three. Furthermore, if the second deviation amount is 4.0 or greater, the result evaluation unit 106 sets the number of input images to four.

[0121] The number of images P1 input to the camera 10 can be changed by, for example, an operator changing the inclination of the calibration chart 20 or the distance of the calibration chart 20 from the camera 10.

[0122] 21 is a diagram showing an example of the installation position of the calibration chart 20 relative to the camera 10. The calibration chart 20 is installed so as to fall within the range of the angle of view of the camera 10. In the example shown in Fig. 21, the calibration chart 20(1) is installed at a position closest to the camera 10 at a distance L1, the calibration chart 20(2) is installed at a position next closest to the camera 10 at a distance L2, and the calibration chart 20(3) is installed at a position farthest from the camera 10 at a distance L3. It is also assumed that the image P1 captured by the camera 10 is equally divided into nine 3 x 3 regions.

[0123] 22A is a diagram showing the position of the calibration chart 20(1) that appears in the entire image P1. The calibration chart 20(1) shown in FIG. 22A is captured across the entire nine divided regions of the image P1. However, because the area where the calibration chart 20(1) appears is small on the left and upper sides of the image P1, there are not enough markers on the calibration chart 20(1). In such positions, it is expected that the second deviation amount will be large.

[0124] 22B is a diagram showing the position of the calibration chart 20(2) that appears off to the left of the image P1. The calibration chart 20(2) shown in FIG. 22B is captured off to the left of the image P1. Therefore, the markers of the calibration chart 20(1) that were missing on the left side of the image P1 shown in FIG. 22A are captured in sufficient quantity.

[0125] 22C is a diagram showing the position of the calibration chart 20(3) that appears near the top of the image P1. The calibration chart 20(3) shown in FIG. 22C was captured near the top of the image P1. As a result, a sufficient number of markers from the calibration chart 20(1) that were missing at the top of the image P1 shown in FIG. 22A are captured.

[0126] <Example of Camera Calibration Method According to Second Embodiment> Fig. 23 is a flowchart showing an example of the procedure of camera calibration processing according to the second embodiment. Note that the camera calibration processing according to the second embodiment also performs the processing of steps S1 to S8 shown in Fig. 17 performed in the first embodiment, and the processing of steps S9 to S12 or S13 shown in Fig. 18. Fig. 23 shows the procedure of processing from step S9 onwards in Fig. 18.

[0127] The processes of steps S9 to S12 in Fig. 23 are the same as those shown in Fig. 18 , and therefore redundant description will be omitted. If it is determined in step S12 in Fig. 18 that the second misalignment amount in the re-divided area is equal to or greater than the first threshold (NO in step S12), the result evaluation unit 106 evaluates whether the number of images P1 corresponding to the second misalignment amount has been input (S21). If it is determined in step S21 that the number of images P1 corresponding to the second misalignment amount has not been input (NO in step S21), the result evaluation unit 106 instructs the misalignment amount calculation unit 105 to change the number of images input for each area according to the weight set for the divided area (step S22). By issuing the instruction in step S22, the process of step S1 in Fig. 17, i.e., the process of photographing the calibration chart, is performed. After the process of step S22, the process returns to step S9 in Fig. 18 and continues.

[0128] The weights for the divided regions are set by the result evaluation unit 106. For example, if the refractive layer 1 is disposed approximately perpendicular to the light beam of the camera 10, the result evaluation unit 106 sets weights for the peripheral (lower left and right) regions of the image P1. If the refractive layer 1 is disposed at an angle to the light beam of the camera 10, the result evaluation unit 106 sets weights for the lower region of the image P1. Furthermore, if the camera 10 is disposed offset to the left or right side, the result evaluation unit 106 sets weights for the region of the image P1 on the opposite side from the position where the camera 10 is disposed. Then, for the divided regions for which weights have been set, a large number of input images P1 are set.

[0129] If it is determined in step S21 that the number of images P1 corresponding to the second shift amount has been input (YES in S21), the correction amount calculation unit 107 calculates the correction amount for each re-divided area (S23). Next, the correction amount calculation unit 107 calibrates the camera using the correction amount calculated in step S23 (step S24). After the processing of step S24, the camera calibration process according to the second embodiment ends.

[0130] In the second embodiment, for a re-divided area where a local change or the like has occurred and the second deviation amount has increased, an instruction is given to increase the number of images P input to the camera 10, and the number of markers captured in that re-divided area also increases. Therefore, according to the second embodiment, it is possible to improve the accuracy of calculation of the correction amount in an area where a local change or the like has occurred, and also improve the accuracy of camera calibration.

[0131] In the second embodiment described above, an example has been given in which an instruction to increase the number of images P1 input to the camera 10 is given after the process of re-dividing the divided area has been performed, but the present invention is not limited to this. The process of re-dividing the divided area may be performed after an instruction to increase the number of images P1 input to the camera 10 has been given.

[0132] Third Embodiment Next, a camera calibration method according to a third embodiment of the present invention, which is performed by the camera calibration device 100, will be described with reference to FIGS. 24 and 25. FIG.

[0133] The configuration of the camera calibration device 100 according to the third embodiment can be the same as that of the camera calibration device 100 according to the first embodiment. The evaluation unit (result evaluation unit 106) of the camera calibration device 100 according to the third embodiment pre-sets a distortion model calculation formula for each divided region based on the second deviation amount calculated during initial setting. If the evaluation unit (result evaluation unit 106) determines that the second deviation amount is equal to or greater than the first threshold, it determines whether the distortion model calculation formula (distortion model calculation formula) corresponds to the second deviation amount. If the evaluation unit (result evaluation unit 106) determines that the distortion model calculation formula does not correspond to the second deviation amount, it changes the distortion model calculation formula set for the divided region to a distortion model calculation formula corresponding to the second deviation amount. Then, the correction amount calculation unit 107 performs the processing according to the first embodiment using the changed distortion model calculation formula. That is, it generates re-divided regions and calculates the correction amount for each generated re-divided region.

[0134] Fig. 24 is a diagram showing an example of setting a distortion model calculation formula according to the first displacement amount. The upper part of Fig. 24 shows the first displacement amount [px] for each divided region, and the lower part of Fig. 24 shows distortion model calculation formulas a to d assigned to each divided region.

[0135] The result evaluation unit 106 assigns the distortion model calculation formula to each divided region. For example, if the first deviation amount calculated in a divided region is less than 1.0, the result evaluation unit 106 assigns calculation formula a to the divided region. If the first deviation amount is 1.0 or more and less than 2.5, the result evaluation unit 106 assigns calculation formula b to the divided region. If the first deviation amount is 2.5 or more and less than 3.5, the result evaluation unit 106 assigns calculation formula c to the divided region. If the first deviation amount is 3.5 or more, the result evaluation unit 106 assigns calculation formula d to the divided region. For calculation formulas a to d shown in the lower part of FIG. 24 , for example, the above-mentioned formulas (1) and (4), or other formulas not shown, can be used.

[0136] <Example of Camera Calibration Method According to Third Embodiment> Fig. 25 is a flowchart showing an example of the procedure of camera calibration processing according to the third embodiment. Note that the processing of steps S1 to S7 in the flowchart shown in Fig. 25 is the same as that shown in Fig. 17, and therefore redundant explanations will be omitted. After the first deviation amount is calculated in step S7 of Fig. 17, the result evaluation unit 106 determines whether the distortion model calculation formula set for the divided region is a calculation formula corresponding to the first deviation amount (step S31).

[0137] If the result evaluation unit 106 determines that the distortion model formula is not a formula corresponding to the first deviation amount (NO in step S31), the process returns to S3 and repeats model generation until a formula corresponding to the first deviation amount is generated. The deviation amount calculation unit 105 then calculates the first deviation amount again using the formula corresponding to the first deviation amount. For example, the result evaluation unit 106 sets a weight for each divided region and instructs the deviation amount calculation unit 105 to change the formula used to calculate the deviation amount for each region in accordance with the weight. This instruction is input to the camera 10, and is then input to the deviation amount calculation unit 105 via the camera 10. The deviation amount calculation unit 105 then recalculates the deviation amount in the region for which the formula change was instructed, using the formula changed in accordance with the instruction.

[0138] For example, if the refractive layer 1 is positioned approximately perpendicular to the light beam from the camera 10, the result evaluation unit 106 sets weights for the peripheral regions (bottom left and right) of the image P1 and issues an instruction to generate a calculation formula that matches this weight. Also, if the refractive layer 1 is positioned tilted with respect to the light beam from the camera 10, a weight is set for the lower region of the image P1 and an instruction to generate a calculation formula that matches this weight is issued. Also, if the camera 10 is positioned offset to the left or right, the result evaluation unit 106 sets weights for the region on the opposite side of the image P1 from where the camera 10 is positioned and issues an instruction to generate a calculation formula that matches this weight.

[0139] On the other hand, in step S31, if the result evaluation unit 106 evaluates that the formula corresponds to the amount of deviation (YES in S31), the process proceeds to step S9 and subsequent steps in Fig. 18. That is, the area deviation amount calculation unit 105 further divides the divided area to create re-divided areas.

[0140] According to the camera calibration device 100 of the third embodiment described above, an appropriate distortion model calculation formula is set for each divided region in accordance with the first amount of deviation. Therefore, even when, for example, the placement position of the camera 10 with respect to the refractive layer 1 is not known, a distortion model calculation formula corresponding to the first amount of deviation is set for the divided region. As a result, regardless of the placement position of the camera 10 with respect to the refractive layer 1, the correction amount is calculated based on the optimal distortion model calculation formula, thereby improving the calculation accuracy of the correction amount.

[0141] The present invention is not limited to the above-described embodiments, and various other applications and modifications are possible without departing from the spirit and scope of the present invention as defined in the claims. For example, the above-described embodiments provide detailed and specific descriptions of the device configurations to clearly explain the present invention, and are not necessarily limited to devices that include all of the described configurations. Furthermore, it is possible to replace some of the configurations of the embodiments described herein with configurations of other embodiments, and it is also possible to add configurations of other embodiments to the configurations of one embodiment. Furthermore, it is also possible to add, delete, or replace other configurations with respect to some of the configurations of each embodiment. Furthermore, the control lines and information lines shown are those considered necessary for the explanation, and not all control lines and information lines are necessarily shown in the actual product. In reality, it is reasonable to assume that almost all components are interconnected.

[0142] REFRACTION LAYER 10 Camera 20, 21, 22, 27 Calibration chart 100 Camera calibration device 101 Marker position detection unit 102 Model generation unit 103 Image position calculation unit 104 Parameter estimation unit 105 Displacement amount calculation unit 106 Result evaluation unit 107 Correction amount calculation unit

Claims

1. A camera calibration device comprising: a deviation amount calculation unit that calculates a first deviation amount between a detected position of an object for calibration, which is detected for each divided area of ​​a predetermined size in an image in which the camera captures an image of the object through a refractive layer, and a first calculated position of the object for each area in an image that the camera can capture without passing through the refractive layer, and a second calculated position of the object for each sub-divided area obtained by subdividing the divided area, which is calculated using a calculation formula of a distortion model that models the camera and the refractive layer, and a second deviation amount between the detected position of the object; an evaluation unit that evaluates the second deviation amount for each sub-divided area and changes the calculation formula of the distortion model in accordance with the evaluation result; and a correction amount calculation unit that calculates a correction amount for each sub-divided area based on the changed calculation formula to correct the detected position of the object to the first calculated position, and calibrates the camera based on the correction amount.

2. The camera calibration device of claim 1, wherein the evaluation unit evaluates the second amount of deviation by determining whether the second amount of deviation is less than a first threshold value, and the correction amount calculation unit calculates the amount of correction for each of the re-divided areas using a calculation formula for the distortion model set for the divided area when the evaluation unit determines that the second amount of deviation is less than the first threshold value.

3. The camera calibration device described in claim 2, wherein, when the evaluation unit determines that the second deviation is equal to or greater than the first threshold and that the size of the re-divided area is equal to or greater than the second threshold, the deviation amount calculation unit further divides the re-divided area to generate sub-sub-divided areas, calculates the second deviation amount for each of the sub-sub-divided areas, and changes the calculation formula for the distortion model based on the second deviation amount.

4. The camera calibration device described in claim 3, wherein when the evaluation unit determines that the second deviation amount is equal to or greater than the first threshold value and the size of the re-divided area is less than the second threshold value, the correction amount calculation unit calculates the correction amount for each re-divided area using the calculation formula of the distortion model set for the divided area.

5. The camera calibration device according to claim 4, wherein the evaluation unit, when determining that the first deviation amount is equal to or greater than a third threshold, changes the calculation formula for the distortion model based on the first deviation amount.

6. The camera calibration device described in claim 4, wherein the evaluation unit pre-sets the number of images that need to be input for each divided area based on the second amount of deviation calculated in the initial setting, and when the evaluation unit determines that the second amount of deviation is equal to or greater than the first threshold, the evaluation unit determines whether the number of images to be input is a number that corresponds to the second amount of deviation, and when the evaluation unit determines that the number is not a number that corresponds to the second amount of deviation, outputs an instruction to input a number of images that corresponds to the second amount of deviation.

7. The camera calibration device according to claim 4, wherein the evaluation unit pre-sets a calculation formula for the distortion model for each of the divided areas based on the second amount of deviation calculated in an initial setting, and when the evaluation unit determines that the second amount of deviation is equal to or greater than the first threshold, the evaluation unit determines whether the calculation formula for the distortion model is a calculation formula corresponding to the second amount of deviation, and when the evaluation unit determines that the calculation formula is not a formula corresponding to the second amount of deviation, the evaluation unit changes the calculation formula for the distortion model set for the divided area to a calculation formula for the distortion model corresponding to the second amount of deviation.

8. The camera calibration device according to claim 5, wherein the calibration object is at least one of a two-dimensional calibration chart and a three-dimensional calibration chart.

9. A camera calibration method comprising the steps of: a deviation amount calculation unit calculating a first deviation amount between a detected position of the object detected for each divided area of ​​a predetermined size in an image in which the camera captures an image of the object for calibration through a refractive layer and a first calculated position of the object for each area in an image that the camera can capture without passing through the refractive layer; and a second deviation amount between a second calculated position of the object for each sub-divided area obtained by subdividing the divided area, the second deviation amount being calculated using a calculation formula of a distortion model that models the camera and the refractive layer, and the detected position of the object; an evaluation unit evaluating the second deviation amount for each sub-divided area and changing the calculation formula of the distortion model in accordance with the evaluation result; and a correction amount calculation unit calculating a correction amount for each sub-divided area based on the changed calculation formula to correct the detected position of the object to the first calculated position, and calibrating the camera based on the correction amount.

Images