Camera calibration device and camera calibration method

Abstract

This camera calibration device includes an amount-of-misalignment calculation unit for calculating the amount of misalignment between a detected position of a target for calibration detected in each area with a predetermined size in an image of the target captured by a camera via a refraction layer and a calculated position of the target calculated for each area of the image that can be captured by the camera without intervention of the refraction layer, an evaluation unit for evaluating the amount of misalignment for each area, and an amount-of-correction calculation unit that calculates the amount of correction for correcting the detected position of the target to the calculated position of the target, for each area according to an evaluation result for the amount of misalignment and calibrates the camera on the basis of the amount of correction for the amount of misalignment calculated for each area.

Description

[Technical Field]

[0001] The present invention relates to a camera calibration device and a camera calibration method. [Background technology]

[0002] There is a growing need to improve safe driving at intersections, with the New Car Assessment Program (NCAP), a representative automobile assessment program, introducing Autonomous Emergency Braking (AEB) at intersections from 2020. For this reason, when a vehicle turns right or left at an intersection, it becomes necessary to detect objects with a wide horizontal angle of view in order to identify pedestrians and other objects around the intersection.

[0003] Generally, cameras are installed in the center of the vehicle interior, either on the left or right side. In this case, the camera recognizes the outside world through the windshield in front of the camera (hereafter referred to as the "refractive layer"). For this reason, the angle of incidence of light rays on the refractive layer becomes large in the wide-angle portion. If the effect of refraction becomes large, the environmental recognition device, which recognizes the environment using images captured by the camera, will be unable to accurately detect objects.

[0004] As a countermeasure to this problem, Patent Document 1 discloses a technology that states, "A first image obtained by capturing an image of a calibration chart without using the windshield is compared with a second image obtained by capturing an image of the calibration chart using the windshield, thereby measuring the pixel shift amount in a first angle of view range." [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2022-109555 Summary of the Invention [Problem to be solved by the invention]

[0006] The technology disclosed in Patent Document 1 obtains the parallax at the center by capturing an image of a calibration chart with a stereo camera, estimates the amount of deviation in the wide-angle portion from the result, and corrects the influence of refraction in the wide-angle portion. Here, the image output by the camera capturing an image of the calibration chart through the windshield will be described with reference to Figure 1.

[0007] FIG. 1 is a diagram showing an example of an image of a calibration chart captured through a windshield. Conventional cameras can capture images of a calibration chart with a horizontal angle of view ranging from −20 degrees to +20 degrees and a vertical angle of view ranging from −20 degrees to +20 degrees. Here, the markers (an example of feature points) on a calibration chart captured on a vehicle without a windshield are represented by a plurality of white dots 200 spaced at approximately equal intervals in the horizontal and vertical directions. On the other hand, the markers on a calibration chart captured on a vehicle with a windshield are represented by a plurality of black dots 210 that are shifted relative to the white dots 200. When the horizontal and vertical angles of view are near 0 degrees (the center of FIG. 1 ), the black dots 210 are positioned approximately the same relative to the white dots 200, with almost no shift. On the other hand, as shown in the lower right of FIG. 1 , as the horizontal and vertical angles of view increase, the black dots 210 shift, resulting in a difference d1 in the vertical direction. Although not shown, a difference between the black dots 210 and the white dots 200 also occurs in the vertical direction.

[0008] The technology disclosed in Patent Document 1 directly measures the amount of pixel shift in the vertical direction at a predetermined angle of view, for example, a horizontal angle of view of -20 degrees to +20 degrees, depending on the presence or absence of a refractive layer. This measurement requires capturing an image of the calibration chart when the refractive layer is not mounted on the vehicle, and then capturing an image of the calibration chart when the refractive layer is mounted on the vehicle, which makes simply capturing an image of the calibration chart extremely time-consuming.

[0009] Furthermore, special equipment and tools are required when installing or removing the refractive layer on a vehicle. Furthermore, affine transformation processing information without the refractive layer is required for parallax calculations, and special equipment is required to obtain this information. This makes it difficult to calibrate the camera.

[0010] The present invention has been made in view of the above circumstances, and has as its object to facilitate the calibration of a camera. [Means for solving the problem]

[0011] The camera calibration device according to the present invention is configured to provide a calibration image for each region of a predetermined size in an image of an object for calibration captured by a camera through a refractive layer. a model generation unit that generates a model including the refraction layer based on the set weights and a plurality of images of the calibration object captured through the refraction layer; For each detected object position and the area of ​​the image that the camera can capture without passing through the refractive layer Using the modeled refractive layer calculation formula A displacement calculation unit calculates the displacement between the calculated position of the object and the calculated position of the object. By setting weights, it is possible to determine whether the number of images corresponding to the amount of deviation has been input for each region. evaluation If it is determined that the number of images inputted is not appropriate for the amount of deviation, the camera is instructed to change the number of images inputted to the deviation amount calculation unit for each area. an evaluation unit; If the evaluation unit evaluates that the number of images corresponding to the amount of deviation has been input, The apparatus is provided with a correction amount calculation unit that calculates, for each region, a correction amount for correcting the detected position of the object to the calculated position of the object, and calibrates the camera based on the correction amount for the deviation calculated for each region. The camera calibration device according to the present invention also includes a model generation unit that models the object for calibration, including the refraction layer, based on weights that are set for each region of a predetermined size in an image in which the camera captures the object for calibration through the refraction layer and multiple images in which the object for calibration is captured through the refraction layer; a deviation calculation unit that calculates the amount of deviation between the detected position of the object detected for each region and the calculated position of the object calculated using a formula for the refraction layer modeled for each region of the image that the camera can capture without passing through the refraction layer; an evaluation unit that sets a weight for each region, evaluates for each region whether the formula is appropriate for the deviation amount, and if it is determined that the formula is not appropriate for the deviation amount, instructs the camera to change the formula used by the deviation calculation unit for each region; and if the evaluation unit determines that the formula is appropriate for the deviation amount, calculates a correction amount for each region to correct the detected position of the object to the calculated position of the object, and calibrates the camera based on the correction amount for the deviation calculated for each region. The camera calibration device according to the present invention also includes a model generation unit that generates a model including the refraction layer based on weights set for each region of a predetermined size of an image of the calibration object captured by the camera through the refraction layer and multiple images of the calibration object captured through the refraction layer; a deviation amount calculation unit that calculates the amount of deviation between the detected position of the object detected for each region and the calculated position of the object calculated using a calculation formula of the refraction layer modeled for each region of the image that the camera can capture without passing through the refraction layer; and a weight set for each region, evaluates for each region whether a number of images corresponding to the amount of deviation have been input, and determines whether the calculation formula is in accordance with the amount of deviation. and an evaluation unit that evaluates the number of images input for each region and, if it determines that the number of images corresponding to the amount of deviation has not been input, instructs the camera to change the number of images input to the deviation amount calculation unit for each region, and, if it determines that the calculation formula is not corresponding to the amount of deviation, instructs the camera to change the calculation formula used by the deviation amount calculation unit for each region; and, if the evaluation unit determines that the number of images corresponding to the amount of deviation has been input and that the calculation formula is corresponding to the amount of deviation, calculates a correction amount for each region to correct the detected position of the object to the calculated position of the object, and calibrates the camera based on the correction amount for the amount of deviation calculated for each region. [Effects of the Invention]

[0012] According to the present invention, the amount of correction is calculated according to the amount of deviation for each area, which makes it possible to easily calibrate the camera. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 10 is a diagram showing an example of an image of a calibration chart captured through a windshield. [Figure 2] 1 is a block diagram showing a schematic configuration example of a camera calibration device according to a first embodiment of the present invention. [Figure 3A] FIG. 1 is a perspective view showing an example of a two-dimensional calibration chart on which circular markers are arranged according to a first embodiment of the present invention. [Figure 3B] 1 is a perspective view showing an example of a two-dimensional calibration chart on which square markers are arranged according to a first embodiment of the present invention. FIG. [Figure 3C]FIG. 1 is a perspective view showing an example of a three-dimensional calibration chart on which circular markers are arranged according to a first embodiment of the present invention. [Figure 4A] 3 is a horizontal cross-sectional view of a refractive layer in a state where a camera is mounted on an automobile according to the first embodiment of the present invention. FIG. [Figure 4B] 3 is a vertical cross-sectional view of a refractive layer in a state where a camera is mounted on an automobile according to the first embodiment of the present invention. FIG. [Figure 5] FIG. 2 is a diagram illustrating an example of a sub-pixel according to the first embodiment of the present invention. [Figure 6] 4 is a diagram showing how a model generating unit according to the first embodiment of the present invention obtains a ray displacement from an approximate curved surface of a refractive layer modeled. FIG. [Figure 7] FIG. 2 is a diagram showing an example of a refractive layer whose curved surface is approximated by a quadratic polynomial according to the first embodiment of the present invention. [Figure 8] 3A and 3B are diagrams showing examples of markers that appear in an image obtained by capturing a calibration chart on which black dots are arranged in a grid pattern according to the first embodiment of the present invention. [Figure 9] 1 is a diagram showing a state in which a calibration chart according to a first embodiment of the present invention is imaged by a camera. [Figure 10] 5 is a diagram showing the relationship between the detected position and the calculated position of a marker on a calibration chart shown in an image according to the first embodiment of the present invention. FIG. [Figure 11] FIG. 4 is a diagram showing the amount of deviation for each divided region according to the first embodiment of the present invention. [Figure 12] 1 is a flowchart illustrating an example of a camera calibration method according to a first embodiment of the present invention. [Figure 13] 1 is a block diagram showing an example of the hardware configuration of a computer according to a first embodiment of the present invention. [Figure 14] 10 is a flowchart illustrating an example of a camera calibration method according to a second embodiment of the present invention. [Figure 15] FIG. 10 is a diagram showing how the number of input images is instructed according to the amount of deviation according to the second embodiment of the present invention. [Figure 16]FIG. 10 is a diagram showing a position where a calibration chart according to a second embodiment of the present invention is installed. [Figure 17A] FIG. 10 is a diagram showing the position of a calibration chart that appears in the entire image according to the second embodiment of the present invention. [Figure 17B] FIG. 10 is a diagram showing the position of a calibration chart that appears to be shifted to the left side of an image according to a second embodiment of the present invention. [Figure 17C] FIG. 10 is a diagram showing the position of a calibration chart that appears near the top of an image according to a second embodiment of the present invention. [Figure 18] 10 is a flowchart illustrating an example of a camera calibration method according to a third embodiment of the present invention. [Figure 19] FIG. 11 is a diagram showing how the number of input images is instructed according to the amount of displacement according to the third embodiment of the present invention. [Figure 20] 10 is a flowchart illustrating an example of a camera calibration method according to a fourth embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functions or configurations are designated by the same reference numerals, and redundant description will be omitted. The present invention is applicable to, for example, a computing device for vehicle control capable of communicating with an on-board ECU (Electronic Control Unit) for an Advanced Driver Assistance System (ADAS) or Autonomous Driving (AD).

[0015] [First embodiment] <Example of the outline of the camera calibration device> 2 is a block diagram showing a schematic configuration example of a camera calibration device 100 according to a first embodiment of the present invention. The camera calibration device 100 calibrates a camera 10 by executing the camera calibration method according to the first embodiment.

[0016] The camera calibration device 100 is configured to include, for example, a marker position detection unit 101, a model generation unit 102, an image position calculation unit 103, a parameter estimation unit 104, a deviation amount calculation unit 105, a result evaluation unit 106, and a correction amount calculation unit 107.

[0017] The camera 10 includes a lens and an image sensor (not shown). The camera 10 may be either a monocular camera or a stereo camera. The camera 10 captures an image of an object through the lens with the image sensor to obtain an image P1. The image P1 is input to the camera calibration device 100 and used as an input image.

[0018] <Example of a calibration chart> 3A and 3B, a calibration chart, which is an example of an object captured by the camera 10, will be described. The calibration object is at least one of a two-dimensional calibration chart and a three-dimensional calibration chart. FIG. 3A is a perspective view showing an example of a two-dimensional calibration chart 20 on which circular markers are arranged. FIG. 3B is a perspective view showing an example of a two-dimensional calibration chart 21 on which square markers are arranged. FIG. 3C is a perspective view showing an example of a three-dimensional calibration chart 22 on which circular markers are arranged.

[0019] The calibration chart is configured with markers such as squares or circles arranged at equal intervals in two or three dimensions. In the following description, the calibration chart 20 will be used as an example.

[0020] The markers placed on the calibration chart 20 are often placed at equal intervals on a flat, non-flexible surface to simplify the calculations used for calibration in the camera calibration device 100 and to simplify the manufacture of the calibration chart 20. However, as long as the camera calibration device 100 can accurately determine the positional relationship of each marker in advance, there is no need to place any restrictions on the spacing or shape of the markers. The worker moves while holding the calibration chart 20. Then, an image of the calibration chart 20 is captured according to the position to which the calibration chart 20 has moved.

[0021] Next, the positional relationship between the refractive layer 1 and the camera 10 will be described with reference to Figures 4A and 4B. This calibration method includes a process in which the camera 10 captures an image of the calibration chart 20 through the refractive layer 1 to obtain an image P1. The refractive layer 1 is assumed to be, for example, the windshield of an automobile, but it may also be the rear window of an automobile, a transparent resin part, or the like.

[0022] <Relationship between refractive layer and light rays> 4A and 4B are schematic diagrams showing the relationship between the refractive layer 1 and light rays. Fig. 4A is a horizontal cross-sectional view of the refractive layer 1 when the camera 10 is mounted on a car. Fig. 4B is a vertical cross-sectional view of the refractive layer 1 when the camera 10 is mounted on a car.

[0023] 4A, when the refractive layer 1 is located in front of the camera 10 and the front of the refractive layer 1 has a convex shape, the influence of the refractive layer 1 is small for a light ray R11 near the front. On the other hand, the influence of the refractive layer 1 is large for a light ray R12 that enters the camera 10 from the wide-angle portion in the horizontal direction.

[0024] 4B, when the refractive layer 1 having a convex shape facing forward is located in front of the camera 10 and the front side of the refractive layer 1 is tilted toward the camera 10, the influence of the refractive layer 1 is small for the light ray R21 near the front. On the other hand, the influence of the refractive layer 1 is large for the light ray R22 that enters the camera 10 from below in the vertical direction.

[0025] Returning to the explanation of Figure 2. The marker position detection unit 101 shown in FIG. 2 detects the markers of the calibration chart 20 that appear in the image P1.

[0026] Although detailed description will be omitted as this is not the essence of the present invention, the marker position detection unit 101 has the function of detecting markers on the calibration chart 20 appearing in the image P1 using methods such as the Hough transform, corner detection, and brightness centroid calculation, and detecting in sub-pixel units at which pixel on the image P1 the detected marker is located.

[0027] <Example of pixels and sub-pixels> A sub-pixel is a pixel that is a unit of one pixel or less, and the pixels of image P1 have integer values. Here, sub-pixels will be described with reference to FIG. Fig. 5 is a diagram showing an example of a sub-pixel, in which a plurality of pixels are arranged.

[0028] 5 are arranged in the X-axis and Y-axis directions. Each pixel is, for example, an arrangement of a plurality of pixels 31 whose position coordinates are (2,1). The length of this pixel 31 in the X-axis direction is px [mm], and the length in the Y-axis direction is py [mm].

[0029] Here, an example will be described in which the point of interest 32 in the figure is expressed in pixel units and sub-pixel units. In pixel units, the position coordinates of the point of interest 32 are expressed as (4, 4). On the other hand, in sub-pixel units, the position coordinates of the point of interest 32 are expressed as (3.2, 3.8). In this way, in sub-pixel units, the position coordinates of the point of interest 32 can be specified in more detail than in pixel units.

[0030] Returning to the explanation of Figure 2. The model generation unit (model generation unit 102) shown in FIG. 2 models the refraction layer (refraction layer 1) based on weights set for each divided region of image P1 and multiple images captured by a camera (camera 10) of a calibration target and input to a deviation calculation unit (deviation calculation unit 105). Specifically, the model generation unit 102 has a function of modeling and generating internal parameters of the camera 10, lens distortion, translational and rotational components of the calibration chart 20, and the approximated curved surface, translational and rotational components of the refraction layer 1. The modeling performed by the model generation unit 102 makes it possible to calculate where three-dimensional points are theoretically located on the image. Thereafter, the result evaluation unit 106 (described later) evaluates the deviation between the calculated three-dimensional points and points on image P1 captured by the camera 10, thereby confirming the validity of the model and parameter settings and estimation. Here, the modeling of the refraction layer 1 performed by the model generation unit 102 will be described.

[0031] <Example of approximate curved surface of refractive layer and ray displacement> FIG. 6 is a diagram showing how the model generating unit 102 obtains the displacement of a light ray from the approximate curved surface of the refractive layer 1 modeled.

[0032] 6 shows how the camera 10 captures an image of an object including the calibration chart 20 via the refractive layer 1. Here, the curved surface on the front side of the refractive layer 1 as seen from the camera 10 is called the front-side approximated curved surface 41, and the curved surface on the back side of the refractive layer 1 is called the back-side approximated curved surface 42. The refractive layer 1 is configured to be sandwiched between the front-side approximated curved surface 41 and the back-side approximated curved surface 42 with respect to the camera 10, and has a parameter of refractive index n. The height direction Z with respect to the xy plane is expressed as a function of x and y.

[0033] As an example of modeling an approximate curved surface, the model generating unit 102 approximates and models the curved surface of the refractive layer 1 using a quadratic polynomial shown in the following equation (1).

[0034]

number

[0035] p in Eq. (1) 00 ,p 10 ,p 01 ,p 20 ,p 11 ,p 02 are the coefficients of each term, and x 2 ,y 2 is a square term. The subscripts of each P in equation (1) (e.g., 10, 20) represent the order of x in front and the order of y in back. Therefore, p 20 represents the coefficient of the term where x is the square and y is the zeroth power. The near-side approximated curved surface 41 and the far-side approximated curved surface 42 expressed by equation (1) have a translation component T and a rotation component R, and each surface is positioned with respect to the camera 10. However, the method of selecting variables in modeling is not limited to that shown in equation (1), and a higher-order polynomial (for example, a 5th-order polynomial shown in equation (6) described later) may also be used.

[0036] The model generation unit 102 identifies, with respect to the camera 10, the point at which a ray emitted in the direction of the refractive layer 1 is incident on the near-side approximated surface 41 as the intersection point. The model generation unit 102 then calculates the normal (near-front side) at this intersection point based on equation (1) of the modeled approximated surface, the translation component T, and the rotation component R. Because the model generation unit 102 can grasp the incident ray, the intersection point, the normal, and the refractive index with respect to the near-side approximated surface 41 through modeling, it can calculate the exit ray by substituting this information into the equation for Snell's law.

[0037] Furthermore, the emergent ray from the front-side approximated surface 41 is regarded as an incident ray for the back-side approximated surface 42. Then, the model generation unit 102 performs processing to obtain the emergent ray from the back-side approximated surface 42 using Snell's law, similar to the calculation for the front-side approximated surface 41, to obtain the emergent ray from the back-side approximated surface 42. This emergent ray is the emergent ray from the refractive layer 1. Then, the model generation unit 102 obtains the normal (back surface) at the intersection point based on equation (1) of the modeled approximated surface, the translation component T, and the rotation component R.

[0038] As a result, the image position calculation unit 103 can calculate the position of the marker from the difference between the position of the marker in the image P1 captured without the refractive layer 1 (the emitted light ray shown by the dotted line in the figure) and the position of the marker in the image P1 captured through the refractive layer 1 (the emitted light ray shown by the solid line in the figure).

[0039] <Example of approximate curved surface of refractive layer> The refractive layer 1 with an approximated curved surface is expressed by equation (1) in FIG. FIG. 7 is a diagram showing an example of the refractive layer 1 whose curved surface is approximated by a quadratic polynomial.

[0040] Even if the refractive layer 1 has a complex curved surface as shown in Fig. 7, it is possible to approximate the curved surface of the refractive layer 1 using formula (1). In addition, by substituting a specific numerical value for the coefficient p in formula (1), it is also possible to express the refractive layer 1 as a plane. Formula (1)' is expressed by substituting a numerical value for the coefficient p in formula (1).

[0041]

number

[0042] <Example of an approximate plane of a refractive layer> FIG. 8 is a diagram showing an example of markers that appear in an image P1 obtained by capturing a calibration chart 20 on which black dots 51 are arranged in a grid pattern. The center of FIG. 8 is defined as position coordinates of x=0, y=0. When a black dot 51 at a position coordinate (4, -2) is captured by the camera 10, it is displayed at the position of a white dot 52 due to the distortion of the refractive layer 1. Here, the position coordinates of the white dot 52 are defined as (x distortion, y distortion). In this way, by displaying the refractive layer 1 as a plane, the influence of the distortion of the refractive layer 1 becomes clear.

[0043] Returning to the explanation of Figure 2. 2 converts the marker positions on the three-dimensional calibration chart 20 into positions on the image. Here, the process of converting the marker positions will be described with reference to FIGS.

[0044] <Explanation of Zhang's method> FIG. 9 is a diagram showing a state when the calibration chart 20 is imaged by the camera 10. Here, the parameters of the camera 10 are modeled, and Zhang's method in which the camera 10 is considered as a pinhole camera will be described. By this method, an object in the world coordinate system is converted into an image in the camera coordinate system.

[0045] An example of the calibration chart 20 represented in world coordinates is shown above FIG. 9. World coordinates are a coordinate system used in the three-dimensional space of the real world, and the position of the marker is represented in millimeters [mm].

[0046] A plurality of markers are arranged at equal intervals on the calibration chart 20. In world coordinates, the position coordinates of each marker represented by a black dot on the calibration chart 20 are represented by the row number and column number attached to Mx and My with the upper left of the calibration chart 20 as the origin. For example, since the marker 25 on the calibration chart 20 is at the position of the 3rd row and 6th column in world coordinates, the position coordinates (Mx 36 , My 36 ) are represented.

[0047] A state where the calibration chart 20 is imaged by the camera 10 is shown below FIG. 9. Here, a method of converting the marker position of the calibration chart in three dimensions into the position of the image coordinates on the image is shown. Image coordinates are a coordinate system used in the image, and the position of the image is represented in pixels [px].

[0048] The calibration chart 20 is installed by rotating or translating with respect to the camera 10. Here, the rotation of the calibration chart 20 is represented by "R", and the translation of the calibration chart 20 is represented by "t". And the position coordinates of the marker 25 on the calibration chart 20 are P ij (X ij , Y ij , Z ij ) are represented. The position of the camera 10 in camera coordinates is represented by (Xcamera, Ycamera, Zcamera). The unit of camera coordinates is mm.

[0049] The position of any point in the image P1 captured by the camera 10 is specified by the image coordinates. Here, the position of the origin (0,0) of the image coordinates is defined as the optical center through which the Z axis (Zcamera) indicating the optical axis of the camera 10 passes, and is called the principal point (Cx, Cy). The marker 25 of the calibration chart 20 captured by the camera 10 is on a straight line 26 that passes through the marker 28 of the calibration chart 27 shown in the image P1. Furthermore, the coordinates of any position on the image coordinates can be expressed as (x ij ,y ij On the other hand, the coordinates of any position on the calibration chart 27 imaged through the refractive layer 1 are expressed as (u ij ,v ij )

[0050] Therefore, the model generation unit 102 models the distortion of the lens of the camera 10 and sets parameters used in the following equations (2) and (3). The equation (2) is a set of coordinates P ij (X ij ,Y ij ,Z ij ) is rotated (R) and translated (t) to indicate where in the three-dimensional space the marker will be placed. The calibration chart parameters are as follows: 2D chart plane coordinates: Mx ij ,My ij Rotation component: R ·Translational component: T

[0051]

number

[0052] For example, the term labeled "external parameters" in equation (2) represents the influence of rotation (R) and translation (t) on the calibration chart 20. Also, the term labeled "two-dimensional chart plane coordinates" in equation (2) represents the position of the marker in world coordinates shown in the upper part of FIG.

[0053] The following equation (3) is the three-dimensional point (X ij ,Y ij ) is used to determine the position in image P1 where the object appears. Here, the coordinate system of image P1 is also called image coordinates.

[0054]

number

[0055] For example, the term labeled "image coordinates" on the left side of equation (3) represents the position in image coordinates of the marker 28 of the calibration chart 27 shown in image P in Fig. 9. The term labeled "internal parameters" on the right side of equation (3) represents camera parameters. The camera parameters include the following: ·Focal length: f Focal length in the x direction: f x Focal length in the y direction: f y Optical center (principal point) coordinates: c x , c y Pixel size: x , p y Shear modulus: s

[0056] where f x =f / p x and f y =f / p y is. Pixel size p x , p y represents the size of a pixel. The shear coefficient s is calculated from the pixel tilt α of the CMOS sensor, which is the image receiving part of the camera, using a predetermined formula (f x Represents the value obtained by the formula (tan α).

[0057] Using such an equation, it is possible to obtain the position of a marker that can be captured by the camera 10 without passing through the refractive layer 1, from an image captured by the camera 10 through the refractive layer 1. The following equation (4) is an example of a camera lens distortion model generated by the model generation unit 102.

[0058]

number

[0059] In equation (4), p1 and p2 represent coefficients in the lens distortion model. Furthermore, x and y in equation (4) represent coordinates on the image when a distortion-free camera (pinhole camera) is used. In reality, a discrepancy occurs between the x and y coordinates due to lens distortion of camera 10. The coordinates in this case are expressed as (x distortion present, y distortion present).

[0060] The image position calculation unit 103 converts the position of each marker on the calibration chart 20 into a position on the image by calculation, by setting each parameter for the model of the refractive layer 1 and lens distortion generated by the model generation unit 102.

[0061] <Relationship between detected and calculated marker positions> 10 is a diagram showing the relationship between the detected positions and calculated positions of the markers on the calibration chart 20 that appear in the image. In FIG. 10, the positions of the markers on the calibration chart 20 are represented by "X."

[0062] The position of the marker on the calibration chart 20 captured by the camera 10 is detected as a detected position Q in the image P1 shown in FIG. ij (u ij ,v ij ) On the other hand, the image position calculation unit 103 shown in FIG. 2 calculates the position of the marker based on the formulas (2) and (3) to obtain the calculated position P' ij (x ij ,y ij Then, the image position calculation unit 103 calculates the calculated positions P' ij (x ij ,y ij) is calculated.

[0063] The parameter estimation unit 104 compares the detection result of the marker detected from the image P1 by the marker position detection unit 101 with the calculation result of the marker calculated by the image position calculation unit 103, and calculates the amount of deviation of the marker. Then, the parameter estimation unit 104 estimates parameters that minimize the amount of deviation. When the parameter estimation unit 104 estimates the parameters, the following equation (5) is used.

[0064]

number

[0065] Q in equation (5) ij and P' ij represents the following position coordinates: Position coordinates of the marker detected by the marker position detection unit 101: Q ij (u ij , v ij ) Position coordinates of the marker calculated by the image position calculation unit 103: P' ij (x ij , y ij ) S(Rt) in equation (5) is the sum of the distances between all detected points and calculated points. The parameter estimation unit 104 then estimates parameters that minimize the sum of the distances between each detected position and calculated position.

[0066] 2 has a function of creating a correction table (not shown) that indicates the positions at which points on image P1 would appear if there were no influence from the distortion of the refractive layer 1 or the lens, from each parameter value calculated by the model generation unit 102 and the parameter estimation unit 104. This correction table is used to simultaneously perform geometric correction to correct the distortion of the lens and correction of the influence of the refractive layer 1 on the image, using only image P1 captured through the refractive layer 1.

[0067] The deviation amount calculation unit 105 has a function of dividing the image P1 into several regions and calculating the amount of deviation that occurs for each of the regions depending on the presence or absence of distortion of the refractive layer 1 or the lens. It is desirable that all of the divided regions have the same size. For this reason, the deviation amount calculation unit (deviation amount calculation unit 105) calculates the amount of deviation between the detected position of the object detected for each region of a predetermined size in an image in which the camera (camera 10) captures the calibration object through the refractive layer (refractive layer 1), and the calculated position of the object calculated for each region of the image that the camera (camera 10) can capture without passing through the refractive layer (refractive layer 1).

[0068] <Example of calculating the amount of deviation> Here, how the image P1 is divided and the amount of deviation is calculated will be described with reference to FIG. 11 is a diagram showing the amount of deviation for each region obtained by dividing image P1. Image P1 is divided into equal vertical and horizontal intervals. A deviation amount calculation unit (deviation amount calculation unit 105) calculates the calculated position of the object using a calculation formula for the modeled refractive layer (refractive layer 1).

[0069] A surface located at an appropriate distance in front of the camera 10 and directly facing the lens surface of the camera 10 is defined as an infinite plane 61. Rectangular areas are shown on the infinite plane 61 according to the sizes of the areas into which the image P1 is divided. As an example, a white point 63 passing through the center of each area is set. The deviation amount calculation unit 105 calculates a black point 64 that corresponds to the white point 63 set in each area and is affected by the distortion of the refractive layer 1 and the lens.

[0070] Next, the deviation amount calculation unit 105 calculates where the white points 63 and black points 64 on the infinite plane 61 are located on the image coordinates 62. The image coordinates 62 indicate white points 65 and black points 66 that correspond to the white points 63 and black points 64 on the infinite plane 61. The deviation amount calculation unit 105 then calculates the distance "deviation amount 67" between the white points 65 and black points 66 for each region.

[0071] The evaluation unit (result evaluation unit 106) evaluates the amount of deviation (amount of deviation 67) for each region calculated by the deviation amount calculation unit 105. Here, the evaluation unit (result evaluation unit 106) sets a weight for each region and evaluates whether or not the amount of deviation has been calculated according to the weight for each region. If the amount of deviation has not been calculated according to the weight, the evaluation unit (result evaluation unit 106) issues an instruction to cause the amount of deviation to be calculated according to the weight. On the other hand, if the amount of deviation has been calculated according to the weight, the evaluation unit (result evaluation unit 106) outputs the amount of deviation to the correction amount calculation unit. For example, if the evaluation result of the amount of deviation is poor, the result evaluation unit 106 instructs the camera 10 to re-photograph the calibration chart 20 or outputs a message prompting the calibration chart 20 to be moved.

[0072] 4A and 4B, assuming that the camera 10 is mounted inside a typical passenger car, it is assumed that the refractive layer 1 (for example, the windshield) in front of the camera 10 is tilted at about 30 degrees relative to the camera 10, and that the camera 10 is installed in the center. The amount of deviation for each region calculated by the deviation amount calculation unit 105 differs depending on the position of the region within the image P1.

[0073] For example, if the focus on the marker is incorrect and the marker is blurred, or if the calibration chart 20 is poorly positioned and the reflected light from the calibration chart 20 is captured, the deviation amount calculation unit 105 cannot accurately determine the deviation amount, and the result evaluation unit 106 evaluates the deviation amount evaluation result as poor. Therefore, the result evaluation unit 106 can instruct the camera 10 to re-photograph the calibration chart 20 or display a message urging the calibration worker to move the calibration chart 20.

[0074] Furthermore, it is expected that the amount of deviation in the four corners, wide-angle portion, and left and right lower portions of image P1 captured by camera 10 will be greater than the amount of deviation in the center and upper portion of image P1. Therefore, result evaluation unit 106 can assign weights to areas where the amount of deviation is greater. Then, for weighted areas, as will be described in the embodiment below, the result evaluation unit 106 can instruct camera 10 to increase the number of images P1 of the area to be input to deviation amount calculation unit 105 or change the calculation formula.

[0075] If the result evaluation unit 106 evaluates the amount of deviation as acceptable, it calculates a correction amount for each region to correct the detected position of the object to the calculated position of the object according to the evaluation result of the amount of deviation, and calibrates the camera (camera 10) based on the correction amount for the amount of deviation calculated for each region. For example, the correction amount calculation unit 107 calculates a correction amount for moving the displaced black dot 66 in the image coordinates 62 to the position of the white dot 65 in the center of each region according to the amount of deviation 67 calculated by the deviation amount calculation unit 105. After calculating the correction amount, the camera 10 is calibrated by performing a process to correct the position of the image captured by the camera 10 for each region using the calculated correction amount. Therefore, the image output from the camera calibration device 100, with the influence of the refractive layer 1 removed, is input to a downstream environment recognition device (not shown) or the like, and is used to recognize the environment around the vehicle.

[0076] <Example of camera calibration method according to the first embodiment> Next, a camera calibration method according to the first embodiment performed by the camera calibration device 100 will be described with reference to FIG. FIG. 12 is a flowchart illustrating an example of a camera calibration method according to the first embodiment.

[0077] First, the camera calibration device 100 captures an image of the calibration chart 20 through the refractive layer 1 (S1). Next, the marker position detection unit 101 detects the positions of the markers from the image P1 output from the camera 10 (S2). Next, the model generation unit 102 generates the above-mentioned model (S3).

[0078] Next, the image position calculation unit 103 calculates the image position from the position of each marker on the calibration chart (S4). The parameter estimation unit 104 estimates parameters that minimize the sum of the distances between the detected and calculated positions of the markers (S5).

[0079] Next, the deviation amount calculation unit 105 calculates the deviation amount for each divided region of the image P1 (S6). The result evaluation unit 106 evaluates the deviation amount for each divided region (S7). The result evaluation unit 106 then determines whether the evaluation result is good or not (S8). If the evaluation result is not good (NO in S8), the result evaluation unit 106 instructs the camera 10 to re-image the calibration chart 20, and the process returns to step S1. If the evaluation result is good (YES in S8), the correction amount calculation unit 107 calculates the correction amount for each divided region (S9), and the process ends.

[0080] <Example of computer hardware configuration> Next, the hardware configuration of the computer 70 that constitutes the camera calibration device 100 will be described. 13 is a block diagram showing an example of the hardware configuration of the calculator 70. The calculator 70 is an example of hardware used as a computer that can operate as the camera calibration device 100 according to this embodiment. The camera calibration device 100 according to this embodiment realizes a camera calibration method in which the functional blocks of the camera calibration device 100 shown in FIG. 2 cooperate with each other by causing the calculator 70 (computer) to execute a program.

[0081] The computer 70 includes a CPU (Central Processing Unit) 71, a ROM (Read Only Memory) 72, and a RAM (Random Access Memory) 73, each connected to a bus 74. The computer 70 further includes a non-volatile storage 75 and a network interface 76.

[0082] The CPU 71 reads out program code of software that realizes each function according to this embodiment from the ROM 72, loads it into the RAM 73, and executes it. Variables, parameters, etc. generated during the calculation processing of the CPU 71 are temporarily written to the RAM 73, and these variables, parameters, etc. are read out by the CPU 71 as appropriate. However, an MPU (Micro Processing Unit) may be used instead of the CPU 71, or the CPU 71 may be used in combination with a GPU (Graphics Processing Unit). The functions of each part of the camera calibration device 100 shown in FIG. 2 are realized by the CPU 71.

[0083] The nonvolatile storage 75 may be, for example, a hard disk drive (HDD), a solid state drive (SSD), a flexible disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, or a nonvolatile memory. The nonvolatile storage 75 stores an operating system (OS), various parameters, and programs for operating the computer 70. The ROM 72 and the nonvolatile storage 75 store programs and data necessary for the CPU 71 to operate, and are used as examples of computer-readable, non-transitory storage media that store programs executed by the computer 70. The nonvolatile storage 75 also stores, for example, an image P1. The nonvolatile storage 75 also stores the model generated by the model generation unit 102, parameters estimated by the parameter estimation unit 104, the deviation amount calculated by the deviation amount calculation unit 105, the deviation amount evaluation result evaluated by the result evaluation unit 106, and the correction amount calculated by the correction amount calculation unit 107.

[0084] The network interface 76 may be, for example, a network interface card (NIC), and various data may be transmitted and received between devices via an in-vehicle local area network (LAN) or dedicated line connected to the terminal of the NIC.

[0085] In the camera calibration device 100 according to the first embodiment described above, a model is generated by detecting marker positions from an image captured by the camera 10 of the calibration chart 20 through the refractive layer 1. The camera calibration device 100 then calculates the positions of the markers on the image and estimates parameters that minimize the deviation between the detected and calculated positions of the markers, thereby creating a correction table that can correct the positions at which points on the image P1 appear when there is no influence from the refractive layer 1 or lens distortion. As a result, the position of an object appearing in the image P1 captured by the camera 10 is converted to a position that is not influenced by the refractive layer 1 or lens distortion using the correction table, allowing an environment recognition device or the like to accurately grasp the position of the object.

[0086] Furthermore, the image P1 input to the camera calibration device 100 from a camera capable of capturing images at a wider angle than conventional cameras is used, for example, to detect pedestrians when a car turns right or left at an intersection. Even in this case, by using the camera calibration method according to the first embodiment, the influence of image shifts due to the refractive layer in the wide-angle area can be suppressed, thereby enabling reliable detection of pedestrians.

[0087] Therefore, in a monocular camera or a stereo camera, geometric image correction can be performed without relying on dedicated equipment or a location, simply by using at least one calibration chart 20. Furthermore, since the correction amount for image P1 is calculated, it is also possible to provide an imaging device and an image correction device that can simulate a situation where there is no refractive layer 1 from image P1 captured through refractive layer 1.

[0088] [Second embodiment] Next, a camera calibration method according to a second embodiment of the present invention, which is performed by the camera calibration device 100, will be described with reference to FIGS.

[0089] The configuration of the camera calibration device 100 according to the second embodiment may be the same as that of the camera calibration device 100 according to the first embodiment. The camera calibration device 100 according to the second embodiment calibrates the camera 10 by changing the number of images P1 input from the camera 10 according to the amount of deviation for each region calculated by the deviation amount calculation unit 105. For example, when configuring the camera 10 using a calibration chart 20 that covers a wide range, it is not possible to cover the entire range with just one image capture, so it is necessary to input multiple images P1 captured at different positions.

[0090] <Example of camera calibration method according to the second embodiment> Fig. 14 is a flowchart showing an example of a camera calibration method according to the second embodiment. Note that in the flowchart shown in Fig. 14, it is assumed that the processes of marker position detection (S2), model generation (S3), image position calculation (S4), and parameter estimation (S5) have already been performed among the steps in the flowchart of the camera calibration method according to the first embodiment shown in Fig. 12, and detailed description thereof will be omitted.

[0091] First, the camera 10 captures an image of the calibration chart 20 (S11). Next, the deviation amount calculation unit 105 calculates the amount of deviation for each divided region (S12). Next, the result evaluation unit 106 evaluates whether or not the number of images P1 corresponding to the amount of deviation has been input (S13). If the calculation of the amount of deviation according to the weight has not been performed, the evaluation unit (result evaluation unit 106) issues an instruction to change the number of images input to the deviation amount calculation unit (deviation amount calculation unit 105) for each region according to the weight set for the region. When this instruction is input to the camera 10, the camera 10 re-images the calibration chart 20. Thereafter, the re-imaged image P1 is input to the deviation amount calculation unit 105, and the amount of deviation is calculated using the number of images P1 set for the region.

[0092] For example, if the refractive layer 1 is disposed approximately perpendicular to the light beam of the camera 10, the result evaluation unit 106 sets a weight to the peripheral regions (bottom left and right) of the image P1, and issues an instruction to increase the number of images P1. Similarly, if the refractive layer 1 is disposed at an angle to the light beam of the camera 10, a weight is set to the lower region of the image P1, and an instruction to increase the number of images P1 is issued. Furthermore, if the camera 10 is disposed biased to the left or right, the result evaluation unit 106 sets a weight to the region of the image P1 opposite the position where the camera 10 is disposed, and an instruction to increase the number of images P1 is issued.

[0093] If the result evaluation unit 106 determines that the number of images P1 corresponding to the amount of misalignment has not been input (NO in S13), the process returns to S11 and repeats the process until the number of images P1 corresponding to the amount of misalignment has been input.

[0094] On the other hand, if the result evaluation unit 106 evaluates that the number of images P1 corresponding to the amount of deviation has been input (YES in S13), the correction amount calculation unit 107 calculates the amount of correction for each divided area (S14) and terminates this process.

[0095] Fig. 15 is a diagram showing how the number of input images according to the amount of displacement is indicated. The upper part of Fig. 15 shows the amount of displacement [px] for each divided area, and the upper part of Fig. 15 shows the number of input images (images P1) required for each divided area.

[0096] The result evaluation unit 106 determines the number of input images required relatively according to the amount of misalignment in each region. The minimum number of input images is one. For example, if the amount of misalignment is less than 1.0, the number of input images is one. On the other hand, if the amount of misalignment is 1.0 or more and less than 3.0, the number of input images is two; if the amount of misalignment is 3.0 or more and less than 4.0, the number of input images is three; and if the amount of misalignment is 4.0 or more, the number of input images is four.

[0097] A necessary condition here is that the image contains a predetermined number of markers on the calibration chart 20 in each region. For example, if an operator changes the tilt of the calibration chart 20 and the camera 10 captures the calibration chart 20 multiple times, a large number of input images are input, which increases the accuracy of the correction amount for each region into which the image P1 is divided. As a result, the camera calibration device 100 can also improve the accuracy of the calibration of the camera 10.

[0098] Here, how the position of the calibration chart 20 is changed and an image is captured will be described with reference to FIG. 16 and FIGS. 17A to 17C. FIG. 16 is a diagram showing the position where the calibration chart 20 is set.

[0099] The calibration chart 20 is placed so that it fits within the angle of view of the camera 10. Here, it is assumed that the calibration chart 20(1) is placed at a distance L1 closest to the camera 10, the calibration chart 20(2) is placed at a distance L2 next closest to the camera 10, and the calibration chart 20(3) is placed at a distance L3 farthest from the camera 10. It is also assumed that the image P1 captured by the camera 10 is divided into nine equal regions.

[0100] FIG. 17A is a diagram showing the position of the calibration chart 20(1) that appears in the entire image P1. 17A is an image of the calibration chart 20(1) captured across all nine divided regions of image P1. However, because the calibration chart 20(1) is captured in a small area on the left and upper sides of image P1, there are not enough markers on the calibration chart 20(1). Therefore, the position of the calibration chart 20 is instructed to the camera 10 by the result evaluation unit 106.

[0101] FIG. 17B is a diagram showing the position of the calibration chart 20(2) that appears to be shifted to the left side of the image P1. The calibration chart 20(2) shown in Figure 17B was captured on the left side of the image P1, so that the markers of the calibration chart 20(1) that were missing on the left side of the image P1 shown in Figure 17A are captured in sufficient quantity.

[0102] FIG. 17C is a diagram showing the position of the calibration chart 20(3) that appears near the top of the image P1. The calibration chart 20(3) shown in Figure 17C was captured near the top of the image P1, so a sufficient number of markers of the calibration chart 20(1) that were lacking in the upper part of the image P1 shown in Figure 17A are captured.

[0103] In this way, by varying the position and inclination of one calibration chart 20 in various ways, the markers on the calibration chart 20 are captured evenly in the image P1. This allows the correction amount calculation unit 107 to calculate the amount of correction for area misalignment more accurately than in the camera calibration method according to the first embodiment, using the positions of the markers detected from each image P1 by the marker position detection unit 101.

[0104] [Third embodiment] Next, a camera calibration method according to the third embodiment of the present invention, which is performed by the camera calibration device 100, will be described with reference to FIGS.

[0105] The configuration of the camera calibration device 100 according to the third embodiment may be the same as that of the camera calibration device 100 according to the first embodiment. The camera calibration device 100 according to the third embodiment calibrates the camera 10 by changing the calculation formula according to the amount of deviation for each region calculated by the deviation amount calculation unit 105. The calculation formula before the change may be the above formula (1) or the above formula (4), or may be both.

[0106] As a specific example, it is assumed that the calculation formula representing at least one of the front-side approximated curved surface 41 and the back-side approximated curved surface 42 shown in Fig. 6 is changed to the following formula (6). For example, the deviation amount calculation unit 105 can change the approximated curved surface of the model shown in formula (4) to the following formula (6).

[0107]

number

[0108] Similarly, the deviation amount calculation unit 105 may change the order of the lens distortion model, or may change to a model combined with another approximated surface. The following equation (7) is an example of another model that the deviation amount calculation unit 105 can change.

[0109]

number

[0110] <Example of camera calibration method according to the third embodiment> Fig. 18 is a flowchart showing an example of a camera calibration method according to the third embodiment. Note that in the flowchart shown in Fig. 18, it is assumed that the processes of marker position detection (S2), image position calculation (S4), and parameter estimation (S5) have already been performed among the steps in the flowchart of the camera calibration method according to the first embodiment shown in Fig. 12, and detailed description thereof will be omitted.

[0111] First, the camera 10 captures an image of the calibration chart 20 (S21). Next, the model generation unit 102 generates a model, that is, generates a calculation formula (S22).

[0112] Next, the deviation amount calculation unit 105 calculates the deviation amount for each divided region (S23). Next, the result evaluation unit 106 evaluates whether the calculation formula corresponds to the deviation amount (S24). If the deviation amount has not been calculated according to the weight, the evaluation unit (result evaluation unit 106) instructs the deviation amount calculation unit (deviation amount calculation unit 105) to change the calculation formula for calculating the deviation amount for each region according to the weight set for the region. This instruction is input to the camera 10, and is then input to the deviation amount calculation unit 105 via the camera 10. Then, the deviation amount calculation unit 105 uses the calculation formula changed in accordance with the instruction to recalculate the deviation amount for the region for which the change in calculation formula was instructed.

[0113] For example, if the refractive layer 1 is disposed approximately perpendicular to the light beam from the camera 10, the result evaluation unit 106 sets weights for the peripheral regions (bottom left and right) of the image P1, and issues an instruction to generate a calculation formula that matches this weight. Similarly, if the refractive layer 1 is disposed tilted with respect to the light beam from the camera 10, the result evaluation unit 106 sets weights for the lower region of the image P1, and issues an instruction to generate a calculation formula that matches this weight. Furthermore, if the camera 10 is disposed offset to the left or right, the result evaluation unit 106 sets weights for the region of the image P1 opposite the position where the camera 10 is disposed, and issues an instruction to generate a calculation formula that matches this weight.

[0114] If the result evaluation unit 106 determines that the formula is not appropriate for the deviation amount (NO in S24), the process returns to S22 and the model generation is repeated until a formula appropriate for the deviation amount is generated. Then, the deviation amount calculation unit 105 calculates the deviation amount using the formula appropriate for the deviation amount.

[0115] On the other hand, if the result evaluation unit 106 evaluates that the calculation formula corresponds to the amount of deviation (YES in S24), the correction amount calculation unit 107 calculates the amount of correction for each divided area (S25), and the process ends.

[0116] Fig. 19 is a diagram showing how the number of input images according to the amount of deviation is specified. The upper part of Fig. 19 shows the amount of deviation [px] for each divided area, and the upper part of Fig. 19 shows the appropriate calculation formula for each divided area.

[0117] The result evaluation unit 106 determines a calculation formula according to the amount of deviation in each region. For example, if the amount of deviation is less than 1.0, calculation formula a is used. On the other hand, if the amount of deviation is 1.0 or more and less than 2.5, calculation formula b is used, if the amount of deviation is 2.5 or more and less than 3.5, calculation formula c is used, and if the amount of deviation is 3.5 or more, calculation formula d is used. The calculation formulas a to d shown here use the above-mentioned formulas (1), (4), etc., or formulas not shown.

[0118] In the camera calibration device 100 according to the third embodiment described above, the calculation formula can be changed arbitrarily. Therefore, by using an appropriate calculation formula selected according to the environment in which the calibration chart 20 is actually imaged, the calculation accuracy of the correction amount can be improved.

[0119] [Fourth embodiment] Next, a camera calibration method according to the fourth embodiment of the present invention, which is performed by the camera calibration device 100, will be described with reference to FIG.

[0120] The configuration of the camera calibration device 100 according to the fourth embodiment may be the same as that of the camera calibration device 100 according to the first embodiment. The camera calibration device 100 according to the fourth embodiment simultaneously performs the camera calibration method according to the second embodiment and the camera calibration method according to the third embodiment. Therefore, calibration is performed by changing the number of input images and changing the calculation formula according to the amount of deviation calculated by the deviation amount calculation unit 105.

[0121] <Example of camera calibration method according to the fourth embodiment> Fig. 20 is a flowchart showing an example of a camera calibration method according to the fourth embodiment. In the flowchart shown in Fig. 20, it is assumed that the processes of marker position detection (S2), image position calculation (S4), and parameter estimation (S5) have already been performed among the steps in the flowchart of the camera calibration method according to the first embodiment shown in Fig. 12, and detailed description thereof will be omitted.

[0122] As described above, the camera calibration method according to the fourth embodiment is a combination of the camera calibration method according to the second embodiment and the camera calibration method according to the third embodiment. If the calculation of the amount of deviation according to the weight has not been performed, the evaluation unit (result evaluation unit 106) issues an instruction to change the number of images input to the deviation amount calculation unit (deviation amount calculation unit 105) for each region according to the weight set for the region, and issues an instruction to the deviation amount calculation unit (deviation amount calculation unit 105) to change the calculation formula used to calculate the amount of deviation for each region. The instruction to change the number of images for each region is input to the camera 10, which causes the camera 10 to re-image the calibration chart 20. Thereafter, the re-imaged image P1 is input to the deviation amount calculation unit 105, and the amount of deviation is calculated using the number of images P1 set for the region. At the same time, the instruction to change the calculation formula is input to the camera 10, which then inputs it to the deviation amount calculation unit 105 via the camera 10. Then, the deviation amount calculation unit 105 uses the calculation formula changed in accordance with the instruction to re-calculate the amount of deviation for the region for which the change of the calculation formula was instructed.

[0123] The processes of steps S31, S33, and S34 shown in Fig. 20 are the same as the processes of steps S11 to S13 shown in Fig. 14. Furthermore, the processes of steps S31 to S33 and S35 shown in Fig. 20 are the same as the processes of steps S21 to S24 shown in Fig. 14.

[0124] Then, in step S35, if it is evaluated that the calculation formula corresponds to the amount of deviation (YES in S35), the correction amount calculation unit 107 calculates the amount of correction for each divided area (S36), and this process ends.

[0125] In the camera calibration device 100 according to the fourth embodiment described above, the number of input images can be changed according to the amount of deviation, and the calculation formula can be changed, thereby enabling more accurate correction of the amount of deviation.

[0126] The present invention is not limited to the above-described embodiments, and it goes without saying that various other applications and modifications are possible without departing from the gist of the present invention as set forth in the claims. For example, the above-described embodiments have described the configuration of the device in detail and specifically in order to clearly explain the present invention, and are not necessarily limited to devices that include all of the described configurations. Furthermore, it is possible to replace part of the configuration of the embodiments described here with the configuration of other embodiments, and it is also possible to add the configuration of one embodiment to the configuration of another embodiment. Furthermore, it is also possible to add, delete, or replace part of the configuration of each embodiment with other configurations. In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0127] 1...refractive layer, 10...camera, 20...calibration chart, 100...camera calibration device, 101...marker position detection unit, 102...model generation unit, 103...image position calculation unit, 104...parameter estimation unit, 105...shift amount calculation unit, 106...result evaluation unit, 107...correction amount calculation unit

Claims

1. a model generation unit that generates a model including the refraction layer based on weights set for each region of a predetermined size in an image of the calibration object captured by a camera through the refraction layer and a plurality of images of the calibration object captured through the refraction layer; and a deviation amount calculation unit that calculates a deviation amount between a detected position of the object detected for each of the regions and a calculated position of the object calculated using a calculation formula of the modeled refraction layer for each of the regions of an image that can be captured by the camera without passing through the refraction layer; an evaluation unit that sets the weight for each of the regions, evaluates for each of the regions whether or not the number of images corresponding to the amount of deviation have been input, and, when it is determined that the number of images corresponding to the amount of deviation have not been input, instructs the camera to change the number of images input to the amount of deviation calculation unit for each of the regions; a correction amount calculation unit that, when the evaluation unit evaluates that the number of images corresponding to the amount of deviation has been input, calculates, for each of the regions, a correction amount for correcting the detected position of the object to the calculated position of the object, and calibrates the camera based on the correction amount for the amount of deviation calculated for each of the regions. Camera calibration equipment.

2. A model generation unit that models the calibration object including the refraction layer based on weights set for each region of a predetermined size in an image of the calibration object captured by a camera through the refraction layer and multiple images of the calibration object captured through the refraction layer; a deviation amount calculation unit that calculates a deviation amount between a detected position of the object detected for each of the regions and a calculated position of the object calculated using a calculation formula of the modeled refraction layer for each of the regions of an image that can be captured by the camera without passing through the refraction layer; an evaluation unit that sets the weight for each of the regions, evaluates for each of the regions whether the calculation formula corresponds to the amount of deviation, and, if it is determined that the calculation formula does not correspond to the amount of deviation, instructs the camera to change the calculation formula used by the deviation amount calculation unit for each of the regions; a correction amount calculation unit that, when the evaluation unit evaluates that the calculation formula corresponds to the amount of deviation, calculates, for each of the regions, a correction amount for correcting the detected position of the object to the calculated position of the object, and calibrates the camera based on the correction amount for the amount of deviation calculated for each of the regions. Camera calibration equipment.

3. A model generation unit that models the object for calibration including the refraction layer based on weights set for each region of a predetermined size in an image of the object for calibration captured by a camera through the refraction layer and multiple images of the object for calibration captured through the refraction layer; a deviation amount calculation unit that calculates a deviation amount between a detected position of the object detected for each of the regions and a calculated position of the object calculated using a calculation formula of the modeled refraction layer for each of the regions of an image that can be captured by the camera without passing through the refraction layer; an evaluation unit that sets the weight for each of the regions, evaluates for each of the regions whether the number of images corresponding to the amount of deviation have been input, evaluates for each of the regions whether the calculation formula is corresponding to the amount of deviation, and if it is determined that the number of images corresponding to the amount of deviation have not been input, instructs the camera to change the number of images input to the deviation amount calculation unit for each of the regions, and if it is determined that the calculation formula is not corresponding to the amount of deviation, instructs the camera to change the calculation formula used by the deviation amount calculation unit for each of the regions; a correction amount calculation unit that calculates, for each of the regions, a correction amount for correcting the detected position of the object to the calculated position of the object when the evaluation unit evaluates that the number of images corresponding to the amount of deviation has been input and when the calculation formula is evaluated to be corresponding to the amount of deviation, and calibrates the camera based on the correction amount for the amount of deviation calculated for each of the regions. Camera calibration equipment.

4. The calibration object is at least one of a two-dimensional calibration chart and a three-dimensional calibration chart. The camera calibration device according to any one of claims 1 to 3.

5. a step of modeling the calibration object including the refraction layer based on weights set for each region of a predetermined size in an image of the calibration object captured by a camera through the refraction layer and a plurality of images of the calibration object captured through the refraction layer; calculating a deviation amount between a detected position of the object detected for each of the regions and a calculated position of the object calculated using a calculation formula of the modeled refraction layer for each of the regions of an image that can be captured by the camera without passing through the refraction layer; setting the weight for each of the regions, evaluating for each of the regions whether or not the number of images corresponding to the amount of deviation have been input, and if it is evaluated that the number of images corresponding to the amount of deviation have not been input, issuing an instruction to the camera to change the number of images input to the amount of deviation calculation unit for each of the regions; a step of calculating, for each of the regions, a correction amount for correcting the detected position of the object to the calculated position of the object when it is evaluated that the number of images corresponding to the amount of deviation has been input; and calibrating the camera based on the amount of correction of the amount of deviation calculated for each of the regions. Camera calibration method.

6. A step of modeling including the refractive layer based on weights set for each region of a predetermined size in an image of an object for calibration captured by a camera through the refractive layer and multiple images of the object for calibration captured through the refractive layer; calculating a deviation amount between a detected position of the object detected for each of the regions and a calculated position of the object calculated using a calculation formula of the modeled refraction layer for each of the regions of an image that can be captured by the camera without passing through the refraction layer; setting the weight for each of the regions, evaluating for each of the regions whether the calculation formula corresponds to the amount of deviation, and if it is evaluated that the calculation formula does not correspond to the amount of deviation, issuing an instruction to the camera to change the calculation formula used by the deviation amount calculation unit for each of the regions; a step of calculating, for each of the regions, a correction amount for correcting the detected position of the object to the calculated position of the object when the calculation formula is evaluated to be in accordance with the amount of deviation; and calibrating the camera based on the amount of correction of the amount of deviation calculated for each of the regions. Camera calibration method.

7. A step of modeling including the refractive layer based on weights set for each region of a predetermined size in an image of an object for calibration captured by a camera through the refractive layer and multiple images of the object for calibration captured through the refractive layer; calculating a deviation amount between a detected position of the object detected for each of the regions and a calculated position of the object calculated using a calculation formula of the modeled refraction layer for each of the regions of an image that can be captured by the camera without passing through the refraction layer; setting the weight for each of the regions, evaluating for each of the regions whether the number of images corresponding to the amount of deviation have been input, and evaluating for each of the regions whether the calculation formula is corresponding to the amount of deviation, and if it is evaluated that the number of images corresponding to the amount of deviation have not been input, instructing the camera to change the number of images input to the deviation amount calculation unit for each of the regions, and if it is evaluated that the calculation formula is not corresponding to the amount of deviation, instructing the camera to change the calculation formula used by the deviation amount calculation unit for each of the regions; a step of calculating, for each of the regions, a correction amount for correcting the detected position of the object to the calculated position of the object when it is evaluated that the number of images corresponding to the amount of deviation has been input and when it is evaluated that the calculation formula corresponds to the amount of deviation; and calibrating the camera based on the amount of correction of the amount of deviation calculated for each of the regions. Camera calibration method.

Images