Camera correction device and camera correction method

Abstract

A camera correction device according to the present invention is provided with: an offset amount calculation unit that calculates an offset amount between a detected position of an object, which is detected for each region of a predetermined size of an image obtained by capturing an image of a correction object via a refractive layer by a camera, and a calculated position of the object; the calculation position of the object is calculated according to each area of an image which can be shot by the camera under the condition that the camera does not penetrate through the refraction layer; an evaluation unit that evaluates the amount of shift for each region; and a correction amount calculation unit that calculates a correction amount for correcting the detected position of the object to the calculated position of the object for each region on the basis of the evaluation result of the offset amount, and corrects the camera on the basis of the correction amount of the offset amount calculated for each region.

Description

Technical Field

[0001] The invention relates to a camera correction device and a camera correction method. Background Art

[0002] In NCAP (New Car Assessment Program), a representative automobile evaluation, there is a growing demand to enhance safe driving at intersections, such as the introduction of AEB (Autonomous Emergency Braking) at intersections from 2020. Therefore, when a car turns left or right at an intersection, it is necessary to detect objects with a wide horizontal field of view to grasp pedestrians and other objects around the intersection.

[0003] Generally speaking, cameras are mostly installed in the center of the left and right sides of the vehicle. In this case, the camera recognizes the outside world through the windshield in front of the camera (hereinafter referred to as the "refractive layer"). Therefore, in the wide-angle part, the incident angle of the light toward the refractive layer is large. When the influence of refraction increases, the environment recognition device that uses the image captured by the camera to recognize the environment can no longer accurately detect the object.

[0004] As a countermeasure, Patent Document 1 discloses a technology described as follows: "A first image obtained by photographing a correction image without a windshield is compared with a second image obtained by photographing the correction image through the windshield, thereby measuring the pixel shift amount within a first field of view angle range." Prior art literature Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2022-109555 Summary of the invention Problem that the invention aims to solve

[0006] Patent document 1 discloses a technique of using a stereo camera to take a calibration image, thereby obtaining the parallax of the central portion, and inferring the offset at the wide-angle portion based on the result, thereby correcting the influence of refraction at the wide-angle portion. Figure 1 The following describes an image that is outputted by a camera photographing a calibration chart through the windshield.

[0007] Figure 1 The diagram shows an example of an image of a calibration chart captured through the windshield. Conventional cameras can capture calibration images within a horizontal field of view range of -20 to +20 degrees and a vertical field of view range of -20 to +20 degrees. Here, the mark (an example of a feature point) of the calibration image captured on a vehicle without a windshield is represented by a plurality of white dots 200 arranged at approximately equal intervals in the vertical and horizontal directions. On the other hand, the mark of the calibration image captured on a vehicle with a windshield is represented by a plurality of black dots 210 offset from the white dots 200. Near the horizontal field of view angle of 0 degrees and the vertical field of view angle of 0 degrees ( Figure 1 The black dot 210 is in the same position relative to the white dot 200, and there is almost no deviation. Figure 1 As shown in the lower right of FIG. 1 , when the horizontal and vertical viewing angles increase, the black dot 210 shifts and a difference d1 is generated in the vertical direction. Although not shown, a difference between the black dot 210 and the white dot 200 is also generated in the vertical direction.

[0008] The technology disclosed in Patent Document 1 directly measures the pixel shift amount in the vertical direction within a predetermined viewing angle, for example, a horizontal viewing angle of -20 degrees to +20 degrees, by the presence or absence of a refractive layer. In this measurement, a calibration chart must be photographed without the refractive layer mounted on the vehicle, and then photographed with the refractive layer mounted on the vehicle, so photographing the calibration chart alone is quite laborious.

[0009] In addition, when the refractive layer is mounted on or removed from the vehicle, dedicated equipment and tools are required. In addition, in order to realize parallax calculation, affine transformation processing information in the state without the refractive layer is required, and obtaining this information also requires dedicated equipment. Therefore, the operation of calibrating the camera is not easy.

[0010] The present invention has been made in view of such a situation, and an object of the present invention is to facilitate calibration of a camera. Technical means of solving problems

[0011] The camera calibration device of the present invention comprises: an offset calculation unit that calculates the offset between a detected position of an object and a calculated position of the object, wherein the detected position of the object is detected for each area of ​​a predetermined size of an image obtained by photographing the object for calibration by a camera through a refractive layer, and the calculated position of the object is calculated for each area of ​​an image that can be photographed by the camera without penetrating the refractive layer; an evaluation unit that evaluates the offset for each area; and a correction amount calculation unit that calculates a correction amount for correcting the detected position of the object to the calculated position of the object for each area based on the evaluation result of the offset, and calibrates the camera based on the correction amount of the offset calculated for each area. Effects of the Invention

[0012] According to the present invention, a correction amount corresponding to the offset amount of each area can be calculated, so that calibration of the camera can be facilitated. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 The diagram shows an example of an image of a calibration chart captured through the windshield. Figure 2 It is a block diagram showing a schematic configuration example of a camera calibration device according to the first embodiment of the present invention. Figure 3A It is a perspective view showing an example of a two-dimensional calibration chart on which circular marks are arranged according to the first embodiment of the present invention. Figure 3B This is a perspective view showing an example of a two-dimensional calibration chart on which square marks are arranged according to the first embodiment of the present invention. Figure 3C This is a perspective view showing an example of a three-dimensional calibration chart on which circular marks are arranged according to the first embodiment of the present invention. Figure 4A This is a horizontal cross-sectional view of the refractive layer in a state where a camera is mounted on the vehicle according to the first embodiment of the present invention. Figure 4B This is a vertical cross-sectional view of the refractive layer in a state where a camera is mounted on the vehicle according to the first embodiment of the present invention. Figure 5 A diagram showing an example of sub-pixels according to the first embodiment of the present invention. Figure 6 This is a diagram showing a state in which light beam displacement is obtained using an approximate curved surface of a refractive layer modeled by a model generation unit according to the first embodiment of the present invention. Figure 7 This is a diagram showing an example of a refractive layer in which a curved surface is approximated by a quadratic polynomial according to the first embodiment of the present invention. Figure 8 This is a diagram showing an example of a mark that appears on an image obtained by photographing a calibration chart in which black dots are arranged in a grid pattern according to the first embodiment of the present invention. Fig. 9 It is a diagram showing a state when a calibration chart is photographed by a camera according to the first embodiment of the present invention. Fig.10 It is a diagram showing the relationship between the detected position and the calculated position of the mark of the calibration chart projected on the video image according to the first embodiment of the present invention. Fig.11 It is a diagram showing the offset amount for each divided area according to the first embodiment of the present invention. Fig.12This is a flowchart showing an example of the camera calibration method according to the first embodiment of the present invention. Fig.13 It is a block diagram showing an example of the hardware configuration of a computer according to the first embodiment of the present invention. Fig.14 This is a flowchart showing an example of a camera calibration method according to the second embodiment of the present invention. Fig.15 It is a diagram showing a state of instructing the number of input images according to the offset amount according to the second embodiment of the present invention. Fig.16 It is a diagram showing the position where a correction chart is provided according to the second embodiment of the present invention. Fig.17A It is a diagram showing the position of the correction chart projected on the entire image according to the second embodiment of the present invention. Fig. 17B It is a diagram showing the position of the correction chart that appears on the left side of the deviated image according to the second embodiment of the present invention. Fig. 17C The diagram shows the position of the correction chart that is projected on the upper side of the image according to the second embodiment of the present invention. Fig.18 This is a flowchart showing an example of a camera calibration method according to the third embodiment of the present invention. Fig.19 It is a diagram showing a state of instructing the number of input images according to the offset amount according to the third embodiment of the present invention. Fig. 20 This is a flowchart showing an example of a camera calibration method according to a fourth embodiment of the present invention. DETAILED DESCRIPTION

[0014] The specific embodiments of the present invention are described below with reference to the accompanying drawings. In this specification and the accompanying drawings, repeated descriptions are omitted by marking the same symbols on the components having substantially the same function or structure. The present invention can be applied, for example, to a computing device for vehicle control that can communicate with an on-board ECU (Electronic Control Unit) for an advanced driver assistance system (ADAS) or autonomous driving (AD).

[0015] [First embodiment] <Schematic Configuration Example of Camera Calibration Device> Figure 21 is a block diagram showing a schematic configuration example of a camera calibration device 100 according to a first embodiment of the present invention. The camera calibration device 100 calibrates the camera 10 by executing the camera calibration method according to the first embodiment.

[0016] As an example, the camera calibration device 100 includes a marker position detection unit 101 , a model generation unit 102 , an image position calculation unit 103 , a parameter estimation unit 104 , an offset calculation unit 105 , a result evaluation unit 106 , and a correction amount calculation unit 107 .

[0017] The camera 10 includes a lens and an image sensor (not shown). The camera 10 may be a monocular camera or a stereo camera. Then, the camera 10 captures an object with the image sensor via the lens and acquires an image P1. The image P1 is input to the camera calibration device 100 and used as an input image.

[0018] 〈An example of a calibration chart〉 Here, reference Figure 3A and Figure 3B A description will be given of a calibration chart as an example of an object captured by the camera 10. The calibration object is at least one of a two-dimensional calibration chart and a three-dimensional calibration chart. Figure 3A To indicate the two-dimensional calibration with circular marks. Fig. 20 A stereogram of an example of . Figure 3B It is a stereoscopic diagram showing an example of a two-dimensional calibration diagram 21 in which square marks are arranged. Figure 3C It is a perspective view showing an example of a three-dimensional calibration diagram 22 in which circular marks are arranged.

[0019] The calibration diagram is composed of marks such as squares or circles arranged at equal intervals in two or three dimensions. Fig. 20 As an example.

[0020] In order to simplify the calculation for calibration in the camera calibration device 100 and to make the calibration Fig. 20 The manufacturing becomes simple, and the calibration Fig. 20 The marks arranged on the camera are mostly arranged at equal intervals on a flat surface without curvature. However, as long as the camera calibration device 100 can accurately grasp the positional relationship of each mark in advance, the interval and shape of the marks are not limited. Fig. 20 Then, according to the calibration Fig. 20 Move the position to shoot for calibration Fig. 20 .

[0021] Next, refer to Figure 4A and Figure 4B The arrangement relationship between the refractive layer 1 and the camera 10 is described below. Fig. 20 The refractive layer 1 is processed to obtain the image P1. For example, a front windshield of a car is assumed as the refractive layer 1, but it may be a rear windshield of a car, a transparent resin part, or the like.

[0022] 〈Relationship between refraction layer and light〉 Figure 4A and Figure 4B Schematic diagram showing the relationship between the refractive layer 1 and the light. Figure 4A It is a horizontal cross-sectional view of the refractive layer 1 in a state where the camera 10 is mounted on the vehicle. Figure 4B It is a vertical cross-sectional view of the refractive layer 1 in a state where the camera 10 is mounted on the vehicle.

[0023] In the image Figure 4A In this case, when the refractive layer 1 is located in front of the camera 10 and the front of the refractive layer 1 is convex, the refractive layer 1 has little influence on the light ray R11 near the front. On the other hand, the refractive layer 1 has a great influence on the light ray R12 incident on the camera 10 from the wide angle portion in the horizontal direction.

[0024] also, Figure 4B Similarly, when the refractive layer 1 having a convex shape in front is located in front of the camera 10 and the front side of the refractive layer 1 is inclined toward the camera 10, the refractive layer 1 has a small influence on the light ray R21 near the front. On the other hand, the refractive layer 1 has a large influence on the light ray R22 incident on the camera 10 from the lower part in the vertical direction.

[0025] Back to Figure 2 Description. Figure 2 The mark position detection unit 101 shown in FIG. 1 detects the correction mark shown in the image P1. Fig. 20 Mark.

[0026] The mark position detection unit 101 is not described in detail because it is not essential to the present invention. However, it has the following functions: detecting the correction point reflected in the image P1 by using methods such as Hough transform, corner point detection, and brightness centroid calculation. Fig. 20 The detected mark is detected in sub-pixel units to detect at which pixel on the image P1 the detected mark is located.

[0027] 〈Examples of pixels and sub-pixels〉 A sub-pixel is a pixel whose unit is less than 1 pixel, and the pixel of the image P1 is an integer value. Figure 5 Sub-pixels will be described. Figure 5 A diagram showing an example of sub-pixels. Figure 5 Shows how to configure multiple pixels.

[0028] Figure 5 The pixels shown are arranged along the X-axis direction and the Y-axis direction. Each pixel is, for example, a plurality of pixels 31 shown by the position coordinates (2, 1). The length of the pixel 31 in the X-axis direction is px [mm], and the length in the Y-axis direction is py [mm].

[0029] Here, an example of expressing the focus point 32 in the figure in pixel units and sub-pixel units is described. In the pixel unit, the position coordinates of the focus point 32 are expressed as (4, 4). On the other hand, in the sub-pixel unit, the position coordinates of the focus point 32 are expressed as (3.2, 3.8). In this way, in the sub-pixel unit, the position coordinates of the focus point 32 can be determined in more detail than in the pixel unit.

[0030] Back to Figure 2 Description. Figure 2 The model generation unit (model generation unit 102) shown in FIG. 1 includes the refractive layer (refractive layer 1) and performs modeling based on the weights set for each region obtained by dividing the image P1 and the plurality of images of the object for calibration captured by the camera (camera 10) and input to the offset calculation unit (offset calculation unit 105). Specifically, the model generation unit 102 has a function of generating the internal parameters of the camera 10, the distortion of the lens, the calibration object, etc., by modeling. Fig. 20 The translation component and rotation component of the refractive layer 1 and the approximate curved surface, translation component and rotation component of the refractive layer 1 are functions. By modeling by the model generation unit 102, it is possible to calculate where the three-dimensional point is theoretically arranged on the image. Thereafter, the result evaluation unit 106 described later evaluates the offset between the calculated three-dimensional point and the point on the image P1 captured by the camera 10, thereby confirming the appropriateness of the setting and inference of the model and parameters. Here, the modeling related to the refractive layer 1 performed by the model generation unit 102 is described.

[0031] 〈Example of approximate curved surface of refractive layer and displacement of light rays〉 Figure 6 1 is a diagram showing a state in which light beam displacement is obtained using the approximate curved surface of the refractive layer 1 modeled by the model generation unit 102 .

[0032] Figure 6 The camera 10 is shown in the figure, taking pictures through the refractive layer 1, including the calibration Fig. 20Here, the curved surface on the near side of the refractive layer 1 as viewed from the camera 10 is referred to as the near side approximate curved surface 41, and the curved surface on the depth side of the refractive layer 1 is referred to as the depth side approximate curved surface 42. Furthermore, the refractive layer 1 is configured to be sandwiched between the near side approximate curved surface 41 and the depth side approximate curved surface 42 with respect to the camera 10, and has a parameter of a refractive index n. Furthermore, Z in the height direction relative to the xy plane is represented by a function of x and y.

[0033] As an example of modeling of an approximate curved surface, the model generation unit 102 approximates the curved surface of the refractive layer 1 using a quadratic polynomial represented by the following equation (1).

[0034] [Formula 1] Z=p 00 +p 10 *x+p 01 *y +p 20 *x 2 +p 11 *x*y+p 02 *y 2 …(1)

[0035] p in formula (1) 00 、p 10 、p 01 、p 20 、p 11 、p 02 is the coefficient of each term, x 2 ,y 2 The subscript numbers (e.g., 10, 20) of each P in formula (1) indicate the degree of x in the front and the degree of y in the back. Therefore, p 20 The coefficients of the terms where x is the square and y is the zeroth power. The near-front approximate surface 41 and the depth-side approximate surface 42 shown in equation (1) have a translation component T and a rotation component R, and each surface is arranged relative to the camera 10. However, the method of selecting variables in modeling is not limited to that shown in equation (1), and a higher-order polynomial (for example, a quintic polynomial shown in equation (6) described later) may also be selected.

[0036] The model generation unit 102 determines the point where the light emitted to the camera 10 in the direction of the refractive layer 1 is incident on the near-front approximate curved surface 41 as the intersection point. Then, the model generation unit 102 obtains the normal line (near-front face) on the intersection point based on the modeled approximate curved surface equation (1) and the translation component T and the rotation component R. The model generation unit 102 can grasp the incident light, the intersection point, the normal line, and the refractive index on the near-front approximate curved surface 41 through modeling, and can obtain the emitted light by substituting this information into the formula of Snell's law.

[0037] In addition, the outgoing light from the near-front approximate curved surface 41 is regarded as the incident light to the depth-side approximate curved surface 42. Therefore, the model generation unit 102 performs processing to obtain the outgoing light from the depth-side approximate curved surface 42 by Snell's law in the same manner as the calculation on the near-front approximate curved surface 41, thereby obtaining the outgoing light from the depth-side approximate curved surface 42. This outgoing light is the outgoing light from the refractive layer 1. Then, the model generation unit 102 obtains the normal (depth surface) at the intersection point based on the equation (1) of the modeled approximate curved surface and the translation component T and the rotation component R.

[0038] As a result, the image position calculation unit 103 can calculate the position of the mark based on the difference between the position of the mark shown in the image P1 captured without the refractive layer 1 (the outgoing light represented by the single-dotted chain line in the figure) and the position of the mark shown in the image P1 captured with the refractive layer 1 (the outgoing light represented by the solid line in the figure).

[0039] 〈Example of approximate curved surface of refractive layer〉 The surface is approximated by the refractive layer 1 and is shown in the following formula (1): Figure 7 To express. Figure 7 This is a diagram showing an example of the refractive layer 1 whose curved surface is approximated by a quadratic polynomial.

[0040] Even if it's like Figure 7 As shown, the refractive layer 1 has a complex curved surface, and the curved surface of the refractive layer 1 can also be approximated by formula (1). In addition, the refractive layer 1 can be represented by a plane by inserting a specific numerical value into the coefficient p in formula (1). Formula (1)' is expressed by inserting a numerical value into the coefficient p of formula (1).

[0041] [Formula 2] Z=1+2*x+3*y+4*x 2 +5*x*y+6*y 2 …(1)'

[0042] 〈Example of a nearly flat surface of a refractive layer〉 Figure 8 The black dots 51 are arranged in a grid pattern for correction. Fig. 20 This figure shows an example of a mark that appears on the image P1 obtained by imaging. Will Figure 8The center of is set as the position coordinates of x=0, y=0. Then, when the black dot 51 at the position coordinates (4, -2) is photographed by the camera 10, it is represented at the position of the white dot 52 due to the distortion of the refractive layer 1. Here, the position coordinates of the white dot 52 are set as (x has distortion, y has distortion). In this way, by representing the refractive layer 1 in a plane, the influence of the distortion of the refractive layer 1 becomes clear.

[0043] Back to Figure 2 Description. Figure 2 The image position calculation unit 103 shown in FIG. Fig. 20 The marker position is transformed to the position on the image. Here, refer to Figures 9 to 11 The process of changing the marker position will be described.

[0044] 〈Explanation of Zhang's Method〉 Fig. 9 The calibration image is captured by the camera 10. Fig. 20 Here, the Zhang method of modeling the parameters of the camera 10 and treating the camera 10 as a pinhole camera is described. By this method, an object in the world coordinate system can be transformed into an image in the camera coordinate system.

[0045] Fig. 9 The upper side shows the calibration data in world coordinates. Fig. 20 The world coordinates are a coordinate system used in the three-dimensional space of the real world, and the position of the marker is expressed in millimeters [mm].

[0046] Calibration Fig. 20 Multiple markers are arranged at equal intervals on the top. In world coordinates, the calibration Fig. 20 The position coordinates of each mark represented by a black dot are used for calibration Fig. 20 The upper left corner is the origin and the row and column numbers of Mx and My are used to represent them. Fig. 20 The marker 25 is located at the 3rd row and 6th column of the world coordinate system, so it is expressed as the position coordinate (Mx 36 ,My 36 ).

[0047] Fig. 9 The lower side shows the calibration image taken by the camera 10. Fig. 20 Here, we will show you how to transform the marker position of the calibration chart in three dimensions into the position of the image coordinates on the image. Image coordinates are a coordinate system used on the image, and the position of the image is expressed in pixels [px].

[0048] Calibration Fig. 20 The camera 10 is set in a rotational or translational manner. Fig. 20 This rotation is indicated as "R" and the correction Fig. 20 This translation is denoted as "t". Fig. 20 The position coordinates of the marker 25 are represented as P ij (X ij ,Y ij ,Z ij ). The position of the camera 10 in the camera coordinate is represented by (Xcamera, Ycamera, Zcamera). The unit of the camera coordinate is mm.

[0049] The position of any point in the image P1 captured by the camera 10 is determined by the image coordinates. Here, the position of the origin (0,0) of the image coordinates is set to the optical center through which the Z axis (Zcamera) representing the optical axis of the camera 10 passes, which is called the principal point (Cx, Cy). Fig. 20 The mark 25 is located on the straight line 26 of the mark 28 of the calibration diagram 27 projected on the image P1. In addition, the coordinates of any position on the image coordinates are expressed as (x ij ,y ij On the other hand, the coordinates of any position in the calibration image 27 photographed through the refractive layer 1 are represented by uv coordinates with u as the horizontal axis and v as the vertical axis (u ij ,v ij ) to indicate.

[0050] Therefore, the model generation unit 102 models the distortion of the lens of the camera 10 and sets the parameters for the following equations (2) and (3). Fig. 20 The position of the marker in relation to the coordinates P ij (X ij ,Y ij ,Z ij ) rotation (R) and translation (t) to determine the three-dimensional position at which the marker is placed. The following parameters are available as parameters of the calibration chart. ·2D graph plane coordinates: Mx ij ,My ij Rotational component: R Translational component: T

[0051] [Formula 3]

[0052] For example, the term marked as external parameters in equation (2) represents the effect of rotation (R) and translation (t) on the correction factor Fig. 20 In addition, the terms marked as the two-dimensional graph plane coordinates in formula (2) represent Fig. 9 The position of the marker in world coordinates is shown on the upper side.

[0053] The following formula (3) is used to find the three-dimensional point (X ij ,Y ij Here, the coordinate system of the image P1 is also referred to as image coordinates.

[0054] [Formula 4]

[0055] For example, the term labeled as image coordinates on the left side of equation (3) represents Fig. 9 The position of the mark 28 in the calibration diagram 27 shown in the image P in the image coordinates is shown. The terms marked as internal parameters on the right side of the equation (3) represent the camera parameters. The following parameters are included in the camera parameters. Focal length: f Focal length in the x direction: f x Focal length in the y direction: f y Coordinates of the optical center (principal point): c x ,c y Pixel size: p x 、p y Shear modulus: s

[0056] Here, f x =f / p x , f y =f / p y . Pixel size p x 、p y Indicates the size of pixels. The shear modulus s is expressed by a predetermined formula (f x tanα) is the value obtained.

[0057] By using such a formula, the position of the mark that can be photographed by the camera 10 without passing through the refractive layer 1 can be obtained using the image photographed by the camera 10 through the refractive layer 1. As an example of the lens distortion model of the camera generated by the model generation unit 102, the following formula (4) is shown.

[0058] [Formula 5]

[0059] p1 and p2 in formula (4) represent coefficients in the lens distortion model. In addition, x and y in formula (4) represent coordinates on the image when a distortion-free camera (pinhole camera) is used. In fact, the distortion of the lens of the camera 10 causes the x coordinate and the y coordinate to shift. The coordinates at this time are expressed as (x is distorted, y is distorted).

[0060] The image position calculation unit 103 sets various parameters for the model of the refractive layer 1 and the lens distortion generated by the model generation unit 102, thereby calculating the correction value. Fig. 20 The positions of each marker on the image are transformed into positions on the image.

[0061] 〈Relationship between the detected and calculated positions of the marker〉 Fig.10 To indicate the correction of the image projection Fig. 20 A graph showing the relationship between the detected position and the calculated position of a marker. Fig.10 In the table, “×” indicates the correction Fig. 20 The location of the marker.

[0062] The calibration image captured by the camera 10 Fig. 20 The position of the marker is Fig.10 The image P1 shown is represented as the detection position Q ij (u ij ,v ij ).on the other hand, Figure 2 The image position calculation unit 103 shown in FIG. 1 calculates the position of the marker according to equations (2) and (3), thereby obtaining the calculated position P′. ij (x ij ,y ij ). In addition, the image position calculation unit 103 calculates the correction Fig. 20 Calculate the position P' of all markers ij (x ij ,y ij ) calculation.

[0063] The parameter estimation unit 104 compares the detection result of the marker detected by the marker position detection unit 101 from the image P1 with the calculation result of the marker calculated by the image position calculation unit 103, and calculates the offset of the marker. Then, the parameter estimation unit 104 estimates a parameter such that the offset is minimized. The parameter estimation unit 104 uses the following formula (5) when estimating the parameter.

[0064] [Formula 6] S(Rt)=∑ ij ||Q ij -P′ ij ||…(5)

[0065] Q in formula (5)ij and P' ij Indicates the following location coordinates. The position coordinates of the marker detected by the marker position detection unit 101: Q ij (u ij ,v ij ) ·The position coordinates of the marker calculated by the image position calculation unit 103: P′ ij (x ij ,y ij ) S(Rt) in equation (5) is the sum of the distances between all detected points and the calculated points. Then, the parameter estimation unit 104 estimates a parameter that minimizes the sum of the distances between each detected position and the calculated position.

[0066] Figure 2 The correction amount calculation unit 107 shown has the function of creating a correction table (not shown) indicating the position where the point on the image P1 appears when there is no influence of the distortion of the refractive layer 1 and the lens, based on the parameter values ​​obtained by the model generation unit 102 and the parameter estimation unit 104. The correction table is used to simultaneously perform geometric correction for correcting the distortion of the lens and correction for the influence of the refractive layer 1 on the image using only the image P1 captured through the refractive layer 1.

[0067] The offset calculation unit 105 has a function of dividing the image P1 into a plurality of regions and calculating the degree of offset generated in each of the above regions due to the presence or absence of distortion of the refractive layer 1 and the lens. The divided regions are ideally all set to the same size. Therefore, the offset calculation unit (offset calculation unit 105) calculates the offset between the detected position of the object detected in each region of a predetermined size of an image obtained by the camera (camera 10) photographing the calibration object through the refractive layer (refractive layer 1) and the calculated position of the object calculated in each region of the image that can be photographed by the camera (camera 10) without passing through the refractive layer (refractive layer 1).

[0068] 〈Example of calculating offset〉 Here, reference Fig.11 A case where the image P1 is divided to calculate the amount of displacement will be described. Fig.11 The figure shows the offset amount of each area divided into the image P1. The image P1 is divided at equal intervals in the vertical direction and at equal intervals in the horizontal direction. The offset amount calculation unit (offset amount calculation unit 105) calculates the calculated position of the object using the calculation formula of the modeled refractive layer (refractive layer 1).

[0069] A surface that is located at an appropriate distance in front of the camera 10 and that is directly opposite to the lens surface of the camera 10 is regarded as an infinite plane 61. Rectangular areas are displayed on the infinite plane 61 according to the sizes of the areas into which the image P1 is divided. As an example, a white point 63 passing through the center of each area is set. The offset calculation unit 105 obtains a black point 64 that is affected by the distortion of the refractive layer 1 and the lens, corresponding to the white point 63 set for each area.

[0070] Next, the offset calculation unit 105 calculates where the white point 63 and the black point 64 on the infinite plane 61 are located on the image coordinates 62. The white point 65 and the black point 66 corresponding to the white point 63 and the black point 64 on the infinite plane 61 are displayed on the image coordinates 62. Then, the offset calculation unit 105 calculates the distance "offset 67" between the white point 65 and the black point 66 for each area.

[0071] The evaluation unit (result evaluation unit 106) evaluates the offset (offset 67) of each area calculated by the offset calculation unit 105. Here, the evaluation unit (result evaluation unit 106) sets a weight for each area, and evaluates whether the offset calculation corresponding to the weight has been performed for each area. Then, if the offset calculation corresponding to the weight has not been performed, the evaluation unit (result evaluation unit 106) issues an instruction for calculating the offset corresponding to the weight. On the other hand, if the offset calculation corresponding to the weight has been performed, the evaluation unit (result evaluation unit 106) outputs the offset to the correction calculation unit. For example, if the evaluation result of the offset is poor, the result evaluation unit 106 instructs the camera 10 to perform a correction. Fig. 20 Re-shoot or output for correction Fig. 20 Mobile news.

[0072] like Figure 4A and Figure 4B As shown, assuming that the camera 10 is installed in a normal passenger car, it is assumed that the front refractive layer 1 (for example, the windshield) is inclined by about 30 degrees with respect to the camera 10 and the camera 10 is installed in the center. The offset amount calculated by the offset amount calculation unit 105 for each area varies depending on the position of the area in the image P1.

[0073] For example, if the mark is blurred due to lack of focus or the correction Fig. 20 The poor position of the correction Fig. 20 When the reflected light is reflected, the offset calculation unit 105 cannot accurately calculate the offset, and the result evaluation unit 106 evaluates the offset evaluation result as poor. Therefore, the result evaluation unit 106 may instruct the camera 10 to correct the error. Fig. 20 Re-shoot or urge the calibration personnel to move the calibration equipment Fig. 20Display indication of the message.

[0074] In addition, it is assumed that the offset of the four corners, wide angle, and lower left and right areas of the image P1 captured by the camera 10 is larger than the offset of the central and upper areas of the image P1. Therefore, the result evaluation unit 106 can weight the areas with large offsets. Then, for the weighted areas, the camera 10 can be instructed to increase the number of images P1 of the area input to the offset calculation unit 105 or change the calculation formula as shown in the embodiment described later.

[0075] If the evaluation result of the offset amount given by the result evaluation unit 106 is good, a correction amount for correcting the detected position of the object to the calculated position of the object is calculated for each area based on the evaluation result of the offset amount, and the camera (camera 10) is calibrated based on the correction amount of the offset amount calculated for each area. For example, the correction amount calculation unit 107 calculates a correction amount for moving the offset black point 66 of the image coordinates 62 to the position of the white point 65 in the center of each area based on the offset amount 67 calculated by the offset calculation unit 105. After the correction amount is calculated, the image captured by the camera 10 is processed to correct the position for each area by the calculated correction amount, thereby calibrating the camera 10. Therefore, the image output from the camera calibration device 100 is input to a subsequent environment recognition device (not shown) in a state where the influence of the refractive layer 1 is removed, and is used for environmental recognition of the surroundings of the vehicle.

[0076] <Example of Camera Calibration Method According to First Embodiment> Next, refer to Fig.12 A camera calibration method according to the first embodiment performed in the camera calibration device 100 will be described. Fig.12 This is a flowchart showing an example of the camera calibration method according to the first embodiment.

[0077] First, the camera calibration device 100 captures the calibration image through the refractive layer 1. Fig. 20 ( S1 ) Next, the marker position detection unit 101 detects the position of the marker on the image P1 output from the camera 10 ( S2 ) Next, the model generation unit 102 generates the above-mentioned model ( S3 ).

[0078] Next, the image position calculation unit 103 calculates the image position based on the position of each marker on the calibration chart (S4). The parameter estimation unit 104 estimates a parameter such that the sum of the distances between the detected position and the calculated position of the marker is minimized (S5).

[0079] Next, the offset calculation unit 105 calculates the offset for each region obtained by dividing the image P1 (S6). The result evaluation unit 106 evaluates the offset for each region obtained by dividing the image P1 (S7). Then, the result evaluation unit 106 determines whether the evaluation result is good or not (S8). If the evaluation result is not good (No in S8), the result evaluation unit 106 instructs the camera 10 to take another calibration image. Fig. 20 , and returns to step S1. If the evaluation result is good (Yes in S8), the correction amount calculation unit 107 calculates the correction amount for each divided region (S9), and ends the present process.

[0080] 〈Computer hardware configuration example〉 Next, the hardware configuration of the computer 70 constituting the camera calibration device 100 will be described. Fig.13 1 is a block diagram showing an example of the hardware configuration of the computer 70. The computer 70 is an example of hardware that can be used as a computer to operate the camera calibration device 100 of this embodiment. The camera calibration device 100 of this embodiment is implemented by the computer 70 (computer) executing a program. Figure 2 The camera calibration method shown is performed by cooperation of the functional blocks of the camera calibration device 100.

[0081] The computer 70 includes a CPU (Central Processing Unit) 71, a ROM (Read Only Memory) 72, and a RAM (Random Access Memory) 73, which are respectively connected to a bus 74. The computer 70 also includes a nonvolatile storage unit 75 and a network interface 76.

[0082] The CPU 71 reads the program code of the software that realizes each function of the present embodiment from the ROM 72 and loads it into the RAM 73 for execution. Variables and parameters generated in the middle of the operation processing of the CPU 71 are temporarily written into the RAM 73, and these variables and parameters are read out as appropriate by the CPU 71. However, an MPU (Micro Processing Unit) may be used instead of the CPU 71, and the CPU 71 and a GPU (Graphics Processing Unit) may be used simultaneously. Figure 2 The functions of the various components of the camera calibration device 100 shown are implemented by the CPU 71 .

[0083] As the nonvolatile storage unit 75, for example, a HDD (Hard Disk Drive), an SSD (Solid State Drive), a floppy disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, or a nonvolatile memory is used. In addition to the OS (Operating System) and various parameters, the nonvolatile storage unit 75 also records a program for making the computer 70 function. The ROM 72 and the nonvolatile storage unit 75 record programs and data required for the CPU 71 to operate, and are used as an example of a non-temporary storage medium that stores programs executed by the computer 70 and can be read by a computer. For example, the image P1 is stored in the nonvolatile storage unit 75. In addition, the nonvolatile storage unit 75 stores a model generated by the model generation unit 102, a parameter estimated by the parameter estimation unit 104, an offset calculated by the offset calculation unit 105, an evaluation result of the offset given by the result evaluation unit 106, and a correction calculated by the correction calculation unit 107.

[0084] The network interface 76 uses, for example, a NIC (Network Interface Card) and can transmit and receive various data between devices via an in-vehicle LAN (Local Area Network) connected to a terminal of the NIC, a dedicated line, or the like.

[0085] The camera calibration device 100 of the first embodiment described above captures the calibration image through the refractive layer 1 from the camera 10. Fig. 20 The model is generated by detecting the position of the mark in the obtained image. Then, the camera calibration device 100 calculates the position of the mark on the image, and estimates the parameter that minimizes the offset between the detected position and the calculated position of the mark, thereby creating a correction table that can correct the projection position of the point on the image P1 without the influence of the distortion of the refractive layer 1 and the lens. As a result, the position of the object projected on the image P1 captured by the camera 10 is converted to a position without the influence of the distortion of the refractive layer 1 and the lens by using the correction table, so that the environment recognition device can accurately grasp the position of the object.

[0086] In addition, the image P1 input to the camera calibration device 100 from a camera that can shoot at a wider angle than conventional cameras is used, for example, to detect pedestrians when a car turns left or right at an intersection. In this case, the effect of the refractive layer on the deviation of the image at the wide-angle portion can be suppressed by using the camera calibration method of the first embodiment, thereby reliably detecting pedestrians.

[0087] Therefore, in a monocular camera or a stereo camera, only at least one correction Fig. 20Furthermore, since the correction amount of the image P1 is calculated, an imaging device and an image correction device can be provided that can simulate a condition without the refractive layer 1 based on the image P1 captured through the refractive layer 1.

[0088] [Second embodiment] Next, refer to Fig.14 and Fig.15 A camera calibration method according to a second embodiment of the present invention performed in the camera calibration device 100 will be described.

[0089] The configuration of the camera calibration device 100 of the second embodiment may be the same as that of the camera calibration device 100 of the first embodiment. The camera calibration device 100 of the second embodiment changes the number of images P1 input from the camera 10 according to the offset amount calculated for each area by the offset amount calculation unit 105, thereby calibrating the camera 10. For example, when using a wide range of calibration objects, Fig. 20 When the camera 10 is configured, the entire range cannot be completely covered by just one shot, so a plurality of images P1 shot at different positions must be input.

[0090] <Example of Camera Calibration Method According to Second Embodiment> Fig.14 FIG. 1 is a flowchart showing an example of a camera calibration method according to the second embodiment. Fig.14 In the flowchart shown, Fig.12 The processing of marker position detection (S2), model generation (S3), image position calculation (S4), and parameter inference (S5) in each step of the flowchart of the camera calibration method of the first embodiment shown is deemed to have been performed and detailed description is omitted.

[0091] First, the camera 10 captures the calibration image. Fig. 20 (S11). Next, the offset calculation unit 105 calculates the offset for each divided area (S12). Next, the result evaluation unit 106 evaluates whether the number of images P1 corresponding to the offset has been input (S13). If the calculation of the offset corresponding to the weight has not been performed, the evaluation unit (result evaluation unit 106) instructs the number of images to be input to the offset calculation unit (offset calculation unit 105) for each area to be changed according to the weight set for the area. This instruction is input to the camera 10, and the camera 10 takes the calibration image again. Fig. 20 Thereafter, the image P1 captured again is input to the shift amount calculation unit 105, and the shift amount is calculated using the number of images P1 set for the region.

[0092] For example, if the refractive layer 1 is arranged substantially vertically with respect to the light of the camera 10, the result evaluation unit 106 sets a weight to the area around (left and right lower parts) the image P1, and instructs to increase the number of images P1. Similarly, if the refractive layer 1 is arranged obliquely with respect to the light of the camera 10, a weight is set to the area below the image P1, and instructs to increase the number of images P1. Furthermore, if the camera 10 is arranged to the left or right, the result evaluation unit 106 sets a weight to the area on the opposite side of the image P1 from the position where the camera 10 is arranged, and instructs to increase the number of images P1.

[0093] If the result evaluation unit 106 evaluates that the number of images P1 corresponding to the offset amount has not been input (No in S13 ), the process returns to S11 and repeats the process until the number of images P1 corresponding to the offset amount is input.

[0094] On the other hand, if the result evaluation unit 106 evaluates that the number of images P1 corresponding to the shift amount has been input (Yes in S13 ), the correction amount calculation unit 107 calculates the correction amount for each divided region ( S14 ), and ends the present process.

[0095] Fig.15 This is a diagram showing a state where the number of input images corresponding to the offset amount is indicated. Fig.15 The upper side shows the offset [px] of each segmented area. Fig.15 The upper side of shows the number of input images (image P1) required for each divided area.

[0096] The result evaluation unit 106 calculates the number of input images required based on the offset in each area. The minimum number of input images is 1. For example, if the offset is less than 1.0, the number of input images is 1. On the other hand, if the offset is greater than 1.0 and less than 3.0, the number of input images is 2, if the offset is greater than 3.0 and less than 4.0, the number of input images is 3, and if the offset is greater than 4.0, the number of input images is 4.

[0097] As a necessary condition here, each area contains a specified number of calibration Fig. 20 For example, the operator changes the calibration Fig. 20 The tilt is corrected by taking pictures with the camera 10 times. Fig. 20 By inputting a large number of input images, the accuracy of the correction amount for each region obtained by dividing the image P1 can be improved. As a result, the camera calibration device 100 can also improve the accuracy of the calibration of the camera 10.

[0098] Here, reference Fig.16 and Figures 17A to 17CFor correction of changes Fig. 20 The following describes the situation where shooting is performed from a position. Fig.16 To indicate the setting of calibration Fig. 20 Map of the location.

[0099] The calibration object is set so as to fall within the range of the field angle of the camera 10. Fig. 20 Here, the calibration point is arranged at a position L1 closest to the camera 10. Fig. 20 (1) A calibration device is arranged at a position L2 which is the second closest distance to the camera 10. Fig. 20 (2) The calibration device is arranged at a position L3 farthest from the camera 10. Fig. 20 (3) In addition, the image P1 captured by the camera 10 is divided into nine equal areas.

[0100] Fig.17A The correction factor is shown in the entire image P1. Fig. 20 (1) The location of the diagram. Fig.17A The calibration shown is Fig. 20 (1) All nine regions of the image P1 are captured. However, the correction Fig. 20 (1) has a small projection range, so the correction is Fig. 20 (1) is insufficiently marked. Therefore, the correction Fig. 20 The position of is indicated to the camera 10 by the result evaluation unit 106 .

[0101] Fig. 17B The correction factor shown on the left side of the deviated image P1 is Fig. 20 (2) Location diagram. Fig. 17B The calibration shown is Fig. 20 (2) is captured to the left of image P1. Therefore, the image is shown with sufficient amount. Fig.17A The left side of the image P1 shown is insufficiently corrected by Fig. 20 (1) Mark.

[0102] Fig. 17C The correction factor shown on the upper side of the deviated image P1 is Fig. 20 (3) Location diagram. Fig. 17C The calibration shown is Fig. 20 (3) is captured on the upper side of image P1. Therefore, the image is displayed in sufficient quantity. Fig.17A The upper side of the image P1 shown is insufficient for correction Fig. 20 (1) Mark.

[0103] By using a correction Fig. 20 Make various changes to the position and inclination of the Fig. 20 Therefore, the correction amount calculation unit 107 can calculate the correction amount of the area shift more accurately than the camera calibration method of the first embodiment, using the positions of the markers detected by the marker position detection unit 101 from each image P1.

[0104] [Third embodiment] Next, refer to Fig.18 and Fig.19 A camera calibration method according to a third embodiment of the present invention performed in the camera calibration device 100 will be described.

[0105] The configuration of the camera calibration device 100 of the third embodiment may be the same as that of the camera calibration device 100 of the first embodiment. The camera calibration device 100 of the third embodiment changes the calculation formula according to the offset amount for each area calculated by the offset amount calculation unit 105, thereby calibrating the camera 10. The calculation formula before the change is the above formula (1), or the above formula (4), or both.

[0106] As a specific example, suppose that Figure 6 The calculation formula of at least one of the front side approximate curved surface 41 and the depth side approximate curved surface 42 is changed to the following equation (6). For example, the offset calculation unit 105 may change the approximate curved surface of the model shown in equation (4) to the following equation (6).

[0107] [Formula 7]

[0108] Similarly, the shift amount calculation unit 105 may also change the order of the lens distortion model, or may change it to a model combined with other approximate curved surfaces. The following equation (7) is an example of another model that the shift amount calculation unit 105 can change.

[0109] [Formula 8]

[0110] <Example of Camera Calibration Method According to Third Embodiment> Fig.18 FIG. 1 is a flowchart showing an example of a camera calibration method according to the third embodiment. Fig.18 In the flowchart shown, Fig.12 The processing of marker position detection ( S2 ), image position calculation ( S4 ), and parameter estimation ( S5 ) in each step of the flowchart of the camera calibration method according to the first embodiment is deemed to have been performed, and detailed description thereof is omitted.

[0111] First, the camera 10 captures the calibration image. Fig. 20 (S21) Next, the model generation unit 102 generates a model, that is, generates a calculation formula (S22).

[0112] Next, the offset calculation unit 105 calculates the offset for each divided area (S23). Next, the result evaluation unit 106 evaluates whether it is a calculation formula corresponding to the offset (S24). If the calculation of the offset corresponding to the weight is not performed, the evaluation unit (result evaluation unit 106) instructs the offset calculation unit (offset calculation unit 105) to change the calculation formula for calculating the offset for each area according to the weight set for the area. The instruction is input to the camera 10, and is thereby input to the offset calculation unit 105 through the camera 10. Then, the offset calculation unit 105 uses the calculation formula changed according to the instruction to calculate the offset in the area where the change of the calculation formula is instructed again.

[0113] For example, if the refractive layer 1 is arranged approximately vertically with respect to the light of the camera 10, the result evaluation unit 106 sets a weight for the area around (left and right lower parts) the image P1, and instructs to generate a calculation formula that matches the weight. Similarly, if the refractive layer 1 is arranged tilted with respect to the light of the camera 10, a weight is set for the area at the lower part of the image P1, and an instruction is given to generate a calculation formula that matches the weight. In addition, if the camera 10 is arranged to the left or right, the result evaluation unit 106 sets a weight for the area on the opposite side of the image P1 from the position where the camera 10 is arranged, and instructs to generate a calculation formula that matches the weight.

[0114] If the result evaluation unit 106 evaluates that the calculation formula is not appropriate for the offset (No in S24), the process returns to S22 to repeat model generation until an appropriate calculation formula is generated. Then, the offset calculation unit 105 calculates the offset using the calculation formula appropriate for the offset.

[0115] On the other hand, if the result evaluation unit 106 evaluates that the calculation formula corresponds to the offset amount (Yes in S24 ), the correction amount calculation unit 107 calculates the correction amount for each divided region ( S25 ), and ends the present process.

[0116] Fig.19 This is a diagram showing a state where the number of input images corresponding to the offset amount is indicated. Fig.19 The upper side shows the offset [px] of each segmented area. Fig.19 The upper side of shows the appropriate calculation formula for each segmented area.

[0117] The result evaluation unit 106 obtains a calculation formula corresponding to the offset in each area. For example, if the offset is less than 1.0, calculation formula a is used. On the other hand, if the offset is greater than 1.0 and less than 2.5, calculation formula b is used, if the offset is greater than 2.5 and less than 3.5, calculation formula c is used, and if the offset is greater than 3.5, calculation formula d is used. The calculation formulas a to d shown here use the above-mentioned formula (1), formula (4), etc. or formulas not shown in the figure.

[0118] In the camera calibration device 100 of the third embodiment described above, the calculation formula can be arbitrarily changed. Fig. 20 The calculation accuracy of the correction amount can be improved by selecting the appropriate calculation formula according to the environment.

[0119] [Fourth embodiment] Next, refer to Fig. 20 A camera calibration method according to a fourth embodiment of the present invention performed in the camera calibration device 100 will be described.

[0120] The configuration of the camera calibration device 100 of the fourth embodiment may be the same as that of the camera calibration device 100 of the first embodiment. In the camera calibration device 100 of the fourth embodiment, the camera calibration method of the second embodiment and the camera calibration method of the third embodiment are performed simultaneously. Therefore, the calibration is performed by changing the number of input images and the calculation formula according to the offset calculated by the offset calculation unit 105.

[0121] <Example of Camera Calibration Method According to Fourth Embodiment> Fig. 20 FIG. 4 is a flowchart showing an example of a camera calibration method according to the fourth embodiment. Fig. 20 In the flowchart shown, Fig.12 The processing of marker position detection ( S2 ), image position calculation ( S4 ), and parameter estimation ( S5 ) in each step of the flowchart of the camera calibration method according to the first embodiment is deemed to have been performed, and detailed description thereof is omitted.

[0122] As described above, the camera calibration method of the fourth embodiment is a combination of the camera calibration method of the second embodiment and the camera calibration method of the third embodiment. If the calculation of the offset corresponding to the weight is not performed, the evaluation unit (result evaluation unit 106) instructs the number of images to be input to the offset calculation unit (offset calculation unit 105) to be changed for each region based on the weight set for the region, and instructs the offset calculation unit (offset calculation unit 105) to change the calculation formula for calculating the offset for each region. The instruction to change the number of images for each region is input to the camera 10, thereby causing the camera 10 to take the calibration image again. Fig. 20 . Then, the image P1 captured again is input to the offset calculation unit 105, and the offset is calculated for the number of images P1 set for the area. At the same time, an instruction to change the calculation formula is input to the camera 10, and is input to the offset calculation unit 105 through the camera 10. Then, the offset calculation unit 105 uses the calculation formula changed according to the instruction to calculate the offset in the area where the change of the calculation formula is instructed.

[0123] Fig. 20 The processing of steps S31, S33, and S34 shown in FIG. Fig.14 The processing of steps S11 to S13 shown is the same. Fig. 20 The processing of steps S31 to S33 and S35 shown in FIG. Fig.14 The processes in steps S21 to S24 shown are the same.

[0124] Next, when it is evaluated in step S35 that the calculation formula corresponds to the offset amount (Yes in S35 ), the correction amount calculation unit 107 calculates the correction amount for each divided region ( S36 ), and ends the present process.

[0125] In the camera calibration device 100 according to the fourth embodiment described above, the number of input images is changed and the calculation formula is changed according to the offset amount, thereby making it possible to correct the offset amount more accurately.

[0126] The present invention is not limited to the above-described embodiments, and various other application examples and modified examples can of course be employed without departing from the gist of the present invention described in the claims. For example, the above-mentioned embodiments are detailed and specific descriptions of the configuration of the device in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to having all the configurations described. In addition, a part of the configuration of the embodiment described here can be replaced with the configuration of other embodiments, and further, the configuration of other embodiments can be added to the configuration of a certain embodiment. In addition, other configurations can be added, deleted, or replaced with part of the configuration of each embodiment. In addition, the control lines and information lines are shown as those considered necessary for explanation, and not all control lines and information lines may be shown on the product. In fact, it can be considered that almost all components are connected to each other. Explanation of symbols

[0127] 1 ... refractive layer, 10 ... camera, 20 ... calibration chart, 100 ... camera calibration device, 101 ... mark position detection unit, 102 ... model generation unit, 103 ... image position calculation unit, 104 ... parameter estimation unit, 105 ... offset calculation unit, 106 ... result evaluation unit, 107 ... correction amount calculation unit.

Claims

1. A camera calibration device, characterized in that: have: an offset calculation unit for calculating an offset between a detected position of an object detected for each region of a predetermined size of an image obtained by photographing the object for calibration via a refractive layer by a camera and a calculated position of the object calculated for each region of the image that can be photographed by the camera without penetrating the refractive layer; an evaluation unit that evaluates the offset amount for each of the regions; as well as A correction amount calculation unit calculates a correction amount for correcting a detected position of the object to a calculated position of the object for each of the regions based on an evaluation result of the offset amount, and calibrates the camera based on the correction amount of the offset amount calculated for each of the regions.

2. The camera calibration device according to claim 1, characterized in that: The evaluation unit sets a weight for each of the areas, and evaluates for each of the areas whether the offset corresponding to the weight has been calculated; if the offset corresponding to the weight has not been calculated, an instruction is given for calculating the offset corresponding to the weight; if the offset corresponding to the weight has been calculated, the offset is output to the correction amount calculation unit.

3. The camera calibration device according to claim 2, characterized in that: If the calculation of the offset amount according to the weight is not performed, the evaluation unit instructs to change the number of images input to the offset amount calculation unit for each region based on the weight set for the region.

4. The camera calibration device according to claim 2, characterized in that: If the calculation of the offset amount according to the weight has not been performed, the evaluation unit instructs the offset amount calculation unit to change a calculation formula for calculating the offset amount for each of the regions, based on the weight set for the region.

5. The camera calibration device according to claim 2, characterized in that: If the calculation of the offset corresponding to the weight is not performed, the evaluation unit instructs the offset calculation unit to change the number of images input to the offset calculation unit for each of the regions based on the weight set for the regions, and instructs the offset calculation unit to change the calculation formula for calculating the offset for each of the regions.

6. The camera calibration device according to claim 4, characterized in that: The object for calibration is at least one of a two-dimensional calibration image and a three-dimensional calibration image.

7. The camera calibration device according to claim 2, characterized in that: A model generating unit is provided for generating a model including the refractive layer based on the weight and the plurality of images input to the offset calculating unit and the object for camera imaging correction, The offset amount calculation unit calculates a calculated position of the object using the calculation formula of the modeled refractive layer.

8. A camera calibration method, characterized in that: The following steps are involved: calculating an offset between a detected position of the object, the detected position of the object being detected for each area of ​​a predetermined size of an image obtained by photographing the object for calibration through a refractive layer by a camera, and a calculated position of the object being calculated for each area of ​​the image that can be photographed by the camera without penetrating the refractive layer; evaluating the offset for each of the regions; calculating, for each of the regions, a correction amount for correcting the detected position of the object to the calculated position of the object based on the evaluation result of the offset amount; as well as The camera is calibrated according to the correction amount of the offset calculated for each of the areas.

Images