How to calibrate your camera

Abstract

To provide an image calibration method capable of easily calibrating a camera with high angle of view for capturing an image through a windshield.SOLUTION: An image for calibration is acquired by photographing an object for calibration having a linear portion extending in one direction that intersects the optical axis of a camera through a windshield with the camera (step S1). A linear portion image corresponding to the linear portion is extracted from the image for calibration (step S2). Based on the linear portion image and an image of imaginary linear portion assumed when the linear portion is photographed without passing through the windshield with the camera, distortion in the image for calibration is corrected (step S3).SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to a camera calibration method and a stereo camera device. [Background technology]

[0002] Conventionally, there have been known inventions relating to a method for calibrating a stereo camera that includes a first camera and a second camera and captures an image of a subject through a transparent body. For example, the calibration method for a stereo camera described in Patent Document 1 below includes a step of calculating a correction parameter that calibrates an absolute positional shift that indicates a shift in the coordinates of the image of the subject caused by the transparent body in at least one of the images captured by the first camera and the second camera.

[0003] This conventional calibration method also includes the steps of photographing the calibration tool with a stereo camera to obtain a first captured image by the first camera and a second captured image by the second camera, and calculating, based on the first and second captured images, a correction parameter for calibrating a relative positional shift that indicates a parallax shift between the image of the subject in the first captured image and the image of the subject in the second captured image. This conventional calibration method also includes the step of storing, in the stereo camera, a correction parameter based on the correction parameter for calibrating the absolute positional shift and the correction parameter for calibrating the relative positional shift (Patent Document 1, Claim 1, etc.). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-132855 Summary of the Invention [Problem to be solved by the invention]

[0005] In the conventional stereo camera calibration method described above, the stereo camera captures an image of a calibration tool to calibrate the relative positional deviation and the absolute positional deviation. Therefore, with this conventional calibration method, if the angle of view of the stereo camera becomes wider, the calibration tool becomes larger, which may make it difficult to calibrate the stereo camera.

[0006] The present disclosure provides a camera calibration method and a stereo camera device that can easily calibrate a wide-angle camera that captures images through a windshield. [Means for solving the problem]

[0007] One aspect of the present disclosure is a method for calibrating a camera that captures an image through a windshield, the method including: photographing a calibration object having a linear portion extending in one direction intersecting an optical axis of the camera through the windshield with the camera to obtain a calibration image; extracting a linear portion image corresponding to the linear portion from the calibration image; and correcting distortion of the calibration image based on a virtual linear portion image that would be expected if the linear portion were photographed by the camera without passing through the windshield and the linear portion image. [Effects of the Invention]

[0008] According to the above aspect of the present disclosure, it is possible to provide an image calibration method that can easily calibrate a wide-angle camera that captures images through a windshield. [Brief explanation of the drawings]

[0009] [Figure 1] 1A and 1B are a plan view and an image diagram illustrating an embodiment of a stereo camera device according to the present disclosure. [Figure 2] FIG. 2 is a block diagram showing an example of the configuration of the stereo camera device of FIG. [Figure 3A] FIG. 4 is a schematic horizontal cross-sectional view illustrating refraction of light that has passed through a windshield. [Figure 3B] 5 is a schematic vertical cross-sectional view illustrating refraction of light transmitted through a windshield. FIG. [Figure 4] 3B is a schematic diagram illustrating the displacement of an image on an image sensor due to the refraction of light in FIGS. 3A and 3B. FIG. [Figure 5] 1 is a flow diagram illustrating an embodiment of a camera calibration method according to the present disclosure. [Figure 6] 6 is a diagram showing an example of a calibration object in the step of acquiring the calibration image in FIG. 5. [Figure 7A] FIG. 7 is a front view of a calibration chart serving as the calibration object shown in FIG. 6. [Figure 7B] FIG. 7B is a diagram showing an image captured by a camera of the calibration chart shown in FIG. 7A. [Figure 8] 6 is an image diagram showing an example of an extraction result in the step of extracting a straight portion image in FIG. 5. [Figure 9A] 10A and 10B are diagrams showing examples of inclinations when a calibration chart is installed. [Figure 9B] 10A and 10B are diagrams showing examples of inclinations when a calibration chart is installed. [Figure 10] 10 is a graph showing the tilt of the image of the straight line portion depending on the tilt at the time of installation of the calibration chart. [Figure 11] FIG. 10 is a schematic vertical cross-sectional view showing pupil deviation of a wide-angle lens. [Figure 12] 10A and 10B are schematic vertical cross-sectional views illustrating the influence of pupil misalignment of a camera lens. [Figure 13] 10 is a graph showing the relationship between the horizontal length of the calibration chart and the installation distance. [Figure 14] FIG. 10 is a schematic side view showing a second embodiment of the camera calibration method of the present disclosure. [Figure 15] FIG. 11 is a perspective view of a calibration object in a camera calibration method according to a third embodiment of the present disclosure. [Figure 16A] FIG. 10 is a front view of a calibration chart according to a fourth embodiment of the camera calibration method of the present disclosure. [Figure 16B] FIG. 16B is a diagram showing an image captured by a camera of the calibration chart shown in FIG. 16A. [Figure 17] FIG. 10 is a front view of a second calibration chart in the fourth embodiment of the camera calibration method of the present disclosure. [Figure 18] FIG. 7B is a front view showing a modified example of the calibration chart shown in FIG. 7A. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of a camera calibration method and a stereo camera device according to the present disclosure will be described with reference to the drawings.

[0011] [Embodiment 1] 1 is a plan view and an image diagram illustrating a stereo camera device according to a first embodiment of the present disclosure, and FIG. 2 is a block diagram illustrating an example of the configuration of the stereo camera device 100 of FIG.

[0012] 1, the stereo camera device 100 of this embodiment is mounted on a vehicle V such as an automobile, detects external environment information including objects around the vehicle V and the distances to those objects through the windshield WS of the vehicle V, and generates a warning as necessary. The external environment information detected by the stereo camera device 100 and the generated warning are used, for example, for automatic driving or advanced driving assistance of the vehicle V by a vehicle control device (not shown).

[0013] 2, the stereo camera device 100 includes left and right cameras 10L and 10R that capture images LI and RI through the windshield WS of the vehicle V. Each camera 10 includes, for example, a lens and an image sensor (not shown), and captures or acquires images of objects around the vehicle V via the lens using the image sensor. Each camera 10 is a wide-angle camera with a horizontal angle of view of, for example, approximately 110° to 150°.

[0014] 1, the imaging area A of the stereo camera device 100 formed by the left and right cameras 10L and 10R includes, for example, a central stereoscopic viewing area As and monocular viewing areas Am on the left and right sides of the stereoscopic viewing area As. The central stereoscopic viewing area As is an area where the imaging areas AL and AR of the left and right cameras 10L and 10R overlap. The monocular viewing area Am on the left side of the stereoscopic viewing area As is the imaging area AL of the left camera 10L and is an area outside the imaging area AR of the right camera 10R. The monocular viewing area Am on the right side of the stereoscopic viewing area As is the imaging area AR of the right camera 10R and is an area outside the imaging area AL of the left camera 10L.

[0015] 2, the stereo camera device 100 includes an image processing unit 20, a stereo parallax image generating unit 30, a road surface cross-sectional shape estimating unit 40, a stereoscopic object detecting unit 50, a left and right monocular image generating unit 60, a hybrid object detecting unit 70, and an alarm control unit 80. Each of these units of the stereo camera device 100 can be configured, for example, by one or more microcontrollers including a central processing unit (CPU), memory such as ROM and RAM, a timer, an input / output unit, and a program recorded in the memory.

[0016] The image processing unit 20 acquires images LI and RI from the left and right cameras 10L and 10R, respectively, performs image processing on them, and outputs the image data generated by the image processing to the stereo parallax image generation unit 30 and the left and right monocular image generation unit 60. For example, as shown in Fig. 2, the image processing unit 20 has an affine processing unit 21, a luminance correction unit 22, a pixel interpolation unit 23, and a luminance information generation unit 24 for each of the left and right cameras 10L and 10R. Each of these units of the image processing unit 20 represents a function realized by, for example, a CPU executing a program recorded in a memory.

[0017] The left and right affine processors 21L and 21R generate transformed images LIa and RIa by affine transforming the image LI from the left camera 10L and the image RI from the right camera 10R, respectively. Furthermore, if the left and right cameras 10L and 10R are equipped with fisheye lenses or wide-angle lenses, each affine processor 21 generates transformed images LIa and RIa by projectively transforming the image height y=f·sinθ of an orthogonal projection lens with a focal length f and a half angle of view θ onto a coordinate system of (f·tanθx, f·tanθy). θx and θy represent the horizontal and vertical angular components of the light incident on the lens.

[0018] Furthermore, the left and right affine processors 21L and 21R generate transformed images LIa and RIa in which pixel shift, which is vertical pixel displacement caused by refraction of light passing through the windshield WS, has been corrected. That is, each affine processor 21 functions as a pixel shift correction unit that corrects pixel shift, for example.

[0019] The left and right luminance correction units 22L and 22R respectively correct the luminance of each pixel of the converted image L1a of the image LI captured by the left camera 10L and the luminance of each pixel of the converted image RIa of the image RI captured by the right camera 10R. Each luminance correction unit 22 corrects the luminance of each pixel based on, for example, the difference in gain between each pixel of the converted images L1a and RIa.

[0020] The left and right pixel interpolation units 23L and 23R respectively perform demosaicing processing on the brightness-corrected converted images LIa and RIa to convert the converted images LIa and RIa from RAW images to color images. The left and right brightness information generation units 24L and 24R respectively generate brightness information for the converted images LIa and RIa converted into color images. More specifically, each brightness information generation unit 24 generates brightness information for generating parallax images from color image information of the converted images LIa and RIa.

[0021] The stereo parallax image generating unit 30 generates a stereo parallax image of the stereo viewing area As by using the images of the stereo viewing area As among the converted images LIa and RIa input from the image processing unit 20. The stereo parallax image generating unit 30 includes, for example, an exposure adjusting unit 31, a sensitivity correcting unit 32, a geometric correcting unit 33, a matching unit 34, a noise removing unit 35, a pixel shift amount calculating unit 36, and a correction function deriving unit 37. Each of these units of the stereo parallax image generating unit 30 represents a function realized by, for example, a CPU executing a program recorded in a memory.

[0022] The exposure adjustment unit 31 and the sensitivity correction unit 32 can perform feedback control by feeding back the exposure amount and sensitivity of the left and right cameras 10L and 10R to the left and right cameras 10L and 10R, respectively. The geometric correction unit 33, the matching unit 34, and the noise removal unit 35 perform geometric correction, matching processing, and noise removal on the left and right converted images LIa and RIa, respectively. The pixel shift amount calculation unit 36 ​​and the correction function derivation unit 37 calculate the pixel shift amount of the left and right converted images LIa and RIa, and derive a correction function (described later), respectively. In other words, the pixel shift amount calculation unit 36 ​​and the correction function derivation unit 37, together with the affine processing unit 21, for example, constitute a pixel shift correction unit.

[0023] The road surface cross-sectional shape estimation unit 40 estimates, for example, the cross-sectional shape of the road surface along which the vehicle V will travel, ahead in the optical axis direction of the left and right cameras 10L, 10R of the stereo camera device 100. The road surface cross-sectional shape estimation unit 40 can estimate the cross-sectional shape of the road surface by, for example, a known method.

[0024] The stereoscopic object detection unit 50 detects objects in the stereoscopic area As using the stereo disparity images generated by the stereo disparity image generation unit 30. The stereoscopic object detection unit 50 further applies stereo matching to the detected objects to detect disparity, and also applies pattern matching to identify the type of object, such as a pedestrian, bicycle, vehicle, or building.

[0025] The left and right monocular image generation unit 60 generates images of the left and right monocular regions Am in the left and right converted images LIa and RIa as left and right monocular images. The left and right monocular image generation unit 60 performs projective transformation on the images of the left and right monocular regions Am in the left and right converted images LIa and RIa so that, for example, the unit length of the left and right monocular images along the same horizontal line and the unit length of the stereo parallax image represent the same distance.

[0026] The hybrid object detection unit 70 generates a single hybrid image by combining, for example, the stereo parallax image generated by the stereo parallax image generation unit 30 and the left and right monocular images generated by the left and right monocular image generation unit 60. That is, the hybrid image has, for example, a stereo parallax image in the horizontal center and monocular images on the left and right of the stereo parallax image.

[0027] The hybrid object detection unit 70 further detects an object based on the generated hybrid image. More specifically, the hybrid object detection unit 70 detects an object from the monocular image of the hybrid image, and detects the distance to the object in the monocular image based on information such as parallax in the stereo parallax image of the hybrid image.

[0028] More specifically, the hybrid object detection unit 70 identifies the type of object detected in the monocular vision area by pattern matching, similar to the stereoscopic object detection unit 50. The hybrid object detection unit 70 also estimates the distance to an object in the monocular vision image based on, for example, the ground position of the object in the monocular vision image and position information from a stereo parallax image on the same horizontal line as the ground position. Therefore, if a pixel shift occurs, in which the image position is shifted vertically due to refraction of light passing through the windshield WS, a distance measurement error occurs in the monocular vision area.

[0029] 3A and 3B are schematic horizontal and vertical cross-sectional views, respectively, illustrating the refraction of light that passes through the windshield WS and enters the right camera 10R of the stereo camera device 100. FIG. 4 is a schematic diagram showing the displacement of the image on the image sensor of the right camera 10R due to the refraction of light that passes through the windshield WS. In FIG. 4, object images Im1-Im4 captured without passing through the windshield WS are shown by two-dot chain lines, and object images Im1'-Im4' captured through the windshield WS are shown by solid lines.

[0030] As shown in Figures 3A and 3B, light rays L1-L4 incident on the windshield WS are refracted according to Snell's law. If the entrance and exit surfaces of the windshield WS through which light rays L1-L4 pass are parallel, the incident and exiting light rays L1-L4 will be parallel. Therefore, as shown in Figure 3A, the angle of light ray L1 incident on the front portion of the windshield WS remains almost unchanged even after passing through the windshield WS. Therefore, as shown in Figure 4, the horizontal displacement of object images Im1' and Im3' captured through the windshield WS near the optical axis OA of camera 10 is small relative to the object images Im1 and Im3 captured without passing through the windshield WS.

[0031] However, as shown in Fig. 3A, the inclination angle of camera 10 with respect to the optical axis direction is large, and light ray L2 incident on windshield WS from the side of vehicle V changes angle relatively greatly after passing through windshield WS. As a result, as shown in Fig. 4, object images Im2' and Im4' captured through windshield WS at a wide-angle position distant from optical axis OA of camera 10 in the horizontal direction exhibit relatively large horizontal displacements relative to object images Im2 and Im4 captured without passing through windshield WS.

[0032] 3B, the windshield WS is tilted relative to the vertical direction so that the upper portion is positioned more rearward. Therefore, light rays L3 and L4 entering the camera 10 through the windshield WS are incident at an angle relative to the incident surface of the windshield WS. As a result, the light ray L3 that passes through the windshield WS at a position near the optical axis OA of the camera 10 and the light ray L4 that passes through the windshield WS at a wide-angle position away from the optical axis OS of the camera 10 both experience a relatively large change in angle after passing through the windshield WS. On the other hand, the change in the vertical ray angle due to the windshield WS in the horizontal center region has little dependence on the horizontal angle of view.

[0033] In this way, the position of the object image formed on the image sensor by the lens of camera 10 when there is no windshield WS changes when the image is captured through the windshield WS. Therefore, as shown in FIG. 4, the object images Im1'-Im4' captured through the windshield WS are displaced relatively significantly downward in the vertical direction from the positions of the object images Im1-Im4 when the object images are captured without the windshield WS. This vertical displacement from the original positions of the object images Im1-Im4 to the positions of the images Im1'-Im4' is called pixel shift. The magnitude of this pixel shift varies depending on the position on the image sensor, and therefore appears as image distortion.

[0034] As described above, the stereo camera device 100 of this embodiment estimates the distance to an object in the monocular vision region Am using the ground position of the object. Therefore, the vertical pixel shift described above causes a distance measurement error in the monocular vision region Am. The magnitude of the pixel shift depends on the shape and tilt of the windshield WS. In particular, the pixel shift becomes large when the curvature of the windshield WS is large and the tilt of the windshield WS relative to the vertical direction is large. The camera calibration method and stereo camera device 100 of this embodiment calibrate the camera 10 to correct this pixel shift.

[0035] 5 is a flow diagram showing an embodiment of a camera calibration method according to the present disclosure. The camera calibration method CM of this embodiment is a method for calibrating a camera 10 that captures an image through a windshield WS. The camera calibration method CM includes a step S1 of acquiring a calibration image, a step S2 of extracting a straight portion image, and a step S3 of correcting distortion in the calibration image. The camera calibration method CM can be implemented by, for example, a stereo camera device 100.

[0036] In step S1, the stereo camera device 100 acquires a calibration image by, for example, using the left and right cameras 10L and 10R to capture an image of a calibration object having a linear portion extending in one direction that intersects with the optical axis OA of the camera 10 through the windshield WS. Note that the camera calibration method CM of this embodiment can also be applied to, for example, a monocular camera.

[0037] Fig. 6 is a diagram showing an example of a calibration object in step S1 of acquiring a calibration image. Fig. 7A is a front view of the calibration object shown in Fig. 6. Fig. 7B is an image of the calibration object shown in Fig. 7A photographed by camera 10 before calibration through a windshield WS. Note that Fig. 7B shows pixel shifts enlarged compared to the actual size.

[0038] 6, 7A, and 7B, the calibration object photographed through the windshield WS by the camera 10 of the stereo camera device 100 is, for example, a calibration chart 200 having a striped calibration pattern 210. Such a calibration chart 200 can be used, for example, when the camera calibration method CM of this embodiment is implemented in a manufacturing plant for the vehicle V, a maintenance plant for the vehicle V, a manufacturing plant for the stereo camera device 100, or the like.

[0039] The striped calibration pattern 210 of the calibration chart 200 is, for example, a pattern in which stripe-shaped dark color portions 211 extending horizontally and stripe-shaped light color portions 212 are alternately arranged in the vertical direction. The dark color portions 211 are, for example, black, and the light color portions 212 are, for example, white. Note that the colors of the dark color portions 211 and the light color portions 212 are not particularly limited as long as edge detection, which will be described later, is possible.

[0040] In step S1 of acquiring a calibration image, the calibration chart 200, which serves as a calibration object and is photographed by the camera 10 through the windshield WS, has a linear portion 213 that extends in one direction intersecting the optical axis OA of the camera 10. This linear portion 213 is, for example, a boundary line between a dark color portion 211 and a light color portion 212 of a striped calibration pattern 210 drawn on the calibration chart 200, and extends in a horizontal direction perpendicular to the optical axis OA of the camera 10.

[0041] Furthermore, in step S1 of acquiring a calibration image, the calibration chart 200 is photographed so that the calibration image Im200, which is an image of the calibration chart 200, includes a straight-line portion image Im213, which is an image of the straight-line portion 213, from one end to the other in one direction intersecting the optical axis OA of the camera 10. In this embodiment, for example, in step S1, the calibration chart 200 is photographed so that the calibration image Im200 includes the straight-line portion image Im213 from one end to the other in the horizontal direction perpendicular to the optical axis OA of the camera 10.

[0042] 7B, in calibration image Im200, the pixel shift downward in the vertical direction becomes larger as the horizontal angle of view becomes wider and further away from the center where the optical axis OA of camera 10 is located. This is because the angle of incidence of light on windshield WS becomes larger as the horizontal angle of view becomes wider.

[0043] In step S1 of acquiring the calibration image, for example, as shown in Fig. 6, a calibration chart 200 is photographed as a calibration object that is placed in front of the windshield WS in the direction of the optical axis OA of the camera 10 and above the vehicle V in the vertical direction. More specifically, the calibration chart 200 is placed, for example, on a body of the vehicle V, such as a hood, in front of the windshield WS of the vehicle V. That is, the calibration chart 200 is placed, for example, between the front end of the vehicle V and the windshield WS in the longitudinal direction of the vehicle V.

[0044] Next, the stereo camera device 100 performs step S2 of extracting a straight line portion image, as shown in Fig. 5. In step S2, the stereo camera device 100 extracts, for example, a straight line portion image Im213, which is an image of the straight line portion 213, from the calibration image Im200. As described above, when the calibration chart 200 is photographed by the camera 10 while the calibration chart 200 and the stereo camera device 100 are positioned close to each other, the resolution of the camera 10 decreases, and the edges, which are the boundaries between the dark color portions 211 and light color portions 212 of the calibration pattern 210, tend to become unclear.

[0045] In contrast, the stereo camera device 100 of this embodiment detects the edges between the dark color portions 211 and the light color portions 212, for example, by the following method. First, a change in signal intensity occurring when the calibration pattern 210 changes from the dark color portions 211 to the light color portions 212, or from the light color portions 212 to the dark color portions 211, is detected. Next, the vertical position of a pixel at which a signal having an intensity half the value of the change in signal intensity is detected is identified, and the edge that is the boundary between the dark color portions 211 and the light color portions 212 is detected based on that position.

[0046] 8 is a diagram showing an example of the straight line portion image Im213 extracted in step S2. For example, as described above, the stereo camera device 100 detects the edge that is the boundary between the dark color portion 211 and the light color portion 212, and extracts the straight line portion image Im213 corresponding to the straight line portion 213, as shown in FIG.

[0047] Next, the stereo camera device 100 executes step S3 of correcting the calibration image, as shown in Fig. 5. In step S3, the stereo camera device 100 corrects distortion of the calibration image Im200 based on a virtual straight portion image that would be assumed if the straight portion 213 were photographed by the camera 10 without passing through the windshield WS, and the straight portion image Im213 acquired in step S1.

[0048] The step S3 of correcting the calibration image includes, for example, a step S31 of creating a correction table for pixel shift, a step S32 of creating a correction table for lens pupil shift, and a step S33 of correcting distortion of the calibration image.

[0049] In step S31 of creating a pixel shift correction table, the stereo camera device 100, for example, first corrects the inclination of the straight line portion image Im213, then calculates the vertical pixel shift amount of the straight line portion image Im213, and creates the pixel shift correction table.

[0050] 9A and 9B are diagrams showing examples of the tilt when the calibration chart 200 is installed. The calibration chart 200 is installed so that, for example, the linear portion 213, which is the boundary between the dark color portion 211 and the light color portion 212 of the striped calibration pattern 210, is horizontal, and the display surface on which the calibration pattern 210 is displayed is perpendicular to the optical axis of the camera 10.

[0051] However, due to an installation error of the calibration chart 200, tilting may occur due to rotation about the normal to the display surface of the calibration chart 200 as shown in Fig. 9A or tilting may occur due to rotation about the vertical center line of the calibration chart 200 as shown in Fig. 9B. Such tilting of the calibration chart 200 may cause tilting in the straight line portion image Im213 extracted in the previous step S2.

[0052] Fig. 10 is a graph showing the gradient of the straight line portion image Im213 extracted in the previous step S2. In Fig. 10, the x-axis represents the horizontal pixel position, and the y-axis represents the vertical displacement of the pixel. Fig. 10 shows one straight line portion image Im213 out of the multiple straight line portion images Im213 shown in Fig. 8, flipped upside down and enlarged.

[0053] 3A, the influence of light refraction by the windshield WS is small near the position centered on the optical axis OA of the camera 10, i.e., near the position where the horizontal angle of view is 0°. Therefore, in the process of correcting the tilt of the straight portion image Im213, an approximate straight line La of the straight portion image Im213 in a pixel range Rp near the position where the horizontal angle of view is 0° is obtained, and the amount of pixel displacement in the vertical direction that makes the tilt of this approximate straight line La zero is obtained, thereby correcting the tilt of the straight portion image Im213. The pixel range Rp can be set to the number of pixels corresponding to a horizontal angle of view of ±10° to ±20°, for example.

[0054] For example, the stereo camera device 100 performs the above-described tilt correction of the straight line portion images Im213 on all straight line portion images Im213 extracted in the previous step S2. As a result, even if the display surface of the calibration chart 200 is tilted as shown in Figures 9A and 9B, the multiple straight line portion images Im213 extracted in the previous step S2 can be corrected to an untilted state.

[0055] Next, the stereo camera device 100 creates a pixel shift correction table based on, for example, the vertical pixel displacement amounts of the pixels of each of the vertically aligned straight portion images Im213 after tilt correction. Here, the vertical pixel displacement amounts of the pixels of each of the straight portion images Im213 are, for example, the vertical pixel displacement amounts relative to the pixels of a virtual straight portion image, which is an image of the straight portion 213 when the windshield WS does not pass through. In the pixel shift correction table, for example, horizontal pixels, vertical pixels, and pixel displacement amounts are set on the X-axis, Y-axis, and Z-axis, respectively. Next, the stereo camera device 100 performs, for example, step S32 of creating a pupil misalignment correction table.

[0056] FIG. 11 is a schematic vertical cross-sectional view showing pupil shift of a wide-angle lens. In a wide-angle lens, the position of the pupil changes depending on the angle of view. As shown in FIG. 11, light rays B1-B4 incident on lens 11 of camera 10 form an image on image sensor 12 of camera 10. Here, on plane 11a, whose origin is the intersection point of extensions of light rays B1 and B2 incident on lens 11, wide-angle light ray B4 intersects with surface 11a at a position a distance d away from the origin. In this embodiment, this distance d is referred to as the amount of pupil shift. The amount of pupil shift changes depending on the angle of view of light incident on lens 11, the design of lens 11, and other factors.

[0057] 12 is a schematic vertical cross-sectional view illustrating the effect of pupil shift of lens 11 of camera 10. When camera 10 detects linear portion 213 from calibration pattern 210 of calibration chart 200, which is a distance D1 away from surface 11a, the vertical angle of view of light ray B4 incident at a wide vertical angle of view is angle θ1 if pupil shift of lens 11 is not taken into consideration. However, when lens 11 is a wide-angle lens, pupil shift occurs, and therefore the vertical angle of view of light ray B4 must be detected as angle θ2, which takes pupil shift into consideration.

[0058] In step S32 of creating a pupil shift correction table, the stereo camera device 100 uses, for example, design values ​​of the lens 11 of the camera 10 that have been stored in advance in memory. More specifically, the stereo camera device 100 calculates, for example, the angle θ2 from the distance D1 and the amount of pupil shift, calculates the pixel position on the image sensor based on the lens projection, and then calculates the pixel displacement amount for the calculated pixel position relative to the pixel position on the image sensor when pupil shift is not taken into account.

[0059] For example, the stereo camera device 100 performs the above process for all directions (angles of view) in the vertical and horizontal directions to create a correction table for pixel displacement amounts due to pupil shift. The X-axis, Y-axis, and Z-axis of this correction table represent horizontal pixels, vertical pixels, and pixel displacement amounts, respectively.

[0060] The influence of the pupil shift of the lens can be ignored when the distance D1 is sufficiently large. However, as shown in Fig. 6, when the calibration chart 200, which is the calibration object, is placed close to the stereo camera device 100, the influence of the pupil shift cannot be ignored. For this reason, the camera calibration method CM and the stereo camera device 100 of this embodiment perform step S32 of creating a correction table for the pupil shift of the lens 11 of the camera 10 using the design values ​​of the lens 11.

[0061] Next, the stereo camera device 100 performs step S33 of correcting distortion in the calibration image. In step S33, the stereo camera device 100 derives a correction derivative for correcting distortion in the calibration image Im200 acquired in step S1, based on the pixel shift correction table and pupil shift correction table created in the previous steps S31 and S32.

[0062] More specifically, the stereo camera device 100 creates a distortion correction table by, for example, adding or subtracting the pixel displacement amount in the pixel shift correction table and the pixel displacement amount in the pupil misalignment correction table. Furthermore, the stereo camera device 100 calculates, for example, a correction derivative that corrects the pixel displacement amount for each pixel in the created distortion correction table. The stereo camera device 100 then calibrates the camera 10 by applying the calculated correction derivative to an image captured by the camera 10.

[0063] The camera calibration method CM and the operation of the stereo camera device 100 of this embodiment will be described below.

[0064] As described above, the camera calibration method CM of this embodiment is a method for calibrating the camera 10 that captures an image through the windshield WS. The camera calibration method CM includes acquiring a calibration image by capturing an image of a calibration object having a linear portion 213 extending in one direction intersecting the optical axis OA of the camera 10 through the windshield WS (step S1). The camera calibration method CM also includes extracting a linear portion image Im213 corresponding to the linear portion 213 from the calibration image (step S2). The camera calibration method CM also includes correcting distortion of the calibration image based on the linear portion image Im213 and a virtual linear portion image that would be expected if the linear portion 213 were captured by the camera 10 without passing through the windshield WS (step S3).

[0065] According to the camera calibration method CM of this embodiment, for example, a straight-line portion image Im213 of a straight-line portion 213 extending in one direction intersecting the optical axis OA of the camera 10 can be extracted from the calibration pattern 210 of a calibration object, such as the calibration chart 200. This straight-line portion image Im213 includes vertical pixel displacement due to refraction of light passing through the windshield WS, i.e., pixel shift, which is significant in a camera 10 with a wide angle of view. Furthermore, by capturing an image of a calibration object, such as the calibration chart 200 having the straight-line portion 213, with the camera 10, a virtual straight-line portion image to be compared with the straight-line portion image Im213 captured through the windshield WS can be easily generated, for example, as an image of a straight line. Therefore, according to the camera calibration method CM of this embodiment, pixel shift due to the influence of the windshield WS can be corrected, thereby easily calibrating the wide-angle camera 10 that captures images through the windshield WS.

[0066] Furthermore, in the camera calibration method CM of this embodiment, acquiring a calibration image (step S1) includes photographing a calibration object such as the calibration chart 200 so that the calibration image Im200 includes a linear portion image Im213 from one end to the other in one direction intersecting the optical axis OA of the camera 10. By using this method, the camera calibration method CM of this embodiment can more reliably correct pixel shift in an image caused by light passing through both horizontal and vertical ends of the windshield WS, where pixel shift becomes noticeable in a camera 10 with a wide angle of view.

[0067] Furthermore, in the camera calibration method CM of this embodiment, correcting distortion of the calibration image (step S3) includes correcting the tilt of the straight line portion image Im213 based on the tilt of the straight line portion image Im213 with respect to the horizontal or vertical direction in a predetermined range (pixel range Rp) of the calibration image (step S31). With this method, the camera calibration method CM of this embodiment makes it possible to correct the tilt of the straight line portion image Im213 due to an installation error of the calibration chart 200 as a calibration object as shown in Figures 9A and 9B, and to more accurately correct pixel shift.

[0068] Furthermore, in the camera calibration method CM of this embodiment, acquiring a calibration image (step S1) includes photographing a calibration chart 200 as a calibration object positioned in front of the windshield WS in the optical axis OA direction of the camera 10 and above the vehicle V in the vertical direction.

[0069] In this way, the camera calibration method CM of this embodiment can prevent an increase in the length of the calibration chart 200 used to calibrate the wide-angle camera 10, making it easier to install the calibration chart 200 in a manufacturing factory or a maintenance shop. This effect will be described in more detail below with reference to FIG. 13.

[0070] 13 is a graph comparing the relationship between the horizontal length of calibration chart 200 required for calibrating camera 10 and the distance from camera 10 to calibration chart 200 when the horizontal angle of view of camera 10 is 40° and when it is 150°. In stereo camera device 100 of this embodiment, the horizontal angle of view of camera 10 is widened to, for example, about 110° to 150°.

[0071] For example, for a conventional camera with a horizontal angle of view of 40°, the horizontal length required for a calibration chart installed 5 m away from the camera is approximately 3.6 m. However, for example, if the horizontal angle of view of camera 10 of stereo camera device 100 of this embodiment is 150° and calibration chart 200 is installed 5 m away from the camera, the horizontal length of calibration chart 200 required for calibrating camera 10 becomes extremely large. This could make it difficult to install calibration chart 200 in a manufacturing plant or a maintenance shop.

[0072] However, as shown in Fig. 6, in the camera calibration method CM of this embodiment, in step S1 of acquiring a calibration image, an image of a calibration chart 200 is captured as a calibration object that is placed in front of the windshield WS in the direction of the optical axis OA of the camera 10 and above the vehicle V in the vertical direction. This reduces the distance between the camera 10 and the calibration chart 200, making it possible to place the calibration chart 200 within 1 meter of the camera 10, for example.

[0073] As a result, for example, the horizontal length of the calibration chart 200 required to calibrate a camera 10 with a wide horizontal angle of view of 150° can be made approximately the same as the length of the calibration chart required when the camera is installed 5 m in front of a camera with a horizontal angle of view of 40°. Therefore, the calibration chart 200 used to calibrate the wide-angle camera 10 can be easily installed in manufacturing plants and maintenance factories.

[0074] Furthermore, in the camera calibration method CM of this embodiment, correcting distortion in the calibration image (step S3) includes correcting distortion due to lens characteristics of the camera 10, such as pupil shift of the lens 11 (step S32). By using this method, the camera calibration method CM of this embodiment can more accurately correct pixel shift due to the influence of the windshield WS.

[0075] The stereo camera device 100 of this embodiment also includes left and right cameras 10L and 10R that capture images through a windshield WS, and an image processing unit 20, a stereo parallax image generation unit 30, a road surface cross-sectional shape estimation unit 40, a stereoscopic object detection unit 50, a left and right monocular image generation unit 60, or a hybrid object detection unit 70, which function as an acquisition unit, an extraction unit, and a correction unit, as described below. The acquisition unit acquires a calibration image Im200 by capturing an image of a calibration object having a straight portion 213 extending in one direction intersecting the optical axis OA of the camera 10 through the windshield WS using the camera 10. The extraction unit extracts a straight portion image Im213 corresponding to the straight portion 213 from the straight portion image Im213. The correction unit corrects distortion of the calibration image Im200 based on the straight portion image Im213 and a virtual straight portion image that would be expected if the camera 10 captured the straight portion 213 without using the windshield WS.

[0076] With this configuration, the stereo camera device 100 of this embodiment can implement the camera calibration method CM described above, and can achieve the same effects as the camera calibration method CM.

[0077] As described above, according to this embodiment, it is possible to provide a camera calibration method CM and a stereo camera device 100 that can easily calibrate a wide-angle camera 10 that captures images through a windshield WS.

[0078] [Embodiment 2] 1 to 5, a second embodiment of the camera calibration method according to the present disclosure will be described below using Fig. 14. Fig. 14 is a schematic side view showing the second embodiment of the camera calibration method according to the present disclosure.

[0079] The camera calibration method CM of this embodiment differs from the camera calibration method CM of the first embodiment in that the calibration object includes a wire or bar 214, and in that step S1 of acquiring a calibration image includes relatively moving the camera 10 and the wire or bar 214 as the calibration object. Since the other points of the camera calibration method CM of this embodiment are the same as those of the camera calibration method CM of the first embodiment, similar parts will be assigned the same reference numerals and descriptions thereof will be omitted.

[0080] The camera calibration method CM of this embodiment can be implemented, for example, on a production line for a vehicle V. Various parts including the stereo camera device 100 are attached to the vehicle V while the vehicle V is moving, for example, to the right in FIG. 14 . Once the attachment of the stereo camera device 100 to the vehicle V is complete, the camera calibration method CM of this embodiment is implemented. In the camera calibration method CM of this embodiment, a wire or bar 214 and a calibration chart 200 for the stereo viewing area As of the left and right cameras 10L and 10R are used as calibration objects in step S1 of acquiring a calibration image.

[0081] The wire or bar 214 is, for example, supported at both ends so that they are parallel to the horizontal direction and is arranged above the production line of the vehicle V, and is provided so that it can be raised and lowered up and down by a lifting mechanism (not shown). The calibration chart 200 has, for example, any calibration pattern that is generally used in the calibration of a stereo camera, and, like the wire or bar 214, is provided so that it can be raised and lowered up and down by a lifting mechanism (not shown).

[0082] The wire or bar 214 itself has a straight portion, for example, by extending horizontally intersecting the optical axis OA of the camera 10. The wire or bar 214 also has, as a straight portion, an outline or shape extending in one direction intersecting the optical axis OA of the camera 10. Strictly speaking, the wire or bar 214 may bend downward at its center relative to both ends due to gravity, but such bending is slight, and therefore the wire or bar 214 can be used as a calibration object having a straight portion.

[0083] In the camera calibration method CM of this embodiment, when the vehicle V moves down the production line to below the wire or bar 214, the lifting mechanism is activated and the wire or bar 214 is lowered in front of the stereo camera device 100. In step S1 of acquiring an image for calibration, the stereo camera device 100 continuously captures images of the descending wire or bar 214 multiple times using the camera 10. The stereo camera device 100 synthesizes multiple images of the wire or bar 214 to acquire the image for calibration.

[0084] Thereafter, the stereo camera device 100 performs step S2 of extracting a straight portion image corresponding to a straight portion from the calibration image, similar to the above-described embodiment 1. As a result, a straight portion image similar to the straight portion image Im213 shown in Fig. 8 of the above-described embodiment 1 is extracted. Thereafter, the stereo camera device 100 performs step S3 of correcting distortion of the calibration image based on a virtual straight portion image that would be expected if the straight portion of the wire or bar 214 were photographed by the camera 10 without passing through the windshield WS, and the straight portion image extracted in step S2, similar to the above-described embodiment 1.

[0085] As a result, the camera calibration method CM and stereo camera device 100 of this embodiment can achieve the same effects as the camera calibration method CM and stereo camera device 100 of the above-described embodiment 1. Furthermore, when the wire or bar 214 has finished descending to the predetermined position, the calibration chart 200 is lowered to a predetermined position in front of the calibration chart 200 by the lifting device.

[0086] The calibration chart 200 of this embodiment is captured in order to calibrate the camera 10 using an image of the stereo viewing area As of the camera 10. Therefore, the horizontal length of the calibration chart 200 of this embodiment is approximately the same as the horizontal length of a calibration chart used to calibrate a normal stereo camera that does not use a wide-angle camera. The stereo camera device 100 calibrates the stereo viewing areas As of the left and right cameras 10L and 10R by capturing images of the calibration chart 200 through the windshield WS with the left and right cameras 10L and 10R.

[0087] Thereafter, the wire or bar 214 and the calibration chart 200 are raised by the lifting device, and the vehicle V passes under the wire or bar 214 and the calibration chart 200 and is sent to the next process in the production line. In this way, according to the camera calibration method CM of this embodiment, the camera 10 of the stereo camera device 100 can be calibrated in the production line for the vehicle V without reducing the efficiency of the production line.

[0088] Furthermore, according to the camera calibration method CM of this embodiment, by photographing the wire or bar 214, it is possible to calibrate the monocular vision areas Am of the left and right cameras 10L, 10R, similarly to the camera calibration method CM of the first embodiment. In addition, according to the camera calibration method CM of this embodiment, after the calibration of the monocular vision areas Am of the left and right cameras 10L, 10R is completed, by photographing the calibration chart 200 with the left and right cameras 10L, 10R, it is possible to continuously calibrate the stereo vision areas As of the left and right cameras 10L, 10R. This allows the calibration of the left and right cameras 10L, 10R of the stereo camera device 100 to be performed efficiently.

[0089] As described above, according to this embodiment, it is possible to provide a camera calibration method CM and a stereo camera device 100 that can easily calibrate a wide-angle camera 10 that captures images through a windshield WS.

[0090] [Embodiment 3] 1 to 5 and using Fig. 15, a third embodiment of the camera calibration method according to the present disclosure will be described below. Fig. 15 is a perspective view showing an example of a calibration object in the third embodiment of the camera calibration method according to the present disclosure.

[0091] The camera calibration method CM of this embodiment differs from the camera calibration method CM of the first embodiment in that the calibration object photographed by the camera 10 through the windshield WS in step S1 of acquiring a calibration image is a building BLD. Other aspects of the camera calibration method CM of this embodiment are similar to those of the camera calibration method CM of the first embodiment, and therefore similar parts are designated by the same reference numerals and description thereof will be omitted.

[0092] In this embodiment, when the stereo camera device 100 starts step S1 of acquiring a calibration image, the camera 10 captures an image of a building BLD having a linear portion 213 extending in one direction intersecting the stereo viewing area As of the camera 10 through the windshield WS. In the example shown in Fig. 15, the building BLD is, for example, a factory having window frame portions extending horizontally as the linear portion 213. Note that the building BLD is not particularly limited as long as it has the linear portion 213, and may be an office building, a shopping mall, or the like.

[0093] Thereafter, the stereo camera device 100 performs step S2 of extracting a straight portion image Im213 corresponding to the straight portion 213 from a calibration image obtained by photographing a building BLD having the straight portion 213, similar to the above-described embodiment 1. Furthermore, the stereo camera device 100 performs step S3 of correcting distortion of the calibration image based on the straight portion image Im213 and a virtual straight portion image that would be assumed if the straight portion 213 were photographed by the camera 10 without passing through the windshield WS.

[0094] The camera calibration method CM of this embodiment not only achieves the same effects as the camera calibration method CM of the first embodiment, but also can be implemented without modifying the production line because it does not require the calibration chart 200. Furthermore, the camera calibration method CM of this embodiment can also correct image shifts that occur due to changes in the stereo camera device 100 over time or changes in temperature.

[0095] [Embodiment 4] 1 to 5, and using FIGS. 16A, 16B, and 17, a fourth embodiment of the camera calibration method according to the present disclosure will be described. FIG. 16A is a front view of a first calibration chart 200 in the fourth embodiment of the camera calibration method according to the present disclosure. FIG. 16B is an image diagram of a camera 10 capturing an image of the first calibration chart shown in FIG. 16A. FIG. 17 is a front view of a second calibration chart 300 in the camera calibration method CM according to this embodiment.

[0096] The camera calibration method CM of this embodiment detects and calibrates horizontal pixel shifts caused by refraction of light passing through the windshield WS. Furthermore, in this embodiment, the stereo camera device 100 does not have, for example, the monocular vision area Am shown in FIG. 1, and the imaging areas AL and AR of the left and right cameras 10L and 10R entirely overlap in the stereo vision area As. Furthermore, the stereo camera device 100 does not have, for example, the left and right monocular image generation unit 60 and the hybrid object detection unit 70 shown in FIG. 2.

[0097] The first calibration chart 200 is a calibration chart 200 having a striped calibration pattern 210, similar to the calibration chart 200 of the first embodiment. The striped calibration pattern 210 of the calibration chart 200 is, for example, a pattern in which striped dark color portions 211 and striped light color portions 212 extending along the vertical direction are alternately arranged in the horizontal direction. The calibration chart 200 has a linear portion 213 extending in one direction intersecting the optical axis OA of the camera 10. This linear portion 213 is, for example, a boundary line between the dark color portions 211 and light color portions 212 of the striped calibration pattern 210 drawn on the calibration chart 200, and extends in the vertical direction perpendicular to the optical axis OA of the camera 10.

[0098] In step S1 of acquiring a calibration image, the calibration image Im200 of the calibration chart 200 captured by the camera 10 through the windshield WS exhibits a larger horizontal pixel shift as the angle of view becomes wider in the horizontal and vertical directions, as shown in Fig. 16B. Therefore, in this modification, similar to the first embodiment, the horizontal pixel shift can be corrected by performing step S2 of extracting a straight portion image Im213 and step S3 of correcting distortion in the calibration image Im200, thereby achieving the same effects as the first embodiment.

[0099] Thereafter, the stereo camera device 100 calibrates the absolute value of the parallax of the stereo viewing area As of the left and right cameras 10L and 10R using a second calibration chart 300 shown in FIG. 17. The stereo camera device 100 captures an image of the calibration chart 300, which is placed at a predetermined distance from the left and right cameras 10L and 10R. Next, the distance between the left and right cameras 10L and 10R and the calibration chart 300 is changed. Thereafter, the stereo camera device 100 captures an image of the calibration chart 300.

[0100] The horizontal dimension of the calibration chart 300 is larger than the vertical dimension, and the calibration chart 300 is photographed over the entire stereo viewing area As. Next, the stereo camera device 100 detects distortion based on the change in the center of gravity of the calibration pattern 301 of the calibration chart 300 photographed at different distances and the amount of change in distance, and calibrates the left and right cameras 10L and 10R using the detected distortion.

[0101] According to this embodiment, the entire field of view can be calibrated by correcting the relative pixel shift in the horizontal direction of the straight-line portion image Im213 using the first calibration chart 200 and calibrating the absolute value of distortion using the second calibration chart 300. Note that instead of the calibration chart 200, it is also possible to use, as the calibration object, a building BLD having vertical straight-line portions 213 or a wire or bar 214 arranged in the vertical direction.

[0102] The above has described in detail embodiments of a camera calibration method and a stereo camera device according to the present disclosure using the drawings, but the specific configurations are not limited to these embodiments, and design changes and the like that do not deviate from the gist of the present disclosure are also included in the present disclosure.

[0103] For example, instead of the striped calibration pattern 210 of the calibration chart 200 shown in Fig. 7A, the calibration object used in step S1 of acquiring the calibration image may be a calibration chart 200 on which only straight lines corresponding to the straight portion 213 are drawn. Alternatively, the calibration object may be one in which straight lines corresponding to the straight portion 213 are projected onto the wall of a building or a screen by a projector. Alternatively, the calibration object may be one in which a plurality of wires or bars 214 extending horizontally are arranged at equal intervals in the vertical direction.

[0104] 18 may be used as the calibration object used in step S1 of acquiring the calibration image. In this case, by detecting the boundary between the dark color portion 211 and the light color portion 212 as the linear portion 213, the linear portion 213 that crosses the entire calibration chart 200 can be extracted.

[0105] In contrast, in a conventional camera calibration method that uses a checkered pattern calibration chart, for example, dark color areas 211 are located in the upper left and lower right, and light color areas 212 are located in the upper right and lower left, and a point CP is detected at the center of four sections. Therefore, if the calibration chart 200 is brought closer to the lens 11, the resolution decreases, causing variations in the detection of the point CP at the center of the four sections, which could result in errors in the calibration. [Explanation of symbols]

[0106] 10 Camera 20 Image processing unit (acquisition unit, extraction unit, correction unit) 100 Stereo camera device 200 Calibration Chart (Calibration Object) 210 Calibration Pattern 213 Straight section 214 Wire or bar (calibration object) BLD building (calibration object) How to calibrate a CM camera Im200 calibration image Im213 Linear section image OA optical axis WS Windshield

Claims

1. 1. A method for calibrating a camera that captures images through a windshield, comprising: acquiring a calibration image by photographing a calibration object, which is a wire or a bar, having a linear portion extending in one direction intersecting an optical axis of the camera through the windshield with the camera; extracting a straight line portion image corresponding to the straight line portion from the calibration image; A camera calibration method including correcting distortion of the calibration image based on a virtual straight section image that would be expected if the straight section were photographed by the camera without passing through the windshield and the straight section image.

2. 2. The camera calibration method of claim 1, wherein acquiring the calibration image includes photographing the calibration object so that the calibration image includes an image of the straight portion from one end to the other end in one direction intersecting the optical axis of the camera.

3. 2. The camera calibration method of claim 1, wherein correcting the distortion of the calibration image includes correcting the tilt of the straight portion image based on the tilt of the straight portion image relative to the horizontal or vertical direction within a predetermined range of the calibration image.

4. 2. The camera calibration method of claim 1, wherein acquiring the calibration image includes photographing the calibration object positioned in a position forward of the windshield of the vehicle in the optical axis direction of the camera and above the vehicle in the vertical direction.

5. The camera calibration method according to claim 1 , wherein correcting the distortion of the calibration image includes correcting distortion caused by lens characteristics of the camera.

6. The camera calibration method according to claim 1 , wherein acquiring the calibration image includes moving the camera and the calibration object relative to each other.

7. 1. A method for calibrating a camera that captures images through a windshield, comprising: capturing an image of a calibration object, which is a building, through the windshield and has a linear portion extending in one direction intersecting an optical axis of the camera, to obtain a calibration image; extracting a straight line portion image corresponding to the straight line portion from the calibration image; A camera calibration method that includes correcting distortion of the calibration image based on a virtual straight section image that would be expected if the straight section were photographed by the camera without passing through the windshield and the straight section image.

8. 1. A method for calibrating a camera that captures images through a windshield, comprising: acquiring a calibration image by photographing a calibration object having a linear portion extending in one direction intersecting an optical axis of the camera through the windshield with the camera; extracting a straight line portion image corresponding to the straight line portion from the calibration image; correcting distortion of the calibration image based on a virtual straight-line portion image that would be assumed if the straight-line portion were photographed by the camera without passing through the windshield and the straight-line portion image; A method for calibrating a camera, wherein acquiring the calibration image includes moving the camera and the calibration object relative to each other.

Images