Stereo camera device and calibration method

Abstract

To provide a stereo camera device capable of calibrating parallax of a stereo camera without a chart, despite that attachment of front glass and attachment of the stereo camera fluctuate from a reference attachment position.SOLUTION: A stereo camera device 10 includes: a storage unit 103 for storing correlation information of a parallax error in a horizontal direction relative to a vertical shift of a plurality of images that are imaged respectively by a stereo camera 21 through a refractive body for refracting light, obtained based on refractive body characteristics containing a posture of the refractive body and a shape of the refractive body mounted on a vehicle, and a posture of the stereo camera 21 arranged on the vehicle; a matching unit 104 for mutually matching the plurality of images obtained by the stereo camera 21 having imaged the same subject, in order to obtain a vertical shift of the plurality of images; a parallax error estimation unit 106 for estimating a parallax error of the plurality of matched images, on the basis of the vertical shift and the correlation information; and a distance calculation unit 107 for calibrating the parallax of the matched images, by using the parallax error.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a stereo camera device and a calibration method. [Background technology]

[0002] With the development of computer vision technology, stereo cameras are being used more and more widely. Meanwhile, with the recent trend toward AI in automobiles, technology for recognizing images captured by stereo cameras is also widely used in ADAS (Advanced Driver Assistance Systems) and autonomous driving. Unlike ordinary cameras, stereo cameras can achieve 3D object recognition using the images they capture. 3D object recognition is a technology that uses two cameras installed a certain distance apart (baseline length) to capture images of the same object, and then calculates the distance from the two cameras to the object based on the parallax of the object using the principle of triangulation, thereby creating a 3D image.

[0003] Conventional stereo cameras primarily support vehicle avoidance technologies by acquiring distance information to objects in front of the vehicle. This technology is primarily used when a vehicle is traveling straight ahead, allowing the vehicle to avoid objects in front of the vehicle. However, as safety standards become more stringent, there is also a demand for technology that can detect and avoid pedestrians and cyclists at intersections before the vehicle enters the intersection.

[0004] In order for a vehicle to avoid pedestrians and cyclists at intersections, a wider angle of view than conventional stereo cameras for ADAS is required. However, using a wide-angle lens to widen the angle of view increases the distortion of the captured image. Therefore, if the calibration to correct the image distortion is imperfect, pixel misalignment occurs in the images captured by the left and right cameras of the stereo camera. As a result, problems occur in the process of measuring the distance to and detecting objects in the images, which can result in incorrect distance measurements. In the following explanation, vertical pixel misalignment in the images captured by the left and right cameras is referred to as "vertical misalignment." Horizontal pixel misalignment is referred to as "parallax error" because it affects the parallax detection of the stereo camera.

[0005] To address such a wide angle of view, the following problem arises. In the wide-angle range, the angle of incidence on the vehicle's windshield increases, resulting in increased pixel shift, or parallax, due to the influence of refraction on the windshield. On the other hand, with conventional narrow-angle stereo cameras, the angle of incidence on the windshield is small, so pixel shift caused by the windshield hardly needed to be considered. For this reason, a method for calibrating parallax for a wide-angle stereo camera that can capture the outside world through the windshield was needed.

[0006] As a calibration method for such a wide angle of view, the techniques described in Patent Documents 1 and 2 below have been proposed. Patent Document 1 addresses the issue that "the wider the angle of view, the greater the parallax shift caused by the refraction of the light path when passing through the windshield," and describes that "parallax shift correction parameters are compiled as a function or table for the horizontal position within the angle of view, and this is stored in a storage device as optical axis characteristics," and that "the image is further corrected using the optical axis characteristics obtained by aiming."

[0007] Patent Document 2 addresses the issue that "windshields differ for each vehicle model, and therefore the glass distortion also differs for each vehicle model. If an attempt is made to evaluate the effect of glass distortion on captured images, actual driving and simulations would need to be performed for all vehicle models, which is inconvenient as it requires a huge amount of man-hours and expenses." It then goes on to state that "multiple geometric tables are combined, and the geometric distortion correction data stored in each of the geometric tables is used to remove the geometric distortion between the stereo camera and each protective hood, or to reproduce the geometric distortion of different protective hoods." [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Patent Publication No. 2021-25868 [Patent Document 2] Japanese Patent Application Laid-Open No. 2017-62198 Summary of the Invention [Problem to be solved by the invention]

[0009] The technology disclosed in Patent Document 1, which corrects images based on the design shape of the windshield (vehicle interior window), works if all windshields installed in a vehicle have the same shape. However, in reality, the positional relationship between the windshield and the camera varies depending on their installation positions. For this reason, if there are manufacturing variations in the horizontal and vertical curvature radii of the windshield, the thickness of the windshield, or the relative installation positions of the windshield and the stereo camera, the tolerance (for example, a parallax error of 0.25 pixels or less, as shown in Figure 5, described below) cannot be met.

[0010] Furthermore, the technology disclosed in Patent Document 2 has a problem in widening the angle of the stereo camera. In this technology, an image is corrected using a calibration chart board (hereinafter referred to as "chart"), but as the angle of the stereo camera becomes wider, a larger chart needs to be used.

[0011] The present invention has been made in view of the above-mentioned problems, and has as its object to perform calibration for correcting the parallax of a stereo camera that captures an image of the outside world through a refractive body without using a chart. [Means for solving the problem]

[0012] The stereo camera device of the present invention includes a memory unit that stores correlation information of horizontal parallax error with respect to vertical shift of multiple images captured by multiple cameras through a refracting body that refracts light, the correlation information being calculated based on the attitude of the refracting body attached to the vehicle, refracting body characteristics including the shape of the refracting body, and the attitudes of the multiple cameras arranged on the vehicle; a matching unit that matches multiple images of the same subject captured by the multiple cameras to determine the vertical shift of the multiple images; a parallax error estimating unit that estimates the parallax error of the matched multiple images based on the vertical shift and the correlation information; and a distance calculation unit that corrects the parallax of the matched images using the parallax error. [Effects of the Invention]

[0013] According to the present invention, it is possible to perform calibration to correct the parallax of a stereo camera that captures an image of the outside world through a refractive body without using a chart. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0014] [Figure 1] This shows the relationship between the angle of view of the stereo camera and the chart width. [Figure 2] FIG. 1 is an explanatory diagram showing an example in which a stereo camera according to an embodiment of the present invention captures an image of a subject. [Figure 3] 1 is a block diagram showing an example of the overall configuration of a stereo camera device according to an embodiment of the present invention; [Figure 4] FIG. 2 is a block diagram illustrating an example of the hardware configuration of a computer according to an embodiment of the present invention. [Figure 5]FIG. 10 is a diagram showing an example of a simulation result of a parallax error calculated when a windshield is sandwiched between a stereo camera and an object according to an embodiment of the present invention. [Figure 6] 10 is a graph showing an example of a δ parallax error versus a horizontal angle of view of a stereo camera. [Figure 7] 1 is a flowchart illustrating an example of a calibration method according to an embodiment of the present invention. [Figure 8] FIG. 10 is a diagram illustrating an example of horizontal stereo matching of left and right images according to an embodiment of the present invention. [Figure 9] FIG. 10 is a diagram illustrating an example of vertical matching of left and right images according to an embodiment of the present invention. [Figure 10] FIG. 10 is a schematic diagram illustrating an example of a procedure for calculating a parallax error according to an embodiment of the present invention. [Figure 11] FIG. 2 is a diagram illustrating a configuration of a correlation table according to an embodiment of the present invention. [Figure 12] 5A and 5B are diagrams illustrating the principle by which a parallax error estimating unit according to an embodiment of the present invention estimates a parallax error from a vertical deviation. [Figure 13] FIG. 10 is a diagram showing an example of nine types of variation according to one embodiment of the present invention. [Figure 14] FIG. 10 is a diagram illustrating the correlation between δ vertical shift and δ parallax error according to an embodiment of the present invention. [Figure 15] FIG. 10 is a diagram showing simulation results of residual values ​​of parallax error obtained for horizontal angles of view of 20 degrees, 40 degrees, and 60 degrees according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In this specification and drawings, when there are multiple components that have substantially the same or similar functions, they may be described using the same reference numerals with different subscripts. Furthermore, when it is not necessary to distinguish between these multiple components, the subscripts may be omitted. The embodiments of the present invention described below are examples for explaining the present invention, and some omissions and simplifications have been made as appropriate for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural. The present invention is also applicable to a vehicle control computing device capable of communicating with an on-board ECU (Electronic Control Unit) for, for example, an Advanced Driver Assistance System (ADAS) or Autonomous Driving (AD).

[0016] [One embodiment] A stereo camera device and a calibration method according to an embodiment of the present invention are an example of an invention that solves the above-mentioned problems. The details of the stereo camera device and the calibration method according to this embodiment will be described below. In the following description, correcting the parallax between two images will be referred to as "calibration."

[0017] The stereo camera is calibrated in four stages: (1) the manufacturing plant, (2) the vehicle plant, (3) the dealership, and (4) while the vehicle is in operation. (1) Calibration at the manufacturing plant When a stereo camera is manufactured at a manufacturing factory, the optical axis is adjusted at the manufacturing factory. (2) Calibration at the vehicle factory After the stereo camera is shipped from the manufacturing plant, it is installed on the vehicle at the vehicle factory and calibrated. Calibration performed at the vehicle factory is also called "aiming." Information such as correction tables obtained during aiming is stored in the stereo camera.

[0018] (3) Calibration at the dealer After the calibration is completed at the vehicle factory, the vehicle is sent to the dealer for dealer calibration, which is mainly performed after replacing the windshield during vehicle repairs. (4) Calibration while the vehicle is running Finally, calibration is performed while the vehicle is in motion, which is called "auto-tuning." The calibration method according to this embodiment is mainly used (3) during dealer aiming and (4) while the vehicle is running. Dealers often have limited space to install devices such as charts, so calibration without a chart according to this embodiment is desirable.

[0019] Here, the chart width of the chart used for calibration will be described with reference to FIG. Figure 1 shows the relationship between the stereo camera's angle of view and chart width. The horizontal axis of Figure 1 represents the angle of view [degrees], and the vertical axis represents the required chart width [m]. In this example, the distance between the stereo camera and the chart is 3 m. As shown in Figure 1, for example, if the angle of view is 40 degrees, the required chart width is approximately 2 m. On the other hand, if the angle of view is 120 degrees, the required chart width exceeds 10 m. Charts with such a wide chart width are difficult to handle, and require major modifications to vehicle manufacturing plants and dealer inspection plants.

[0020] FIG. 2 is an explanatory diagram showing an example in which the stereo camera 21 captures an image of a subject 24. As shown in FIG. The stereo camera 21 is mounted inside the windshield 22 (inside the vehicle 20) and is composed of two cameras arranged horizontally, one on the left and one on the right. The refractive body (windshield 22) has the property of refracting light that passes through it, and may be either transparent or translucent. Of the two cameras that make up the stereo camera 21, the left camera is referred to as the left camera 21L, and the right camera is referred to as the right camera 21R. The left camera 21L and the right camera 21R are sometimes referred to as the "left and right cameras" for short, and are sometimes collectively referred to as the stereo camera 21.

[0021] 2 shows a scene in which stereo camera 21 detects subject 24 through windshield 22. Fig. 2 shows that the horizontal angle of view of stereo camera 21 (approximately 120 degrees) is wider than the horizontal angle of view of a conventional camera (approximately 40 degrees). Note that left camera 21L and right camera 21R are each capable of capturing an image of the outside world including subject 24 at a horizontal angle of view of approximately 120 degrees.

[0022] The orientation of the windshield 22 when attached to a vehicle (vehicle 20) varies depending on the vehicle model. Even for the same vehicle model, the orientation of the windshield 22 when attached to a vehicle (vehicle 20) may vary slightly.

[0023] Next, an example of the configuration of the stereo camera device 10 according to an embodiment will be described with reference to FIG. FIG. 3 is a block diagram showing an example of the overall configuration of the stereo camera device 10 according to an embodiment. The stereo camera device 10 includes an imaging unit 100, an external input unit 101, a correction unit 102, a storage unit 103, a matching unit 104, a noise removal unit 105, a parallax error estimation unit 106, a distance calculation unit 107, and a control unit 108. The stereo camera device 10 may be configured as a part of an ECU (Electronic Control Unit) mounted on a vehicle.

[0024] The imaging unit 100 is connected to the correction unit 102. The imaging unit (imaging unit 100) acquires multiple images captured by multiple cameras (stereo cameras 21) arranged on a vehicle (vehicle 20) through a refractive body (windshield 22). The imaging unit 100 is a device capable of acquiring images of the vehicle's external environment, such as a visible light camera or an infrared camera, and is composed of multiple cameras arranged horizontally. Therefore, the imaging unit 100 may be configured not only as a camera capable of detecting the external environment through visible light, but also as a camera capable of detecting the external environment based on light rays other than visible light, such as infrared light. However, the imaging unit 100 is not limited to a visible light camera or an infrared camera. In this case, instead of the imaging unit 100, an external environment detection unit capable of detecting the external environment using light rays of a predetermined wavelength may be configured. Furthermore, the object (e.g., subject 24) captured by the imaging unit 100 may be a person, another vehicle, a landscape, or a conventional calibration chart.

[0025] The imaging unit 100 is composed of a left camera 21L and a right camera 21R (stereo camera 21) shown in FIG. 2. The imaging unit 100 outputs two images, left and right, of the scenery ahead of the vehicle to the correction unit 102. Each pixel in the image indicates a luminance value. The image is not limited to an RGB image separated from a color image, but may also be a monochrome image.

[0026] The external input unit 101 is connected to the storage unit 103. The external input unit 101 receives correlation information between the parallax error and vertical deviation between two images, which information has been calculated in advance based on design values ​​such as variations in the mounting orientation of the stereo camera 21 and the mounting orientation and shape of the windshield 22, and sends the information to the storage unit 103. The storage unit 103 stores the correlation information between the parallax error and vertical deviation as a correlation table shown in FIG. 11, which will be described later. In this specification, image deviation in the horizontal direction is referred to as "parallax error." The correlation information between the parallax error and vertical deviation is illustrated in an image diagram in FIG. 10, which will be described later, and an example of a proportional relationship is shown in FIG. 14, which will be described later.

[0027] Furthermore, external input unit 101 inputs a reference parallax error and a reference vertical displacement that have been calculated in advance by an external computing device based on design values ​​such as variations in the mounting attitude of stereo camera 21 and the mounting attitude and shape of windshield 22, and sends these to storage unit 103. Here, the reference parallax error and the reference vertical displacement are values ​​that arise from refraction at windshield 22 (design values), and are different from the above-mentioned correlation information between parallax error and vertical displacement calculated from the mounting attitude, shape, etc. of stereo camera 21 and windshield 22. Storage unit 103 stores the reference parallax error and the reference vertical displacement. The reference parallax error and the reference vertical displacement will be described later using Figures 8 and 9.

[0028] Furthermore, the external input unit 101 inputs a correction table obtained by aiming at the factory and sends it to the storage unit 103. The storage unit 103 stores the correction table obtained by aiming at the factory.

[0029] The correction unit 102 is connected to the imaging unit 100, the storage unit 103, and the matching unit 104. The correction unit 102 corrects both the left and right images acquired from the imaging unit 100 using a correction table read from the storage unit 103. The correction of the left and right images here is a process of correcting the influence of the distortion of the lenses of the left camera 21L and the right camera 21R, and is not a process of correcting the parallax of the left and right images. Then, the correction unit 102 transmits the corrected left and right images to the matching unit 104.

[0030] The storage unit 103 is connected to the external input unit 101, the correction unit 102, the matching unit 104, the noise removal unit 105, the parallax error estimation unit 106, and the distance calculation unit 107. The storage unit 103 is a non-volatile memory configured inside the stereo camera 21 and can store various types of information. For example, the storage unit (storage unit 103) stores a reference parallax error and a reference vertical shift that are calculated in advance based on the design values ​​of the refractive body (windshield 22). The reference parallax error and the reference vertical shift are information input from the external input unit 101 and represent the parallax error and the vertical shift before the windshield 22 is attached to the vehicle 20. The reference parallax error and the reference vertical shift are stored in the storage unit (storage unit 103) when the multiple cameras (stereo camera 21) are shipped.

[0031] The storage unit 103 also stores a correction table input from the external input unit 101 and sends the correction table to the correction unit 102. The storage unit (storage unit 103) also stores correlation information of horizontal parallax error with respect to vertical shifts of multiple images captured by multiple cameras (left camera 21L and right camera 21R) of the vehicle (vehicle 20) through a refractive body (windshield 22) that refracts light. This correlation information is calculated based on the orientation of the refractive body (windshield 22) attached to the vehicle (vehicle 20), refractive body characteristics (characteristics of the windshield 22) including the shape of the refractive body (windshield 22), and the orientations of the multiple cameras (left camera 21L and right camera 21R) arranged on the vehicle (vehicle 20). The windshield 22 has refractive body characteristics (characteristics of the windshield 22) such as the horizontal curvature of the windshield 22, the vertical curvature of the windshield 22, and the thickness of the windshield 22.

[0032] The storage unit 103 also stores the disparity and vertical displacement input from the matching unit 104. Here, the storage unit 103 transmits the vertical displacements of multiple frames input from the matching unit 104 through multiple processes to the noise removal unit 105. The storage unit 103 also transmits correlation information between the vertical displacement and the disparity error to the disparity error estimation unit 106. The storage unit 103 also transmits the disparity input from the matching unit 104 to the distance calculation unit 107.

[0033] The reference parallax error and the reference vertical shift may be stored in the storage unit (storage unit 103) before the refractive body (windshield 22) is attached to the vehicle (vehicle 20). In this case, the amount of change with respect to the reference parallax error after the refractive body (windshield 22) is actually attached to the vehicle (vehicle 20) is the δ parallax error, and the amount of change with respect to the reference vertical shift is the δ vertical shift.

[0034] The matching unit 104 is connected to the correction unit 102 and the storage unit 103. The matching unit (matching unit 104) matches multiple images of the same subject captured by multiple cameras (stereo cameras 21) to determine the vertical deviation of the multiple images. For example, the matching unit 104 acquires the corrected left and right images from the correction unit 102, matches the left and right images, and calculates the parallax and vertical deviation. The matching unit 104 transmits the calculated parallax and vertical deviation to the storage unit 103, and stores the parallax and vertical deviation in the storage unit 103.

[0035] The noise removal unit 105 is connected to the storage unit 103 and the parallax error estimation unit 106. The noise removal unit 105 calculates the average value of the vertical displacements of multiple frames read out from the storage unit 103. Next, the noise removal unit 105 uses a filter to remove invalid vertical displacements that are mixed in as noise from the calculation of the average value, and transmits the average value of the vertical displacement to the parallax error estimation unit 106. Note that if noise removal is not necessary, the stereo camera device 10 does not need to be provided with the noise removal unit 105.

[0036] The parallax error estimation unit 106 is connected to the storage unit 103, the noise reduction unit 105, and the distance calculation unit 107. The parallax error estimation unit (parallax error estimation unit 106) estimates the parallax error of the matched images based on the vertical shift and correlation information read from the storage unit 103. To this end, upon receiving the average value of the vertical shift from the noise reduction unit 105, the parallax error estimation unit 106 reads correlation information between the parallax error and the vertical shift from the storage unit 103. Furthermore, the parallax error estimation unit 106 reads the reference parallax error and the reference vertical shift from the storage unit 103, calculates the δ vertical shift based on the average value of the vertical shift received from the noise reduction unit 105, and then calculates the δ parallax error.

[0037] Therefore, the parallax error estimating unit (parallax error estimating unit 106) calculates a δ vertical shift, which is the amount of change in vertical shift caused by the refractive body (windshield 22) actually attached to the vehicle (vehicle 20), based on the reference vertical shift and the vertical shift read out from the memory unit (memory unit 103). Next, the parallax error estimating unit (parallax error estimating unit 106) calculates a δ parallax error, which is the amount of change in parallax error caused by the refractive body (windshield 22) actually attached to the vehicle (vehicle 20), based on the calculated δ vertical shift and the correlation information read out from the memory unit (memory unit 103). Then, the parallax error estimating unit 106 transmits the estimated δ parallax error to the distance calculating unit 107.

[0038] Furthermore, the parallax error estimation unit 106 transmits the calculated δ parallax error and δ vertical shift to the storage unit 103, where the δ parallax error and δ vertical shift are stored. As described above, the δ parallax error and δ vertical shift represent the amount of change before and after the windshield 22 is attached to the vehicle 20. That is, the δ parallax error is the amount of change from the reference parallax error after the shipment of the multiple cameras (stereo camera 21), and the δ vertical shift is the amount of change from the reference vertical shift. The δ vertical shift and δ parallax error will be described in detail later with reference to FIGS. 8 and 9.

[0039] If the stereo camera device 10 is configured not to include the noise removal unit 105, the parallax error estimation unit (parallax error estimation unit 106) calculates the average value of the vertical shift by accumulating the vertical shifts determined by the matching unit 104 multiple times at the same position in multiple images captured at different times. Then, the parallax error estimation unit (parallax error estimation unit 106) estimates the parallax error based on the average value of the vertical shift and the correlation information. By calculating the average value of the vertical shift in this way, even if the vertical shift calculated only once is an outlier, it is possible to prevent the parallax error estimated due to this vertical shift from becoming too large.

[0040] The distance calculation unit 107 is connected to the storage unit 103 and the parallax error estimation unit 106. The distance calculation unit (distance calculation unit 107) corrects the parallax of the matched images using the parallax error. At this time, the distance calculation unit (distance calculation unit 107) corrects the parallax using the reference parallax error and the δ parallax error. Therefore, the distance calculation unit 107 receives the δ parallax error from the parallax error estimation unit 106 and the parallax and the reference parallax error from the storage unit 103. Then, the distance calculation unit 107 performs parallax correction to remove the reference parallax error and the δ parallax error from the parallax. Furthermore, the distance calculation unit (distance calculation unit 107) calculates the distance from the vehicle (vehicle 20) to an external object using the parallax corrected using the parallax error. Thereafter, the distance calculation unit 107 transmits distance information including the distance calculated for each object to the control unit 108.

[0041] The control unit 108 is connected to the distance calculation unit 107. When the control unit 108 receives distance information from the distance calculation unit 107, it performs automatic control such as issuing a warning to alert the driver or applying the brakes, depending on the distance to each object.

[0042] The above-mentioned functional blocks 100 to 106 are responsible for processing that is performed before the vehicle starts running, for example, by a vehicle dealer, while the functional blocks 100 to 107 are responsible for processing that is performed while the vehicle is running.

[0043] <Example of computer hardware configuration> Next, an example of the hardware configuration of the computer 50 that constitutes the stereo camera device 10 will be described. 4 is a block diagram showing an example of the hardware configuration of the calculator 50. The calculator 50 is an example of hardware used as a computer that can operate as the stereo camera device 10 according to this embodiment. The stereo camera device 10 according to this embodiment realizes a parallax error correction method in which the functional blocks shown in FIG. 3 cooperate with each other by causing the calculator 50 (computer) to execute a program.

[0044] The computer 50 includes a CPU (Central Processing Unit) 51, a ROM (Read Only Memory) 52, and a RAM (Random Access Memory) 53, each connected to a bus 54. The computer 50 further includes a non-volatile storage 55 and a network interface 56.

[0045] The CPU 51 reads out program code of software that realizes each function according to this embodiment from the ROM 52, loads it into the RAM 53, and executes it. Variables, parameters, etc. generated during the calculation process of the CPU 51 are temporarily written to the RAM 53, and these variables, parameters, etc. are read out by the CPU 51 as appropriate. However, the CPU 51 may be used in combination with a GPU (Graphics Processing Unit). The functions of the functional blocks 100 to 107 shown in FIG. 3 are realized by the CPU 51.

[0046] The nonvolatile storage 55 may be, for example, a hard disk drive (HDD), a solid state drive (SSD), a flexible disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, or a nonvolatile memory. The nonvolatile storage 55 stores an operating system (OS), various parameters, and programs for operating the computer 50. The ROM 52 and the nonvolatile storage 55 store programs, data, and the like required for the CPU 51 to operate, and are used as examples of computer-readable, non-transitory storage media that store programs executed by the computer 50. The function of the storage unit 103 shown in FIG. 3 is realized by the RAM 53, but may also be realized by the nonvolatile storage 55.

[0047] The network interface 56 may be, for example, a network interface card (NIC), and various data can be transmitted and received between devices via an in-vehicle local area network (LAN) or dedicated line connected to the terminal of the NIC.

[0048] Here, the conventional problems will be quantitatively explained. Figure 5 shows an example of the simulation results of the parallax error calculated when the windshield is sandwiched between the stereo camera and the subject. The vertical axis of Figure 5 represents the parallax error [pixels], and the horizontal axis represents the horizontal angle of view [degrees] of the stereo camera. The calculation conditions for the parallax error simulation process are as follows. The allowable value for the parallax error is 0.25 pixels. <Calculation conditions (design conditions)> Lens focal length: 5.41mm Lens projection: ftanθ Glass curvature radius (horizontal): 5.5m Glass curvature radius (vertical): 3.5m Glass refractive index: 1.52 Glass thickness: 4.7mm Glass tilt: 30 degrees Sensor pixel pitch: 2.25μm Camera baseline length: 180mm Lens-to-glass distance: 40mm

[0049] 5, for example, when the horizontal angle of view exceeds 70 degrees (35 degrees left + 35 degrees right), the parallax error exceeds the allowable value of 0.25 pixels. Therefore, as disclosed in Patent Document 1, it is conceivable to correct the image based on the designed shape of the windshield.

[0050] Figure 6 is a graph showing an example of the δ parallax error versus the horizontal field of view of a stereo camera. The deviation relative to the horizontal parallax determined from the design conditions described above is called the "δ parallax error." The vertical axis of Figure 6 represents the δ parallax error [pixels], and the horizontal axis represents the horizontal field of view [degrees]. The figure also shows the allowable values ​​41, where the δ parallax error is -0.25 pixels and +0.25 pixels, with dashed lines.

[0051] Graph 40 in Fig. 6 shows the simulation results of the change in the parallax error shown in Fig. 5, calculated when the stereo camera is rotated 3 degrees in the yaw direction, in order to represent variations in the mounting orientation of the stereo camera. The results shown in Fig. 6 reveal that even if the image is corrected using the technology disclosed in Patent Document 1, variations in the mounting orientation of the stereo camera cause the δ parallax error to fail to satisfy the tolerance range of ±0.25 pixels, represented by tolerance value 41. This is a particular issue with imaging unit 100, such as a stereo camera, which captures the outside world with a wide angle of view.

[0052] On the other hand, in the calibration method according to this embodiment, the parallax error at the design value of the windshield (hereinafter referred to as the "reference parallax error") and the δ parallax error due to variations in the mounting orientation of the stereo camera are calculated separately. Then, the δ parallax error is used to correct the parallax, thereby achieving highly accurate calibration. Therefore, the calibration method according to this embodiment will be described in detail.

[0053] 7 is a flowchart showing an example of a calibration method according to this embodiment. Here, the processing performed by each functional block shown in FIG. 3 will be described using an example in which the stereo camera 21 in FIG. 2 captures an image of the subject 24.

[0054] Before performing the calibration according to this embodiment, an external computing device is used to calculate correlation information between parallax error and vertical displacement based on the mounting posture of stereo camera 21, the design values ​​of the mounting posture and shape of windshield 22, and the reference parallax error according to the vehicle model. Then, external input unit 101 stores a correlation table (see FIG. 11, which will be described later) in which correlation information between parallax error and vertical displacement is recorded in storage unit 103 (S1). In step S1, external input unit 101 also stores the reference vertical displacement and reference parallax error, which are design values ​​according to the vehicle model, in storage unit 103.

[0055] Next, the stereo camera device 10 starts calibration to reduce the influence of the windshield 22 on the parallax between the two images captured by the stereo camera 21 (S2). Then, the stereo camera device 10 starts the stereo camera 21 of the imaging unit 100.

[0056] Next, the imaging unit 100 performs an image acquisition process to acquire two left and right images (S3). At this time, the imaging unit 100 acquires two images of the same subject 24 or scenery captured by the left camera 21L and the right camera 21R. In the following description, the two images captured by the left camera 21L and the right camera 21R are also referred to as "left and right images."

[0057] Next, the correction unit 102 performs a correction process to correct the two images acquired from the imaging unit 100 using the correction table read out from the storage unit 103 (the correction table obtained by aiming at the factory) (S4).

[0058] Next, the matching unit 104 performs a matching process to calculate the parallax and vertical shift between the two images corrected by the correction unit 102 (S5). Here, the matching unit 104 calculates the parallax by performing horizontal stereo matching on the two corrected left and right images. The matching unit 104 also calculates the vertical shift by matching the left and right images in the vertical direction. The matching unit 104 then stores the calculated parallax and vertical shift in the storage unit 103.

[0059] Thereafter, the processes of steps S3 to S5 are repeated (S5A). Matching unit 104 stores the vertical deviations of multiple frames calculated in the repeated processes in storage unit 103. The repeated process of step S5A is performed a number of times corresponding to the number of frames required to add the vertical deviations in step S6, which will be described later.

[0060] Next, the noise removal unit 105 adds up the vertical displacements of the multiple frames read out from the storage unit 103 and performs a process of removing noise (S6). To do this, the noise removal unit 105 repeatedly performs steps S3 to S5 and reads out the vertical displacements of the multiple frames stored in the storage unit 103 from the storage unit 103. The noise removal unit 105 applies a filter to the vertical displacements before adding them up, thereby removing invalid vertical displacements that would cause noise in the calculation result. Then, the noise removal unit 105 adds up the multiple vertical displacements from which the invalid vertical displacements have been removed, and calculates the average value of the vertical displacements.

[0061] Next, the parallax error estimating unit 106 performs a process of calculating a δ vertical displacement using the vertical displacement from which noise has been removed by the noise removal unit 105 and the reference vertical displacement read out from the storage unit 103 (S7). Next, the parallax error estimating unit 106 performs a process of calculating a δ parallax error from the δ vertical displacement using a correlation table in which correlation information between parallax error and vertical displacement is recorded (S8). Next, the parallax error estimating unit 106 stores the calculated δ parallax error in the storage unit 103 as correction information (S9).

[0062] Then, the distance calculation unit 107 corrects the parallax by removing the reference parallax error and the δ parallax error from the parallax read out from the storage unit 103, and then performs a process of calculating distance information to the captured object using the corrected parallax (S10). Then, the calibration process shown in FIG. 7 ends.

[0063] After this process is completed, distance information to the captured object is output to the control unit 108. Then, the control unit 108 recognizes the outside world based on the distance information and controls automatic driving, such as avoiding obstacles ahead of the vehicle.

[0064] 8 is a diagram showing an example of horizontal stereo matching of left and right images. Here, the left camera 21L and the right camera 21R constituting the stereo camera 21 each capture an image of a subject 24. In the following description, the windshield 22 will be abbreviated to "glass."

[0065] The upper part of FIG. 8 shows image PL of subject 24 captured by left camera 21L, and the lower part of FIG. 8 shows image PR of subject 24 captured by right camera 21R. Image PL1 is an image of subject 24 captured by left camera 21L without glass. Image PR1 is an image of subject 24 captured by right camera 21R without glass. Parallax DPX is the true value of parallax (horizontal deviation) calculated by horizontal stereo matching of image PL1 and image PR1 captured without glass. Parallax DPX is represented as "Dx" in the equations explained below. Furthermore, the glass when installed in the designed position on the vehicle is called "glass (design value)."

[0066] Image PL2 is an image of subject 24 captured by left camera 21L through glass when glass (design value) is present. Image PR2 is an image of subject 24 captured by right camera 21R through glass when glass (design value) is present. Image PL2 is shifted to the lower right relative to image PL1, and image PR2 is shifted to the lower right relative to image PR1.

[0067] Pixel shift LS1 indicates the parallax error of image PL2 captured by left camera 21L through glass (design value) relative to image PL1. Pixel shift RS1 indicates the parallax error of image PR2 captured by right camera 21R through glass (design value) relative to image PR1. Therefore, when horizontal stereo matching is performed on images PL2 and PR2, a parallax DPXE1 occurs. Parallax DPXE1 represents the parallax error caused by the influence of glass (design value).

[0068] Here, if the parallax DPXE1 when glass (design value) is present is expressed as "DEx" in the formula, and the reference parallax error is expressed as "εx", which is the parallax error caused by the influence of the glass (design value), then the formula "DEx = Dx + εx" can be obtained. That is, the parallax DEx when glass (design value) is present is the sum of the true parallax value Dx and the reference parallax error εx caused by the influence of the glass (design value).

[0069] Incidentally, the case where there is glass (design value) here is an ideal state. In reality, the mounting orientation of the camera relative to the glass (design value) varies, and the mounting orientation and shape of the glass vary. Here, the glass in the state where it is actually mounted on the vehicle is called the "glass (actual value)." Figure 8 shows an image PL3 captured by the left camera 21L through the glass (actual value) and an image PR3 captured by the right camera 21R through the glass (actual value). Image PL3 is shifted to the lower right with respect to image PL2, and image PR3 is shifted to the lower right with respect to image PR2.

[0070] Pixel shift LS2 indicates the parallax error of image PL3 captured by the left camera 21L through the glass (actual value) relative to image PL2. Pixel shift RS2 indicates the parallax error of image PR3 captured by the right camera 21R through the glass (actual value) relative to image PR2. Therefore, when horizontal stereo matching is performed on images PL3 and PR3, a parallax DPXE2 occurs. The parallax DPXE2 represents the parallax error caused by the influence of the glass (actual value).

[0071] Here, the parallax DPXE2 when glass (actual value) is present is expressed as "DE'x" in the formula, and the δ parallax error caused by variations in the glass and camera compared to the design value is expressed as "δεx", resulting in the formula "DE'x=Dx+εx+δεx". That is, the parallax DE'x when there is glass (actual value) is the sum of the above-mentioned true parallax value Dx, the reference parallax error εx caused by the influence of the glass (design value), and the δ parallax error δεx. That is, the formula is (parallax) = (true parallax value) + (reference parallax error) + (δ parallax error). Note that the δ parallax error δεx may be a negative value.

[0072] FIG. 9 is a diagram showing an example of vertical matching between left and right images. The left side of Fig. 9 shows image PL of subject 24 captured by left camera 21L, and the right side of Fig. 9 shows image PR of subject 24 captured by right camera 21R. Images PL1, PL2, PL3 shown in image PL, pixel shifts LS1, LS2, images PR1, PR2, PR3 shown in image PR, and pixel shifts RS1, RS2 are as described with reference to Fig. 8.

[0073] The vertical deviation DPY is the true value of the vertical deviation calculated by stereo matching in the vertical direction between the images PL1 and PR1 captured without passing through glass. The vertical deviation DPY is represented as "Dy" in the formula. The true value of the vertical deviation when there is no glass is adjusted before shipping the stereo camera 21. Therefore, the true value of the vertical deviation between the left camera 21L and the left camera 21L is zero, and is expressed by the formula Dy=0.

[0074] On the other hand, when glass (design value) is present, the positions of the light rays incident through the glass (design value) differ between the left camera 21L and the right camera 21R. As a result, the vertical pixel shift LS1 of the image captured by the left camera 21L differs from the vertical pixel shift RS1 of the image captured by the right camera 21R. Therefore, when glass (design value) is present, vertical stereo matching is performed on the images PL2 and PR2, resulting in a vertical shift DPYE1. In other words, the vertical shift DPYE1 represents the vertical shift caused by the influence of the glass (design value).

[0075] Here, if the vertical deviation DPYE1 when glass (design value) is present is expressed as "DEy" in the formula, and the reference vertical deviation is expressed as "εy", which is the vertical deviation caused by the influence of the glass (design value), then since Dy = 0, the formula "DEy = εy" can be obtained. That is, the vertical deviation DEy when there is glass (design value) is the sum of the true vertical deviation Dy (=0) and the reference vertical deviation εy caused by the influence of the glass (design value).

[0076] As shown with reference to Fig. 8, the case where there is glass (design value) is an ideal state. Fig. 9 shows an image PL3 captured by the left camera 21L through the glass (actual value) and an image PR3 captured by the right camera 21R through the glass (actual value).

[0077] Pixel shift LS2 indicates the vertical shift of image PL3 captured by left camera 21L through glass (actual value) relative to image PL2. Pixel shift RS2 indicates the vertical shift of image PR3 captured by right camera 21R through glass (actual value) relative to image PR2. Therefore, when vertical stereo matching is performed on images PL3 and PR3, a vertical shift DPYE2 occurs. Vertical shift DPYE2 represents the vertical shift value caused by the influence of glass (actual value).

[0078] Here, the vertical deviation DPYE2 when glass (actual value) is present is expressed as "DE'y" in the formula, and the δ vertical deviation caused by variations in the glass and camera compared to the design value is expressed as "δεy", resulting in the formula "DE'y = εy + δεy". In other words, the vertical deviation DE'y when there is glass (actual value) is the sum of the true vertical deviation Dy (=0), the reference vertical deviation εy caused by the influence of the glass (design value), and the δ vertical deviation. In other words, the formula is (vertical deviation) = (reference vertical deviation) + (δ vertical deviation). Note that the δ vertical deviation (δεy) may also be a negative value.

[0079] Incidentally, the value of DE'x in the horizontal direction changes depending on the distance from the camera to the subject 24. For this reason, it has been necessary to perform measurements under predetermined conditions, such as when the distance to the object is known.

[0080] On the other hand, the stereo camera device 10 according to this embodiment is characterized by using a vertical displacement that does not depend on distance, rather than a horizontal displacement. When there is no glass, the vertical displacement is zero, so even if the distance to the subject 24 changes, the vertical displacement does not change and remains the same. Therefore, the stereo camera device 10 can directly calculate the vertical displacement, and can use the vertical displacement to calculate the parallax error.

[0081] Next, the procedure for calculating the parallax error will be described. FIG. 10 is a schematic diagram showing an example of a procedure for calculating a parallax error according to this embodiment. (1) Photographing 1001 landscapes First, the stereo camera 21 captures an image of the scenery 1001 through the windshield 22. At this time, the processes of steps S3 to S5 in FIG.

[0082] (2) Vertical stereo matching Next, the matching unit 104 performs stereo matching in the vertical direction between the two images captured by the left and right cameras to detect the vertical deviation of 10 DPY. At this time, the processing of step S5 in FIG. 7 is performed. Here, the vertical deviation in the image is represented by a vertical arrow in the vertical deviation of 10 DPY. The direction of each arrow indicates the direction of the vertical deviation, and the length indicates the amount of vertical deviation.

[0083] (3) Calculation of parallax error The parallax error estimating unit 106 then estimates a horizontal parallax error of 10 DPX. Here, the parallax error within the image is represented by a horizontal arrow with a parallax error of 10 DPX. The direction of each arrow indicates the direction of the parallax error, and the length indicates the amount of parallax error shift.

[0084] An arrow of the parallax error 10DPX at the same position as an arrow of the vertical displacement 10DPY is correlated. The correlation between these arrows is stored as correlation information c1 and c2 in a correlation table in the storage unit 103. As shown in the correlation information c1 and c2, the vertical displacement and the parallax error are not significantly different near the center of the image. However, at the lower side where the curvature of the windshield 22 is greater, the parallax error is larger than the vertical displacement. For this reason, multiple correlations are stored in the storage unit 103 as correlation information. As will be described later, the correlation between the vertical displacement and the parallax error can be used as the correlation between the δ vertical displacement and the δ parallax error. The parallax error estimation unit 106 can estimate the δ parallax error from the δ vertical displacement based on the correlation information including multiple correlations read from the storage unit 103.

[0085] If the windshield 22 is configured as a part of a spherical surface, it is expected that the correlation between the vertical shift and the parallax error will be constant throughout the entire windshield 22. In this case, one correlation is stored as correlation information in the storage unit 103. Then, the parallax error estimating unit 106 can estimate the δ parallax error from the δ vertical shift based on the correlation information including one correlation read from the storage unit 103.

[0086] As described above, if the parallax error estimating unit 106 can calculate the δ vertical shift, it can estimate the δ parallax error at each angle of view indicated by the parallax error 10DPX. At this time, the processes of steps S6 to S8 in FIG.

[0087] 11 is a diagram showing the structure of the correlation table, which records correlation information between parallax error and vertical deviation.

[0088] The correlation table is expressed, for example, as a table of the horizontal angle of view [degrees] and the vertical angle of view [degrees] of the stereo camera 21. In the cell where each angle of view intersects, two values ​​are stored as correlation information: the parallax error "e(m)" and the vertical displacement "v(n)." Here, "m" in the parallax error e(m) corresponds to the horizontal angle of view and takes a value between "-60" and "+60." Furthermore, "n" in the vertical displacement v(n) corresponds to the vertical angle of view and takes a value between "-30" and "+30."

[0089] The storage unit (storage unit 103) stores, in a correlation table, a plurality of pieces of correlation information corresponding to the imaging angles of view (horizontal and vertical angles of view) of the camera (stereo camera 21). For example, the correlation table shown in FIG. 11 stores correlation information when the horizontal and vertical angles of view are changed by 1 degree each within a horizontal angle of view range of −60 degrees to +60 degrees and a vertical angle of view range of −30 degrees to +30 degrees. This allows the parallax error estimating unit 106 to easily read out correlation information between parallax error and vertical deviation at certain horizontal and vertical angles of view from the correlation table. The parallax error estimating unit (parallax error estimating unit 106) can then estimate a parallax error using the plurality of pieces of correlation information read out from the storage unit (storage unit 103) according to the imaging angles of view.

[0090] The correlation table may store correlation information when the horizontal and vertical angles of view are changed by 0.5 degrees, or may store correlation information when the horizontal and vertical angles of view are changed by 10 degrees.

[0091] Furthermore, the storage unit (storage unit 103) may store one piece of correlation information calculated for the refractive body (windshield 22) when the shape of the refractive body is uniform. If the correlation information is the same regardless of the horizontal angle of view and the vertical angle of view, only one piece of correlation information for a parallax error and a vertical shift may be stored in the correlation table. The parallax error estimation unit (parallax error estimation unit 106) can estimate the parallax error using one piece of correlation information read out from the storage unit (storage unit 103).

[0092] Next, the reason why the parallax error estimating unit 106 can estimate the horizontal parallax error from the vertical pixel shift will be explained. FIG. 12 is a diagram illustrating the principle by which the parallax error estimating unit 106 estimates the parallax error from the vertical displacement.

[0093] The explanatory diagram 12P0 shown at the top of Figure 12 is a three-dimensional diagram showing how light rays enter glass without variations and exit. Here, an incident light ray 11in enters the windshield 22 and exits as an exiting light ray 11out. The X-axis in the diagram indicates the horizontal angle of view, the Y-axis indicates the vertical angle of view, and the Z-axis indicates the direction of travel of the vehicle. When there is no need to distinguish between the incident light ray 11in and the exiting light ray 11out, they are referred to as "light rays."

[0094] When the incident light ray 11in is projected onto the XY plane, it is expressed as an XY plane projected light ray 1111. When the outgoing light ray 11out is projected onto the XY plane, it is expressed as an XY plane projected light ray 1113. Here, the cut surface 1112 is the location where the light ray hits the glass 22 projected onto the XY plane.

[0095] Next, a case where light passes through glass with no variation in mounting position will be described. An example of a ray projected onto the XY plane shown in diagram 12P0 is shown in diagram 12P1 shown in the lower left of Fig. 12. Here, rays 1115 and 1114 obtained by decomposing the XY plane projected ray 1113 into the X axis and Y axis represent the horizontal and vertical components of the XY plane projected ray 1113, respectively.

[0096] The case where light passes through glass with variations in installation position will be described. Diagram 12P2 shown in the lower right of Figure 12 shows an example of a light ray passing through glass with variations in installation position projected onto the XY plane shown in diagram 12P0. Diagram 12P2 also shows the state of displacement of the light ray that changes at the cutting surface 1112, the same as diagram 12P1.

[0097] For example, when glass has roll rotation variation B1130, the cut surface 1122 (solid line) when there is no variation can be represented by the position where it changes to cut surface 1122 (dashed line). Also, the exit ray 1123 (dashed line) is a change from the XY plane projected ray (solid line).

[0098] Additionally, ray 1124 (dashed line) and ray 1125 (dashed line) represent the vertical and horizontal components, respectively, of the changed output ray 1123 (dashed line). The vertical difference between ray 1124 (dashed line) and ray 1114 (solid line) is the δ vertical shift δεy caused by glass variations. Additionally, the horizontal difference between ray 1125 (dashed line) and ray 1115 (solid line) is the δ parallax error δεx caused by glass variations.

[0099] The actual variation in the mounting orientation of the glass is much smaller than the design value. Therefore, when there is variation in the mounting orientation of the glass, the ratio of the δ parallax error ε'x to the δ vertical shift ε'y does not change significantly from the ratio of the reference parallax error εx to the reference vertical shift εy calculated using the design values ​​of the glass. In other words, there is a relationship of |εx| / |εy|≒|ε'x| / |ε'y|.

[0100] For example, if the correlation between the reference parallax error εx and the reference vertical shift εy is calculated in advance using the design values ​​of the glass, it can be expressed as |εx| / |εy|=k. On the other hand, if the δ parallax error is "ε'x" and the δ vertical shift is "ε'y" when there is variation in the glass installation posture, the correlation between the δ parallax error ε'x and the δ vertical shift ε'y can be expressed as |ε'x|≒k*|ε'y|. That is, since |ε'x|-|εx|≈k*(|ε'y|-|εy|), the relationship between the δ parallax error δεx and the δ vertical shift δεy can be expressed as δεx≈k*δεy.

[0101] Therefore, the parallax error estimating unit 106 can estimate the actual parallax error using the calculation formula: (actual parallax error) = (reference parallax error (glass design value)) + (δ parallax error (variable)). Here, the reference parallax error and correlation coefficient k are values ​​obtained from the glass design value. Also, as described above, the δ parallax error δεx can be calculated as (δ vertical shift δεy * k).

[0102] In this way, even if there is variation in the mounting orientation of the glass, the stereo camera device 10 according to this embodiment calculates the δ parallax error (with variation) using the correlation between the δ vertical shift and the δ parallax error, i.e., the correlation between the vertical shift and the parallax error, and corrects the parallax using the actual parallax error, thereby achieving highly accurate calibration. Note that although the correlation has been described as linear here, the correlation becomes nonlinear as the amount of change increases. Needless to say, the method is effective even in this case.

[0103] Next, we will confirm the correlation between the δ vertical shift and the δ parallax error when variations in the mounting orientation of the glass are taken into consideration. Below, we will explain examples of nine types of variations that take into account the variation conditions. 13 is a diagram showing examples of nine types of variations. The variations (1) to (3) shown in FIG. 13 are based on the following calculation conditions.

[0104] <Calculation conditions (variation)> X, Y, Z shift: +2mm Pitch, yaw, roll rotation: +3 degrees Glass curvature radius (horizontal, vertical): -1.5m Glass thickness: +1mm

[0105] Variation (1) in Fig. 13 shows an example of shift variation of the stereo camera relative to the windshield 22. From the left in Fig. 13, examples of shift in the X-axis direction (X shift), shift in the Y-axis direction (Y shift), and shift in the Z-axis direction (Z shift) are shown. Variation (2) in Fig. 13 shows an example of rotation variation of the stereo camera relative to the windshield 22. From the left in Fig. 13, examples of rotation in the pitch direction (pitch rotation), rotation in the yaw direction (yaw rotation), and rotation in the roll direction (roll rotation) are shown. Variation (3) in Fig. 13 shows an example of variation in the characteristics of the windshield 22. From the left in Fig. 13, examples of the horizontal curvature of the windshield 22, the vertical curvature of the windshield 22, and the thickness of the windshield 22 are shown.

[0106] Fig. 14 is a diagram showing the correlation between the δ vertical shift and the δ parallax error. The horizontal axis of Fig. 14 represents the δ vertical shift, and the vertical axis represents the δ parallax error. The correlation between the δ vertical shift and the δ parallax error is calculated under the following calculation conditions shown in the legend of Fig. 14.

[0107] <Calculation conditions> ·Left-right translation (X-axis): +2mm ·Vertical translation (Y axis): +2mm ·Back and forth translation (Z axis): +2mm Pitch angle (X axis rotation): +3 degrees Yaw angle (Y-axis rotation): +3 degrees Roll angle (Z axis rotation): +3 degrees ·Horizontal radius of curvature: -1500mm ·Vertical radius of curvature: -1500mm Thickness: +0.2mm

[0108] Here, the correlation between the δ vertical shift and the δ parallax error is found by calculating the δ vertical shift and the δ parallax error when the stereo camera device 10 detects an object 24 50 m away. Graph RT20 in the upper part of Fig. 14 shows the correlation between the δ vertical shift and the δ parallax error when the horizontal angle of view is 20 degrees. Graph RT40 in the middle of Fig. 14 shows the correlation between the δ vertical shift and the δ parallax error when the horizontal angle of view is 40 degrees. Graph RT60 in the lower part of Fig. 14 shows the correlation between the δ vertical shift and the δ parallax error when the horizontal angle of view is 60 degrees.

[0109] As shown in the graphs RT20, RT40, and RT60, it can be seen that there is a high correlation between the δ vertical shift and the δ parallax error. Furthermore, the δ parallax error shifts with variations in the windshield 22, but the δ vertical shift also changes along with the shift in the δ parallax error. Therefore, it can be seen that the δ parallax error can be found by calculating the δ vertical shift.

[0110] It can also be seen that the correlation coefficient k, which represents the correlation between the δ vertical shift and the δ parallax error, varies depending on the horizontal angle of view. For example, if the horizontal angle of view is 60 degrees, the correlation coefficient k is approximately +1. As the horizontal angle of view changes to 40 degrees, 20 degrees, and so on, the correlation coefficient k becomes smaller than +1. For this reason, to perform highly accurate calibration, it is desirable to use a different correlation coefficient k for each horizontal angle of view of the stereo camera 21.

[0111] Fig. 15 shows the correction effect under combined conditions that take into account all of the calculation conditions (variations) shown in Fig. 14. Fig. 15 shows the simulation results of parallax error obtained for horizontal angles of view of 20 degrees, 40 degrees, and 60 degrees. Here, the parallax error before correction is indicated by a black circle, the parallax error corrected using the correlation coefficient k for each horizontal angle of view is indicated by a black triangle, and the parallax error corrected using one correlation coefficient at a horizontal angle of view of 60 degrees is indicated by a white square.

[0112] It can be seen that the parallax error corrected using one correlation coefficient at a horizontal angle of view of 60 degrees is greater than 0 pixels at horizontal angles of view of 40 degrees and 60 degrees. On the other hand, the parallax error corrected using the correlation coefficient k for each horizontal angle of view according to this embodiment is almost 0 pixels even at a horizontal angle of view of 40 degrees. However, when the horizontal angle of view is 60 degrees, the parallax error after correction is about 0.5 pixels.

[0113] As can be seen, the parallax error can be significantly reduced by performing the correction according to this embodiment. Furthermore, although the residual parallax error can be reduced by performing correction using the correlation coefficient k for each horizontal angle of view, even the correlation coefficient k at a horizontal angle of view of 60 degrees is still effective compared to the parallax error before correction, which exceeded one pixel.

[0114] In the stereo camera device 10 according to the embodiment described above, the reference parallax error calculated using the design values ​​of the windshield 22 and the δ parallax error due to variations in the windshield 22 are calculated separately. Therefore, the stereo camera device 10 can accurately correct (calibrate) the parallax of the stereo camera 21 with a wide angle of view even without a calibration chart. For example, the stereo camera device 10 can accurately correct the parallax of the stereo camera 21 even if there are manufacturing variations in the radius of curvature in the horizontal and vertical directions of the windshield 22 or variations in the relative value relationship between the windshield 22 and the stereo camera 21.

[0115] For example, when replacing the windshield 22 at a dealer's repair shop, even if there are variations in the characteristics such as the mounting posture of the stereo camera 21 and the mounting posture and shape of the windshield 22, high distance measurement accuracy can be obtained by performing the calibration according to this embodiment.

[0116] Furthermore, the method according to this embodiment does not require a large chart for calibration, so calibration can be performed using an image of a road scene, and the influence of the windshield 22 can be corrected even while the vehicle 20 is moving.

[0117] Furthermore, the user can use calibration to correct the influence of the windshield 22 with the stereo camera 21 while the vehicle 20 is stopped or moving. For example, in calibration mode, the stereo camera 21 performs correction in real time while the vehicle 20 is stopped or moving. Calibration while the vehicle 20 is moving is performed according to the flowchart shown in FIG. 7, and multiple vertical deviations are accumulated using multiple frames of images acquired while the vehicle 20 is moving. For example, the vertical deviations are accumulated based on changes in the scenery captured in images captured while the vehicle 20 is moving or stopped.

[0118] For example, the vehicle 20 may be parked in a certain direction, capture an image of the scenery, and then park the vehicle 20 in a different direction and capture an image of the scenery in that direction, thereby accumulating the vertical shift based on the change in the scenery. For example, the stereo camera device 10 may acquire left and right images of the same subject 100 times, perform stereo matching 100 times, and accumulate the vertical shifts for the 100 times to calculate the average value. The parallax error estimating unit 106 then estimates the parallax error from the vertical shift using a correlation table, and the distance calculating unit 107 performs calibration to remove the parallax error from the parallax.

[0119] 10 is not limited to people, animals, etc. For example, it may be any scenery from which parallax can be acquired, such as a road surface or a chart.

[0120] 7. Furthermore, the repetitive process (S5A) such as steps S3 to S5 shown in FIG. 7 is not limited to accumulating vertical displacement. For example, multiple δ parallax errors may be calculated and the process of correcting parallax multiple times may be performed. Furthermore, other parameters may be accumulated to improve the accuracy of parallax correction.

[0121] The invention made by the present inventor has been specifically described above based on the embodiments, but it goes without saying that the invention is not limited to the above embodiments and can be modified in various ways. For example, the present invention can be applied to calibration in a factory, making it possible to correct parallax without using a chart. Furthermore, even when calibration is performed using a chart in a factory, a chart with a narrower chart width can be used compared to conventional calibration, eliminating the need to remodel the factory or prepare space for a large chart.

[0122] Furthermore, the stereo camera device 10 can correct the influence on the glass even when the vehicle 20 is moving. Furthermore, since calibration can be performed even while the vehicle 20 is moving, it becomes possible to appropriately correct the parallax error even when the mounting posture of the stereo camera 21 is shifted due to vibration of the vehicle 20.

[0123] Furthermore, the stereo camera 21 may be installed on the inside of the rear window if it is installed inside the vehicle 20. In this case, the stereo camera device 10 can correct parallax from multiple images of the external world behind the vehicle 20 captured by the stereo camera 21, and calculate the distance to an object in the external world behind the vehicle 20.

[0124] In addition, in the above-described embodiment, the stereo camera 21 is installed on the inside of the windshield 22 as an example of a refractive body, but a transparent organic compound such as transparent plastic may also be used as the refractive body other than glass.

[0125] As such, the present invention is not limited to the above-described embodiments, and it goes without saying that various other applications and modifications are possible without departing from the gist of the present invention as set forth in the claims.

[0126] The position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc., in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc., disclosed in the drawings. Furthermore, the following embodiments describe the system configuration in detail and specifically in order to clearly explain the present invention, and are not necessarily limited to systems that include all of the described configurations. Furthermore, it is also possible to add, delete, or replace part of the configuration of the present embodiment with other configurations. In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0127] 10... stereo camera device, 20... vehicle, 21... stereo camera, 22... windshield, 100... imaging unit, 101... external input unit, 102... correction unit, 103... storage unit, 104... matching unit, 105... noise removal unit, 106... parallax error estimation unit, 107... distance calculation unit, 108... control unit

Claims

1. a storage unit that stores correlation information of horizontal parallax error with respect to vertical shift of a plurality of images captured by a plurality of cameras through the refracting body that refracts light, the correlation information being calculated based on the attitude of the refracting body attached to the vehicle, refracting body characteristics including the shape of the refracting body, and the attitudes of a plurality of cameras arranged on the vehicle; a matching unit that matches a plurality of images captured by the plurality of cameras of the same subject and determines a vertical shift of the plurality of images; a parallax error estimating unit that estimates a parallax error of the matched plurality of images based on the vertical shift and the correlation information; a distance calculation unit that corrects the parallax of the matched images using the parallax error. Stereo camera device.

2. the storage unit stores a plurality of pieces of correlation information corresponding to imaging angles of view of the camera; The parallax error estimating unit estimates the parallax error using a plurality of pieces of correlation information read from the storage unit in accordance with an imaging angle of view of the camera. The stereo camera device according to claim 1 .

3. the storage unit stores one piece of correlation information obtained for the refractive body when the shape of the refractive body is uniform; The parallax error estimating unit estimates the parallax error using one of the correlation information read out from the storage unit. The stereo camera device according to claim 1 .

4. The parallax error estimating unit accumulates the vertical displacements determined by the matching unit a plurality of times at the same position in the plurality of images captured at different times, calculates an average value of the vertical displacements, and estimates the parallax error based on the average value of the vertical displacements and the correlation information. The stereo camera device according to claim 2 .

5. the storage unit stores a reference parallax error and a reference vertical deviation calculated in advance based on design values ​​of the refractive body; the parallax error estimation unit calculates a δ vertical shift, which is a change in the vertical shift caused by the refractor actually attached to the vehicle, based on the reference vertical shift and the vertical shift read out from the storage unit, and calculates a δ parallax error, which is a change in the parallax error caused by the refractor actually attached to the vehicle, based on the δ vertical shift and the correlation information read out from the storage unit; The distance calculation unit corrects the parallax using the reference parallax error and the δ parallax error. The stereo camera device according to claim 3 .

6. the reference parallax error and the reference vertical deviation are stored in the storage unit at the time of shipping of the plurality of cameras, The amount of change with respect to the reference parallax error after shipment of the plurality of cameras is the δ parallax error, and the amount of change with respect to the reference vertical shift is the δ vertical shift. The stereo camera device according to claim 5 .

7. the reference parallax error and the reference vertical deviation are stored in the storage unit before the refractive body is attached to the vehicle; The amount of change with respect to the reference parallax error after the refraction body is actually mounted on the vehicle is the δ parallax error, and the amount of change with respect to the reference vertical shift is the δ vertical shift. The stereo camera device according to claim 5 .

8. The distance calculation unit calculates a distance from the vehicle to an object in the outside world using the parallax corrected using the parallax error. The stereo camera device according to claim 2 .

9. A process of acquiring a plurality of images taken by a plurality of cameras arranged on the vehicle through a refracting body attached to the vehicle that refracts light; A process of matching a plurality of images captured by the plurality of cameras of the same subject to determine vertical deviation of the plurality of images; a process of estimating a parallax error of the matched images based on correlation information of horizontal parallax errors with respect to the vertical shifts of the plurality of images, the correlation information being calculated based on refractive body characteristics including the attitude and shape of the refractive body and the attitudes of the plurality of cameras; and and correcting the parallax of the matched images using the parallax error. Calibration method.

Images