Method for rectifying images and / or image points, system based on a camera and vehicle

Abstract

The invention relates to a method for rectifying images and / or image points acquired by at least one camera (2) of a camera-based system (1) of a vehicle (14) with a windshield (6). First, an original image of a scene is acquired with the camera (2). From the original image, original image data is selected, wherein the original image data is the complete original image, a part of the original image or a plurality of original image points of the original image. Then, intermediate image data is calculated based on the original image data and camera parameters, wherein the intermediate image data comprises an intermediate image or a plurality of intermediate image points, wherein the intermediate image data resembles an image or image points of the relevant scene acquired by a pinhole camera (8) through the windshield (6). Finally, a set of points in the relevant scene space is calculated, wherein the set of points corresponds to the intermediate image pixels or intermediate image points. The calculation is performed using a parallel shift (11) of the optical path (7) caused by the windshield (6) based on windshield parameters. Furthermore, the invention also relates to a camera-based system (1) for a vehicle (14) and a vehicle (14).

Description

Technical Field

[0001] The present invention relates to a method for correcting images and / or image points acquired by at least one camera device in a camera-based system for a vehicle with a windshield, and to a camera-based system and vehicle for a vehicle. Background Technology

[0002] For example, computer vision used in Advanced Driver Assistance Systems (ADAS) assigns points in space to image pixels or image regions acquired by a camera device. For ADAS, accurately estimating the camera's projection is crucial for accurately estimating the distance to objects in front of the vehicle. Conversely, accurately estimating the distance to objects in front of the vehicle is essential for camera-based traffic safety applications such as automatic braking or adaptive cruise control.

[0003] To achieve the goals of Advanced Driver Assistance Systems (ADAS) in vehicles, cameras are typically mounted in the mirror housing behind the windshield. This obviously solves the problem of camera lenses being obscured by dirt or rain. However, the camera's unique position introduces the optical characteristics of the windshield into the projection geometry of the entire optical system. The windshield's influence results in significantly different optical path behavior compared to a lens coaxially positioned within the camera's optical system. Therefore, the optical system of a camera behind the windshield can produce a noticeable image deviation compared to the same camera without a windshield. This type of optical distortion is known as windshield distortion of the camera image.

[0004] The problem of estimating the projection characteristics of an optical system, including distortions of any type, is known as camera device calibration. Typically, this post-processing is parametric. That is, there is a family of functions adjusted by parameter vectors, i.e., a camera device model, which describes a physical camera device for each fixed parameter vector—that is, how points in space are projected onto the image by that particular camera device. These vectors are called camera device parameters.

[0005] The application of camera device models in Advanced Driver Assistance Systems (ADAS) typically addresses the inverse problem of projection. That is, given a pixel location in an image, the goal is to estimate the set of points in space that image that location. Camera device calibration is then used as a tool to eliminate the negative effects of the optics, thus describing the optical system with a computationally simple model. This simple model is essentially a pinhole camera device with a central projection. Further elimination of the windshield's influence is called image correction. Especially for stereo imaging, correction is a primary tool for effectively estimating dense distance maps of a scene. Stereo correction modifies images from two different camera device systems to appear as if they originated from two optically identical pinhole camera devices in parallel systems. In these systems, corresponding object points can be found on the same horizontal image line, which greatly simplifies the search for these points and thus also simplifies parallax calculations.

[0006] Ignoring the effects of windshield distortion, especially at short distances, can severely negatively impact distance estimation of objects in front of the vehicle. Therefore, modeling and estimating windshield distortion is crucial.

[0007] As an example, European patent application EP 3 293 701 A1 includes a method for correcting an image acquired by at least one camera in a camera-based system for a vehicle with a windshield. The camera-based system is calibrated by setting a plate-shaped image target with a known pattern in the field of view of the camera, thereby allowing the camera to acquire a calibrated image of the plate through the windshield. The windshield distortion caused by the windshield is then calculated. An image is then acquired using the camera, and the acquired windshield distortion is used to calculate a set of points spatially projected onto a location on the image. However, this method is computationally expensive and treats the camera and windshield as an optical system. Therefore, a new calibration is required for each combination of camera and windshield. Summary of the Invention

[0008] One object of the present invention is to provide a method for correcting images and / or image points obtained by at least one camera device of a camera-based system for a vehicle with a windshield, a method that is computationally fast and separates the optical effects of the windshield and the camera device. Another object of the present invention is to provide a camera-based system suitable for vehicles performing the method, and a class of vehicles having a windshield and a camera-based system.

[0009] This task is solved by the subject matter of the appended independent claims. The dependent claims and the following description and drawings provide embodiments.

[0010] According to a first aspect of the present invention, a method for correcting an image and / or image points is provided. That is, the method is applicable both to correcting the entire image and to correcting some image points of the image, such as image points relating to an object of interest, image points relating to highly characteristic objects in the image, or image points relating to bounding boxes such as other vehicles or pedestrians.

[0011] These images or image points are acquired by at least one camera in a camera-based system for vehicles with windshields. If the camera-based system includes more than one camera, the camera can acquire, for example, stereoscopic information. The camera is positioned behind the windshield, that is, the windshield is positioned between the camera and the vehicle's surrounding environment. Thus, the camera is protected from environmental factors such as rain or dirt.

[0012] As a first step of the method, a camera device acquires an original image of a scene. The scene can be a traffic scene such as a road, road signs, buildings, pedestrians, and / or other vehicles. Since the windshield is positioned in front of the camera device, light from the scene is first deflected by the windshield and then focused onto an image sensor by an objective lens. The objective lens can be, for example, a fisheye lens or a linear wide-angle lens, a wide-angle lens. The image sensor can be, for example, a complementary metal-oxide-semiconductor (CMOS) sensor or a charge-coupled device (CCD) sensor.

[0013] Then, raw image data is selected from the raw image, where the raw image data is the complete raw image, a portion of the raw image, or multiple raw image points of the raw image. A portion of the raw image can be a region of particular interest, and raw image points can relate to objects of interest, highly characteristic objects in the image, or bounding boxes relating to other vehicles or pedestrians.

[0014] Intermediate image data is calculated based on the original image data. The intermediate image data includes an intermediate image or multiple intermediate image points, which is similar to a scene image or image points acquired by a pinhole camera through the windshield. In other words, the objective lens effect is removed from the intermediate image and replaced by the pinhole camera. Furthermore, for this purpose, the intermediate image data is calculated based on the camera device parameters, i.e., based on the parameters characterizing the objective lens. The details of this calculation are known to those skilled in the art. By eliminating the objective lens effect in this step, the optical effects of the camera device and the windshield are separated from each other. Therefore, if a different camera device is used behind a specific windshield, only new camera device parameters need to be provided; if a camera device with known camera device parameters is installed behind different windshields, only the effect of different windshields needs to be evaluated.

[0015] As the final step of the method, a series of points in the relevant scene space are calculated. These points correspond to pixels or points in the intermediate image. In other words, the points in the relevant scene space are points in space that, if projected through the windshield using a pinhole camera, would produce pixels or points in the intermediate image. To calculate the spatial points of the relevant scene, the optical path translation caused by the windshield is used. That is, the effect of the windshield is modeled as an optical path translation. This also applies perfectly to flat windshields and provides an excellent approximation for windshields with only slight curvature in the camera's area. The above calculations are based on windshield parameters and are computationally fast.

[0016] Therefore, the method provides correction of images and / or image points acquired by at least one camera device, the correction separating the effects of the windshield and the camera device, and is computationally fast.

[0017] According to one embodiment, a camera device acquires a sequence of original images of a scene. The sequence is, in particular, a temporal sequence of images of the scene. In the sequence, the scene changes with the movement of objects and / or vehicles within the scene, as well as the movement of the camera device. As the relevant scene changes, original image points and / or intermediate image points are tracked. The set of points in space also changes with the original image sequence and corresponds to the tracked original image points and / or intermediate image points. By tracking the original image points and / or intermediate image points, the distance to the corresponding spatial point can be determined.

[0018] According to one embodiment, windshield parameters include a normal vector n of the windshield in the region near the camera device, a thickness t of the windshield, and / or a refractive index ν of the windshield. The region near the camera device is particularly the windshield region around the visual center of the camera device. The normal vector n and the windshield thickness t can be determined, for example, through geometric measurements.

[0019] According to one embodiment, the parallel offset / translation is equal to the slope offset σ multiplied by the normal vector n. This choice of parallel offset results in highly efficient and rapid calculations, which can be directly calculated and demonstrated via the optical path.

[0020] According to one embodiment, the slab shift σ is approximated as a constant. Approximating the slab shift as a constant allows for very quick calculations and provides good results for relatively small viewing angles. The slab shift σ is specifically equal to t(ν–1) / ν, which is an accurate solution for optical paths perpendicular to the windshield.

[0021] According to one embodiment, the slope offset σ is calculated as a quartic equation g(σ)=a4σ 4 +a3σ 3 +a2σ 2+a1σ+a0, the root of σ0, where 0≤σ0≤t, a4=ν 2 a3=–2ν 2 (w+t), a2=(v 2 –1)(u 2 +t 2 )+ν 2 w(w+4t), a1=–2t(ν) 2 (w 2 +u 2 )+tw(ν 2 –1)–u 2 ), a0=(ν 2 –1)t 2 (w 2 +u 2 ), w = n·s and u = √(s·s–w 2 Point s is a spatial point in the relevant scene space, corresponding to a pixel or point in the intermediate image. The quartic equation can be derived by calculating the light path from point s to the origin, assuming a pinhole camera at the origin. Rotating the coordinates by n along the e3 axis and s along the e1-e3 plane s simplifies the calculation. The quartic equation can be solved accurately using the Ferrari solution, providing a unique solution for σ0 within a given range. While this solution is accurate and offers the highest accuracy, its computational cost is very high.

[0022] According to one embodiment, the slope offset σ is calculated as the fixed point of the fixed point / fixed point equation σ=φ(σ), where φ(σ)=t(1–1 / √(ν 2 –1)(u 2 / (w–σ) 2 +1)+1)), w=n·s, u=√(s·s–w 2 Similarly, s is a spatial point in the relevant scene space, corresponding to a pixel or point in the intermediate image. The fixed-point equation can also be derived by calculating the light path from point s to the origin, assuming a pinhole camera at the origin. To reiterate, rotating the coordinates by n along the e3 axis and by s in the e1-e3 plane simplifies the calculation. It can be seen that the fixed-point equation is a convergent equation with a Lipschitz constant, the upper limit of which is defined by c = t / (wt), which is less than 1 in all practical applications. Therefore, the fixed-point equation converges, and a specific accuracy can be achieved through a finite number of iterations of the fixed-point equation.

[0023] According to one implementation, only the first or second iteration of the fixed-point equation is calculated. This makes the calculation of the slope offset σ extremely fast, while providing excellent accuracy.

[0024] According to one embodiment, the camera device parameters are determined by acquiring a calibration image of a known pattern, positioned within the camera device's field of view without a windshield. The known pattern can be, for example, a checkerboard pattern. The calibration image is then compared to the known pattern, specifically identifying points in the calibration image using points from the known pattern. The camera device parameters are determined based on this comparison. The camera device parameters can also be determined by performing the above steps on another camera device similar to the one described. The camera device parameters of the other camera device are expected to be very similar to those of the described camera device, and for all practical purposes, the camera device parameters of the other camera device are almost identical to those of the described camera device.

[0025] According to one embodiment, windshield parameters are determined by acquiring a calibration image of a known pattern positioned in the field of view of a camera device, with the windshield in front of the camera device. The known pattern may be, for example, a checkerboard pattern. An intermediate calibration is calculated based on the calibration image and the camera device parameters. The intermediate calibration image is similar to an image of the known pattern captured by a pinhole camera through the windshield. In other words, the influence of the camera device's objective lens is eliminated in the intermediate calibration image. The intermediate calibration image is compared with the known pattern, specifically identifying points in the intermediate calibration image using points from the known pattern. The windshield parameters are determined based on this comparison. The influence of the windshield on the optical path can be calculated as described above. The windshield parameters can also be determined by performing the above steps on another windshield similar to the windshield and / or on another camera device similar to the aforementioned camera device. In all cases, the obtained windshield parameters are expected to be very similar and almost identical to the stated windshield parameters for all practical purposes.

[0026] According to one embodiment, the windshield parameters are determined or improved based on an automatic calibration of the camera system. This automatic calibration is explained in German patent application DE 10 2018 204 451A1 and will not be detailed here. The automatic calibration, for example, can correct for minor variations that occur during the mounting of the camera behind the windshield.

[0027] According to one embodiment, the calculation of the scene space midpoint corresponding to an intermediate image pixel or point is performed through bundle adjustment. This technique is described, for example, in the second edition of the paper "Multiple View Geometry in Computer Vision" published in Cambridge in 2003 by R. Hartley and A. Zisserman. The fast image correction method described above facilitates the rapid calculation of the spatial midpoint.

[0028] According to one embodiment, the calculated set of points in the scene space is used for object recognition, object tracking, and / or advanced driver assistance systems. These applications benefit greatly from increased accuracy due to rapid calculations and consideration of the windshield's influence.

[0029] According to another aspect of the present invention, a camera-based system for a vehicle is provided. The camera-based system includes at least one camera mounted behind the vehicle's windshield and a computing unit. It is suitable for performing the methods described above, and thus benefits from rapid calculations and separation of the optical effects of the windshield and the camera.

[0030] According to another aspect of the invention, a vehicle is provided, which includes a windshield and a camera-based system as described above.

[0031] These and other aspects of the invention will become apparent and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0032] Exemplary embodiments of the present invention are described below with reference to the accompanying drawings:

[0033] Figure 1 A schematic diagram of a camera-based system installed behind the windshield is shown.

[0034] Figure 2 A schematic diagram of a pinhole camera device installed behind a windshield is shown, and

[0035] Figure 3 A schematic diagram of the calibration settings is shown.

[0036] The accompanying drawings are merely illustrative and are used only to illustrate embodiments of the present invention. In principle, identical or equivalent parts have the same reference numerals. Detailed Implementation

[0037] Figure 1 A schematic diagram of a camera-based system 1 for a vehicle is shown. The camera-based system 1 includes a camera device 2 and a computing unit 3. The camera device 2 includes an objective lens 4 and an image sensor 5. The objective lens 4 may be, for example, a fisheye lens or a linear wide-angle lens. The image sensor 5 may be a complementary metal-oxide-semiconductor (CMOS) sensor or a charge-coupled device (CCD) sensor. The light acquired by the objective lens 4 is detected by the image sensor 5. The resulting raw image is then further processed by the computing unit 3.

[0038] The camera-based system 1 is mounted behind the windshield 6. The windshield 6 has a thickness t and a direction n given by the windshield normal vector. The windshield 6 also has a refractive index ν.

[0039] Figure 1 The image also shows a spatial point s, which is a spatial point within a scene. Spatial point s can be a point of particular interest, or a corner of a bounding box such as a pedestrian or other vehicle. In addition, a light path 7 is shown, which originates from spatial point s, is deflected by windshield 6, and is then focused onto image sensor 5 by objective lens 4.

[0040] Intermediate image data is calculated to correct the original image or some points in the original image. The intermediate image data includes an intermediate image or intermediate image points. The intermediate image data is calculated to eliminate the influence of objective lens 4, wherein the calculation is based on the parameters of the imaging device, particularly the parameters of objective lens 4.

[0041] like Figure 2 As shown, eliminating the influence of objective lens 4 results in an image acquired by pinhole camera device 8 instead of camera device 2. The light path 7 passes through the pinhole 9 of pinhole camera device 8 and terminates at sensor 5.

[0042] Figure 2 The image also shows another virtual ray 10, which passes through the pinhole 9 and, like the light path 7, terminates at the same point on the sensor 5. The virtual ray 10 is drawn in the absence of the windshield 6.

[0043] On the side of the windshield 6 opposite to the pinhole camera 8, the optical path 7 has a parallel offset 11 relative to the virtual ray. The parallel offset 11 is equal to the product of the slope offset σ and the normal vector n.

[0044] The slope offset σ can be approximated as a constant, especially as t(ν-1) / ν. This approximation is very quick to calculate, but it is only sufficiently accurate for small incident angles.

[0045] As an alternative, the slope offset σ can be calculated as the root σ0 of the quartic equation.

[0046] g(σ)=a4σ 4 +a3σ 3 +a2σ 2 +a1σ+a0,

[0047] Where 0≤σ0≤t, a4=ν 2 a3=–2ν 2 (w+t), a2=(v 2 –1)(u 2 +t 2 )+ν 2 w(w+4t), a1=–2t(ν) 2 (w 2 +u 2 )+tw(ν 2 –1)–u2 ), a0=(ν 2 –1)t 2 (w 2 +u 2 ), w = n·s and u = √(s·s–w 2 This calculation leads to an extremely accurate result, but the computational cost is very high.

[0048] As an alternative, the slope offset σ can be calculated as a fixed point of the fixed point equation σ=φ(σ), where φ(σ=t(1–1 / √((ν 2 –1)(u 2 / (w–σ) 2 +1)+1)), w and u are as described above. One or two iterations of this convergent fixed-point equation can yield extremely accurate results, and the computation is rapid.

[0049] Once the parallel offset 11 is known, the optical path 7 can be traced back, and further methods such as bundle adjustment or stereo vision can be used to calculate the point s in space.

[0050] Figure 3 The setup for determining windshield parameters is illustrated in Figure 12. A known pattern 13 is positioned at the front of a vehicle 14, which includes a windshield 6 and a camera-based system 1 positioned behind the windshield 6. The camera-based system 1 acquires a calibration image of the known pattern 13. Using known camera parameters, an intermediate calibration image is calculated from the calibration image, wherein the intermediate calibration image is similar to an image of the known pattern 13 captured by a pinhole camera 8 through the windshield 6. Then, a set 15 of points of the known pattern 13 is identified and compared with pixels in the intermediate calibration image. Based on the comparison, the windshield parameters are obtained.

[0051] Other variations of the disclosed embodiments can be understood and implemented by a person skilled in the art when practicing the claimed invention, by studying the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other parts or steps, and the indefinite article "a" does not exclude multiple. In fact, the specific measures cited in mutually different dependent claims do not imply that combinations of these measures cannot be applied in a beneficial manner. All reference numerals in the claims should not be construed as limiting the scope of the claims.

Claims

1. A method for correcting images and / or image points acquired by at least one camera device (2) of a camera-based system for a vehicle (14) with a windshield (6), wherein, The method includes the following steps: The camera device (2) acquires the original image of a scene. The camera device (2) includes an objective lens (4) and an image sensor (5). A windshield (6) is set in front of the camera device (2). Light from the scene is first deflected by the windshield (6) and then focused onto the image sensor (5) by the objective lens (4). Select original image data from the original image, where the original image data is the complete original image, a portion of the original image, or multiple original image points of the original image; Intermediate image data is calculated based on the original image data and the camera device parameters of the characterizing objective lens (4). The intermediate image data includes intermediate images or multiple intermediate image points, which are equivalent to relevant scene images or scene image points obtained by the pinhole camera device (8) through the windshield (6). The parallel offset (11) of the optical path (7) caused by the windshield (6) is obtained based on the windshield parameters, and the obtained parallel offset (11) is used to calculate a set of points in the scene space corresponding to intermediate image pixels or intermediate image points, wherein the windshield parameters include the normal vector n of the windshield (6) in the area near the camera device (2), the thickness t of the windshield (6) and / or the refractive index ν of the windshield (6).

2. The method according to claim 1, wherein, A sequence of original images of a scene is acquired using a camera device (2); Tracking points in the original image and / or intermediate image, and The set of points in space is matched with the original or intermediate image points being tracked.

3. The method according to claim 1 or 2, wherein, Parallel offset (11) is equal to the product of slope offset σ and normal vector n.

4. The method according to claim 3, wherein, Set the slope offset σ to a constant.

5. The method according to claim 3, wherein, The slope offset σ is equal to t (ν - 1) / ν.

6. The method according to claim 3, wherein, The slope offset σ is calculated as a quartic equation g(σ) = a⁴σ 4 +a3 σ 3 + a2 σ 2 + a1 σ + root σ0 of a0, where 0 ≤ σ0 ≤ t, a4 = ν 2 a3 = – 2ν 2 (w + t), a2 = (v 2 – 1)(u 2 + t 2 ) + ν 2 w(w + 4t), a1 = – 2t(ν) 2 (w 2 + u 2 ) + tw(ν 2 – 1) – u 2 ), a0 = (ν 2 – 1)t 2 (w 2 + u 2 ), w = n·s, u = √(s·s – w 2 ) and s is a spatial point in the relevant scene space.

7. The method according to claim 3, wherein, The slope offset σ is calculated as the fixed point of the fixed point equation σ = φ(σ), where φ(σ) = t(1 – 1 / √((ν)). 2 – 1)(u 2 / (w – σ) 2 + 1) + 1)), w = n·s, u = √(s·s – w 2 ) and s is a spatial point in the relevant scene space.

8. The method according to claim 7, wherein, Only calculate the first or second iteration of the fixed-point equation.

9. The method according to claim 1 or 2, wherein, The camera device parameters are determined for camera device (2) or another camera device similar to camera device (2) by means of... Acquire a calibration image of a known pattern (13) positioned in the field of view of the camera device (2) or another camera device without a windshield (6). The calibrated image is compared with a known pattern (13), and The camera device parameters are determined from the comparison.

10. The method according to claim 1 or 2, wherein, The windshield parameters are determined by the following method for windshield (6) or another windshield similar to said windshield (6): Acquire a calibration image of a known pattern (13) set within the field of view of the camera device (2) or another camera device similar to the camera device (2), wherein a windshield (6) or other windshield is positioned in front of the camera device (2) or other camera device. An intermediate calibration image is calculated based on the calibration image and camera device parameters, wherein the intermediate calibration image is equivalent to an image of a known pattern (13) captured by a pinhole camera (8) through the windshield (6). The intermediate calibration image is compared with the known pattern (13), and The windshield parameters are determined from the comparison.

11. The method according to claim 1 or 2, wherein, Based on the automatic calibration of the camera-based system, windshield parameters are determined or improved.

12. The method according to claim 1 or 2, wherein, The calculation of points in scene space corresponding to intermediate image pixels or intermediate image points is performed through bundle adjustment.

13. The method according to claim 1 or 2, wherein, The calculated set of points in the scene space is used for object recognition, object tracking, and / or advanced driver assistance systems.

14. A camera-based system for a vehicle (14), comprising at least one camera (2) disposed behind the windshield (6) of the vehicle (14) and a computing unit (3), wherein, The system based on the camera device is suitable for performing the method according to any one of claims 1 to 13.

15. A vehicle, wherein, The vehicle includes a windshield (6) and a camera-based system for the vehicle (14) according to claim 14.

Images