Method for rectifying images and / or image points, camera-based system and vehicle - Patents.com

Abstract

The present invention relates to a method for rectifying images and / or image points acquired by at least one camera (2) of a camera-based system (1) of a vehicle (14) having a windshield pane (6). First, a raw image of a scene is acquired using the camera (2). Raw image data is selected from the raw image, where the raw image data is the entire raw image, a portion of the raw image, or multiple raw image points of the raw image. Next, intermediate image data is calculated based on the raw image data and camera parameters, where the intermediate image data comprises an intermediate image or multiple intermediate image points, and the intermediate image data resembles an image or image points of the scene acquired by a pinhole camera (8) through the windshield pane (6). Finally, a set of points in the space of the scene is calculated, where the set of points corresponds to pixels or intermediate image points of the intermediate image. This calculation is performed using a parallel shift (11) of an optical path (7) guided by the windshield pane (6) based on the windshield pane parameters. The present invention also relates to a camera-based system (1) for a vehicle (14) and the vehicle (14).

Description

[Technical Field]

[0001] The present invention relates to a method for rectifying images and / or image points acquired by at least one camera of a camera-based system for a vehicle having a windshield glazing, a camera-based system for a vehicle and a vehicle. [Background technology]

[0002] For example, computer vision used in advanced driver assistance systems (ADAS) can assign points in space to pixels or regions in images acquired by a camera. For ADAS, accurate evaluation of the camera's projection is essential for accurate estimation of the distance to objects in front of the vehicle, while the latter is essential for camera-based road safety applications, such as automatic braking or automatic cruise control.

[0003] For ADAS applications in vehicles, cameras are typically mounted on the mirror cover behind the windshield pane. Obviously, this solves the problem of the camera lens being blocked by dirt or rain. Unfortunately, the camera's particular location introduces the optical properties of the windshield pane into the projection geometry of the entire optical system. The influence of the windshield pane can cause the optical path to behave significantly differently compared to the behavior of coaxially arranged lenses in the camera optics. As a result, a camera's optical system, along with the windshield pane, can produce significant image deviations compared to the same camera without the windshield pane in front of it. This type of optical distortion is referred to as windshield pane distortion aberration of the camera image.

[0004] The task of evaluating the projection characteristics of an optical system, including any type of distortion aberration that occurs, is called camera calibration. Generally, the latter process has a parametric character. That is, there exists a family of functions (camera models) that are customized by a vector of parameters, and for each constraint vector of parameters, they describe the physical camera device, that is, how a point in space is projected onto an image by this particular camera. In this case, these vectors are called camera parameters.

[0005] Typically, the application of camera models in ADAS addresses the inverse problem of projection. That is, given a pixel location in an image, the goal is to estimate the set of points in space that are imaged at this particular pixel location. In this case, camera calibration is used as a tool to remove undesirable effects of optics in order to describe the optical system with a computationally simple model. This simple model is a pinhole camera, which is inherently a central projection. Further removing the effects of windshield panes is called image rectification. In particular, with regard to stereo imaging, rectification is a key tool for efficiently estimating a dense distance map of a scene. Stereo rectification rectifies images from two different camera systems as if they were produced by two parallel systems of optically identical pinhole cameras. In such systems, corresponding object points can be found on the same horizontal image line, which greatly simplifies the search for corresponding points and, consequently, the disparity calculation.

[0006] Ignoring the effects of windshield panel distortion can significantly obscure distance estimation for objects in front of the vehicle, especially at short distances. Therefore, it is essential to model and estimate windshield panel distortion.

[0007] As an example, U.S. Patent No. 6,277,949 (Patent Document 1) includes a method for rectifying images acquired by at least one camera of a camera-based system of a vehicle having a windshield window pane. The camera-based system is calibrated by providing an imaging target in the form of a board with a known pattern in the field of view of the camera of the camera-based system so that the camera can acquire a calibration image of the board through the windshield window pane. Based on this, the windshield distortion aberration introduced by the windshield window pane is calculated. Next, an image is acquired using the camera, and a set of points in space that are projected onto positions on the image are calculated using the calculated windshield distortion aberration. However, this method is computationally intensive and treats the camera and windshield window pane as a single optical system. Therefore, a new calibration must be performed for every combination of camera and windshield window pane. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] European Patent Application Publication No. 3293701 [Patent Document 2] German Patent Application Publication No. 102018204451 [Non-patent literature]

[0009] [Non-Patent Document 1] R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, Cambridge, 2nd edition, 2003 Summary of the Invention [Problem to be solved by the invention]

[0010] It is an object of the present invention to provide a method for rectifying images and / or image points acquired by at least one camera of a camera-based system of a vehicle having a windshield glazing, which is computationally fast and separates the optical effects of the windshield glazing from the optical effects of the camera.It is a further object of the present invention to provide a camera-based system for a vehicle configured to perform this method, and a vehicle comprising a windshield glazing and this camera-based system. [Means for solving the problem]

[0011] This problem is solved by the subject matter of the independent claims. Embodiments are provided by the dependent claims, the following description and the accompanying figures.

[0012] According to a first aspect of the present invention, there is provided a method for rectifying images and / or image points, i.e. the method is applied to both rectifying the entire image and to rectifying some image points of the image, for example image points relating to an object of interest, image points relating to a highly distinctive object in the image or image points relating to the bounding boxes of other vehicles or pedestrians, for example.

[0013] The images or image points are acquired by at least one camera of a camera-based system of a vehicle having a windshield window pane. If the camera-based system includes two or more cameras, for example, stereoscopic information may be acquired by the cameras. The cameras are located behind the windshield window pane, i.e., the windshield is positioned between the cameras and the surroundings of the vehicle. The cameras are therefore protected from environmental influences such as rain or dirt.

[0014] As a first step of the method, a camera is used to acquire a raw image of a scene. The scene may be a traffic scene including roads, road signs, buildings, pedestrians, and / or other vehicles. A windshield window pane is positioned in front of the camera, so that light rays from the scene are first deflected by the windshield window pane and then focused onto the image sensor by an objective lens. The objective lens may be a wide-angle lens, such as a fisheye lens or a rectilinear wide-angle lens. The image sensor may be, for example, a CMOS sensor or a CCD sensor.

[0015] Next, RAW image data is selected from the RAW image, the RAW image data being the entire RAW image, a portion of the RAW image, or a number of RAW image points of the RAW image, where the portion of the RAW image may be a particular region of interest and the RAW image points may relate to an object of interest, a highly distinctive object in the image, or a bounding box of, for example, another vehicle or pedestrian.

[0016] The intermediate image data is calculated based on the raw image data. The intermediate image data comprises an intermediate image or a plurality of intermediate image points and is similar to an image or image points of a scene acquired by a pinhole camera through a windshield pane. In other words, the influence of the objective lens is removed in the intermediate image and replaced by a pinhole camera. To do so, the calculation of the intermediate image data is further based on camera parameters, i.e., parameters characterizing the objective lens. Details of this calculation are well known to those skilled in the art. By removing the influence of the objective lens in this step, the optical influence of the camera and the optical influence of the windshield pane are separated from each other. Therefore, if a different camera is used behind a given windshield pane, only the new camera parameters need to be provided, and if a camera with known camera parameters is installed behind a different windshield pane, only the influence of the different windshield pane needs to be evaluated.

[0017] As the final step of the method, a set of points in scene space is calculated. This set of points in scene space corresponds to pixels or intermediate image points of the intermediate image. That is, a point in scene space is a point in space that, when projected by a pinhole camera through a windshield pane, results in a pixel or intermediate image point of the intermediate image. To calculate the points in scene space, a translational shift of the light path induced by the windshield pane is used. That is, the effect of the windshield pane is modeled as a translational shift of the light path. This applies accurately to planar windshield panes and is a very good approximation for windshield panes that have only slight curvature in the field of the camera. The calculation is based on windshield pane parameters and is computationally fast.

[0018] The method therefore provides a rectification of images and / or image points acquired by at least one camera that separates the effects of the windshield pane and the effects of the camera and is fast to compute.

[0019] According to an embodiment, a sequence of RAW images of a scene is acquired using a camera. This sequence is in particular a time-series image of the scene. In the sequence, the scene changes as objects in the scene move and / or as the vehicle, and therefore the camera, moves. As the scene changes, RAW image points and / or intermediate image points are tracked. A set of points in space also changes with the sequence of RAW images and corresponds to the tracked RAW image points and / or tracked intermediate image points. By tracking the RAW image points and / or intermediate image points, the distance of the point in space corresponding to this image point may be determined.

[0020] According to an embodiment, the windshield pane parameters are a parameter of the windshield pane in the area near the camera.

number

number

[0021] According to an embodiment, the parallel shift is

number

[0022] According to an embodiment, the slab shift σ is approximated as a constant. Approximating the slab shift as a constant allows for very fast calculations and produces good results for relatively small angles of view. In particular, the slab shift σ is equal to t(ν−1) / ν, which is the exact solution for a light path perpendicular to the windshield pane.

[0023] According to an embodiment, the slab shift σ is expressed by a quartic equation g(σ)=a4σ 4 +a3σ 3 +a2σ 2 +a1σ+a0, where 0≦σ0≦t, a4=ν 2 , a3=-2ν 2 (w+t),a2=(v 2 -1)(u 2 +t 2 )+ν 2 w(w+4t), a1=-2t(ν 2 (w 2 +u 2 )+tw(ν 2 -1)-u 2 ), a0=(ν 2 -1)t 2 (w 2 +u 2 ),

number

number

number

[0024] According to an embodiment, the slab shift σ is calculated as a fixed point of the fixed point equation σ=φ(σ), where φ(σ)=t(1−1 / √((ν 2 -1)(u 2 / (w-σ) 2 +1)+1)),

number

number

number

[0025] According to an embodiment, only one or two iterations of the fixed-point equation are calculated, which makes the calculation of the slab shift σ very fast while still providing very good accuracy.

[0026] According to an embodiment, the camera parameters are determined for a camera by acquiring a calibration image of a known pattern in the camera's field of view, without the windshield pane present. This known pattern may be, for example, a checkerboard pattern. The calibration image is then compared with the known pattern, in particular identifying points in the calibration image as points in the known pattern. Based on this comparison, the camera parameters are determined. The camera parameters may also be determined by performing the above steps for another camera similar to the camera. The camera parameters of this other camera are expected to be very similar to the camera parameters of the camera, and in fact nearly identical.

[0027] According to an embodiment, the windshield pane parameters are determined for a windshield pane by acquiring a calibration image of a known pattern in front of a camera and in the camera's field of view, where the windshield pane is located. The known pattern may be, for example, a checkerboard pattern. An intermediate calibration image is calculated based on the calibration image and the camera parameters. The intermediate calibration image resembles an image of the known pattern acquired through the windshield pane with a pinhole camera. In other words, the influence of the camera objective lens is removed in the intermediate calibration image. The intermediate calibration image is compared with the known pattern, and in particular, points in the intermediate calibration image are identified as points of the known pattern. The windshield pane parameters are determined based on this comparison. The influence of the windshield pane on the optical path may be calculated as described above. The windshield pane parameters may also be determined by performing the above steps for other windshield panes similar to the windshield pane and / or for other cameras similar to the camera. In all cases, the acquired windshield pane parameters are expected to be very similar to the windshield pane parameters, and in fact, substantially identical to the windshield pane parameters.

[0028] According to an embodiment, the windshield pane parameters are determined or improved based on an automatic calibration of the camera-based system, which is described, for example, in U.S. Patent No. 6,277,999 and will not be described in detail herein. The automatic calibration may, for example, correct for slight variations in mounting the camera behind the windshield pane.

[0029] According to an embodiment, the calculation of points in the space of the scene corresponding to pixels or points of the intermediate images is performed by bundle adjustment, as described, for example, in [1]. Such a fast method for rectifying images allows the points in space to be calculated quickly.

[0030] According to an embodiment, the calculated set of points in the space of the scene is used for object recognition, object tracking and / or advanced driver assistance systems, which are applications that greatly benefit from fast computation and the additional accuracy that comes from taking into account the effects of windshield panes.

[0031] According to another aspect of the present invention, there is provided a camera-based system for a vehicle, comprising at least one camera arranged behind a windshield pane of a vehicle and a computing unit, the camera-based system being adapted to perform the method and thus benefiting from fast computation and separation of the optical effects of the windshield pane and the camera.

[0032] According to yet another aspect of the present invention, there is provided a vehicle comprising a windshield pane and the above camera-based system.

[0033] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter.

[0034] Exemplary embodiments of the present invention are described below with reference to the following drawings: [Brief explanation of the drawings]

[0035] [Figure 1] FIG. 1 shows a schematic diagram of a camera-based system mounted behind a windshield pane. [Figure 2] FIG. 2 shows a schematic diagram of a pinhole camera mounted behind the windshield window pane. [Figure 3] Figure 3 shows a schematic diagram of the calibration setup.

[0036] The figures are only schematic representations and serve only to illustrate embodiments of the invention, and in principle identical or similar elements are provided with the same reference signs. DETAILED DESCRIPTION OF THE INVENTION

[0037] FIG. 1 shows a schematic diagram of a camera-based system 1 for a vehicle. The camera-based system 1 comprises a camera 2 and a computing unit 3. The camera 2 comprises an objective lens 4 and an image sensor 5. The objective lens 4 may be, for example, a fisheye lens or a rectilinear wide-angle lens. The image sensor 5 may be a CMOS sensor or a CCD sensor. Light captured by the objective lens 4 is detected by the image sensor 5. The RAW image thus captured is then further processed by the computing unit 3.

[0038] The camera-based system 1 is mounted behind a windshield pane 6. The windshield pane 6 has a thickness t.

number

[0039] Also, in Figure 1, we see a point in the space of the scene

number

number

[0040] In order to rectify the RAW image or some RAW image points, intermediate image data is calculated, which comprises an intermediate image or intermediate image points, which is calculated by removing the influence of the objective lens 4, and which calculation is based on the camera parameters, in particular the parameters of the objective lens 4.

[0041] By removing the influence of the objective lens 4, the image would be captured by a pinhole camera 8 as shown in Figure 2 instead of the camera 2. The optical path 7 of the light passes through a pinhole 9 in the pinhole camera 8 and terminates at the sensor 5.

[0042] Also shown in Figure 2 is another virtual ray 10 that passes through pinhole 9 and terminates at the same point on sensor 5 as light path 7. Virtual ray 10 is illustrated as if windshield pane 6 were not present.

[0043] On the opposite side of the windshield pane 6 from the pinhole camera 8, there is a parallel shift 11 of the optical path 7 for the virtual ray. This parallel shift 11 is

number

[0044] The slab shift σ may be approximated as a constant, specifically as t(ν−1) / ν, which, while very fast to compute, is only accurate enough for small angles of incidence.

[0045] Alternatively, the slab shift σ can be expressed as a quartic equation g(σ)=a4σ 4 +a3σ 3 +a2σ 2 +a1σ+a0, where 0≦σ0≦t, a4=ν 2 , a3=-2ν 2 (w+t),a2=(v 2 -1)(u 2 +t 2 )+ν 2 w(w+4t), a1=-2t(ν 2 (w 2 +u 2 )+tw(ν 2 -1)-u 2 ), a0=(ν 2 -1)t 2 (w 2 +u 2 ),

number

[0046] As yet another alternative, the slab shift σ may be calculated as a fixed point of the fixed point equation σ = φ(σ), where φ(σ) = t(1 − 1 / √((ν 2 -1)(u 2 / (w-σ) 2 +1)+1)), w, and u are as above. One or two iterations of this convergent fixed-point equation give very accurate results and are fast to compute.

[0047] Once the parallel shift 11 is obtained, the light path 7 can be traced and

number

[0048] 3 shows a setup for determining windshield pane parameters. A chart 12 having a known pattern 13 is mounted in front of a vehicle 14 with a windshield pane 6 and a camera-based system 1 mounted behind the windshield pane 6. A calibration image of the known pattern 13 is acquired by the camera-based system 1. Using the known camera parameters, an intermediate calibration image is calculated from the calibration image, which resembles an image of the known pattern 13 acquired by the pinhole camera 8 through the windshield pane 6. A set of points 15 of the known pattern 13 is then identified and compared to pixels in the intermediate calibration image. Based on this comparison, the windshield pane parameters are obtained.

[0049] Other variations of the disclosed embodiments can be understood and realized by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure and the appended claims. The word "comprising" in the claims does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope of the claims. The present application relates to the invention described in the claims, but also includes the following as other aspects. 1. 1. A method for rectifying images and / or image points acquired by at least one camera (2) of a camera-based system (1) of a vehicle (14) having a windshield window pane (6), comprising: capturing a RAW image of a scene using the camera (2); selecting RAW image data from the RAW image, the RAW image data being the entire RAW image, a portion of the RAW image, or a plurality of RAW image points of the RAW image; calculating intermediate image data based on the raw image data and camera parameters, the intermediate image data comprising an intermediate image or a plurality of intermediate image points, the intermediate image data resembling an image or image points of a scene captured by a pinhole camera (8) through the windshield window pane (6); and calculating a set of points in scene space corresponding to pixels or points of the intermediate image using a parallel shift (11) of the light path (7) induced by the windshield pane (6) based on windshield pane parameters. 2. a sequence of RAW images of a scene is acquired using said camera (2); the raw image points and / or the intermediate image points are tracked; 2. The method of claim 1, wherein the set of points in space corresponds to the tracked raw image points or the tracked intermediate image points. 3. The windshield pane parameters are the parameters of the windshield pane (6) in the area near the camera (2).

number

number

number

number

Claims

1. 1. A method for rectifying images and / or image points acquired by at least one camera (2) of a camera-based system (1) of a vehicle (14) having a windshield window pane (6), comprising: acquiring a RAW image of a scene using said camera (2); selecting RAW image data from the RAW image, wherein the RAW image data is the entire RAW image, a portion of the RAW image, or a plurality of RAW image points of the RAW image; calculating intermediate image data based on the RAW image data and camera parameters, the intermediate image data comprising an intermediate image or a plurality of intermediate image points, the intermediate image data resembling an image or image points of a scene captured by a pinhole camera (8) through the windshield window pane (6); and calculating a set of points in the space of the scene corresponding to pixels or points of the intermediate image using a parallel shift (11) of the light path (7) induced by the windshield pane (6) based on windshield pane parameters. In the method, The windshield pane parameters are the parameters of the windshield pane (6) in the area near the camera (2). [Equation 1] a thickness t of the windshield pane (6) and / or a refractive index ν of the windshield pane (6), and The parallel shift (11) [Equation 2] is equal to the slab shift σ times The slab shift σ is calculated as the root σ 0 of the quartic equation g(σ) = a 4 σ 4 + a 3 σ 3 + a 2 σ 2 + a 1 σ + a 0 , where 0≦σ 0 ≦t, a 4 =ν 2 , a 3 =-2ν 2 (w+t), a 2 = (ν 2 -1)(u 2 +t 2 )+ν 2 w(w+4t), a 1 =-2t(ν 2 (w 2 +u 2 )+tw(ν 2 -1)-u 2 ), a 0 = (ν 2 -1)t 2 (w 2 +u 2 ), [Equation 3] is the spatial point of the point in the space of the scene, A method characterized by:

2. A method for rectifying images and / or image points acquired by at least one camera (2) of a camera-based system (1) of a vehicle (14) having a windshield window pane (6), comprising: acquiring a RAW image of a scene using said camera (2); selecting RAW image data from the RAW image, wherein the RAW image data is the entire RAW image, a portion of the RAW image, or a plurality of RAW image points of the RAW image; calculating intermediate image data based on the RAW image data and camera parameters, the intermediate image data comprising an intermediate image or a plurality of intermediate image points, the intermediate image data resembling an image or image points of a scene captured by a pinhole camera (8) through the windshield window pane (6); and calculating a set of points in the space of the scene corresponding to pixels or points of the intermediate image using a parallel shift (11) of the light path (7) induced by the windshield pane (6) based on windshield pane parameters. In the method, The windshield pane parameters are the parameters of the windshield pane (6) in the area near the camera (2). [Equation 4] a thickness t of the windshield pane (6) and / or a refractive index ν of the windshield pane (6), and The parallel shift (11) [Equation 5] is equal to the slab shift σ times The slab shift σ is calculated as a fixed point of the fixed point equation σ=φ(σ), where φ(σ)=t(1−1 / √((ν 2 −1)(u 2 / (w−σ) 2 +1)+1)), [Equation 6] is the spatial point of the point in the space of the scene, A method characterized by:

3. a sequence of RAW images of a scene is acquired using said camera (2); the raw image points and / or the intermediate image points are tracked; The method of claim 1 or 2, wherein the set of points in space correspond to the tracked raw image points or the tracked intermediate image points.

4. The method of claim 2 , wherein the fixed-point equation iteration is calculated only once or twice.

5. Determining the camera parameters for the camera (2) or for other cameras similar to the camera (2) comprises: acquiring a calibration image of a known pattern (13) in the field of view of the camera (2) or the other camera without the windshield pane (6); comparing said calibration image with said known pattern (13); The method of any one of claims 1 to 4, wherein said step of determining said camera parameters is carried out by:

6. Determining the windshield pane parameters for the windshield pane (6) or for other windshield panes similar to the windshield pane (6) comprises: the windshield pane (6) or the other windshield pane is in front of the camera (2) or another camera similar to the camera (2), and acquiring a calibration image of a known pattern (13) in the field of view of the camera (2) or the other camera; calculating an intermediate calibration image based on the calibration image and the camera parameters, the intermediate calibration image resembling an image of the known pattern (13) acquired through the windshield pane (6) by a pinhole camera (8); comparing said intermediate calibration image with said known pattern (13); and determining said windshield pane parameters from said comparison.

7. The method according to any one of the preceding claims, wherein the windshield pane parameters are determined or improved based on an automatic calibration of the camera-based system (1).

8. Method according to any one of claims 1 to 7, wherein the calculation of the points in the space of the scene corresponding to the pixels or intermediate image points of the intermediate images is done by bundle adjustment.

9. The method according to any one of claims 1 to 8, wherein the calculated set of points in the space of the scene is used for object recognition, object tracking and / or advanced driver assistance systems.

10. A camera-based system (1) for a vehicle (14), comprising at least one camera (2) arranged behind a windscreen window (6) of the vehicle (14), and a computing unit (3), the camera-based system (1) being configured to perform the method according to any one of claims 1 to 9.

11. A vehicle comprising a windscreen (6) and a camera-based system (1) according to claim 10.

Images