Dynamic Auto-Calibration of Behind-the-Wind Vehicle Camera Systems

Abstract

The present invention relates to a method and device for automatically calibrating a vehicle camera (1; 31; 40) while the vehicle (33) is moving, which can be used as a sensor system for driver assistance systems and autonomous driving. The vehicle camera (1; 31; 40) images an area of ​​the vehicle's (33) surroundings (45; 46) through a glass (32; 44). The method comprises: a) providing a projection model of the vehicle camera (1; 31; 40), the projection model being based on parameters (θ j ,int,ψ) as multiple external parameters (θ j ), at least one internal parameter (int) of the vehicle camera (1; 31; 40) and at least one parameter (ψ) characterizing the glass (32; 44), b) capturing a sequence of images of a vehicle camera (1; 31; 40) while the vehicle (33) is travelling around a curve (5); c) determining the curve type based on the actual movement of the vehicle (33) while traveling through the curve (5) by means of a curve estimator; d) - image points of an image sequence of stationary objects (45) in the surrounding environment of the vehicle (33), - the actual movement of the vehicle (33) and -Determined curve type Considering the image points determined from the image sequence corresponding to the stationary object (45) in the vehicle's surrounding environment and the image points (p ij ) and the error between estimating parameters by minimizing TIFF2024542542000040.tif6170; e) Estimated parameters (θ j , ψ, int). The present solution offers the advantage of significantly simplifying the test systems on the vehicle manufacturer side in the production of camera-based driver assistance systems, and on the repair shop side in the testing or calibration systems when replacing camera systems or vehicle glass.

Description

[Technical Field]

[0001] The present invention relates in particular to a method and apparatus for automatically calibrating a vehicle camera system that can be used in a vehicle as a sensor system for driver assistance systems and autonomous driving, capturing the surrounding environment through glass. [Background technology]

[0002] Patent Document 1 describes a method and apparatus for automatically calibrating a monocular vehicle camera. a) capturing a sequence of images from a vehicle camera, the vehicle camera mapping an area of ​​the surrounding environment in front of the vehicle; b) detecting a curve run by the vehicle suitable for automatic calibration if the curve radius described by the vehicle is less than or equal to a defined maximum radius or the curve angle passed by the vehicle is greater than or equal to a defined minimum angle; c) performing an automatic calibration when at least one curve suitable for automatic calibration is detected, d) the automatic calibration is performed taking into account the movement of stationary objects in the vehicle's surrounding environment when driving a curve suitable for the automatic calibration.

[0003] Camera calibration is an essential element for Advanced Driver Assistance Systems (ADAS) or Automated Driving (AD) systems to capture the surrounding environment using camera systems mounted in or on the vehicle. The camera system tracks the vehicle's movement or travels similarly. For camera calibration, estimation methods are used to calculate the parameters of the formal relationship (projection) between 3D spatial points and the corresponding image points of the physical camera system. The calculated parameters are then stored in the ADAS system for further use. The projection policy includes a description of the light propagation path within the camera's optical system (intrinsic parameters) and its position and orientation with respect to a fixed reference coordinate system in the vehicle (extrinsic parameters).

[0004] If the projection policy parameters are calculated accurately, the vehicle's surroundings can be captured and measured while driving using a structure-from-motion (SfM) or multi-perspective method. If the projection policy parameters deviate slightly from the actual projection policy, subsequent method processes that use the distance (e.g., adaptive cruise control (ACC), emergency braking assist (EBA), or automatic emergency braking (AEB)) or angle (e.g., lane keeping assist (LKA) or head lamp assist (HLA)) calculated from the camera image data for the mapped objects may produce inaccurate results. If the projection policy parameters deviate significantly from the actual projection policy, the ADAS / AD system may be limited or become unavailable (short-term or permanent failure).

[0005] Typically, ADAS cameras are mounted in vehicles behind the vehicle's windshield or protective glass. The (windshield) glass constitutes an additional optical system and significantly changes the projection policy of the camera compared to a configuration without the windshield glass between the scene and the camera. To ensure accurate functionality of subsequent method processes, the windshield needs to be taken into account during camera calibration.

[0006] The existing camera calibration process for ADAS camera systems involves multiple steps, including camera calibration at the end of the camera production line, camera calibration at the end of the vehicle production line, and automatic camera calibration while driving. Calibration in production (whether in camera production or vehicle production) is fundamental for robust environment capture in the vehicle, as it serves as the initial solution or foundation for the further two steps. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] German Patent Application Publication No. 102018204451 [Patent Document 2] European Patent No. 3293701 [Non-patent literature]

[0008] [Non-Patent Document 1] Hartley, Zissermann, Multiple View Geometry in Computer Vision, 2000, Cambridge University Press, ISBN: 0521623049 (1st edition). [Non-patent document 2] Peter Sturm: Critical motion sequences for monocular self-calibration and uncalibrated euclidean reconstruction, CVPR, 1997, p. 1100-1105. [Non-patent document 3] C. Wu: Critical configurations for radial distortion self-calibration. In CVPR, 2014. Summary of the Invention [Problem to be solved by the invention]

[0009] It is basically conceivable to calibrate the camera together with the windshield glass at the end of the camera production line. However, due to the variety of windshield glass types and the individual mounting positions of each camera behind each windshield glass, the corresponding projection policies are also individual. Therefore, from a practical point of view, calibration during camera production is impossible or not meaningful.

[0010] It is also possible to calibrate the camera together with a special windshield glass at the final stage of vehicle production. This method is also the most frequently used conventional technique. Obviously, this option requires a new calibration of the optical system when the glass is replaced. Due to the high accuracy required for the camera calibration, calibration using the above method is very costly and therefore expensive. For this purpose, targets manufactured with high precision are used. Furthermore, the vehicle is adjusted with high precision for calibration using special, expensive equipment. Generally, only external parameters can be readjusted using the above method.

[0011] The influence of the windshield glass on the optical path of the light through the camera is often not taken into account. Even if prior art calibrations solve part of this problem, they do not address the possibility that the optical system consisting of the ADAS camera and the windshield glass is changed during driving. Thus, there is always a risk that an end user, i.e., a driver of a vehicle equipped with a driver assistance system, will use an ADAS system with misestimated parameters and therefore with limited functionality. The object of the present invention is to provide a solution for simplifying and improving the process of camera calibration. [Means for solving the problem]

[0012] This problem is solved by the subject matter having the features of the independent claims. Preferred embodiments are the subject matter of the dependent claims.

[0013] One aspect of this solution is that the entire optical system based on glass and cameras is calibrated not at the end of the vehicle production line, but dynamically in the vehicle while it is in motion and continuously improved. Dynamic auto-calibration can also be called online calibration.

[0014] One aspect is that the process of calibrating ADAS camera systems mounted behind windshield glass or protective glass during camera production can be greatly simplified.

[0015] One aspect of the present invention relates to an apparatus and method for estimating a set of parameters of an ADAS camera system while the vehicle is in motion. One advantage of the present invention is that highly accurate manufactured targets are not required for calibration.

[0016] The method uses a fixed part of the (unknown) traffic scene located in front of the vehicle to calibrate the effect of the windshield. To achieve this, a special optical model of the windshield and camera combination is required.

[0017] It is known from experience in the prior art that parameter estimation of such models is not always possible. It is also known from the prior art that certain parameters can be calibrated during a particular advantageous run to perform camera calibration. An advantage here is, for example, non-vanishing (e.g., radial) distortion. In one embodiment of the present invention, the estimation of the parameters that can be estimated during the run trajectory can be learned during the run based on previous experience. These two properties allow calibration quality comparable to industry standards to be achieved after a few measurements.

[0018] One aspect of the present invention relates to an apparatus (eg, a control device) that computes the calibration of a vehicle-mounted camera system. One aspect of the present invention relates to estimating the driving movement or driving geometry of a vehicle while driving around a curve (ie, curve movement estimation) and taking this into account when estimating the parameters of a camera system. A further aspect relates to considering at least one parameter that characterizes the glass.

[0019] One aspect of the present solution relates to the realization or assumption that automatic calibration of the entire windshield and camera-based optical model during special curve maneuvers is possible using bundle adjustment methods. The method can be configured to learn correlations between curve types and calculable parameters. This method represents a significant innovative step compared to the current state of the art.

[0020] One aspect of the present invention relates to the construction of a library or some kind of rule-of-thumb data bank where information is stored that allows for that type of driving or that type of curve to be updated or recalibrated. The use of a library allows for more curve driving to be used for calibration compared to the method of the '600 patent, and provides a way to specifically find out which parameters can be recalibrated and when.

[0021] The apparatus and method are configured to provide the necessary data to initialize the bundle adjustment algorithm with a good initial solution. From the results of many curve runs, filtering can be used to obtain significant improvements in accuracy comparable to that of production calibration.

[0022] This solution provides the advantage that vehicle manufacturers can greatly simplify the test system in the production of ADAS systems, and repair shops can greatly simplify the test or calibration system when replacing camera systems or vehicle glass, thereby greatly simplifying and reducing the cost of the production and maintenance process of ADAS systems.

[0023] In the following, aspects of the present solution are first categorized into the context known from the prior art literature. Prior art list: Non-patent document 1 Non-patent document 2 Non-patent document 3

[0024] Targetless camera calibration is well known in the prior art (Non-Patent Document 1). Calibration methods are subdivided into methods that estimate a (more or less rough) initial solution for the parameters and methods that improve the available solution. The former methods are algebraic in nature. Due to the high algorithm cost and poor robustness, the former methods are only suitable for practical solutions in special cases. Furthermore, such methods are less relevant for ADAS applications, since in the ADAS world, very good initial solutions are generally known from production. Practical applications for automotive applications are often limited to continuously improving the estimated calibration, with the final estimated parameters being a very good initial solution for the algorithm. A type of optimal method known as the "gold standard" (Non-Patent Document 1) is called bundle adjustment (Non-Patent Document 1 mentions the concept of "gold standard" in Section 10.4.1 in connection with bundle adjustment in the algorithm in Section 10.3).

[0025] The available prior art documents do not adequately cover cases where a windshield glass is incorporated into the optical system. Related prior art includes Patent Document 2, which describes a method for calibrating a camera-based vehicle system with a windshield glass. An imaging target having the shape of a plate with a known pattern is positioned within the field of view of a camera of the camera-based system so that the camera can capture a calibration image of the plate through the windshield glass. The camera is used to accurately capture a calibration image of the plate. The calibration image is compared with the known pattern. The windshield glass distortion aberration caused by the windshield glass is calculated using a camera model with parameters that describe the distortion aberration characteristics of the windshield glass. The internal parameters of the camera are given as known. The windshield glass distortion aberration is stored in the camera-based system.

[0026] It has already been recognized that real-world implementations present a series of non-trivial problems that have not previously been clearly answered in the prior art. The problems of the prior art lie in so-called "critical configurations" for calibration. These critical configurations can be viewed as unfavorable combinations of optical model, scene geometry, and vehicle motion, such that the bundle adjustment problem does not have a unique solution or the true solution lies in the vicinity of multiple solutions. In all of these cases, misestimations of internal parameters can occur, which can be essentially arbitrary and deviate significantly from the true parameters.

[0027] In Non-Patent Document 2, Peter Stoum describes a general taxonomy of critical configurations for pinhole camera auto-calibration. From this basic work, it is clear that the entire movement in a plane (e.g., along a curve) is critical for a pinhole camera, regardless of the scene. However, vehicles equipped with ADAS systems in practice perform near-planar movements over short periods of time (a few seconds). In summary, if a pure pinhole camera model is used to model the camera, short-term internal auto-calibration in the vehicle is difficult, if not impossible.

[0028] Since Peter Stoum (1990s), there have been only a few prior art publications addressing the uniqueness problem of automatic calibration for complex camera models. The only paper on the critical configuration of radial distortion and its corresponding functions is by C. Wu (3). Part of the failure is due to the lack of available formalisms for analyzing the critical configurations.

[0029] On the other hand, it is known from the prior art that the evaluation of the results of a bundle adjustment process is possible based on higher derivatives of the error function, and this property can be advantageously utilized in embodiments of the present invention.

[0030] For windshield glass models, no analysis of critical configurations currently exists in the prior art literature because no practically established mathematical model for such optical systems exists to date. The above techniques have not been applied to this emerging optical model to date. In the following paragraphs, a practical approach for implementing automatic calibration is presented and explained in detail.

[0031] A method for automatically calibrating a vehicle camera that maps an area of ​​the vehicle's surrounding environment through (transparent) glass while the vehicle is moving is described. a) providing a projection model of a vehicle camera, the projection model comprising as parameters a plurality of external parameters, one or more internal parameters of the vehicle camera and at least one parameter characterizing the glass; b) capturing a sequence of vehicle camera images while the vehicle is navigating a curve; c) determining the curve type based on the actual movement of the vehicle while navigating the curve, for example using a curve estimator; d) - image points or image feature points corresponding to stationary objects in the vehicle's environment, i.e., corresponding points in the image sequence; -Actual movement of the vehicle and -Determined curve type estimating the parameters (to be recalibrated) taking into account the estimation of the parameters is performed by minimizing an error function (or "loss function") that indicates the error between image points / image feature points determined from the image sequence corresponding to stationary objects in the vehicle's environment and image points of the stationary objects projected using the projection model; e) outputting at least one of the estimated parameters.

[0032] The term vehicle camera may also refer to a vehicle camera system. In the simplest case, the vehicle camera corresponds to a monocular camera. If the vehicle camera system comprises multiple monocular cameras, these may be calibrated as individual cameras. However, optionally, the reciprocity of the multiple cameras can be taken into account during calibration. If all cameras of a vehicle system are fixedly mounted on the vehicle, for example, vehicle movement is the same for all cameras.

[0033] The glass may be a (glass) pane in the optical path of the camera, for example a vehicle glass, such as a windshield glass, rear glass or side glass of a vehicle, through which the camera captures the vehicle's surrounding environment.

[0034] The parameters depend on the camera's projection model. "Providing a projection model" effectively determines the mathematical model used to model the world. Here, there are external parameters, one or more internal parameters of the camera, and at least one parameter characterizing the glass.

[0035] Extrinsic parameters define the position and orientation of the camera in the world. That is, they provide information about the relationship between world coordinates and camera coordinates. When the camera moves, the camera pose changes through translation and rotation. In the case of a camera fixedly mounted on a vehicle, the camera movement is determined by the vehicle movement. To initialize the extrinsic parameters in the case of bundle adjustment, which estimates all parameters, the estimated curve information or the odometry data used for that purpose can be used. At the end of bundle adjustment, often only the intrinsic and glass parameters are important, and the rest are discarded.

[0036] Intrinsic parameters allow mapping camera coordinates to image pixel positions, such as the focal length, the principal point (or image center), the horizontal and vertical extent of a pixel, distortion coefficients giving information about the (e.g. radial) distortion, etc.

[0037] The at least one parameter characterizing the glass may be, for example, the thickness of the glass. Also, the orientation of the glass may be a characterizing parameter. The orientation of the glass may be described by the normal vector of the glass. The orientation of the glass relative to the camera's line of sight or optical axis defines the angle of incidence on the light path. A further glass parameter may be the refractive index of the glass material.

[0038] The error function comprises as variables the spatial point (stationary object), the external camera parameters or camera pose, the internal camera parameters and the glass parameters. The coordinates of the spatial point are alternatively estimated jointly.

[0039] In one embodiment, after the parameter estimation step, the success of the calibration is checked based on a threshold on the minimized error function.

[0040] According to one embodiment, if the automatic calibration is successful, a variance analysis or verification is performed to determine which one or more parameters have been estimated sufficiently accurately, and the corresponding one or more parameters are output. Alternatively, all parameters and information on which one or more parameters were estimated sufficiently accurately can be output. In a variant of one embodiment, updating of all variable parameters is considered only if the covariance analysis does not indicate variance. If the estimate includes some variance parameters, there is a risk that other parameters may also be determined inaccurately. If this is the case (i.e., if variance is not indicated), only parameters that are estimated "sufficiently well" by a second covariance analysis are updated.

[0041] In one embodiment, information is output to a library in addition to the estimated parameters. The library comprises the assignment of the traversed curve type to each parameter that can be estimated (well or uniquely according to previous information) at that time. The library can be integrated into the driving geometry estimator. The output information indicates whether the estimation was successful (and unique) for the actually traversed curve type with respect to one or more parameters (to be recalibrated) and (if successful) with respect to which one or more parameters. In other words, this means that the actual (and successful) estimated curve maneuver is assigned to a curve type for which certain parameters can be estimated well (as a result of the variance analysis).

[0042] According to one embodiment, the estimability of the parameters to be recalibrated for the actual curve type is evaluated, in particular by checking a library. One or more parameters to be recalibrated that are expected to be estimable for a particular curve type are thus calculated. These parameters can be referred to as free parameters. One or more other parameters are fixed. An actual estimation of the parameter or parameters to be recalibrated is only performed if there is a given probability of estimation, otherwise the method is executed again (at the next time step).

[0043] In one embodiment, the significance analysis includes a covariance assessment.

[0044] According to one embodiment, the initialization of the interior and / or glass parameters is performed by carrying over the interior and / or glass parameters from a factory calibration of the vehicle camera.

[0045] According to one example embodiment, an initial solution for the extrinsic parameters is based on the actual movement of the vehicle while negotiating a curve (e.g., as defined by odometry-based movement data), thus allowing the geometry of the driving movement to be incorporated into the parameter estimation.

[0046] In one embodiment, an optical flow estimator and / or flow tracker is used to determine image points / image features from the image sequence that correspond to stationary objects in the vehicle's environment.

[0047] According to one example embodiment, the glass is a vehicle windshield glass.

[0048] A further subject of the invention relates to an apparatus for automatically calibrating a vehicle camera while the vehicle is moving, the apparatus comprising a vehicle camera, a calculation unit, a curve estimator or curve estimation unit and an output unit. The vehicle camera is configured to image an area of ​​the vehicle's surroundings through the vehicle's glass. The calculation unit is configured to provide a projection model of the vehicle camera, the projection model comprising as parameters a plurality of external parameters, at least one internal parameter of the vehicle camera and at least one parameter characterizing the glass. The vehicle camera is configured to capture a sequence of images while the vehicle is navigating a curve. The computation unit is image points of a sequence of images of stationary objects in the vehicle's environment; -Actual movement of the vehicle and - (Actual) determined curve type The parameter is estimated taking into account The parameters are estimated by minimizing an error function that indicates the error between image points determined from an image sequence corresponding to stationary objects in the vehicle's environment and the image points of the stationary objects projected using the projection model. The output unit is configured to output the estimated parameters.

[0049] The device and / or computing unit may in particular comprise a microcontroller or microprocessor, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), etc., together with software for performing the corresponding method steps.

[0050] A further subject of the invention relates to a vehicle equipped with a vehicle camera and a corresponding automatic calibration device.

[0051] A further subject of the invention relates to a computer program element which, when a data processing unit or a control device is configured to execute the program, commands a data processing unit to carry out the method according to the invention.

[0052] A further subject of the invention relates to a computer-readable storage medium on which a computer program element according to the invention is stored.

[0053] Accordingly, the invention may be implemented in digital electronic circuitry, in computer hardware, firmware, or software.

[0054] Exemplary embodiments will now be described, and certain aspects will be explained in detail with reference to the drawings. [Brief explanation of the drawings]

[0055] [Figure 1] FIG. 1 shows a schematic diagram of an apparatus, for example a control device, and the flow of auto-calibration in the control device. [Figure 2] FIG. 2 shows a schematic representation of the geometry of a vehicle driving process including cornering. [Figure 3] Figure 3 shows an ADAS camera mounted inside a vehicle behind the windshield glass. [Figure 4] FIG. 4 shows a schematic of an ADAS camera imaging the scene outside the vehicle through the windshield. [Figure 5] FIG. 5 outlines the start of the auto-calibration method. [Figure 6] FIG. 6 shows the iterative flow of the calibration method through parameter determination. [Figure 7] Figure 7 shows the details of the parameter estimation. DETAILED DESCRIPTION OF THE INVENTION

[0056] Figure 3 shows the starting situation. A vehicle camera 31 of the driver assistance system is mounted inside the vehicle 33, behind the windshield glass 32, approximately in the area (above) of the interior rearview mirror. The vehicle camera 31 faces generally forward, i.e., it captures the surrounding environment or surroundings in front of the vehicle 33.

[0057] FIG. 4 shows a highly schematic view of a vehicle 33 in motion. The vehicle camera 40 of the vehicle 33 comprises a housing 42, camera optics 43, and electrical connections 41 to a computing unit. The windshield 44 of the vehicle 33 can be modeled as a plane-parallel, transparent glass. The camera optics 43 focuses on an area outside the vehicle 33, and the windshield 44 can be considered as a substantially plane-parallel glass. Through the windshield 44, the vehicle camera 40 captures a scene of the actual vehicle environment. The camera optics 43 or camera lens may comprise, for example, a fisheye lens or a rectilinear wide-angle lens. The camera optics focuses the scene outside the vehicle onto the image sensor of the camera 40. The image sensor may be, for example, a CMOS sensor or a CCD sensor. The raw image captured by the image sensor is further processed by the computing unit.

[0058] The scene includes fixed parts 45 ("stationary objects"), such as trees, which are shown schematically, or structures, traffic signs, bridges, buildings, etc., which are not shown, and dynamic parts, such as moving pedestrians. As the vehicle 33 moves, the vehicle camera 40 (e.g., the windshield 44) ​​moves with it.

[0059] The windshield glass in front of the camera 40, or more generally a protective glass behind which another camera can be placed, is characterized in that the refractive medium is a substantially plane-parallel plate with thickness b. This includes, for example, the entire protective glass of a camera with a flat exit aperture, and the glass windows of a vehicle.

number

number

[0060] The offset of the light path 47 due to the windshield glass can be represented or approximated by a parallel shift 49 of a virtual ray 48 (dotted line) that would otherwise be incident on the windshield unimpeded. The parallel shift 49 is calculated by combining the glass shift or plate shift (in English "slab shift") σ and

number

[0061] The glass shift σ can be approximated as a constant, specifically b(ν−1) / ν, which, while fast to compute, is only sufficiently accurate for small angles of incidence.

[0062] Alternatively, the glass shift σ can be calculated as the root σ 0 of a quartic function.

[0063]

number

[0064] This operation produces highly accurate results, but is very computationally expensive.

[0065] As a further alternative, the Glass shift σ can be calculated by the fixed-point equation σ = φ(σ)(φ(σ) = b(1 − 1 / √[(ν 2 -1)(u 2 / {w-σ} 2 +1)+1]), where w and u are defined above. One or two iterations of this (convergent) fixed-point equation give very accurate results and are fast to compute.

[0066] Once the parallel shift 49 is obtained, it can be traced back to the optical path 47, for example, using bundle adjustment or stereo methods.

number

[0067] Below we will explain in detail how the vehicle camera 40 can be calibrated while the vehicle 33 is in motion.

[0068] The interaction and overview of the automatic calibration device will now be described with reference to FIG.

[0069] The vehicle camera 1 is mounted on the rear of a windshield 32 or protective glass in a moving vehicle 33 so that the vehicle camera is pointed forward (in the direction of travel). The vehicle camera 1 supplies images at regular time intervals to a control device 11. In one embodiment of the invention, the vehicle camera 1 may be integrated into the control device 11, or the vehicle camera and the control device may be integrated into a housing, which corresponds to a "smart camera".

[0070] In one embodiment of the invention, the curve sensor 2 is provided on the vehicle 33 to transmit information about the actual speed and yaw rate of the vehicle 33 to the control unit 11. In one embodiment of the invention, the curve sensor 2 may be integrated into the control unit 11. In a further embodiment of the invention, the curve sensor 2 can use the image data (and data for further calculation steps) to determine the yaw rate.

[0071] The control device 11 is provided with a storage device for two successive images 3 and 4, corresponding to times t and t-1, respectively (or vice versa, which will not be considered here). Both images are fed to an optical flow estimator 6 at time t. This results in a so-called time flow from t-1 to t, which describes the movement of image points (infinitesimal space points) of objects in the scene from time t-1 to t. The optical flow is tracked over time in a flow tracker 7, dynamic objects are filtered out, and outliers are removed. The result is a track of points that track one and the same stationary object across multiple images.

[0072] The information from the curve sensor 2 is processed by a driving geometry estimator 5 in the control device 11 .

[0073] The estimation of the driving geometry 26 will now be described with reference to Figure 2. The driving geometry estimator 5 provides an estimate of the driving geometry 26 in a plane. The driving geometry estimator 5 can also provide an estimate of which parameters can be estimated during or based on an actual curve driving 25. The actual curve driving can be characterized by a curve entry point 23, a curve radius 21, a curve angle traversed 22, and a curve exit point 24. If the curve driven corresponds to a particular curve type, the data of the driving section and the estimable parameters are forwarded to a bundle adjustment algorithm 8.

[0074] In one implementation of the present invention, the curve sensor information is extracted from the intrinsic geometry of each frame. The bundle adjustment algorithm 8 optionally takes the last computed results, or production estimates, or nominal data for a given vehicle as an initial solution for parameters characterized as estimable, and refines this using actually acquired flow tracks calculated while navigating a curve.

[0075] The bundle adjustment method 8 can be implemented according to the prior art (Non-Patent Document 1), with the difference that a projection model with glass is used as the projection model instead of a normal camera model. Such a projection model is the subject of Patent Document 2. On the other hand, the target-based method proposed in Patent Document 2 for calibrating a vehicle camera behind the windshield or together with the windshield is not compatible with dynamic autocalibration. The special projection model is based on an approximate solution of the hidden path equation of light propagation (Equation 15 in Patent Document 2) for an unknown projection in the image. The solution is carried out by a series of approximations, ultimately resulting in the solution of a quadratic equation (Equation 43 in Patent Document 2). For the described model, a configuration is presented that can be used to calculate the model parameters (see Figure 3 in Patent Document 2).

[0076] By extending this model of the '661 patent, approximations to the mathematical projection through the windshield are no longer necessary or only introduce very small errors. Further developments of the model are described below. In this way, the further developed model can be used in conjunction with bundle adjustment for dynamic calibration of windshield glass.

[0077] The windshield glass parameters are the plane of the windshield glass near the vehicle camera.

number

number

[0078] The parallel shift of the optical path due to the influence of the windshield glass is called the glass shift or plate shift (in English "slab shift") σ.

number

[0079] The glass shift σ can be approximated as a constant, which allows for very fast computation and achieves good results for small field angles. In particular, the constant can be set equal to b(ν−1) / ν, which corresponds to an exact solution for the optical path perpendicular to the windshield glass plane.

[0080] Alternatively, the glass shift σ can be calculated as the root σ 0 of a quartic function.

[0081]

number

[0082]

number

number

[0083] As a further alternative, the glass shift σ can be expressed as:

number

number

number

[0084] the i-th point at viewpoint j

number

[0085]

number

[0086] Here, the unknown glass parameter ψ and one or more internal parameters int remain constant during a long run, and θ jAt this point, the internal parameter int can be added to the glass parameter ψ to obtain a constant parameter ψ' that approximates the distance for each leg of travel. Here, p ij is not explicitly defined, but rather the measured image points are assumed to be perfectly described by the camera mapping k. That is, the corresponding extrinsic parameters θ j ,

number

[0087] If we use the information obtained from the curve geometry estimator (5), this information is j The rest of the method is based on the unknowns, i.e.

number

[0088]

number

[0089] where the subscript n is

number

[0090] We can also formulate the bundle adjustment minimization problem differently. First, we consider all image points p ij of

number

number

number

number

number

[0091]

number

[0092] However, since the image points contain measurement noise, the following probabilistic model is used:

number

[0093]

number

[0094] To obtain an approximation of the covariance matrix of this parameter vector estimator,

number

[0095]

number

[0096] Using the formula for propagation of the covariance matrix, we obtain:

[0097]

number

[0098] In the ideal case, the true parameter vector is

number

number

[0099] The results of the bundle adjustment method8 are a refined vehicle pose, a fixed scene environmental space reconstruction, refined interior parameters, and refined windshield glass parameters. The method can be made more robust by multiple modifications (Non-Patent Document 1). In one implementation of the present invention, the optimization results can be refined by averaging or filtering. Based on the characteristics of the method, this allows it to achieve accuracy equivalent to that of production prior art techniques after a small number of filtering steps.

[0100] In one embodiment of the present invention, the intrinsic parameters int of the camera 40 are jointly estimated. It is advantageous for the camera to have certain properties in order for the bundle adjustment 8 described above to be successful. In one implementation of the present invention, the camera, and therefore the camera model k, has non-vanishing radial distortion on (Non-Patent Document 2). This is typical for current vehicle cameras. In one further implementation, the camera has non-vanishing tangential distortion in addition to this characteristic.

[0101] At the end of the calculation, the result is additionally verified. If positive (successful calibration), the resulting camera parameters are stored in the storage device 10 for further processing, if necessary. If negative, the calibration may be symmetric or contain errors. The symmetric can be evaluated, for example, based on the Jacobian matrix of the vector-valued mapping / residual function and on the Hessian matrix of the error function and calculated at the found minimum point ((J T ΣJ) -1 ) (see above). In principle, we analyze the eigenvectors of the inverse Hessian of the error function. This gives us the approximate covariance matrix COV of the estimated parameters. approx It can also be understood as This information also provides information about which parameters could not be uniquely calculated. In this case, the calculation of a set of parameters or all of them can be marked with a corresponding vehicle movement as unfavorable. In this way, the decision of which parameters to estimate for which curve type (see decision step "Curve" 9 in Figure 1) can be advantageously improved during the journey, without the need to strictly pre-check the corresponding conditions. A failed calibration is recognized from a large value of the error function l. In case of a failed calibration, the result is discarded.

[0102] FIG. 5 exemplarily illustrates the start of the auto-calibration method. In a first step S10, a model is determined, provided, or set to describe the image of the vehicle's surroundings from a camera inside the vehicle behind the windshield. The model comprises both the internal camera parameters int and at least one parameter ψ characterizing the windshield glass.

[0103] In a further step S12, an error function is determined that takes into account the vehicle motion, the mapping model and the location of the features (feature points, e.g., image points) in the image sequence. j is taken into account. The external parameter θ j is basically determined by the movement of the vehicle.

[0104] In a further step S14, the parameters are initialised: the internal parameter int and one or more windshield glass parameters ψ may be carried over from the factory calibration.

[0105] In step S16, a library is provided with the assignment of traversed curve types to the parameters that can be estimated at that time. This library may have been created by previous (test) runs. The method of Patent Document 1 can be used to create a very basic library. The library may also be initially empty. The library serves the purpose of storing content by repeatedly executing the method, i.e., information about which curve types are suitable for determining which parameters.

[0106] In step S18, the iterative portion of the method, shown in FIG. 6, begins. The description of FIG. 6 begins with iterative method S18.

[0107] In step S20, a sequence of camera images captured during actual curve traversal is provided.

[0108] In step S22, the curve type of the actual curve movement is estimated, for example using a curve geometry estimator or a driving geometry estimator. If it is estimated that the vehicle is actually driving in a straight line, a new camera image is requested or provided.

[0109] In step S24, the possibility of estimating parameters to be recalibrated is evaluated for the estimated curve type by checking the library. The possibility of estimating parameters to be potentially recalibrated is evaluated based on the actual curve movement.

[0110] Step S26 concerns the question or decision as to whether an estimability has been given for at least one parameter to be recalibrated.

[0111] Step S20 starts again if the library indicates that the actual curve type is not suitable for parameter estimation, which may be the case for other curve types in addition to straight driving.

[0112] Otherwise, it is checked whether one or more of the parameters are given estimability. If estimability is given (according to the library), for example, for only one parameter, in step S28 the other parameters (parameters that should not be recalibrated) are fixed to the values ​​actually used. In that case, only the one parameter to be recalibrated remains a free parameter and may be updated below. It may be that all parameters should be recalibrated, in which case parameter fixing is omitted.

[0113] In step S30, the free or recalibrated parameters are (newly) estimated, as will be explained in more detail below with reference to Figure 7. Essentially, one or more new parameters are determined by minimizing an error function.

[0114] Step S40 involves determining whether the calibration with the newly estimated parameters was successful. If the error function outputs a value above a threshold, this means that the calibration was not successful and the newly estimated parameters are discarded. Subsequently, step S20 begins again.

[0115] If the error function outputs a value that does not exceed the threshold, the calibration is considered successful and subsequently in step S42 it is determined whether there is any ambiguity in the estimated parameters (parameters that should potentially be recalibrated, i.e., parameters that should have been recalibrated before).

[0116] If one or more estimated parameters are ambiguous, then in step S44 the library for the actual curve type is updated, as this curve type is unsuitable for recalibration of the (non-unique) estimated parameters. Since the parameters could not be uniquely estimated, they are not updated in this case. The method continues with step S20.

[0117] In contrast, if one or more estimated parameters are unique, then in step S46 the library for the actual curve type is updated as this curve type is suitable for one or more recalibrated parameters. A check can be made using covariance evaluation to see which one or more parameters could be estimated better than before or "good enough", as will be exemplarily explained in the next paragraph. In other words, the curve estimator is updated as marking one or more parameters that could be well estimated for this curve type. The one or more parameters are updated, and the method starts anew from step S20. Of course, the updated parameters or all actual parameters can be output in step S46 and can be read out at that time or at any time needed and used for image evaluation functions, detection methods, calibration mechanisms or correction mechanisms.

[0118] Example flow for "good enough" evaluation using covariance matrix: A brief summary of the specifics: For the covariance matrix, we can think of an n-dimensional error super-ellipsoid, which can be scaled larger or smaller depending on which probability masses should be taken (for example, quantiles). Place a central n-dimensional cuboid with edge lengths corresponding to the accepted accuracy tolerance on the n-dimensional error hyperellipsoid. -The extent of the ellipsoid does not exceed the boundaries of the rectangular prism, all coordinate directions (and their corresponding parameters) are suitable, all others are estimated to be too inaccurate. Using an expression:

number

[0119]

number

[0120] Summary of example embodiments and further aspects of the present invention: Drive the car and for each estimated curve type, check whether it is a new curve type or whether it is one that has already been learned. (This distinction is merely illustrative here; the distinction is henceforth obsolete in the method due to the loop, since the new curve type is one in which all parameters can still be estimated from the heuristic state). -New case: Compute parameter estimates as usual (all parameters are kept as variables). - If known: Check the previously collected information on parameters to see if there are any parameters that do not cause variance for this curve type. If yes, fix all parameters of the cost function that, according to previous experience, caused variance for this curve type, and calculate estimates for the free parameters. (Here we fix the values ​​used so far for these parameters) In case of successful calibration (e.g. RMSE (root mean square error) is below a predefined threshold), analyze the covariance matrix. This may be, for example, as follows: COV -1 (P est _approx)=(f'(P0) T Σ -1 Calculate the eigenvalues ​​and eigenvectors of f'(P0) Note: COV -1 If has eigenvalue μ, then COV has eigenvalue 1 / μ and the eigenvector remains the same Eigenvectors vs. eigenvalues ​​"near 0" / "too small" (eigenvectors vs. very large eigenvalues ​​in such a COV will result in very large uncertainties) indicate which parameters cannot be well jointly estimated due to their components being significantly unequal to 0 (i.e., indicating potential groups of parameters that cannot be well jointly estimated). Store this information about the actual curve type If this curve type occurs the next time, fix each parameter of the group that cannot be estimated together (preferably the parameter with the largest eigenvector component, and preferably the one that is located at the average of multiple groups). In that case, after estimation, the same analysis is performed again with the non-fixed, i.e., free, estimated parameters (thus "converging" to a set of estimable parameters).

[0121] As soon as (after several iterations if necessary) for the same curve type the covariance analysis produces an estimate that does not show variability for any of the estimated parameters (still without updating the previous runs that produced variability from the covariance matrix): Update all non-fixed, i.e., free, estimated parameters that are "good enough" (for the assessment of "good enough," covariance can again be used; see embodiment above).

[0122] 7 shows the details of parameter estimation S30. After the parameters that should not be recalibrated are fixed to their actual or previously used values, the parameters that should be recalibrated (step S30) are estimated as follows.

[0123] In step S32, image point correspondences in the camera image sequence are determined, for example using an optical flow estimator or flow tracker. The external parameters or their estimates can be provided by a curve estimator.

[0124] In step S34, an error function is minimized, which takes into account the error between the determined or measured image points and the image points projected according to the parametric model for a number of images of the image sequence.

[0125] As a result, in step S36, new values ​​are determined for the parameters to be recalibrated.

[0126] Subsequently, in step S40 (see FIG. 6), it is determined whether the calibration was successful. The present application relates to the invention described in the claims, but also includes the following as other aspects. 1. A method for automatically calibrating a vehicle camera (1; 31; 40) while the vehicle (33) is moving, the method comprising: imaging an area of ​​the vehicle's (33) surroundings (45; 46) through a windshield (32; 44) by the vehicle camera (1; 31; 40); a) providing a projection model of the vehicle camera (1; 31; 40), the projection model comprising parameters (θ j ,int,ψ) as multiple external parameters (θ j ), at least one internal parameter (int) of the vehicle camera (1; 31; 40) and at least one parameter (ψ) characterizing the glass (32; 44), b) capturing a sequence of images of the vehicle camera (1; 31; 40) while the vehicle (33) is navigating a curve (5); c) determining the curve type based on the actual movement of the vehicle (33) while traveling around the curve (5) by a curve estimator; d) - image points of said image sequence of stationary objects (45) in the environment of said vehicle (33); - the actual movement of said vehicle (33); and the determined curve type , the image points determined from the image sequence corresponding to stationary objects (45) in the vehicle's surroundings and the image points (p ij ) and the error between

number

number

number

Claims

1. A method for automatically calibrating a vehicle camera (1; 31; 40) while the vehicle (33) is moving, the vehicle camera (1; 31; 40) imaging an area of ​​the vehicle's (33) surroundings (45; 46) through a glass (32; 44), the method comprising the steps of: a) providing a projection model of the vehicle camera (1; 31; 40), said projection model being based on parameters (θ ｊ , int, ψ) as external parameters (θ ｊ ), at least one internal parameter (int) of the vehicle camera (1; 31; 40) and at least one parameter (ψ) characterizing the glass (32; 44), b) capturing a sequence of images of the vehicle camera (1; 31; 40) while the vehicle (33) is navigating a curve (5); c) determining the curve type based on the actual movement of the vehicle (33) during the curve maneuver (5) by a curve estimator; d) - image points of said image sequence of stationary objects (45) in the environment of said vehicle (33); - the actual movement of said vehicle (33), and - the determined curve type and the image points (p) of the stationary object (45) projected using the projection model are calculated by taking into account the image points determined from the image sequence corresponding to the stationary object (45) in the vehicle's surrounding environment. ｉｊ ) and the error between [Equation 1] estimating the parameters (θ j , int, ψ) by minimizing [Equation 2] θ m is an external parameter indicating the position and orientation of the camera, int is an internal parameter indicating the focal length, the principal point or center of the image, the horizontal and vertical size of the pixel, and the distortion coefficient that provides information about the distortion; ψ is the glass thickness, glass orientation, and refractive index of the glass material. estimating the parameters (θ j , int, ψ); e) the estimated parameters (θ ｊ , ψ, int), In the method, outputting information to a library in addition to the estimated parameters (θ j ; ψ ; int), the library comprising an assignment of the traversed curve type to each estimable parameter at that time, the output information indicating whether a unique estimation of the actual curve type was successful and for which parameter or parameters (θ j ; ψ ; int) it was successful; A method characterized by:

2. The parameter (θ ｊ 2. The method of claim 1, wherein after the step of estimating (; ψ; int), the success of the calibration is checked based on a threshold on the minimized error function.

3. If the autocalibration is successful, the covariance estimate is used to determine which parameter (θ ｊ ; ψ; int) and one or more corresponding parameters (θ ｊ 3. The method of claim 2, wherein the output is a signal from the input signal.

4. The estimation possibility of the parameters to be recalibrated for the actual curve type is evaluated by checking the library, and one or more of the parameters to be recalibrated (θ ｊ 2. The method of claim 1, wherein the estimation of (; ψ; int) is performed only in the case of a predetermined estimation probability.

5. 2. The method of claim 1, further comprising initializing the interior and / or glass parameters (ψ; int), the initialization of these parameters (ψ; int) being performed from a factory calibration of the vehicle camera (1; 31; 40).

6. Based on the actual movement of the vehicle (33) while traveling along the curve (5), the external parameters (θ ｊ 2. The method of claim 1, wherein an initial solution is found for

7. 2. The method of claim 1, further comprising determining image points or image feature points from the image sequence that correspond to stationary objects in the vehicle's environment using an optical flow estimator and / or a flow tracker.

8. 2. The method of claim 1, wherein the at least one parameter (ψ) characterizing the glass (32; 44) comprises a thickness b of the glass.

9. The at least one parameter (ψ) characterizing the glass is the glass (32; 44) [Equation 3] The method of claim 1 , comprising:

10. 1. A device for automatically calibrating a vehicle camera (1; 31; 40) while the vehicle (33) is moving, comprising: the vehicle camera (1; 31; 40) configured to image an area of ​​the vehicle's (33) surroundings (45; 46) through a glass (32; 44); a calculation unit configured to provide a projection model of the vehicle camera (1; 31; 40), the projection model being based on parameters (θ ｊ , int, ψ) as external parameters (θ ｊ ), at least one internal parameter (int) of the vehicle camera (1; 31; 40) and at least one parameter (ψ) characterizing the glass (32; 44), the vehicle camera (1; 31; 40) is configured to capture a sequence of images while the vehicle (33) is navigating a curve (5); a curve estimator configured to determine a curve type based on actual movement of the vehicle (33) while the vehicle is navigating the curve (5); The computing unit: image points of said image sequence of stationary objects (45) in the environment of said vehicle (33), - the actual movement of said vehicle (33), and - the determined curve type and the image points (p) of the stationary object (45) projected using the projection model are calculated by taking into account the image points determined from the image sequence corresponding to the stationary object (45) in the vehicle's surrounding environment. ｉｊ ) and the error between [Equation 4] is configured to estimate the parameters (θ j , int, ψ) by minimizing [Equation 5] θ k is an extrinsic parameter indicating the position and orientation of the camera; ψ is the glass thickness, glass orientation, and refractive index of the glass material; The estimated parameters (θ ｋ an output unit for outputting the input signal (input signal, int, ψ); In the device, outputting information to a library in addition to the estimated parameters (θ j ; ψ ; int), the library comprising an assignment of the traversed curve type to each estimable parameter at that time, the output information indicating whether a unique estimation of the actual curve type was successful and for which parameter or parameters (θ j ; ψ ; int) it was successful; Device.

11. A vehicle (33) comprising a vehicle camera (1; 31; 40) and a device according to claim 10.

12. A computer program element which, when the data processing unit is configured to execute the program, instructs the data processing unit to carry out the method according to any one of claims 1 to 9.

13. 13. A computer-readable storage medium having stored thereon the computer program element of claim 12.

Images