Dynamic Auto-Calibration of Behind-Glass Vehicle Camera Systems
The dynamic auto-calibration method for vehicle cameras with windshield glass uses curve drives to learn and update parameters, addressing the cost and accuracy issues of existing methods, ensuring robust ADAS system functionality.
Patent Information
- Application Number
- JP2024531109
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-12-09
- Filing Date
- 2022-11-10
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-11-10
AI Technical Summary
Existing camera calibration methods for vehicles with windshield glass are costly and inaccurate, as they require precise recalibration when glass is replaced, and do not account for the optical effects of different windshield types, leading to potential system failures.
A dynamic auto-calibration method that estimates camera and windshield glass parameters during vehicle motion using a bundle adjustment algorithm, leveraging curve drives to learn correlations and update parameters in a heuristic database, simplifying and reducing the need for high-precision targets.
Achieves calibration accuracy comparable to production standards with fewer measurements, reducing costs and simplifying maintenance processes for ADAS systems by continuously improving calibration during vehicle use.
Smart Images

Figure 2024542542000001_ABST
Abstract
Description
[Technical field]
[0001] In particular, the present invention relates to a method and apparatus for automatically calibrating a vehicle camera system that can be used in a vehicle as a sensor system for driver assistance systems and autonomous driving, capturing the surrounding environment through glass. [Background technology]
[0002] Patent Document 1 describes a method and apparatus for automatically calibrating a monocular vehicle camera. a) capturing a sequence of images from a vehicle camera, the vehicle camera mapping an area of a surrounding environment in front of the vehicle; b) detecting a curve run of the vehicle suitable for automatic calibration if the curve radius described by the vehicle is less than or equal to a defined maximum radius or the curve angle passed is greater than or equal to a defined minimum angle; c) performing an automatic calibration when at least one curve suitable for automatic calibration is detected, d) the automatic calibration is performed taking into account the movement of stationary objects in the vehicle's surrounding environment when driving in a curve suitable for the automatic calibration.
[0003] Camera calibration is an essential element for Advanced Driver Assistance Systems (ADAS) or Automated Driving (AD) systems to capture the surrounding environment using a camera system mounted in or on the vehicle. The camera system tracks the movement of the vehicle or similarly passes through it. For camera calibration, estimation methods are used to calculate the parameters of the formal relationship (projection) between the 3D space points and the corresponding image points of the physical camera system. The calculated parameters are then stored in the ADAS system for further use. The projection policy includes a description of the light propagation path in the optical system of the camera (intrinsic parameters) and the position and orientation with respect to a fixed reference coordinate system in the vehicle (extrinsic parameters).
[0004] If the parameters of the projection policy are calculated accurately, it is possible to capture and measure the vehicle's surroundings while driving using Structure from Motion (SfM) or multi-viewpoint methods. If the parameters of the projection policy deviate slightly from the actual projection policy, subsequent method processes using distances (e.g., Adaptive Cruise Control (ACC), Emergency Braking Assist (EBA) or Automatic Emergency Braking (AEB)) or angles (e.g., Lane Keeping Assist (LKA) or Head Lamp Assist (HLA)) calculated from the image data of the camera for the mapped objects may thus produce inaccurate results. If the parameters of the projection policy deviate significantly from the actual projection policy, the ADAS / AD system may be limited or unavailable (short-term or permanent failure).
[0005] Typically, the ADAS camera is mounted in the vehicle behind the windshield glass or protective glass of the vehicle. The (windshield) glass constitutes an additional optical system and significantly changes the projection policy of the camera as described above compared to a configuration in which there is no windshield glass between the scene and the camera. To ensure correct functionality of the subsequent method processes, it is necessary to take the windshield into account during the camera calibration.
[0006] The existing process of camera calibration with ADAS camera systems involves multiple steps, including camera calibration at the end of the camera production line, camera calibration at the end of the vehicle production line, and automatic calibration of the camera while driving. Calibration in production (whether in camera production or vehicle production) is the basis for robust surroundings capture in the vehicle, as it serves as the first solution or basis for the further two steps. [Prior art documents] [Patent documents]
[0007] [Patent Document 1] DE 102018204451 [Patent Document 2] European Patent No. 3293701 [Non-patent literature]
[0008] [Non-Patent Document 1] Hartley, Zissermann, Multiple View Geometry in Computer Vision, 2000, Cambridge University Press, ISBN: 0521623049 (1st edition). [Non-Patent Document 2] Peter Sturm: Critical motion sequences for monocular self-calibration and uncalibrated euclidean reconstruction, CVPR, 1997, p. 1100-1105. [Non-Patent Document 3] C. Wu: Critical configurations for radial distortion self-calibration. In CVPR, 2014. Summary of the Invention [Problem to be solved by the invention]
[0009] It is basically conceivable to calibrate the camera together with the windshield glass at the end of the camera production line. Due to the variety of different windshield glass types and due to the individual mounting positions of each camera behind each windshield glass, the corresponding projection policies are also individual. Therefore, calibration during camera production is impossible or not meaningful from a practical point of view.
[0010] It is also basically conceivable to calibrate the camera together with a special windshield glass at the final stage of vehicle production. This approach is also the most frequently used prior art. It is self-evident that this option requires a new calibration of the optics when the glass is replaced. Due to the precision requirements of the camera calibration, calibration according to the above approach is very costly and therefore expensive. For this purpose, targets produced with high precision are used. Also, the vehicle is adjusted with high precision for the calibration using special, costly and expensive equipment. Generally, according to the above approach, only external parameters can be re-adjusted.
[0011] The influence of the windshield glass on the optical path of the light through the camera is often not taken into account. Even if the calibration according to the prior art solves part of this problem, it is out of the scope of the configuration in which the optical system consisting of the ADAS camera and the windshield glass is changed during driving. In this way, there is always a risk that an ADAS system with misestimated parameters and therefore with limited functionality is used by an end user, i.e. a driver of a vehicle equipped with a driving assistance system. The object of the present invention is to provide a solution for simplifying and improving the process of camera calibration. [Means for solving the problem]
[0012] This problem is solved by the subject matter having the features of the independent claims. Preferred embodiments are the subject matter of the dependent claims.
[0013] One aspect of the solution is that the entire glass and camera based optical system is not calibrated at the end of the vehicle production line, but is dynamically calibrated on the vehicle while it is in motion and continuously improved. Dynamic auto-calibration can also be referred to as online calibration.
[0014] One aspect is that the process of calibrating ADAS camera systems mounted behind windshield glass or protective glass during camera production can be greatly simplified.
[0015] One aspect of the present invention relates to an apparatus and method for estimating a set of parameters of an ADAS camera system while a vehicle is traveling. One advantage of the present invention is that highly accurate manufactured targets are not required for calibration.
[0016] The method uses a fixed part of the (unknown) traffic scene located in front of the vehicle to calibrate the effect of the windshield glass. To achieve this, a special optical model of the windshield glass + camera combination is required.
[0017] It is known from experience in the prior art that parameter estimation of such models is not always possible. It is also known from the prior art that certain parameters can be calibrated during a certain advantageous run and a camera calibration can be performed. An advantage here is for example non-vanishing (e.g. radial) distortion aberration. The estimation that the parameters can be estimated during the run trajectory can be learned during the run, based on previous experience, in one form of the invention. These two properties allow a calibration quality comparable to industry standards to be achieved after a few few measurements.
[0018] One aspect of the present invention relates to an apparatus (eg, a controller) that computes a calibration of a vehicle-mounted camera system. One aspect of the invention relates to estimating the driving motion or geometry of a vehicle while driving around a curve (ie, curve motion estimation) and taking this into account when estimating the parameters of a camera system. A further aspect relates to considering at least one parameter that characterizes the glass.
[0019] One aspect of the solution relates to the realization or assumption that it is possible to automatically calibrate the entire windshield and camera-based optical model during special curve driving using a bundle adjustment method. The method can be configured to learn correlations between curve types and calculable parameters. This method represents a significant innovative step compared to the current state of the art.
[0020] One aspect of the invention relates to the construction of a library or some kind of rule-of-thumb data bank where information is stored that parameters for that type of run or that type of curve can be updated or recalibrated. The use of a library allows more curve runs to be used for calibration compared to the method of US Pat. No. 5,999,433 and allows for a method to specifically find out which parameters can be recalibrated and when.
[0021] The apparatus and method are configured to provide the necessary data to initialize the bundle adjustment algorithm with a good initial solution. From the results of many curve runs, filtering can be used to obtain significant improvements in accuracy that can rival that of a production calibration.
[0022] The solution provides the advantage that the vehicle manufacturer greatly simplifies the test system in the production of the ADAS system, and the repair shop greatly simplifies the test or calibration system when replacing the camera system or the vehicle glass, which greatly simplifies and reduces the cost of the production and maintenance process of the ADAS system.
[0023] In the following, aspects of the present solution are first categorized into the context as known from the prior art literature. Prior art list: Non-patent literature 1 Non-patent literature 2 Non-patent literature 3
[0024] Camera calibration without targets is well known in the prior art (Non-Patent Document 1). Calibration methods are sub-classified into methods that estimate a (more or less rough) initial solution for parameters and methods that improve the available solution. The former methods are algebraic in nature. Due to the high cost of algorithms and poor robustness, the former methods are suitable for practical solutions only in special cases. Also, such methods are less relevant for ADAS applications, since in the ADAS world, a very good initial solution is generally known from production. Practical applications for automotive applications are often limited to successively improving the estimated calibration, with the last estimated parameters being a very good initial solution for the algorithm. A class of optimal methods known as the "gold standard" (Non-Patent Document 1) are called bundle adjustment methods (Non-Patent Document 1 mentions the concept of "gold standard" in section 10.4.1 in conjunction with bundle adjustment in algorithms in section 10.3).
[0025] The available prior art documents do not adequately cover the case where a windshield glass is integrated into the optical system. Related prior art includes Patent Document 2. Patent Document 2 describes a method for calibrating a camera-based vehicle system with a windshield glass. An imaging target having the shape of a plate with a known pattern is placed in the field of view of the camera of the camera-based system so that the camera can take a calibration image of the plate through the windshield glass. The camera is used to take a precise calibration image of the plate. The calibration image is compared with the known pattern. The windshield glass distortion aberration caused by the windshield glass is calculated using a camera model with parameters that represent the distortion aberration characteristics of the windshield glass. The internal parameters of the camera are given as known. The windshield glass distortion aberration is stored in the camera-based system.
[0026] It has already been recognized that in real implementations a series of non-trivial problems arise for which no clear answer has been given so far in the prior art literature. The problem with the prior art lies in so-called "critical configurations" for the calibration. These critical configurations can be seen as unfavourable conglomerations of optical models, scene geometries and vehicle movements, such that the bundle adjustment problem does not have a unique solution or the true solution lies in the vicinity of multiple solutions. In all these cases, misestimations of internal parameters can occur that are essentially arbitrarily located far from the true parameters.
[0027] In Non-Patent Document 2, Peter Sturm describes a general taxonomy of critical configurations for pinhole camera auto-calibration. From this basic work, it is clear that the entire movement in a plane (e.g. along a curved road) is critical for a pinhole camera, regardless of the scene. However, vehicles equipped with ADAS systems in practice perform nearly planar movements for short periods (a few seconds). In summary, if a pure pinhole camera model is used to model the camera, short-term internal auto-calibration in the vehicle is difficult, if not impossible.
[0028] Since Peter Stoum (1990s), there are only a few prior art papers addressing the uniqueness problem of autocalibration for complex camera models. The only paper on the critical configuration of radial distortion aberration and its corresponding functions is by C. Wu (3). Part of the failure is that no formalism is available to analyze the critical configuration.
[0029] On the other hand, it is known from the prior art that the evaluation of the result of the bundle adjustment process is possible based on higher derivatives of the error function, this property can be advantageously exploited in embodiments of the present invention.
[0030] Regarding the windshield glass model, the analysis of the critical configurations does not currently exist in the prior art literature because there is no practically established mathematical model for such an optical system to date. The above techniques have not been applied to this emerging optical model to date. In the following paragraphs, a practical technique for implementing automatic calibration is presented and explained in detail.
[0031] A method for automatically calibrating a vehicle camera that maps an area of the vehicle's surrounding environment through (transparent) glass while the vehicle is moving is provided, comprising: a) providing a projection model of a vehicle camera, the projection model comprising as parameters a number of external parameters, one or more internal parameters of the vehicle camera and at least one parameter characterizing the glass; b) capturing a sequence of vehicle camera images while the vehicle is traveling through a curve; c) determining a curve type based on the actual movement of the vehicle through the curve, for example using a curve estimator; d) - image points or image feature points corresponding to stationary objects in the vehicle's environment, i.e. corresponding points in the image sequence; -Actual movement of the vehicle and -Determined curve type estimating the parameters (to be recalibrated) taking into account the estimation of the parameters is performed by minimizing an error function (or "loss function") that indicates the error between image points / image feature points determined from the image sequence corresponding to stationary objects in the vehicle's environment and image points of the stationary objects projected using the projection model; e) outputting at least one of the estimated parameters.
[0032] A vehicle camera may also denote a vehicle camera system. In the simplest case, the vehicle camera corresponds to a monocular camera. If the vehicle camera system comprises several monocular cameras, these may be calibrated as individual cameras. However, optionally, the reciprocity of the several cameras can be taken into account during the calibration. If all cameras of a vehicle system are fixedly mounted on the vehicle, for example, vehicle movements are identical for all cameras.
[0033] The glass may be a (glass) pane in the optical path of the camera, for example a vehicle glass, for example a windshield glass, a rear glass or a side glass of a vehicle, through which the camera captures the vehicle's surrounding environment.
[0034] The parameters depend on the projection model of the camera. "Filling a projection model" means in effect deciding on a mathematical model to use to model the world. Here, there are external parameters, one or more internal parameters of the camera, and at least one parameter that characterizes the glass.
[0035] The extrinsic parameters define the position and orientation of the camera in the world, i.e. they give information about the relationship between world coordinates and camera coordinates. When moving the camera, the camera pose changes by translation and rotation. In case of a camera fixedly mounted on the vehicle, the camera movement is specified by the vehicle movement. To initialize the extrinsic parameters in case of bundle adjustment to estimate all parameters, the estimated curve information or the odometry data used for it can be used. At the end of the bundle adjustment, often only the intrinsic and glass parameters are important, the rest are discarded.
[0036] Intrinsic parameters allow to map camera coordinates to image pixel positions, e.g. focal length, principal point (or image center), horizontal and vertical extent of a pixel, distortion coefficients giving information about the (e.g. radial) distortion aberration, etc.
[0037] The at least one parameter characterizing the glass may for example be the thickness of the glass. Also, the orientation of the glass may be a characterizing parameter. The orientation of the glass may be described by a normal vector of the glass. The orientation of the glass with respect to the line of sight or optical axis of the camera defines the angle of incidence on the light path. A further glass parameter may be the refractive index of the glass material.
[0038] The error function comprises as variables the spatial points (stationary objects), the external camera parameters or camera pose, the internal camera parameters and the glass parameters. The coordinates of the spatial points are alternatively estimated jointly.
[0039] In one embodiment, after the parameter estimation step, the success of the calibration is checked based on a threshold on the minimized error function.
[0040] According to one embodiment, if the automatic calibration is successful, a variance analysis or verification is performed to determine which one or more parameters have been estimated sufficiently accurately, and the corresponding one or more parameters are output. It is also possible to output all parameters and information on which one or more parameters could be estimated sufficiently accurately. In a variant of one embodiment, the update of all variable parameters is considered only if the covariance analysis does not show variance. If the estimation includes some variance parameters, there is a risk that other parameters are also determined inaccurately. If this is the case (i.e. if variance is not shown), only those parameters that are estimated "sufficiently well" by a second covariance analysis are updated.
[0041] In one embodiment, information is output to the library in addition to the estimated parameters. The library comprises the assignment of the traversed curve type to the respective (successfully or uniquely according to previous information) estimable parameters. The library may be integrated into the driving geometry estimator. The output information indicates whether the estimation is successful (and unique) for the actually traversed curve type with respect to one or more (parameters to be recalibrated) parameters and (if successful) with respect to which one or more parameters. In other words, it means that the actual (and successful) estimated curve driving is assigned to a curve type for which certain parameters can be successfully estimated (as a result of the variance analysis).
[0042] According to an example embodiment, the estimation of the recalibrated parameters for the actual curve type is performed, in particular by checking a library. Thus, one or more recalibrated parameters are calculated, which are expected to be estimable for a particular curve type. These parameters can be called free parameters. One or more other parameters are fixed. An actual estimation of the parameter(s) to be recalibrated is only performed if there is a given estimation probability, otherwise the method is executed again (at the next time step).
[0043] In one embodiment, the significance analysis includes covariance assessment.
[0044] According to one embodiment, the initialization of the interior and / or glass parameters is performed by carrying over the interior and / or glass parameters from a factory calibration of the vehicle camera.
[0045] According to one example embodiment, an initial solution for the extrinsic parameters is based on the actual movement of the vehicle while driving around a curve (e.g., as defined by odometry-based movement data), in this way the geometry of the driving movement can be incorporated into the parameter estimation.
[0046] In one embodiment, an optical flow estimator and / or flow tracker is used to determine image points / image feature points from the image sequence that correspond to stationary objects in the vehicle's environment.
[0047] According to one example embodiment, the glass is a vehicle windshield glass.
[0048] A further subject of the invention relates to an apparatus for automatic calibration of a vehicle camera while the vehicle is moving, the apparatus comprising a vehicle camera, a calculation unit, a curve estimator or curve estimation unit and an output unit. The vehicle camera is configured to image an area of the vehicle's surrounding environment through the (vehicle's) glass. The calculation unit is configured to provide a projection model of the vehicle camera, the projection model comprising as parameters a plurality of external parameters, at least one internal parameter of the vehicle camera and at least one parameter characterizing the glass. The vehicle camera is configured to capture a sequence of images while the vehicle is navigating a curve. The computation unit is image points of a sequence of images of stationary objects in the vehicle's environment, -Actual movement of the vehicle and - (Actual) determined curve type The parameter estimation is performed by taking into account The parameters are estimated by minimizing an error function that indicates the error between image points determined from an image sequence corresponding to stationary objects in the vehicle's environment and the image points of the stationary objects projected using a projection model. The output unit is configured to output the estimated parameters.
[0049] The device and / or computing unit may in particular comprise a microcontroller or microprocessor, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), etc., together with software for performing the corresponding method steps.
[0050] A further subject of the invention relates to a vehicle equipped with a vehicle camera and a corresponding automatic calibration device.
[0051] A further subject of the invention relates to a computer program element which commands a data processing unit or a control unit to carry out the method according to the invention when the data processing unit or the control unit is configured to execute the program.
[0052] A further subject of the invention relates to a computer-readable storage medium on which a computer program element according to the invention is stored.
[0053] Accordingly, the invention may be implemented in digital electronic circuitry, in computer hardware, firmware, or software.
[0054] Exemplary embodiments will now be described, and certain aspects will be explained in detail with reference to the drawings. [Brief description of the drawings]
[0055] [Figure 1] FIG. 1 shows a schematic diagram of an apparatus, for example a controller, and a flow of auto-calibration in the controller. [Diagram 2] FIG. 2 shows a schematic representation of the geometry of a vehicle driving process including cornering. [Diagram 3] FIG. 3 shows an ADAS camera mounted inside a vehicle behind the windshield glass. [Figure 4] FIG. 4 shows a schematic of an ADAS camera imaging the scene outside the vehicle through the windshield glass. [Diagram 5] FIG. 5 outlines the initiation of the auto-calibration method. [Figure 6] FIG. 6 shows the iterative flow of the calibration method through parameter determination. [Figure 7] FIG. 7 shows the details of the parameter estimation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0056] Figure 3 shows the starting situation. A vehicle camera 31 of the driving assistance system is mounted inside the vehicle 33, behind the windshield glass 32, approximately in the area (above) the interior rearview mirror. The vehicle camera 31 faces generally forward, i.e. it captures the surrounding environment or surroundings in front of the vehicle 33.
[0057] FIG. 4 shows very diagrammatically the situation when the vehicle 33 is moving. The (vehicle) camera 40 of the vehicle 33 comprises a housing 42, a camera optics 43 and an electrical connection 41 to the computation unit. The windshield glass 44 of the vehicle 33 can be modelled as a plane-parallel transparent glass. The camera optics 43 focuses on the area outside the vehicle 33 and the windshield glass 44 can be considered as a substantially plane-parallel glass. Through the windshield glass 44 the vehicle camera 40 captures the scene of the actual vehicle surroundings. The camera optics 43 or the camera lens may comprise, for example, a fish-eye lens or a rectilinear wide-angle lens. The camera optics focuses on the scene outside the vehicle on the image sensor of the camera 40. The image sensor may be, for example, a CMOS sensor or a CCD sensor. The raw image captured by the image sensor is further processed by the computation unit.
[0058] The scene includes fixed parts 45 ("stationary objects"), such as trees, which are shown diagrammatically, or structures, traffic signs, bridges, buildings, etc., which are not shown, and dynamic parts, such as moving pedestrians. As the vehicle 33 moves, the vehicle camera 40 (and, for example, the windshield glass 44) moves with it.
[0059] The windshield glass in front of the camera 40, or in general a protective glass behind which another camera can be placed, is characterized in that the refractive medium is a substantially plane-parallel plate having a thickness b. This includes, for example, the entire protective glass of a camera with a flat exit aperture, and also the glass windows of a vehicle.
number
number
[0060] The offset of the light path 47 due to the windshield glass can be represented or approximated by a parallel shift 49 of a virtual ray 48 (dotted line) that enters the windshield unimpeded. The parallel shift 49 can be expressed as a glass shift or plate shift (in English "slab shift") σ and
number
[0061] The glass shift σ can be approximated as a constant, specifically, b(ν−1) / ν. While this approximation is fast to compute, it is only accurate enough for small angles of incidence.
[0062] Alternatively, the glass shift σ can be calculated as the root of a fourth order function σ 0 .
[0063]
number
[0064] This operation produces highly accurate results, but is very computationally expensive.
[0065] As a further alternative, the Glass shift σ can be expressed as a function of the fixed-point equation σ = φ(σ)(φ(σ) = b(1-1 / √[(ν 2 -1)(u 2 / {w-σ} 2 +1)+1]), w and u defined above. One or two iterations of this (convergent) fixed-point equation give very accurate results and are fast to compute.
[0066] Once the parallel shift 49 is obtained, it can be traced back to the optical path 47, for example using bundle adjustment or stereo techniques.
number
[0067] Below, it will be explained in detail how the vehicle camera 40 can be calibrated while the vehicle 33 is moving.
[0068] The interaction and overview of the automatic calibration device will now be described with reference to FIG.
[0069] The vehicle camera 1 is mounted behind a windshield glass 32 or protective glass in a moving vehicle 33 so that the vehicle camera is pointed forward (in the direction of travel). The vehicle camera 1 supplies images at uniform time intervals to the control device 11. In one embodiment of the invention, the vehicle camera 1 may be integrated into the control device 11 or the vehicle camera and the control device may be integrated into a housing, which corresponds to a "smart camera".
[0070] In one embodiment of the invention, the curve sensor 2 is provided on the vehicle 33 to transmit information about the actual speed and yaw rate of the vehicle 33 to the control device 11. In one embodiment of the invention, the curve sensor 2 may be integrated into the control device 11. In a further embodiment of the invention, the curve sensor 2 can use the image data (and data of further calculation steps) to determine the yaw rate.
[0071] The control device 11 is provided with a storage device for two mutually successive images 3 and 4, which correspond respectively to times t and t-1 (or vice versa, which will not be considered below). Both images are fed to an optical flow estimator 6 at time t. This results in a so-called time flow from t-1 to t, which describes the movement of image points (infinitesimal space points) of objects of the scene from time t-1 to t. The optical flow is tracked over time in a flow tracker 7, dynamic objects are filtered and outliers are removed. The result is a track of points that track one and the same stationary object across several images.
[0072] The information of the curve sensor 2 is processed in the control device 11 by a driving geometry estimator 5 .
[0073] The estimation of the driving geometry 26 is explained with reference to Fig. 2. The driving geometry estimator 5 provides an estimate of the driving geometry 26 in a plane. The driving geometry estimator 5 can also provide an estimate of which parameters can be estimated during or based on an actual curve driving 25. The actual curve driving can be characterized by a curve entry point 23, a curve radius 21, a curve angle 22 traversed and a curve exit point 24. If the curve driven corresponds to a special curve type, the data of the driving section and the estimable parameters are forwarded to a bundle adjustment algorithm 8.
[0074] In one implementation of the invention, the curve sensor information is extracted from the intrinsic geometry of each frame. The bundle adjustment algorithm 8 selectively takes the last computed results, or estimates from production, or nominal data for a given vehicle as an initial solution for parameters characterized as estimable, and refines this with the actually acquired flow tracks calculated while driving the curve.
[0075] The bundle adjustment method 8 can be carried out according to the prior art (Non-Patent Document 1) with the difference that a projection model with glass is used as the projection model instead of a normal camera model. Such a projection model is the subject of Patent Document 2. On the other hand, the target-based method proposed in Patent Document 2 for calibrating a vehicle camera behind the windshield or together with the windshield is not compatible with dynamic autocalibration. The special projection model is based on an approximate solution of the hidden path equation of light propagation (equation 15 in Patent Document 2) for an unknown projection in the image. The solution is carried out by a series of approximations, which finally results in the solution of a quadratic equation (equation 43 in Patent Document 2). For the described model, a configuration is presented (see FIG. 3 in Patent Document 2) that can be used to calculate the parameters of the model.
[0076] By developing this model of the '693 patent, it is possible that approximations are no longer necessary or only introduce very small errors in the mathematical projection through the windshield. Further developments of the model are described below. In this way, the further developed model can be used in conjunction with bundle adjustment for dynamic calibration of the windshield glass.
[0077] The windshield glass parameters are the windshield glass plane near the vehicle camera.
number
number
[0078] The parallel shift of the optical path due to the effect of windshield glass is called the glass shift or plate shift (in English "slab shift") σ.
number
[0079] The glass shift σ can be approximated as a constant, which allows very fast computation and achieves good results for small field angles. In particular, the constant can be set equal to b(ν−1) / ν, which corresponds to an exact solution for the optical path perpendicular to the windshield glass plane.
[0080] Alternatively, the glass shift σ can be calculated as the root of a fourth order function σ 0 .
[0081]
number
[0082]
number
number
[0083] As a further alternative, the glass shift σ can be expressed as:
number
number
number
[0084] The i-th
number
[0085]
number
[0086] Here, the unknown glass parameter ψ and one or more internal parameters int remain constant during a long run and change θ from one viewpoint to the next. jIt is assumed that only the difference is . At this point, the internal parameter int can be added to the glass parameter ψ to obtain a constant parameter ψ' that is approximated for each travel section. Here, p ij is not explicitly defined, but rather it is assumed that the measured image points can be perfectly described by the camera mapping k. That is, the corresponding extrinsic parameters θ j ,
number
[0087] When using the information obtained from the curve geometry estimator (5), this information is j This gives a very good initial solution to the problem. The rest of the method relies on all the unknowns, i.e.
number
[0088]
number
[0089] Here, the subscript n is
number
[0090] We can also formulate the bundle adjustment minimization problem differently. First, for all image points p ij of
number
number
number
number
number
[0091]
number
[0092] However, since the image points contain measurement noise, the following probabilistic model is used:
number
[0093]
number
[0094] To obtain an approximation of the covariance matrix of this parameter vector estimator,
number
[0095]
number
[0096] Using the formula for propagation of the covariance matrix, we obtain:
[0097]
number
[0098] In the ideal case, the true parameter vector is
number
number
[0099] The results of the bundle adjustment8 computation are a refined vehicle pose, a reconstruction of the fixed scene ambient space, refined interior parameters, and refined windshield glass parameters. The method can be made more robust by multiple modifications (Non-Patent Document 1). The optimization results can be refined by averaging or filtering in one implementation of the present invention. Based on the characteristics of the method, it can achieve accuracy equivalent to the production prior art after a small number of filtering steps.
[0100] In one embodiment of the invention, the intrinsic parameters int of the camera 40 are jointly estimated. It is advantageous for the camera to have certain properties in order for the bundle adjustment 8 described above to be successful. In one implementation of the invention, the camera, and therefore the camera model k, has a non-vanishing radial distortion aberration on (Non-Patent Document 2). This is typical for current vehicle cameras. In one further implementation, the camera has, in addition to this characteristic, a non-vanishing tangential distortion aberration.
[0101] At the end of the calculation, the result is additionally verified. The resulting camera parameters are, in the positive case (successful calibration), stored in the storage device 10 for further processing, if necessary. In the negative case, the calibration may be ambiguous or contain errors. The ambiguity can be evaluated, for example, based on the Jacobian matrix of the vector-valued mapping / residual function and on the Hessian matrix of the error function and calculated at the determined minimum point ((J T ΣJ) -1 ) (see above). In principle, we analyze the eigenvectors of the inverse Hessian of the error function. This gives us the approximate covariance matrix COV of the estimated parameters. approx It can also be understood as. This information also gives information about which parameters could not be calculated uniquely. In this case, for the calculation of the set of parameters or for all calculations of the set of parameters, the corresponding vehicle movement can be marked as unfavorable. In this way, the decision of which parameters should be estimated for which curve type (see decision step "curve" 9 in Fig. 1) can be advantageously improved during the journey without the corresponding conditions having to be checked rigorously in advance. An unsuccessful calibration is recognized from a large value of the error function l. In case of an unsuccessful calibration, the result is discarded.
[0102] FIG. 5 exemplarily illustrates the start of the auto-calibration method. In a first step S10, a model is determined or provided or set to describe the image of the vehicle environment by a camera inside the vehicle behind the windshield. The model comprises both the internal camera parameters int and at least one parameter ψ characterizing the windshield glass.
[0103] In a further step S12, an error function is determined that takes into account the vehicle movement, the mapping model and the locations of the features (feature points, e.g. image points) in the image sequence. j The external parameters θ j is essentially determined by the movement of the vehicle.
[0104] In a further step S14, the parameters are initialised: the internal parameter int and the windshield glass parameter(s) ψ may be taken over from a factory calibration.
[0105] In step S16, a library is provided with the assignment of traversed curve types to the currently estimable parameters. This library may have been created by previous (test) runs. The method of US 2010 / 0139999 can be used to create a very basic library. The library may also be initially empty. The library serves the purpose of being able to store content by iteratively executing the method, i.e. information about which curve types are suitable for determining which parameters.
[0106] In step S18, the iterative portion of the method, shown in FIG. 6, begins. The description of FIG. 6 begins with iterative method S18.
[0107] In step S20, a sequence of camera images captured during an actual curve traversal is provided.
[0108] In step S22, the curve type of the actual curve movement is estimated, for example using a curve geometry estimator or a driving geometry estimator. If it is estimated that the vehicle is actually driving in a straight line, new camera images are requested or provided.
[0109] In step S24, the possibility of estimating parameters to be recalibrated is evaluated for the estimated curve type by checking the library. The possibility of estimating parameters to be potentially recalibrated is evaluated based on the actual curve movement.
[0110] Step S26 concerns the question or decision as to whether an estimability has been given for at least one parameter to be recalibrated.
[0111] Step S20 starts again if the library shows that the actual curve type is not suitable for parameter estimation, which may be the case for other curve types in addition to straight line driving.
[0112] Otherwise, it is checked whether estimability was given for one or more of the parameters. If estimability was given (according to the library), for example, for only one parameter, in step S28 the other parameters (parameters that should not be recalibrated) are fixed to the actually used values. In that case, only the one parameter to be recalibrated is still a free parameter and may be updated below. It may be that all parameters should be recalibrated, in which case the parameter fixing is omitted.
[0113] In step S30, the free or to be recalibrated parameters are (newly) estimated, as will be explained in more detail below with reference to figure 7. Essentially, the new parameter or parameters are determined by minimizing an error function.
[0114] Step S40 concerns the determination of whether the calibration with the newly estimated parameters was successful or not. If the error function outputs a value above a threshold, this means that the calibration was not successful and the newly estimated parameters are discarded. Then step S20 starts again.
[0115] If the error function outputs a value that does not exceed the threshold, the calibration is considered successful and subsequently in step S42 it is determined whether there is any ambiguity in the estimated parameters (parameters that should potentially be recalibrated, i.e. parameters that should have been recalibrated before).
[0116] If one or more estimated parameters are ambiguous, then in step S44 the library for the actual curve type is updated as this curve type is unsuitable for recalibration of the (non-unique) estimated parameters. Since the parameters could not be uniquely estimated, they are not updated in this case. The method continues with step S20.
[0117] In contrast, if the estimated parameter(s) are unique, then in step S46 the library for the actual curve type is updated as this curve type is suitable for the recalibrated parameter(s). A check can be made using covariance evaluation as to which parameter(s) could be estimated better than before or "good enough". This is exemplarily explained in the next paragraph. In other words, the curve estimator is updated as marking the parameter(s) that could be well estimated for this curve type. The parameter(s) are updated and the method starts anew from step S20. Of course, the updated parameters or all the actual parameters can be output in step S46 and can be read out at that time or at each required time and used for the image evaluation function, detection method, calibration mechanism or correction mechanism.
[0118] Example flow for "good enough" evaluation using covariance matrix: A brief summary of the specifics: For the covariance matrix, one can think of an n-dimensional error super-ellipsoid, which is scaled larger or smaller depending on which probability masses should be taken (for this, one can think of e.g. quantiles). - Place a central n-dimensional cuboid with edge lengths corresponding to the allowed accuracy tolerance on the n-dimensional error hyperellipsoid. -The extent of the ellipsoid does not exceed the boundaries of the rectangular prism, all coordinate directions (and their corresponding parameters) are suitable, all others are estimated to be too inaccurate. Using the formula:
number
[0119]
number
[0120] Summary of exemplary embodiments and further aspects of the present invention: Drive the car and for each estimated curve type check whether it is a new curve type or whether it is one that has already been learned (This distinction is merely illustrative here and is henceforth obsolete in the method due to the loop, since the new curve type is one in which all parameters can still be estimated from the heuristic state). -New case: Compute parameter estimates as usual (all parameters are kept as variables). - If known: Check previously collected information on parameters to see if there are any parameters that do not cause variance for this curve type. If YES, fix all parameters of the cost function that, according to previous rules of thumb, caused variance for this curve type and compute estimates for the free parameters. (Here we fix the values used so far for these parameters) In case of successful calibration (e.g., the root mean square error (RMSE) is below a predefined threshold), analyze the covariance matrix. This may be, for example, as follows: COV -1 (P est _approx)=(f'(P0) T Σ -1 Calculate the eigenvalues and eigenvectors of f'(P0) Note: COV -1 If has eigenvalue μ, then COV has eigenvalue 1 / μ and the eigenvector remains the same Eigenvectors vs. eigenvalues "near 0" / "too small" (eigenvectors in such COVs vs. very large eigenvalues lead to very large uncertainties) indicate which parameters cannot be well jointly estimated due to their components being significantly unequal to 0 (i.e., they indicate potential groups of parameters that cannot be well jointly estimated). Store this information about the actual curve type In the next round, if this curve type occurs, fix each parameter of the groups that cannot be estimated together (preferably the parameters with the largest eigenvector components, and preferably those that lie at the average of multiple groups). In that case, after estimation, the same analysis is performed again with the non-fixed, i.e., free, estimated parameters (thus "converging" to a set of estimable parameters).
[0121] As soon as (after several iterations if necessary) we have estimated for the same curve type that the covariance analysis does not show variability for any of the estimated parameters (still without updating in previous runs where variability appeared from the covariance matrix): Update all non-fixed, i.e., free, estimated parameters that are "good enough" (for the assessment of "good enough" one can again use covariance, see embodiment above)
[0122] Parameter estimation S30 is shown in detail in Figure 7. After the parameters that should not be recalibrated are fixed to their actual or previously used values, the parameters that should be recalibrated (step S30) are estimated as follows.
[0123] In step S32, image point correspondences in the camera image sequence are determined, for example using an optical flow estimator or a flow tracker. The external parameters or their estimates can be provided by a curve estimator.
[0124] In step S34, an error function is minimized, which takes into account the error between the determined or measured image points and the image points projected according to the parametric model for a number of images of the image sequence.
[0125] As a result, in step S36, new values are determined for the parameters to be recalibrated.
[0126] Subsequently, in step S40 (see FIG. 6), it is determined whether the calibration was successful.
Claims
1. A method for automatically calibrating a vehicle camera (1; 31; 40) while the vehicle (33) is moving, the method comprising: imaging an area of the vehicle's (33) surroundings (45; 46) through a glass (32; 44) by the vehicle camera (1; 31; 40), the method comprising: a) providing a projection model of the vehicle camera (1; 31; 40), said projection model being based on parameters (θ j , int, ψ) as external parameters (θ j ), at least one internal parameter (int) of said vehicle camera (1; 31; 40) and at least one parameter (ψ) characterizing said glass (32; 44), b) capturing a sequence of images of the vehicle camera (1; 31; 40) while the vehicle (33) is travelling around a curve (5); c) determining a curve type based on the actual movement of the vehicle (33) during the curve maneuver (5) by a curve estimator; d) - image points of said image sequence of stationary objects (45) in the surrounding environment of said vehicle (33), the actual movement of said vehicle (33), and - the determined curve type Considering the image points determined from the image sequence corresponding to a stationary object (45) in the vehicle's surroundings and the image points (p ij ) and the error between [0010] estimating the parameters by minimizing e) the estimated parameters (θ j , ψ, int).
2. The parameter (θ j 2. The method of claim 1, further comprising the step of estimating (i.e., φ; int), checking whether the calibration is successful or not based on a threshold on the minimized error function.
3. If the auto-calibration is successful, one or more of the parameters (θ j A significance analysis is performed to determine whether the parameters (θ j 3. The method of claim 2, further comprising outputting:
4. The estimated parameters (θ j ; ψ; int), the library comprises an assignment of the traversed curve type to each estimable parameter at that time, and the output information indicates whether a unique estimation of the actual curve type was successful or not, and which of one or more parameters (θ j The method of any one of claims 1 to 3, further comprising indicating success or failure with respect to the first and second iterations of the method.
5. The estimation of the recalibrated parameters for the actual curve type is evaluated by checking the library, and one or more of the recalibrated parameters (θ j 5. The method according to claim 1, wherein the estimation of φ(t) is performed only in the case of a predetermined estimation probability.
6. The method according to any one of claims 3 to 5, wherein the significance analysis comprises a covariance assessment.
7. The method according to any one of claims 1 to 6, further comprising initialising the interior and / or glass parameters (ψ; int), the initialisation of these parameters (ψ; int) being performed from a factory calibration of the vehicle camera (1; 31; 40).
8. Based on the actual movement of the vehicle (33) while traveling along the curve (5), the external parameters (θ j 8. The method according to claim 1, further comprising: obtaining an initial solution for
9. The method according to any one of claims 1 to 8, further comprising determining from the image sequence image points or image feature points corresponding to stationary objects (45) of the vehicle's surrounding environment using an optical flow estimator (6) and / or a flow tracker (7).
10. The method according to any one of the preceding claims, wherein the at least one parameter (ψ) characterising the glass (32; 44) comprises a thickness b of the glass.
11. The at least one parameter (ψ) characterizing the glass is the glass (32; 44) [0025] The method according to any one of claims 1 to 10, comprising:
12. 1. A device for automatically calibrating a vehicle camera (1; 31; 40) while the vehicle (33) is moving, comprising: said vehicle camera (1; 31; 40) configured to image an area of the surrounding environment (45; 46) of said vehicle (33) through a glass (32; 44); A computation unit configured to provide a projection model of the vehicle camera (1; 31; 40), the projection model being based on parameters (θ j , int, ψ) as external parameters (θ j ), at least one internal parameter (int) of said vehicle camera (1; 31; 40) and at least one parameter (ψ) characterizing said glass (32; 44), said vehicle camera (1; 31; 40) is configured to capture a sequence of images while said vehicle (33) is travelling around a curve (5); a curve estimator configured to determine a curve type based on an actual movement of the vehicle (33) while the vehicle is navigating the curve (5); The computing unit: image points of said image sequence of stationary objects (45) in the surrounding environment of said vehicle (33), the actual movement of said vehicle (33), and - the determined curve type Considering the image points determined from the image sequence corresponding to a stationary object (45) in the vehicle's surroundings and the image points (p ij ) and the error between [0030] is configured to estimate the parameters by minimizing The estimated parameters (θ k , int, ψ).
13. A vehicle (33) comprising a vehicle camera (1; 31; 40) and a device according to claim 12.
14. A computer program element which, when the data processing unit is configured to execute the program, instructs the data processing unit to carry out the method according to any one of claims 1 to 11.
15. 15. A computer readable storage medium having stored thereon a computer program element according to claim 14.
Citation Information
Patent Citations
Automotive camera system and its calibration method and calibration program
CN103136747A
On-vehicle camera system, and calibration method and program for same
JP2013115540A
Stereocamera calibration method, parallax calculation device and stereocamera
JP2015163866A
Method and Apparatus for the Autocalibration of a Vehicle Camera System
US20190297314A1
On-vehicle camera, method for adjusting on-vehicle camera, and on-vehicle camera system
WO2018042954A1
