Method for calibrating an interior camera in a vehicle

Abstract

Method for extrinsic calibration of an interior camera in a vehicle, wherein the interior camera is first calibrated with respect to a reference coordinate system and a position of an interior light of the vehicle relative to the reference coordinate system (S1) is known, characterized in that - a three-dimensional environmental model in relation to the vehicle is created based on signals recorded by the vehicle's environmental sensors (S2), - the interior lighting in a section is activated corresponding to a position of a three-dimensional object captured in the environment model (S3), - a projection vector is determined between a light point of the activated section of the interior lighting and the position of the detected object (S4), - a corneal reflection image of an eye of a driver of the vehicle is determined using image data captured by the interior camera to be calibrated (S5), - the determined projection vector is projected onto the corneal reflection image, thus determining a two-dimensional reference vector (S6), - in the corneal reflection image a reflected light point of the interior lighting activated relative to the object is detected (S7), - a deviation between the reflected light point and the two-dimensional reference vector is determined (S8), - the deviation is assessed in relation to at least one specified threshold (S9), and - if it is determined that the deviation exceeds at least one specified threshold, extrinsic calibration parameters of the interior camera are recalibrated (S10).

Description

[0001] The invention relates to a method for the extrinsic calibration of an interior camera in a vehicle, wherein the interior camera is first calibrated with respect to a reference coordinate system and the position of an interior light of the vehicle relative to the reference coordinate system is known.

[0002] From DE 11 2018 005 553 T5, an extrinsic camera calibration system and a method which is carried out by a processor of an extrinsic calibration system are known.The method comprises receiving images via a network interface from a surround-view system comprising four cameras mounted on the front, rear, left and right sides of a vehicle, each captured image containing features of at least one calibration pattern positioned in front of each camera; determining extrinsic calibration parameters for each camera from the image features, mapping the coordinates of the features to camera coordinates and usable for subsequent normal operation of the cameras when capturing and displaying subsequent images; and storing, in a memory of the surround-view system, the calibration parameters, which are accessible to the surround-view system during subsequent normal operation.

[0003] Calibration methods for cameras are known from DE 10 2022 107 909 A1 and US 2016 / 0 134 863 A1. Furthermore, DE 11 2018 006 164 B4 describes a line-of-sight direction calibration method, US 2022 / 0 080 888 A1 a method for displaying information to a vehicle driver, DE 10 2016 001 204 A1 a motor vehicle with linear light sources for direct and indirect illumination of a vehicle interior, and CN 1 13 602 216 A a method for acquiring sensor information, an electronic device, a light source, and an autonomous vehicle.

[0004] Furthermore, in NAGAMATSU, Takashi; HIROE, Mamoru; RIGOLL, Gerhard: Cornealreflection-based wide range gaze tracking for a car. In: Human interface and the management of information. Cham: Springer, 2019 (Lecture notes in computer science; 11570). Title page + imprint + table of contents pp. 385-400. - ISBN 978-3-030-22648-0. URL: https: / / doi.org / 10.1007 / 978-3-030-22649-7_31 [accessed on 06.06.2025] a wide-range gaze tracking system is described that includes a calibration-free function.

[0005] In NITSCHKE, Christian; NAKAZAWA, Atsushi; TAKEMURA, Haruo: Corneal imaging revisited: An overview of corneal reflection analysis and applications. In: IPSJ transactions on computer vision and applications, Vol. 5, 2013, pp. 1-18. ISSN 1882-6695. URL: https: / / doi.org / 10.2197 / ipsjtcva.5.1 [accessed on 06.06.2025], the generation of so-called corneal reflective images is described.

[0006] The invention is based on the objective of providing a novel method for the extrinsic calibration of an interior camera of a vehicle.

[0007] The problem is solved according to the invention by a method which has the features specified in claim 1.

[0008] Advantageous embodiments of the invention are the subject of the dependent claims.

[0009] A method for the extrinsic calibration of an interior camera in a vehicle, wherein the interior camera is first calibrated with respect to a reference coordinate system and the position of an interior light of the vehicle relative to the reference coordinate system is known, provides according to the invention that - a three-dimensional environmental model is created in relation to the vehicle based on signals recorded by the vehicle's environmental sensors, - the interior lighting in a section is activated corresponding to a position of a three-dimensional object captured in the environment model, - a projection vector is determined between a light point of the activated section of the interior lighting and the position of the detected object, - a corneal reflection image is determined based on image data captured by the indoor camera to be calibrated, - the determined projection vector is projected onto the corneal reflection image, thus determining a two-dimensional reference vector, - in the corneal reflection image, a reflected point of light from the interior lighting activated relative to the object is detected, - a deviation between the reflected light point and the two-dimensional reference vector is determined, - the deviation is evaluated in relation to at least one predetermined threshold, and - if it is determined that the deviation exceeds at least one predetermined threshold, extrinsic position parameters of the interior camera are recalibrated.

[0010] By applying this method, it is possible to verify extrinsic calibration parameters of the interior camera, thus enabling a potential recalibration of the interior camera.

[0011] The procedure can be carried out essentially independently of weather and / or lighting conditions in the vehicle's vicinity, and can also be performed regardless of the vehicle's surroundings. This means that the procedure can be carried out regardless of the vehicle's current location.

[0012] The procedure requires no additional human intervention or connection of map data, and no modifications to hardware components are necessary.

[0013] The method can be executed independently of other sensor modalities and can be repeated cyclically, for example within a control loop.

[0014] In particular, the method provides input variables for a variety of assistance functions, such as an intelligent lighting function, augmented reality, a function for determining the vehicle's own movement, etc.

[0015] Exemplary embodiments of the invention are explained in more detail below with reference to a drawing.

[0016] This shows: Fig. Figure 1 schematically shows the process for the extrinsic calibration of an interior camera of a vehicle.

[0017] The only Fig. Figure 1 describes a procedure for the extrinsic calibration of an interior camera in a vehicle, in particular a driver observation camera. Specifically, the procedure presents an approach to verifying the extrinsic calibration of the interior camera and to performing the calibration automatically. Extrinsic calibration of the interior camera, especially the driver observation camera, is necessary to determine the driver's gaze pattern with respect to a reference coordinate system, particularly a vehicle coordinate system. The driver's gaze pattern is relevant, for example, for sensor visualization and / or for adaptive warning functions of certain driver assistance systems.

[0018] When calibrating an indoor camera, a distinction is generally made between extrinsic and intrinsic calibration. An extrinsic parameter is the position of the indoor camera, particularly with regard to orientation and translation, whereas an intrinsic parameter is the focal length of the indoor camera.

[0019] The extrinsic calibration of the interior camera is subject to certain mechanical stresses and / or thermal loads in the field, i.e., during vehicle operation. Such thermal stress occurs, for example, when the vehicle is parked for an extended period, which can result in temperatures exceeding 50 °C. Furthermore, the installation of the interior camera involves adhesives that can potentially dissolve and / or become viscous, thus altering the camera's orientation. Therefore, it is relatively important to determine the camera's extrinsic calibration parameters during vehicle use, particularly while driving, using vehicle-side calibration algorithms and to automatically correct any detected miscalibration.

[0020] The procedure described below verifies and, if necessary, recalibrates the extrinsic alignment of the interior camera. This procedure requires that the interior camera is calibrated with respect to a reference coordinate system (i.e., a vehicle coordinate system) and that the position of the interior lighting (i.e., ambient lighting) relative to this reference coordinate system is known. In particular, the interior camera is a component of at least one of the vehicle's driver assistance systems.

[0021] The calibration of the interior camera with respect to the reference coordinate system constitutes a first process step S1. This reference coordinate system is located, for example, in the center of a front axle of the vehicle.

[0022] In a second process step S2, an environmental model is created in relation to the vehicle based on signals acquired by the vehicle's environmental sensors. For this purpose, the environmental sensors include at least ultrasound-based sensors that continuously acquire signals while the vehicle is in operation.

[0023] The vehicle also features active interior lighting, which is linked, for example, to the vehicle's parking assistance system. This means that signals detected by the environmental sensors are displayed by emitting light from the interior lighting. With interior lighting designed in this way, it is possible to visually indicate to the driver, during a parking maneuver, the direction in relation to the vehicle of an object, particularly a three-dimensional object. This is achieved by highlighting detected objects in the driver's line of sight with color.If such a potential collision object is detected in the side and front area of ​​the vehicle, a different colored light is emitted in a section of the interior lighting corresponding to the position of the detected object, compared to the rest of the interior lighting. This results in points in three-dimensional space based on the created environmental model, and points, specifically positions, in sections of the interior lighting that reflect the position of the detected object.

[0024] In a third process step S3, the interior lighting in a section is activated corresponding to a position of a three-dimensional object detected in the environment model, in particular by emitting light of a different color compared to the rest of the interior lighting present in the vehicle.

[0025] In a fourth process step S4, a projection vector is determined between a light point of the activated section of the interior lighting and the position of the object captured in the environment model.

[0026] Based on the initial calibration of the indoor camera performed in the first process step S1, a relationship is known between the projection vector, which is located in the reference coordinate system, and an image plane of the indoor camera. This is achieved, among other things, by taking the extrinsic calibration parameters into account.

[0027] Subsequently, in a fifth process step S5, a corneal reflective image is determined using image data captured by the interior camera. For example, in this fifth process step S5, the corneal reflective image undergoes post-processing, such as rectification of a spherical projection. This corneal reflective image is essentially a reflection of the surroundings onto the cornea of ​​the driver's eye.

[0028] In a sixth process step, S6, the projection vector is projected onto the corneal reflection image, thereby forming a two-dimensional reference vector. Thus, in the sixth process step, S6, a projection is performed on the corneal reflection image of the interior camera based on intrinsic and extrinsic calibration parameters of the interior camera with respect to the reference coordinate system, i.e., the vehicle coordinate system.

[0029] A seventh process step, S7, involves searching for a response—that is, a reflection of the light point from the interior lighting corresponding to the position of the detected object—within the corneal reflection image using at least one suitable image processing algorithm. This is achieved, for example, by using specific filters and / or templates, since it is generally known that such a light point is being sought.

[0030] In a subsequent eighth process step, S8, a deviation is determined between the light spot of the interior lighting detected in the corneal reflection image and the two-dimensional reference vector. If the interior camera is ideally calibrated, the light spot lies on the two-dimensional projection vector.

[0031] For example, a deviation is determined using suitable Key Performance Indicators, for which a two-dimensional orthogonal distance can be used. During an initial application, a tolerance value for the deviation is specified, whereby, for example, a certain deviation from the light point in relation to the object's position can result, especially since this is based on the 95th percentile of drivers.

[0032] Furthermore, only a subset of the reference vectors is evaluated, because it is known where a driver is located in the vehicle, and it is also known that a representation along a curvature results.

[0033] The procedure then jumps to the fifth procedure step S5 for a predetermined number of repetitions in order to repeat this and the subsequent procedure steps S6, S7, so that the validity of the extrinsic calibration parameters can be verified.

[0034] In a ninth process step S9, the determined deviation is evaluated with respect to at least one threshold value. If the determined deviation exceeds the at least one predefined threshold value, an extrinsic recalibration of the interior camera is performed in a tenth process step S10, taking into account the deviation determined in the ninth process step S9.

[0035] For example, recalibration can be achieved by optimizing a cost function, as in the present embodiment by optimizing the two-dimensional orthogonal distance. This involves repeatedly changing the orientation of the interior camera, i.e., modifying extrinsic calibration parameters and extracting new two-dimensional projection vectors. This process is repeated until the cost function reaches a predefined minimum. The resulting extrinsic calibration parameters are then stored as a valid parameter set in the vehicle and can be used for further functionality.

[0036] Using the procedure described above, the calibration parameters can be verified on the vehicle, and the parameters can be recalibrated on the vehicle if necessary. This does not require any modifications to the vehicle's hardware components.

Images