Device and method for calibrating an interior camera arranged in the interior of a vehicle

Abstract

The invention relates to a device and a method for calibrating an interior camera arranged in the interior of a vehicle, in the context of which at least one image with an image representation of a region of the interior is captured with the aid of the camera arranged in the interior of the vehicle, and image data corresponding to the image are generated and processed. Using the image data from the camera as a starting point, at least one edge of the vehicle visible in the interior is detected. In a first step, the roll angle of the camera is determined using the detected edge as a starting point. In a second step, the pitch angle and / or the yaw angle of the camera are / is determined using the determined roll angle, the image data and CAD data of the interior of the vehicle as a starting point.

Description

[0001] Device and method for calibrating an interior camera arranged in the interior of a vehicle

[0002] The invention relates to a device and a method for calibrating an interior camera arranged in the interior of a vehicle. Using the camera arranged in the interior of the vehicle, at least one image depicting a region of the interior is captured, and image data corresponding to the image is generated and processed. Based on the image data from the camera, at least one edge of the vehicle visible in the interior is detected.

[0003] Document DE 102020 209 613 A1 discloses a method for determining a change in the position of an image capture device attached to an adjustable element of a vehicle relative to a vehicle coordinate system. The method comprises reading in a status signal and determining a change signal.

[0004] Document US 2023 / 0127692 A1 discloses a method for calibrating a vehicle cabin camera with a pitch, yaw, and roll angle; and a field of view that captures features of the vehicle cabin that are symmetric about a vehicle longitudinal axis, wherein points are selected from an image of the vehicle cabin and projected onto a 3D unit circle in accordance with a camera projection model. For each of one or more rotations of a set of candidate yaw and roll rotations, the method comprises rotating the projected points with the rotation; mirroring the rotated points about a pitch axis; counter-rotating the projected points with an inverse of the rotation; and mapping the counter-rotated points back to an image plane to provide a set of transformed points.A rotation candidate is selected that provides the best match between the set of transformed points and the positions of the selected points in the image plane.

[0005] Document EP 2 808 645 A1 discloses a camera calibration device capable of estimating camera parameters with high precision and ease, without installing special calibration equipment or measuring 3D coordinates of reference points used for calibration. A normal vector detection means detects normal vectors perpendicular to a horizontal reference plane from an image of a camera to be calibrated. CAD data of the vehicle can be used to calibrate an interior camera of a vehicle, and even for a 2D interior camera, the perspective-n-point (PnP) problem can be solved using the CAD data and complex image processing. The 3D coordinate system can be matched to the 2D image using antialiasing. This is time-consuming and computationally intensive. The result is 6-dimensional: rotation angle + 3D position.

[0006] In particular, if no position and orientation data are available from a vehicle's interior camera, especially if there are no corresponding sensors for detecting the position, the interior camera must be calibrated regularly, or a determined calibration must be verified. This is especially important after the vehicle is activated and / or after the camera's field of view is obscured, especially with the hand of a vehicle occupant, as occurs when adjusting an interior mirror in which the interior camera may be integrated.

[0007] Based on the known prior art, it is an object of the invention to provide a device and a method for calibrating an interior camera arranged in the interior of a vehicle.

[0008] This object is achieved by a device having the features of claim 1 and by a method having the features of the independent method claim. Advantageous further developments are specified in the dependent claims.

[0009] The device according to claim 1 achieves the determination of the roll angle very easily and with minimal time and computational effort, in particular without the use and processing of position data from vehicle elements. Preferably, no vehicle CAD data is processed to determine the roll angle. Based on the roll angle thus determined and the image data, the pitch angle and the yaw angle can then be determined using position data from elements of the vehicle's interior. For this purpose, precise position data, such as vehicle CAD data, are processed. This enables precise calibration of the interior camera, and the PnP problem is solved with a relatively short expenditure of time.

[0010] The processing unit can be configured to process further images and / or image sequences captured by the camera based on the determined roll angle, particularly to determine the yaw and pitch angles in the second step. This enables a simpler process. The effort required to determine the yaw and pitch angles can thus be further reduced.

[0011] It is also advantageous if the processing unit knows the position of the edge in the vehicle's interior, especially if the position of the edge with respect to its angle of rotation relative to the vehicle's longitudinal axis or relative to an edge parallel to the vehicle's longitudinal axis is known to the processing unit. This reduces the effort required to determine the roll angle.

[0012] Furthermore, it is advantageous if the processing unit is designed to classify an object with the edge in order to determine the position of the edge. This makes it easy to determine an edge suitable for determining the roll angle, wherein the object is preferably a known object with an edge, for which the position of the edge with regard to its angle of rotation in relation to the vehicle's longitudinal axis or in relation to an edge parallel to the vehicle's longitudinal axis is known to the processing unit. The classified object can in particular be a rear window, a B-pillar, a headrest and / or a seat of the vehicle. As a result, the effort required to determine the roll angle is relatively low. The classification can in particular be carried out using a pattern comparison.

[0013] It is particularly advantageous if the edge is essentially vertical and / or horizontal. The edge runs, in particular, in a plane orthogonal to the vehicle's longitudinal axis. This is achieved without determining the edge and / or the edge's position from precise position data, such as the vehicle's CAD data. This also allows the roll angle to be determined with minimal effort.

[0014] The processing unit can be configured to process additional images and / or image sequences captured by the camera based on the determined roll angle, the determined yaw angle, and the determined pitch angle. This enables precise image processing, in particular, the precise determination of the positions of objects in the vehicle interior.

[0015] The processing unit can also be configured to determine the position of an object in the vehicle's interior based on the determined angles and the camera image data. The object is preferably a body part of a vehicle occupant, for example, the head, an ear, the mouth, an eye, and / or a hand of the occupant. The vehicle occupant is, in particular, the driver and / or the front passenger.

[0016] The processing unit can further be configured to determine at least one distance between two key points of a human pose estimation method based on the determined angles and image data. This enables simple and precise positioning of other body parts of the vehicle occupant, in particular the positioning of so-called key points of the vehicle occupant. This allows, in particular, an estimate of the position of the key points to be corrected and / or improved, and / or the accuracy of the positioning and / or tracking of the key points to be increased.Furthermore, the processing unit can be designed to carry out an estimation algorithm for estimating at least one property of the vehicle occupant, in particular for estimating the age, height or weight of one of the vehicle occupants, based on the distance between at least two key points of the vehicle occupant, or to improve this estimation algorithm and / or to correct an already estimated value.

[0017] It is particularly advantageous if the processing unit is configured to determine the position of the vehicle occupant's eyes based on the determined angles and image data. In this case, the processing unit can determine the vehicle occupant's gaze direction based on the determined eye position.

[0018] The camera can be a camera integrated into the vehicle's interior rearview mirror or integrated into the housing of the interior rearview mirror, a camera arranged above the driver and / or the front passenger in the roof area and / or in the B pillar on the driver's side and / or the front passenger side. This makes it possible to detect objects that are contacted by an occupant. Furthermore, it is advantageous if the field of view of the camera has a detection angle in the range of 100° to 150°, whereby the camera is preferably a monocular camera and / or an RGB-IR camera. This allows image processing by the processing unit, in particular, based on the recorded IR images. The use of IR images is advantageous because these images are independent of the ambient light.

[0019] The method with the features of the independent method claim has the same advantages as the claimed device. In particular, the method can be further developed with the features of the dependent claims directed to the device. In particular, the method is always carried out when the position of the camera has been changed or could have been changed. This ensures that the camera is always correctly calibrated.

[0020] Embodiments of the invention are explained in more detail below with reference to the figures, in which:

[0021] Figure 1 is a perspective schematic representation of a cockpit of a vehicle;

[0022] Figure 2 shows a schematic representation of an image of the interior of the vehicle taken by an interior camera of the vehicle; and

[0023] Figure 3 shows a flow chart for calibrating the interior camera.

[0024] Figure 1 shows a perspective schematic representation of a cockpit 100 of a vehicle 102. A driver 104 sits in a driver's seat 106 of the vehicle 102. The cockpit 100 also includes a central information display (CID) 108, a head-up display 110, and a graphical instrument cluster 112, which are arranged in a dashboard 114 of the vehicle 102. The aforementioned display elements 108, 110, 112 each form a functional unit of an output unit of the vehicle 102, which is designed to output information to the driver 104 and / or another occupant 206 (see Figure 2). At least one loudspeaker 116 of an entertainment system of the vehicle 102 is also arranged in the cockpit 100. The loudspeaker 116 also forms a functional unit of the output unit. The functional units of the output unit also serve as a playback unit for audio and / or video playback.

[0025] The cockpit 100 further includes a steering wheel 118 with control elements 120, a gear selector 122, a pedal assembly 124, and an input unit 126 with a rotary dial and a push-button function and / or a touch input field. This input unit 126 is also referred to as the Ergo Commander.

[0026] An interior rearview mirror 132 and an interior camera 134 integrated into this interior rearview mirror 132 are arranged in the upper region of a windshield 128 of the vehicle 102. The camera 134 is designed and integrated into the rearview mirror 132 in such a way that it captures images depicting at least one region of the interior of the vehicle 102. Specifically, the field of view of the camera 134 is directed toward the driver's seat 106 and a passenger seat 202 of the vehicle 102. An exemplary image 200 captured with the aid of the camera 134 is shown in Figure 2. The camera 134 is designed, in particular, to capture a plurality of temporally successive images, in particular in the form of a video stream, to generate image data corresponding to the images 200, and to transmit this data to a control unit 138 serving as a processing unit.

[0027] The control unit 138 has data outputs 140 and data inputs 142, which serve to connect to other units of the vehicle 102, for example to other cameras, sensors, input and output units and control units of assistance systems.

[0028] The control unit 138 further comprises a communication module 144 which is designed to establish a connection to a telecommunications network, in particular a mobile radio network.

[0029] Furthermore, additional cameras are arranged in the interior of the vehicle 102 in the B-pillars and above the driver 104 and the passenger. With the help of the camera 134 and the additional interior cameras, several images can be recorded consecutively, preferably as an image sequence or video sequence, and corresponding image data can be generated for each image, which is transmitted to the control unit 138 and processed by it. In a first step, the control unit 138 detects at least one edge of the vehicle 102 visible in the interior based on the image data from the camera 134 and determines the roll angle of the camera 134 based on the detected edge.

[0030] In a second step, the control unit 138 determines the pitch angle and / or the yaw angle of the camera 134 based on the roll angle, the image data and position data of elements of the interior of the vehicle 102. This determines all three angles required to calibrate the camera 134.

[0031] The roll angle is determined without taking into account the exact positions of objects in the interior. Only the angle of rotation around the vehicle's longitudinal axis or around an axis parallel to the vehicle's longitudinal axis is used by the control unit 138 to determine the camera's roll angle. The yaw angle and pitch angle are then determined in the second step, taking into account the exact positions of objects in the interior. These positions are available, in particular, from CAD data of the vehicle. The CAD data can be processed for this purpose by the control unit 138. Figure 2 shows a schematic representation of an image 200 of the interior of the vehicle 102 recorded by the interior camera 134. The camera 134 has a field of view of 120 degrees purely as an example. It is particularly advantageous if the camera 134 is an RGB-IR interior camera.In other embodiments, the camera 134 may also have a field of view in the range of 100 degrees to 150 degrees or more. The camera 130 is directed toward the driver's seat 106, a passenger seat 202, and a rear seat bench 204 of the vehicle 102. In the situation shown in Figure 2, the vehicle driver 104 is sitting in the driver's seat 106, and the other passenger 206 is sitting in the passenger seat 202. Thus, in the present embodiment, the other passenger is passenger 206. The passenger 206 has a tablet computer in their hands. The vehicle driver 104 is wearing glasses 210, a fitness tracker 212, and headphones 214.

[0032] When processing the image data of the image 200 captured by the camera 134, the control unit 138 detects the position of at least one of the following edges 216 to 226 in the image 200: upper horizontal edge 216 of the rear window of the vehicle 102, vertical edge 218 of the front passenger seat 202, vertical edge 220 of the driver's seat 106, vertical edge 222 of the B-pillar on the driver's side, vertical edge 224 of the headrest of the driver's seat 106, vertical edge 226 of the partition of the split rear seat bench 204.

[0033] Based on the detected edge 216 to 226, the control unit 138 determines the position of the edge 216 to 226 in the vehicle 102 and in the image 200. In particular, the control unit determines the position of the edge 216 to 226 with respect to its rotation around the vehicle's longitudinal axis or around an axis parallel to the vehicle's longitudinal axis. Based on the position of the edge 216 to 226 with respect to its rotation around the vehicle's longitudinal axis or around an axis parallel to the vehicle's longitudinal axis and the position of the edge 216 to 226 in the image 200, the control unit determines the roll angle of the camera 134.

[0034] Figure 3 shows a flowchart for calibrating the interior camera 134. The process begins in step S100. Subsequently, in step S102, based on the image data from the camera 134, at least one edge of the vehicle 102 visible in the interior is detected, and the roll angle of the camera 134 is determined based on the detected edge.

[0035] Subsequently, in step S104, the pitch angle and / or the yaw angle of the camera 134 is determined based on the determined roll angle, the image data, and position data of elements of the interior of the vehicle 102. The position data of the elements can be determined in particular by processing CAD data of the vehicle 102 by the control unit 138. Once the roll angle, the yaw angle, and the pitch angle of the camera are known, in step S106 the image data of all further images 200 recorded by the camera 134 are processed using the determined angles. The process then ends in step S108. In the exemplary embodiments described with reference to Figures 1 to 3, at least the camera 134 and the control unit 138 form a device for calibrating an interior camera 134 arranged in the interior of a vehicle 102. The control unit 138 is also referred to as a processing unit.Further elements and features shown in Figures 1 to 3 and mentioned in the preceding description may be part of the device for calibrating an interior camera 134 arranged in the interior of a vehicle 102. Likewise, method steps described with reference to the device may be part of the claimed method.

[0036] List of reference symbols

[0037] 100 cockpits

[0038] 102 vehicles

[0039] 104 drivers

[0040] 106 Driver's seat

[0041] 108 central information display

[0042] 110 Head-Up Display

[0043] 112 graphic instrument cluster

[0044] 114 Dashboard

[0045] 116 speakers

[0046] 118 Steering wheel

[0047] 120 control element

[0048] 122 gear selector switch

[0049] 124 Pedals

[0050] 126 input unit

[0051] 128 Windscreen

[0052] 130 microphone

[0053] 132 interior rearview mirror

[0054] 134 Interior camera

[0055] 136 emergency bracelet

[0056] 138 Control unit

[0057] 140 Data output

[0058] 142 Data input

[0059] 144 Communication interface

[0060] 200 images

[0061] 202 passenger seat

[0062] 204 rear seat

[0063] 206 passengers

[0064] 208 tablet computers

[0065] 210 glasses

[0066] 212 fitness bracelet

[0067] 214 headphones

[0068] 216 - 226 edges

[0069] S100- S108 Process steps

Claims

Claims:

1. Device for calibrating a sensor arranged in the interior of a vehicle Interior camera, wherein the camera (134) is arranged in the interior of the vehicle (102) and is designed to capture at least one image (200) with a depiction of a region of the interior and to generate image data corresponding to the image (200), with a processing unit (138) designed to process the image data, wherein the processing unit (138) is designed, in a first step, to detect at least one edge of the vehicle (102) visible in the interior based on the image data of the camera (134) and, based on the detected edge, to determine the roll angle of the camera (134), and wherein the processing unit (138) is designed, in a second step, to determine the pitch angle and / or the yaw angle of the camera (134) based on the determined roll angle, the image data, and position data of elements of the interior of the vehicle (102).

2. Device according to claim 1, characterized in that the Processing unit (138) is designed to process further images and / or image sequences recorded with the aid of the camera (134) based on the determined roll angle.

3. Device according to one of the preceding claims, characterized in that the processing unit (138) knows the position of the edge in the interior of the vehicle (102).

4. Device according to one of the preceding claims, characterized in that the processing unit (138) is designed to classify an object with the edge in order to determine the position of the edge.

5. Device according to claim 4, characterized in that the classified object is a rear window, a B-pillar, a headrest and / or a seat of the vehicle (102).

6. Device according to one of the preceding claims, characterized in that the edge is a substantially vertical and / or horizontal edge.

7. Device according to one of the preceding claims, characterized in that the processing unit (138) is designed to process further images and / or image sequences recorded with the aid of the camera (134) based on the determined roll angle, the determined yaw angle and the determined pitch angle.

8. Device according to claim 7, characterized in that the processing unit (138) is designed to determine the position of an object (208, 210, 212, 214) in the interior of the vehicle (102) based on the determined angles and the image data of the camera (134).

9. Device according to claim 7 or 8, characterized in that the processing unit (138) is designed to determine at least one dimension of an object (208, 210, 212, 214) in the interior of the vehicle (102) based on the determined angles and the image data of the camera (134).

10. Device according to one of the preceding claims 7 to 9, characterized in that the processing unit (138) is designed to determine or correct at least one distance between two key points of a human pose estimation method based on the determined angles and the image data of the camera (134), and preferably based on the distance between at least two key points of the vehicle occupant (104, 206), to carry out, improve and / or correct an estimation algorithm for estimating at least one property of the vehicle occupant (104, 206), in particular for estimating the age, height or weight of one of the vehicle occupants (104, 206).

11. Device according to one of the preceding claims 7 to 10, characterized in that the processing unit (138) is designed to determine the position of the eyes of the vehicle occupant (104, 206) based on the determined angles and the image data of the camera (134).

12. Device according to claim 11, characterized in that the processing unit (138) is designed to determine the viewing direction of the vehicle occupant (104, 206) based on the determined position of the eyes.

13. Device according to one of the preceding claims, characterized in that the camera (134) captures a 2D image, wherein the camera (134) is in particular an RGB-IR camera, and / or that the camera (134) is attached to an adjustable element of the vehicle (102).

14. A method for calibrating an interior camera arranged in the interior of a vehicle, in which at least one image (200) with an image of a region of the interior of the vehicle (102) is captured with the aid of the camera (134) arranged in the interior of the vehicle (102), and image data corresponding to the image (200) are generated and processed, in which, based on the image data of the camera (134), at least one edge of the vehicle (102) visible in the interior is detected, in which, in a first step, the roll angle of the camera (134) is determined based on the detected edge, in which, in a second step, the pitch angle and / or the yaw angle of the camera (134) is determined based on the determined roll angle, the image data, and position data of elements of the interior of the vehicle (102).

15. The method according to claim 14, characterized in that the method according to claim 14 is carried out whenever the position of the camera (134) has been changed or could have been changed.

Images