Device and method for calibrating an interior space camera arranged in an interior space of a vehicle

Abstract

In a device and a method for calibrating an interior space camera arranged in an interior space of a vehicle, at least one image of an imaging of a region of the interior space is acquired by means of a camera arranged in the interior space of the vehicle and image data corresponding to the image is generated and processed. At least one edge of the vehicle visible in the interior space is detected on the basis of the image data of the camera. In a first step, a roll angle of the camera is determined on the basis of the detected edge. In a second step, a pitch angle and / or a yaw angle of the camera are determined on the basis of the determined roll angle, the image data and CAD data of the interior space of the vehicle.

Description

Technical Field

[0001] This invention relates to an apparatus and method for calibrating an interior space camera arranged in the interior space of a vehicle. At least one image of a region having the interior space is acquired using the camera arranged in the vehicle's interior space, and image data corresponding to the image is generated and processed. At least one visible edge of the vehicle within the interior space is detected based on the image data from the camera. Background Technology

[0002] Document DE102020209613A1 discloses a method for determining the orientation change of an image acquisition device fixed to an adjustable element of a vehicle relative to the vehicle coordinate system. The method includes reading in a state signal and determining the change signal.

[0003] Document US2023 / 0127692A1 discloses a method for calibrating a vehicle cabin camera, the vehicle cabin camera having pitch, yaw, and roll angles; and a field of view that acquires features of a vehicle cabin symmetrical about the vehicle's longitudinal axis, wherein points are selected from an image of the vehicle cabin and projected onto a 3D unit circle according to a camera projection model. For each of one or more rotations in a set of yaw and roll rotation candidates, the method includes: rotating the projected point using the rotation; mirroring the rotated point about the pitch axis; reversing the projected point using a reverse rotation of the rotation; and imaged the reversed point back into the image plane to provide a set of transformed points. Rotation candidates are selected such that they provide an optimal match between the transformed point set and the positions of the selected points in the image plane.

[0004] Document EP2808645A1 discloses a camera calibration device that can estimate camera parameters with high accuracy and simplicity without installing special calibration equipment or measuring the 3D coordinates of a reference point for calibration. The normal vector acquisition mechanism obtains the normal vector perpendicular to the horizontal reference plane from the image of the camera to be calibrated.

[0005] To calibrate the vehicle's interior space camera, the vehicle's CAD data can be used. Even with a 2D interior space camera, the perspective n-point (PnP) problem can be solved using CAD data and complex image processing. Here, edge smoothing can be used to match the 3D coordinate system to the 2D image. This is time-consuming and computationally intensive. The result is six dimensions: rotation angle + 3D position.

[0006] Especially when no position and orientation data from the vehicle's interior space camera is available, particularly when the use of the corresponding sensor to obtain position is abandoned, the interior space camera must be calibrated periodically, or the determined calibration must be checked. This is especially important after the vehicle is activated and / or, for example, when adjusting the interior space rearview mirror (in which the interior space camera may be integrated), or when the camera's field of view is obstructed, particularly by the hands of a vehicle occupant. Summary of the Invention

[0007] Based on known prior art, the object of the present invention is to provide an apparatus and method for calibrating an interior space camera arranged in the interior space of a vehicle.

[0008] This objective is achieved, respectively, by the apparatus having the features of claim 1 and by the method having the features of the independent method claim. Advantageous improvements are given in the dependent claims.

[0009] The apparatus according to claim 1 achieves the following: Roll angle is determined very simply and with minimal time and computational cost, particularly without using and processing position data of vehicle components. Preferably, the roll angle is determined without processing the vehicle's CAD data. Based on the thus determined roll angle and image data, pitch and yaw angles can then be determined using position data of components within the vehicle's interior space. For this purpose, precise position data, such as the vehicle's CAD data, is processed. This enables accurate calibration of the interior space camera and resolves the PnP problem with relatively little time.

[0010] The processing unit can be configured to process additional images and / or image sequences captured by the camera based on the determined roll angle, in order to determine the yaw and pitch angles, particularly in the second step. This simplifies the process and further reduces the time required to determine the yaw and pitch angles.

[0011] Another advantage is that the processing unit knows the orientation of the edge within the vehicle's interior space, particularly the orientation of the edge with respect to its rotation angle relative to the vehicle's longitudinal axis or relative to an edge parallel to the vehicle's longitudinal axis. This reduces the time spent determining the roll angle.

[0012] Furthermore, it is advantageous that the processing unit is configured to classify objects having said edges to determine the orientation of said edges. This allows for the simple determination of edges suitable for determining the roll angle, wherein the object is preferably a known object with an edge for which the orientation of the edge with respect to its rotation angle relative to the vehicle's longitudinal axis or relative to an edge parallel to the vehicle's longitudinal axis is known to the processing unit. The objects to be classified can in particular be the vehicle's rear window, B-pillar, headrest, and / or seat. Thus, determining the roll angle is relatively inexpensive. This classification can be performed, in particular, through pattern comparison.

[0013] A particular advantage is that the edge is substantially vertical and / or horizontal. The edge extends, in particular, in a plane orthogonal to the vehicle's longitudinal axis. This is especially advantageous when the edge and / or its orientation are not determined from precise vehicle position data, such as CAD data. This also allows for the determination of the roll angle with minimal effort.

[0014] The processing unit can be configured to process additional images and / or image sequences captured by the camera based on a determined roll angle, a determined yaw angle, and a determined pitch angle. This enables precise image processing, particularly accurate determination of the position of objects within the vehicle's interior space.

[0015] The processing unit can also be configured to determine the position of an object within the vehicle's interior space based on the determined angle and image data from the camera. The object is preferably a body part of a vehicle occupant, such as the occupant's head, ears, mouth, eyes, and / or hands. Vehicle occupants are particularly the driver and / or front passenger.

[0016] The processing unit can also be configured to determine at least one distance between two key points in a human pose estimation method based on the determined angle and image data. This allows for a simple and accurate determination of the position of other body parts of a vehicle occupant, particularly the positions of so-called key points of the vehicle occupant. This can, in particular, correct and / or improve the estimation of key point positions and / or increase the accuracy of key point position determination and / or tracking. Furthermore, the processing unit can be configured to execute an estimation algorithm for estimating at least one characteristic of the vehicle occupant, particularly for estimating the age, height, or weight of one of the vehicle occupants, or to improve the estimation algorithm and / or correct already estimated values, based on the distance between at least two key points of the vehicle occupant.

[0017] Of particular advantage is that the processing unit is configured to determine the position of the vehicle occupant's eyes based on the determined angle and image data. In this case, the processing unit can determine the vehicle occupant's viewing direction based on the determined eye position.

[0018] The camera can be integrated into the vehicle's rearview mirror or its housing, located in the headliner area above the driver and / or passenger, or in the B-pillar on the driver's and / or passenger's side. This allows for the detection of objects touched by the occupant. Furthermore, it is advantageous that the camera has an acquisition angle in the range of 100° to 150°, wherein the camera is preferably a monocular camera and / or an RGB-IR camera. This allows for image processing by a processing unit based on the captured IR images. Using IR images is advantageous because these images are independent of ambient light.

[0019] The method having the features of the parallel method claims has the same advantages as the claimed device. In particular, the method can be improved by the features of the dependent claims relating to the device. The method is particularly performed whenever the camera's position has changed or may have changed. This ensures that the camera is always correctly calibrated. Attached Figure Description

[0020] Embodiments of the present invention will now be explained in more detail with reference to the accompanying drawings, wherein:

[0021] Figure 1 A perspective view of the vehicle's cockpit is shown.

[0022] Figure 2 A schematic diagram showing images of the vehicle's interior space taken by an interior space camera; and

[0023] Figure 3 A flowchart for calibrating an interior space camera is shown. Detailed Implementation

[0024] Figure 1 A perspective view of the cockpit 100 of vehicle 102 is shown. The driver 104 is seated in the driver's seat 106 of vehicle 102. The cockpit 100 also includes a central information display (CID) 108, a head-up display 110, and a graphical instrument cluster 112, which are arranged in the instrument panel 114 of vehicle 102. The aforementioned display elements 108, 110, and 112 respectively form functional units of the output unit of vehicle 102, which is configured to output information to the driver 104 and / or another occupant 206 (see [reference]). Figure 2 At least one speaker 116 of the entertainment system of vehicle 102 is also arranged in the cockpit 100. The speaker 116 also forms a functional unit of the output unit. The functional unit of the output unit also serves as a playback unit for audio and / or video playback.

[0025] The cockpit 100 also includes a steering wheel 118 with operating elements 120, a gear selector switch 122, a pedal assembly 124, and an input unit 126 with rotary dial and button functions and / or a touch input area. This input unit 126 is also known as the Ergo Commander.

[0026] An interior rearview mirror 132 and an interior space camera 134 integrated into the rearview mirror 132 are arranged in the upper region of the windshield 128 of the vehicle 102. The camera 134 is constructed and integrated into the rearview mirror 132 in such a way that it acquires an image of at least one area of ​​the interior space of the vehicle 102. Specifically, the field of view of the camera 134 is directed toward the driver's seat 106 and the front passenger seat 202 of the vehicle 102.

[0027] Exemplary image 200 captured by camera 134 Figure 2 As shown in the figure, the camera 134 is specifically configured to acquire multiple images that are sequential in time, particularly in the form of a video stream, generate image data corresponding to the image 200, and transmit these image data to the control unit 138, which serves as a processing unit.

[0028] The control unit 138 has a data output terminal 140 and a data input terminal 142, which are used to connect to other units of the vehicle 102, such as other cameras, sensors, input and output units of the auxiliary system and control unit.

[0029] The control unit 138 also includes a communication module 144 configured to establish a connection with a telecommunications network, particularly a mobile radio network.

[0030] In addition, other cameras are arranged in the B-pillars of the interior space of vehicle 102 and above the driver 104 and the front passenger. Multiple images, preferably as an image sequence or video sequence, can be captured sequentially using camera 134 and the other interior space cameras, and corresponding image data is generated for each image. This image data is transmitted to and processed by control unit 138. In a first step, control unit 138 detects at least one visible edge of vehicle 102 within the interior space based on the image data from camera 134 and determines the roll angle of camera 134 based on the detected edge.

[0031] In the second step, the control unit 138 determines the pitch and / or yaw angle of the camera 134 based on the roll angle, image data, and position data of components within the interior space of the vehicle 102. This determines all three angles required to calibrate the camera 134.

[0032] The roll angle is determined without considering the precise position of the object within the interior space. The roll angle of the camera is determined by the control unit 138 based solely on the rotation angle around the vehicle's longitudinal axis or an axis parallel to the vehicle's longitudinal axis. Then, in a second step, the yaw and pitch angles are determined, taking into account the precise position of the object within the interior space. These positions can be obtained, in particular, from the vehicle's CAD data. For this purpose, the CAD data can be processed by the control unit 138.

[0033] Figure 2 A schematic diagram showing image 200 of the interior space of vehicle 102 captured by interior space camera 134. Camera 134 has an exemplary field of view of 120 degrees. Particularly advantageously, camera 134 is an RGB-IR interior space camera. In other embodiments, camera 134 may also have a field of view ranging from 100 to 150 degrees or greater. Camera 130 is oriented towards the driver's seat 106, front passenger seat 202, and rear seats 204 of vehicle 102. Figure 2 In the illustrated scenario, the vehicle driver 104 is seated in the driver's seat 106, while another occupant 206 is seated in the front passenger seat 202. Therefore, in this embodiment, the other occupant is the front passenger 206. The front passenger 206 holds a tablet computer in their hand. The vehicle driver 104 is wearing glasses 210, a fitness tracker 212, and headphones 214.

[0034] When processing image data of image 200 captured by camera 134, control unit 138 detects the orientation of at least one of the following edges 216 to 226 in image 200:

[0035] - The upper horizontal edge 216 of the rear window of vehicle 102;

[0036] - Vertical edge 218 of the front passenger seat 202;

[0037] - Vertical edge 220 of driver's seat 106;

[0038] - Vertical edge 222 of the B-pillar on the driver's side;

[0039] - Vertical edge 224 of the headrest of the driver's seat 106;

[0040] - The vertical edge 226 of the partition of the split rear seat 204.

[0041] Based on the detected edges 216 to 226, the control unit 138 determines the orientation of the edges 216 to 226 in the vehicle 102 and in the image 200. Specifically, the control unit determines the orientation of the edges 216 to 226 relative to their rotation about the vehicle's longitudinal axis or about an axis parallel to the vehicle's longitudinal axis. Based on the orientation of the edges 216 to 226 relative to their rotation about the vehicle's longitudinal axis or about an axis parallel to the vehicle's longitudinal axis, and the orientation of the edges 216 to 226 in the image 200, the control unit determines the roll angle of the camera 134.

[0042] Figure 3 A flowchart for calibrating the interior space camera 134 is shown. The process begins in step S100. Subsequently, in step S102, at least one visible edge of the vehicle 102 in the interior space is detected based on image data from the camera 134, and the roll angle of the camera 134 is determined based on the detected edge.

[0043] Subsequently, in step S104, the pitch angle and / or yaw angle of the camera 134 are determined based on the determined roll angle, image data, and position data of components within the interior space of the vehicle 102. The position data of the components can be determined, in particular, by processing the CAD data of the vehicle 102 by the control unit 138.

[0044] Then, after knowing the roll angle, yaw angle, and pitch angle of the camera, the image data of all the other images 200 captured by the camera 134 are processed in step S106 using the determined angles. The process then ends in step S108.

[0045] In reference Figures 1 to 3 In the described embodiment, at least camera 134 and control unit 138 form a means for calibrating interior space camera 134 arranged in the interior space of vehicle 102. Control unit 138 is also referred to as a processing unit. Figures 1 to 3 The additional elements and features shown and mentioned in the foregoing description may be part of the apparatus for calibrating the interior space camera 134 arranged in the interior space of vehicle 102. Similarly, the method steps described with reference to the apparatus may be part of the claimed method.

[0046] List of reference numerals

[0047] 100 cockpit

[0048] Vehicle 102

[0049] Driver of vehicle 104

[0050] 106 driver's seat

[0051] 108 Central Information Display

[0052] 110 head-up display

[0053] 112 Graphic Instrument Group

[0054] 114 Dashboard

[0055] 116 speakers

[0056] 118 steering wheel

[0057] 120 operating element

[0058] 122-position selector switch

[0059] 124 pedal assembly

[0060] 126 input units

[0061] 128 windshield

[0062] 130 microphone

[0063] 132 Car interior rearview mirror

[0064] 134 interior space cameras

[0065] 136 Emergency Bracelet

[0066] 138 control unit

[0067] 140 data output terminal

[0068] 142 Data Input Terminal

[0069] 144 communication interface

[0070] 200 images

[0071] 202 Passenger seat

[0072] 204 rear seats

[0073] 206 co-pilots

[0074] 208 tablet computer

[0075] 210 Glasses

[0076] 212 Fitness Tracker

[0077] 214 headphones

[0078] 216-226 edge

[0079] S100-S108 Method Steps

Claims

1. A device for calibrating interior space cameras arranged in the interior space of a vehicle. in, The camera (134) is arranged in the interior space of the vehicle (102) and configured to acquire at least one image (200) of an area having the interior space and generate image data corresponding to the image (200). A processing unit (138) is provided, the processing unit being configured to process the image data. The processing unit (138) is configured to detect at least one visible edge of the vehicle (102) in the interior space based on image data from the camera (134) in a first step and determine the roll angle of the camera (134) based on the detected edge. The processing unit (138) is configured to determine the pitch angle and / or yaw angle of the camera (134) in the second step based on the determined roll angle, the image data, and the position data of the components in the interior space of the vehicle (102).

2. The apparatus according to claim 1, characterized in that, The processing unit (138) is configured to process additional images and / or image sequences captured by the camera (134) based on the determined roll angle.

3. The apparatus according to any one of the preceding claims, characterized in that, The processing unit (138) is aware of the orientation of the edge in the interior space of the vehicle (102).

4. The apparatus according to any one of the preceding claims, characterized in that, The processing unit (138) is configured to classify objects having the edges in order to determine the orientation of the edges.

5. The apparatus according to claim 4, characterized in that, The classified objects are the rear window, B-pillar, headrest and / or seat of the vehicle (102).

6. The apparatus according to any one of the preceding claims, characterized in that, The edge is a substantially vertical and / or horizontal edge.

7. The apparatus according to any one of the preceding claims, characterized in that, The processing unit (138) is configured to process additional images and / or image sequences captured by the camera (134) based on the determined roll angle, the determined yaw angle and the determined pitch angle.

8. The apparatus according to claim 7, characterized in that, The processing unit (138) is configured to determine the position of objects (208, 210, 212, 214) in the interior space of the vehicle (102) based on the determined angle and image data from the camera (134).

9. The apparatus according to claim 7 or 8, characterized in that, The processing unit (138) is configured to determine at least one dimension of the object (208, 210, 212, 214) in the interior space of the vehicle (102) based on the determined angle and image data from the camera (134).

10. The apparatus according to any one of claims 7 to 9, characterized in that, The processing unit (138) is configured to determine or correct at least one distance between two key points of a human posture estimation method based on the determined angle and image data from the camera (134), and preferably to perform, improve and / or correct an estimation algorithm for estimating at least one characteristic of the vehicle occupants (104, 206), particularly for estimating the age, height or weight of one of the vehicle occupants (104, 206), based on the distance between at least two key points of the vehicle occupants (104, 206).

11. The apparatus according to any one of claims 7 to 10, characterized in that, The processing unit (138) is configured to determine the position of the eyes of the vehicle occupants (104, 206) based on the determined angle and the image data from the camera (134).

12. The apparatus according to claim 11, characterized in that, The processing unit (138) is configured to determine the viewing direction of the vehicle occupants (104, 206) based on the determined eye position.

13. The apparatus according to any one of the preceding claims, characterized in that, The camera (134) acquires 2D images, the camera (134) is in particular an RGB-IR camera, and / or the camera (134) is fixed to an adjustable element of the vehicle (102).

14. A method for calibrating an interior space camera arranged in the interior space of a vehicle, wherein, The camera (134) arranged in the interior space of the vehicle acquires at least one image (200) of an area of ​​the interior space of the vehicle (102) and generates and processes image data corresponding to the image (200); Detect at least one visible edge of the vehicle (102) in the interior space based on image data from the camera (134); In the first step, the roll angle of the camera (134) is determined based on the detected edges; In the second step, the pitch angle and / or yaw angle of the camera (134) are determined based on the determined roll angle, the image data, and the position data of the components in the interior space of the vehicle (102).

15. The method according to claim 14, characterized in that, Whenever the position of the camera (134) has changed or may have changed, the method according to claim 14 shall be performed.

Images