Method and device for calibrating a camera in a vehicle

Abstract

The invention relates to a method and a device for estimating the three-dimensional position and the three-dimensional orientation of an image recording unit (1), which is provided in the interior of a vehicle, in a vehicle-internal coordinate system, having the following steps: recording, by means of the image recording unit (1), at least one two-dimensional image (2) of the interior; receiving, by means of a data processing unit (3), the image (1); detecting, by means of an image processing unit (4), a number N of vehicle-specific key points P1 - PN in the image (2); requesting, by means of the data processing unit (3), a vehicle-specific database (5); and calculating, by means of a position determining unit (6), the absolute three-dimensional position [x, y, z] and the three-dimensional position angles [α, β, γ] of the image recording unit (1) by comparing the previously known positions of the key points in the vehicle-internal coordinate system with the projections of the detected key points.

Description

[0001] Method and apparatus for calibrating a camera in a vehicle

[0002] The invention relates to a method and a device for calibrating a camera in a vehicle.

[0003] Methods for analyzing a vehicle's interior using a camera installed in the vehicle are known in the prior art. For example, gestures, poses, gaze directions, and head orientations of passengers can be recognized and linked to vehicle functions. This requires capturing the camera's coordinates within a vehicle-internal coordinate system. This includes not only the camera's three-dimensional position (x, y, and y coordinates) but also its three-dimensional orientation (roll, pitch, and yaw angles).

[0004] However, this presents the problem that the camera used for interior analysis is often installed in moving vehicle components, such as the rearview mirror. The camera can therefore be moved to different positions and directions, meaning that its three-dimensional position and orientation cannot be fixed. The object of the invention is thus to provide a method and a device that enable the determination of the three-dimensional position and orientation of a movable camera in a vehicle without requiring additional sensors. In particular, the invention is intended to allow the use of conventional two-dimensional cameras for interior vehicle analysis.

[0005] These and other problems of the invention are solved by a method according to claim 1.

[0006] A method according to the invention is designed for estimating the three-dimensional positions x, y, z and the three-dimensional attitude angles, in particular the roll angle a, the pitch angle β and the yaw angle y, of an image acquisition unit, in particular a camera, in the interior of a vehicle in a vehicle-internal coordinate system. It comprises the following steps:

[0007] In a first step, an image capture unit takes at least a two-dimensional image of the vehicle interior. This image capture unit can be a conventional 2D camera. The image is then received by a data processing unit.

[0008] The data processing unit can be designed as a microcontroller or microcomputer and may include a central processing unit (CPU), volatile semiconductor memory (RAM), non-volatile semiconductor memory (ROM, SSD hard drive), magnetic storage (hard drive) and / or optical storage (CD-ROM), as well as interface units (Ethernet, USB) and the like. The components of such data processing units are generally known to those skilled in the art.

[0009] In the next step, an image processing unit detects the two-dimensional positions of several, preferably at least N = 3, distinctive key points Pi - PN of the vehicle interior in the captured two-dimensional image. For this purpose, known pattern recognition algorithms, particularly from the field of machine learning, can be used.

[0010] In particular, a neural network can be used for this purpose, which has been trained to recognize prominent key points of a vehicle interior from 2D images and to estimate their coordinates. Training data is created in advance for various vehicle models to ensure the highest possible accuracy in detecting the vehicle-specific key points in the 2D image.

[0011] According to the invention, for this purpose, two-dimensional images of vehicle interiors, whose key points are annotated, can be used as training data for the neural network. However, instead of a neural network, any other method from the field of machine learning and artificial intelligence can also be used.

[0012] According to the invention, three-dimensional coordinates relative to an origin point in an internal vehicle coordinate system are stored in a database for the key landmarks in the vehicle. The origin point can be the manufacturer-specified position of the center point of the vehicle's front axle. The key landmarks are, where possible, selected within the vehicle so that they are not obscured by vehicle parts or by passengers. Furthermore, it is necessary that the points are not movable; that is, they are points directly connected to the vehicle's body, for example, at the windows or on the headliner.

[0013] If necessary, the key landmarks in the interior may be marked with a machine-readable code or other distinctive marking to facilitate their detection.

[0014] The next step involves a query of the vehicle-specific database by the data processing unit. The database contains the known, absolute three-dimensional coordinates of the key points Pi. (abs) - p N (abs) The data is stored in the vehicle's internal coordinate system. The data processing unit transmits the determined key points Pi - PN to the database and receives the corresponding three-dimensional coordinates of these key points.

[0015] In the next step, the absolute three-dimensional position [x, y, z] and the three-dimensional position angles [a, ß, y] of the image acquisition unit are calculated by a position determination unit by comparing the previously known positions of the key points Pi. (abs) - p N < abs ) in the vehicle's internal coordinate system with the projections of the detected key points Pi - PN in the plane of the recorded two-dimensional image.

[0016] To calculate the position and orientation of the image acquisition unit, intrinsic geometric parameters of the unit are used. These parameters can include, in particular, the focal length, the principal point, and the distortion coefficients of a camera. For example, if the camera is known to have a focal length of 50 mm, the position and orientation of the image acquisition unit can be calculated from a reference model and two-dimensional estimates of these key points. Well-known programming libraries such as OpenCV and Dlib have corresponding routines, as described in the online article https: / / learnopencv.com / head-pose-estimation-using-peer-and-djib / .

[0017] As a result of the calculation, the absolute three-dimensional positions and the position angles of the image acquisition unit in the coordinate system of the vehicle are known.

[0018] Both the image processing unit and the position determination unit can be provided as separate hardware units or, preferably, as software modules in the RAM or ROM of the data processing unit. Alternatively, these units can be located externally, for example, on a server on the internet, to which the necessary data is transmitted. According to the invention, the position determination unit can compare the position [x, y, z] and the orientation angles [a, β, y] of the image acquisition unit with previously calculated or initial values. In particular, the position determination unit can calculate a correction vector [Ax, Ay, Az] and correction angles [Aa, Aβ, Ay] for correcting application-specific signals.

[0019] According to the invention, the positions of the key points can be transferred into a topological three-dimensional data model, in particular into a graph. A predefined topology of known vehicle interiors can be used to create the topological data model. For example, it may be known that vehicles always have a rear window, a headliner, and at least two side windows. Thus, the identified position of the headliner can be represented as a key point in the form of a node in a graph and connected to the positions of the side windows via edges.

[0020] The invention further relates to a computer-readable storage medium comprising instructions that cause a data processing unit to execute a method according to the invention.

[0021] The invention further relates to a device for estimating the three-dimensional position and three-dimensional orientation of an image acquisition unit arranged in the interior of a vehicle in a vehicle-internal coordinate system, which is configured to carry out a method according to the invention.

[0022] The invention further relates to a vehicle comprising a device according to the invention.

[0023] Further features of the invention will become apparent from the claims, the exemplary embodiments and the figures.

[0024] The invention is explained below using an exemplary, non-exclusive embodiment. Fig. 1 shows a schematic example of a device according to the invention;

[0025] Fig. 2a shows a schematic representation of the image captured by the image acquisition unit during the execution of a method according to the invention; Fig. 2b shows a schematic representation of the two-dimensional and absolute three-dimensional positions of the vehicle-specific key points during the execution of a method according to the invention.

[0026] Fig. 1 shows a schematic example of a device according to the invention for estimating the three-dimensional position and orientation of an image acquisition unit 1 arranged in the interior of a vehicle within a vehicle-internal coordinate system. The device comprises an image acquisition unit 1, which is configured to capture a two-dimensional image 2 and provides intrinsic geometric parameters 7, in particular the focal length of the image 2. Furthermore, the device comprises a data processing unit 3 with an image processing unit 4, a database 5, and a position determination unit 6.

[0027] Image processing unit 4 comprises a neural network trained to extract vehicle-specific key points Pi - PN from a two-dimensional image. Two-dimensional image data of vehicles with annotated key points were used to train this network; that is, the images used for training contain images of various vehicle interiors whose key points are annotated.

[0028] First, image processing unit 4 is applied to image 2, so that the key points of the vehicle Pi - PN in image 1 are preserved in the two-dimensional image 2. The key points are in the form of Cartesian two-dimensional coordinates and are stored as a data object in the form of a graph.

[0029] Fig. 2a shows a schematic representation of the image 2 captured by the image acquisition unit during the execution of a method according to the invention. In this example, the image processing unit 4 has detected a number N = 3 distinctive key points Pi - P3 in image 2, namely on the headliner and on the two side windows.

[0030] In the next step, data processing unit 3 queries a vehicle-specific database 5 by transmitting the detected key points Pi - PN and the vehicle type. Database 5 provides the predefined three-dimensional coordinates of the key points Pi. (abs) - p3< abs ) in the vehicle's internal coordinate system, which can also be referred to as the reference system. In this vehicle's internal coordinate system, the origin point is located at a convenient position within the vehicle, as determined by the manufacturer, such as the center point of the vehicle's front axle.

[0031] Fig. 2b shows a schematic representation of the two-dimensional and three-dimensional positions of the key points Pi - P3 during the execution of a method according to the invention. The absolute positions of the key points Pi are determined by querying database 5. (abs) - p3< abs ) known; furthermore, the two-dimensional coordinates Pi - P3 of these key points are known as a projection into the image plane of image 2, as shown schematically.

[0032] In the next step, a position determination unit 6 calculates the absolute three-dimensional position [x, y, z] and the three-dimensional position angles [a, ß, y] of the image acquisition unit 1 by comparing the previously known positions of the key points Pi. (abs) - p3< abs) in the vehicle's internal coordinate system with the projections of the detected key points Pi - P3. For this purpose, the focal length, the principal point of the image, and the distortion parameters of the image acquisition unit 1 are used as geometric parameters 7. Knowing these values, as shown in Fig. 2b, the position and the three-dimensional orientation angles of the camera can be determined, so that the projections of the 3D points Pi< abs >, p2< abs ) and PS (abs) with the identified 2D points Pi, P2, and P3. Consequently, the absolute coordinates Pi are (abs) , P2 (abs) and PS (abs)The three selected points, as well as the absolute three-dimensional position [x, y, z] and the three-dimensional orientation angles [a, β, y] of the image acquisition unit 1 in the vehicle's internal coordinate system, are known. Furthermore, Fig. 1 shows that the database 5 provides additional values ​​to the position determination unit 6. These are the initial, i.e., manufacturer-defined, three-dimensional position and orientation angles of the image acquisition unit 1. The position determination unit 6 then compares the determined actual position [x, y, z] and orientation angles [a, β, y] of the image acquisition unit 1 with the initial values ​​and calculates a correction vector [Ax, Ay, Az] and correction angles [Aa, Aβ, Ay], which can subsequently be used to correct application-specific signals.

[0033] This allows, for example, the gaze directions, gestures or poses of the vehicle passengers calculated by a vehicle application to be corrected by this correction vector and these correction angles in order to compensate for differences between the manufacturer-specified position and location of the image capture unit and the actually measured position and location of the image capture unit.

[0034] However, the invention is not limited to this described embodiment, but also includes further embodiments of the present invention within the scope of the following patent claims.

Claims

Patent claims 1. A method for estimating the three-dimensional position and orientation of an image acquisition unit (1) located in the interior of a vehicle in a vehicle-internal coordinate system, comprising the steps: a. Acquisition by the image acquisition unit (1) of at least one two-dimensional image (2) of the interior; b. Receipt by a data processing unit (3) of the image (1); c. Detection by an image processing unit (4) of a number N of vehicle-specific key points Pi - PN in the image (2), characterized in that the following steps are performed: d. Query by the data processing unit (3) of a vehicle-specific database (5) and receipt of previously known absolute three-dimensional coordinates of the key points Pi (abs) - p N < abs)in the vehicle's internal coordinate system; e. calculation, by a position determination unit (6), of the absolute three-dimensional position [x, y, z] and the three-dimensional position angles [a, β, y] of the image acquisition unit (1 ) by comparison of the previously known positions of the key points Pi (abs) - p N < abs ) in the vehicle's internal coordinate system with the projections of the detected key points Pi - PN taking into account intrinsic geometric parameters (7) of the image acquisition unit (1 ), in particular the focal length f and the principal point of the image.

2. Method according to claim 1, characterized in that the position determination unit (6) compares the position [x, y, z] and the position angles [a, ß, y] of the image acquisition unit (1) with previously calculated or initial values ​​and calculates a correction vector [Ax, Ay, Az] and correction angles [Aa, Aß, Ay] to correct application-specific signals.

3. Method according to claim 1 or 2, characterized in that the vehicle-specific key points are predefined, vehicle-specific, preferably immovable points in the interior of the vehicle, in particular points on the headliner, on the side windows or on the rear window.

4. Method according to one of claims 1 to 3, characterized in that the key points are stored by the data processing unit (3) in electronically readable form, for example in the form of a table or a graph.

5. Method according to any one of claims 1 to 4, characterized in that N is greater than or equal to three.

6. Computer-readable storage medium comprising instructions that cause a data processing unit (3) to execute a method according to any one of claims 1 to 5.

7. Device for estimating the three-dimensional position and orientation of an image acquisition unit (1) arranged in the interior of a vehicle in a vehicle-internal coordinate system, comprising: a. an image acquisition unit (1) configured to capture at least one two-dimensional image (2) of the interior; b. a data processing unit (3) for receiving the image (2); c. an image processing unit (4) configured to detect a number N of vehicle-specific key points Pi - PN in the image (2); characterized in that d. the data processing unit (3) is configured to query a vehicle-specific database (5) and to receive previously known, absolute three-dimensional coordinates of the key points Pi (abs) - p N < abs ) is formed in the vehicle's internal coordinate system, wherein e. a position determination unit (6) is provided which is used to calculate the absolute three-dimensional position [x, y, z] and the three-dimensional position angles [a, ß, y] of the image acquisition unit (1) by comparing the previously known positions of the key points Pi (abs) - Pri (abs) in the vehicle's internal coordinate system with the projections of the detected key points Pi - PN taking into account intrinsic geometric parameters (7) of the image acquisition unit (1), in particular the focal length f and the principal point of the image.

8. Vehicle comprising a device according to claim 7.

Images