Calibration of camera coordinate system

Abstract

A method and system for calibrating a coordinate transformation CT between a reference coordinate system and a camera coordinate system, where the reference coordinate system is associated with a vehicle interior in which a camera is mounted. The method includes obtaining a 3D model of the interior of the vehicle, the 3D model including at least one physical structure identifiable in an image, acquiring the image of the interior of the vehicle using a camera, identifying at least one feature in the image corresponding to one of the physical structures, and selecting a set of pixel coordinates for the feature, each pixel coordinate is associated with a particular point in the 3D model, forming a 2D projection of the particular point on the camera image plane by applying a coordinate transformation and a model of the camera optics, and calibrating the CT based on a relationship between the set of selected pixel coordinates and the 2D projection.

Description

Field of the Invention

[0001] The present invention relates to the calibration of the local coordinate system of a camera mounted inside a vehicle. Background of the Invention

[0003] In vehicle imaging applications, such as driver monitoring systems (DMS), cameras are mounted inside the vehicle. The exact location of the camera isn't necessarily fixed and known. For example, if the camera is mounted on the steering column, it will move regularly with at least two degrees of freedom (axial and elevation). If the camera is mounted on the rearview mirror, it will move whenever the mirror is adjusted. And even for fixed locations, such as on the dashboard, the camera can shift unintentionally.

[0004] For this reason, it is important to regularly update or calibrate the camera position and orientation (i.e., the camera's local coordinate system) relative to a known reference coordinate system (i.e., the coordinate system inside the vehicle). The camera's local coordinate system is often referred to as the camera coordinate system (CCS). Calibration of the CCS is crucial for the correct operation of features such as head tracking, eye tracking, and viewing targets. Document WO 2018 / 000037 provides an example of an existing solution.

[0005] Existing solutions are designed to track features on an image using template matching. Using the 2D displacement of the tracked features as input, a linear interpolation of the CCS is performed for the four extremes of the steering column. The interpolation result is considered to be a calibrated CCS. Unfortunately, it is difficult (sometimes impossible) to find suitable features inside a car. Therefore, this approach requires extensive experience and manual work in order to configure the system, e.g., to find appropriate features / templates. In addition, a satisfactory configuration is often application-specific and may not be applicable to different interiors. Even for carefully designed configurations, template matching is not very robust, which leads to erroneous detections (e.g., on the back of the steering wheel). Complex control logic and filtering are required to increase robustness, which in turn increases complexity and cost.

[0006] Furthermore, in conventional systems, camera calibration is typically limited to two degrees of freedom, such as position in / out and pitch up / down. As mentioned above, in more complex situations and in more degrees of freedom, calibration is typically required.

[0007] General Disclosure of the Invention

[0008] It is an object of the present invention to alleviate the above-mentioned problems and to provide a computationally more efficient calibration of the CCS.

[0009] According to a first aspect of the present invention, this and other objects are achieved by a method for calibrating a coordinate transformation (CT) between a reference coordinate system and a camera coordinate system, wherein the reference coordinate system is associated with the interior of a vehicle in which a camera is mounted. The method comprises obtaining a 3D model of the vehicle interior, the 3D model including at least one physical structure identifiable in an image, acquiring an image of the vehicle interior using a camera, identifying at least one feature in the image corresponding to one of the physical structures, selecting a set of pixel coordinates of the feature, each pixel coordinate being associated with a specific point in the 3D model, forming a 2D projection of the specific point on the camera image plane by applying a coordinate transformation and a model of the camera optics, and calibrating the CT based on the relationship between the set of selected pixel coordinates and the 2D projection. The coordinate transformation is represented as a series of nested transformations, each transformation representing a possible movement of the camera, such that the coordinate transformation models the mechanical mounting of the camera.

[0010] Typically, the first transformation represents a fixed position for the installation. This transformation can have up to six degrees of freedom (DOF), representing installation tolerances or displacements relative to the intended position. Other transformations may have limited DOF due to mechanical mounting. Typical movement restrictions are rotation around a single angle, translation along a single axis, etc.

[0011] In one example, the camera is mounted on a steering column. In this case, the nested transformations may include a first transformation representing the (usually known, or at least approximately known) pivot point of the column, a second transformation representing the rotation (pitch) of the column, and a third transformation representing the translation (extension) of the column.

[0012] In another example, the camera is mounted on a 3D pivot point, for example, on a rearview mirror. In this case, the nested transformation can include a first transformation representing the (usually known or at least approximately known) pivot point and a second transformation with three rotational degrees of freedom.

[0013] By expressing the coordinate transformation as a set of nested transformations, calibration of the CT can be made more computationally efficient. For example, the numerical solution can converge more quickly. It is important to note that this is not necessarily a limit on the number of DOFs (the first transformation can typically have 6 DOFs). Rather, this is because movements with a larger expected range will be modeled by a more constrained transformation (e.g., a single DOF).

[0014] The 3D model can include the vehicle's geometric data (e.g., CAD data), which provides a precise description of all mechanical structures. The features to be recognized can then be defined (configured) by specifying the 3D description of their corresponding structures in the CAD data. This ensures more robust recognition and tracking, more independent of specific vehicle interior conditions (such as color, surface characteristics, etc.).

[0015] It should be noted that the 2D projection does not have to be determined explicitly, but can occur implicitly during calibration. For example, the calibration step can include creating a set of equations, each of which defines an element of an error vector as the difference between one of the pixel coordinates and the associated 2D projection, and minimizing the error vector, for example in a least-squares sense. This set of equations can be solved numerically. The equations can be nonlinear.

[0016] The number of equations in this set is determined by the number of features identified in the image and mapped to corresponding features in the projected 3D model. The selected features should be fixed (i.e., non-movable) physical structures in the vehicle that can be reliably identified in the image and easily extracted from the projected 3D model. Examples of such features include the shape and size of B- or C-pillars, door handles, center consoles, portions of the rear seat, etc.

[0017] Calibration is not limited to cameras mounted on the steering column. In fact, CCS (and transformation CT) can generally be described and calibrated in six degrees of freedom (orientation and position).

[0018] According to a second aspect of the present invention, the aforementioned object is achieved by a system for calibrating a coordinate transformation (CT) between a reference coordinate system and a camera coordinate system, wherein the reference coordinate system is associated with a vehicle interior, the system comprising a camera mounted in the vehicle interior and a controller for controlling the camera to acquire an image of the vehicle interior. The system further comprises processing circuitry configured to obtain a 3D model of the vehicle interior, the 3D model comprising at least one physical structure identifiable in the image, identify at least one feature in the image corresponding to one of the physical structures, select a set of pixel coordinates of the feature, each pixel coordinate being associated with a particular point in the 3D model, form a 2D projection of the particular point onto a camera image plane by applying the coordinate transformation (CT) and a model of the camera optics, and calibrate the coordinate transformation (CT) based on a relationship between the set of selected pixel coordinates and the 2D projection. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The present invention will be described in more detail with reference to the accompanying drawings, which show currently preferred embodiments of the invention.

[0021] Figure 1 Schematically shown is an eye tracking system mounted on the dashboard of a vehicle.

[0022] Figure 2 Shown in more detail Figure 1 Eye tracking system in .

[0023] Figure 3 is a flowchart of a method according to an embodiment of the present invention.

[0024] Figure 4 Three nested transformations of a model representing a steering column are shown.

[0025] Figure 5a-5b A calibration process according to an embodiment of the present invention is shown.

[0026] Detailed Description of the Preferred Embodiments

[0027] Embodiments of the invention will now be discussed with reference to an eye tracking system. However, the principles of the invention are equally applicable to any application where a camera is mounted in the cabin of a vehicle, such as any driver monitoring system (DMS) or cab monitoring system (CMS).

[0028] Figure 1 A driver 1 of a vehicle is shown. A camera 2 is mounted in front of the driver 1, here on a steering column 3. Alternatively, the camera 2 may be mounted on the dashboard, fixed to the ceiling, or have any other location suitable for a particular application. The cameras form an imaging system 4 for acquiring images of the driver and / or the interior of the vehicle (see Figure 2 ). For example, system 4 may be a driver monitoring system (DMS).

[0029] refer to Figure 2 , shows the components of imaging system 4 in more detail. Here, camera 2 includes an image sensor 5 (e.g., a CMOS image sensor) and suitable optics 6. The optics are configured to project incident light onto an image plane of sensor 5. In the illustrated case, the system also includes at least one light source 7, which has a known geometric relationship with sensor 5. Light source 7 is typically configured to emit light outside the visible range, such as infrared (IR) or near-infrared (NIR). The light source can be a solid-state light source, such as an LED. In the illustrated example, light source 7 is an LED configured to emit light with a spectrum concentrated in a 50nm band centered around 850 or 940nm (NIR). In addition, an optical bandpass filter (e.g., an interference filter) can be arranged between the user and sensor 5. The filter (not shown) is configured to have a passband that substantially corresponds to the emission spectrum of light source 7. Thus, in the above example, filter 6 would have a passband centered around 850nm or 940nm (e.g., 825-875nm or 915-965nm).

[0030] The controller 8 is connected to the camera 2 and the LED 7 and is programmed to control the sensor 5 to acquire continuous images under the illumination of the LED 7. Typically, the LED 7 is driven at a given duty cycle, and the controller 8 then controls the sensor 5 to acquire images in synchronization with the light pulses from the LED 7.

[0031] The system also includes processing circuitry 9 (also referred to as a processor) and memory 10. The memory stores program code executable by the processor 9, enabling the processor to receive and process images acquired by the sensor 5. The processor 9 can be configured to determine and track eye posture to determine the user's gaze direction, i.e., where the user is looking. Figure 1 The system has many different applications, including automotive applications such as tracking the driver's eyes for safety reasons, and various human-machine interfaces.

[0032] During operation of the eye tracking embodiment, user 1 is illuminated by light source 7, and light reflected from the object (the user's face) passes through camera optics 6 and is received by imaging sensor 5. It should be noted that most of the ambient light will be blocked by the filter, thereby reducing the required light source power.

[0033] The direction of gaze can be determined by determining the head pose (the position and orientation of the head in a reference coordinate system (RCS)) and then determining the eye pose (the position and orientation of the eyes relative to the head). In a simple case, sometimes referred to as estimated eye tracking, the eye pose is determined based on the position of the iris relative to the head. However, in many applications, more accurate eye gaze detection is achieved by using a light source 7. Under illumination from the light source, the acquired image will include reflections (glints) from the cornea of the eye, which can be used to more accurately determine eye gaze.

[0034] In the embodiment shown, a 3D model 11 of the vehicle interior (eg a CAD model) is stored in the memory 10. The 3D model 11 contains geometric data of the vehicle interior expressed in a reference coordinate system RCS.

[0035] The position of camera 2 can also be expressed in terms of a reference coordinate system, and this position is referred to as the camera coordinate system CCS. Knowing the coordinate transformation CT between the RCS and CCS, any point in the 3D model can be mapped to the camera coordinate system by applying the coordinate transformation. Using knowledge of the camera optics, any point in the camera coordinate system can then be projected onto the camera's image plane.

[0036] exist Figure 1 , a reference coordinate system RCS is indicated by 12 , and a camera coordinate system CCS is indicated by 13 .

[0037] Now refer to Figure 3 - Figure 5 discusses the operation of the system. Figure 3 The steps in can be performed by the processing circuit 9.

[0038] In a first initialisation step S10, a 3D model 11 of the vehicle interior is obtained and stored in the memory 10. In embodiments where the camera is mounted on a pivotable steering column 3, the model may include the position of the pivot point P.

[0039] The calibration process begins in step S11 by acquiring an image 20 of the vehicle interior using the camera 1. In step S12, at least one geometric feature 21 is identified in the image 20 using suitable image processing. Each feature corresponds to a fixed physical structure in the vehicle interior that is present in the 3D model. A suitable structure may be, for example, a B-pillar 14 (see Figure 1 ) or C-pillars, door handles, or any other immovable physical structure that can be easily identified in the image of the vehicle interior. It should be noted that movable structures (such as the backrests of the front seats) are not well suited for the calibration process. After the features 21 have been identified, a set of pixel coordinates 22 is selected for each identified feature 21. Each selected pixel coordinate 22 corresponds to a specific point in the 3D model.

[0040] Recognition of features in an image can be achieved by conventional template tracking, however, this requires using actual images from the vehicle interior to create the template. A more sophisticated approach is to use a neural network system that has been trained to recognize a set of features in the image (landmark detection). The neural network system can be trained for many different vehicle interiors, making it robust to vehicle variations (e.g., color, shape, etc.). A feature detector using such a trained neural network can also provide an internal confidence signal that indicates how similar the detected feature is to the feature that the neural network was trained to recognize. A low confidence signal may indicate that a particular feature has not been correctly recognized, for example because it is not visible (an occluded feature). As an alternative, a neural network can be trained to perform pixel segmentation and extract boundaries between stable segments.

[0041] In step S13, the current (or initial) approximation of the transform CT is applied to specific points of the 3D model to map these points to the camera coordinate system. The mathematical model of the camera optics 6 is then used in step S14 to form a 2D projection 23 of the specific point on the image plane of the camera 2. Thus, each 2D projection 23 of the specific point will be a mathematical representation obtained by applying the transform CT and the model of the camera optics to a selected point of the 3D model data.

[0042] In step S15 , the CT is calibrated (updated) based on the relationship between the selected pixel coordinates 22 and the corresponding 2D projections 23 .

[0043] Step S15 can be executed by the processing circuit as a calculation operation. However, for the purpose of understanding the principle, step S15 is divided into sub-steps S16 and S17.

[0044] In step S16, a set of equations is formed, where each equation defines an element of the error vector as the difference between 1) one of the pixel coordinates identified in the image and 2) the 2D projection of the point associated with that pixel coordinate in the 3D point model data. The equations will also include a calibration vector that includes one variable for each degree of freedom.

[0045] In step S17, the error vector is minimized, for example in a least squares sense, thereby providing calibration of the coordinate transformation.

[0046] As just explained, the solution to the minimization problem has six variables, one for each degree of freedom (three positions and three rotations). However, this process can be made more efficient and robust by modeling the camera's mechanical mounting using a series of nested transformations. Each transformation then corresponds to a possible way to move the camera, potentially with a finite number of degrees of freedom.

[0047] In this example, where the camera 2 is mounted on the steering column 3, the coordinate transformation can be modeled as three nested transformations: 1) the position of P, 2) the rotation in the vertical plane (pitch, φ), and 3) expansion / contraction (z). Figure 4 As described above, the pivot point P of the steering column can be assumed to be a known point (known coordinates) in the 3D model data. However, installation tolerances (or unexpected displacements) can be included as an additional six degrees of freedom (see below).

[0048] Graphically, the process of determining CCS can be shown as a two-stage process, as Figure 5a-5b outlined in . Figure 5a An edge 21 of the B-pillar 14 identified in the image 20 is shown, along with a set of pixel coordinates 22 along the edge 21. These pixel coordinates therefore correspond to a specific point in the 3D model 11. Figure 5b The 2D projections 23 of two of these specific points in the image plane are shown. Since the coordinate transformation CT at this point is not calibrated, the 2D projections 23 are not correctly aligned with the corresponding pixel coordinates 22.

[0049] For illustrative purposes, the calibration of CT can be considered as the calibration of the camera coordinate system CCS (e.g. Figure 4 During calibration, the CCS is rotated in the vertical plane to minimize errors as much as possible. This aligns the pixel coordinates 22 and the projections 23 in the plane. In addition, the CCS is translated in the z-direction to further minimize errors.

[0050] In practice, the process of calibrating a CT is typically performed in a single computational procedure. In the case of a camera mounted on the steering column, this process can be formulated as a two-variable minimization problem, minf(φ,z), where the function f involves a first (known) transformation of the steering column pivot point, P, and two additional geometric transformations. The second transformation describes the rotation around the pivot point, while the third describes the translation along the z-axis. The solution to this problem provides both the pitch and the translation z.

[0051] To account for tolerances in the camera mount, limited calibration can also be allowed in the other degrees of freedom. By introducing these additional DOFs in the second transformation, the minimization problem can be expressed as min f(pitch, 6DOF).

[0052] During the calibration process, the most recent calibration will be used as the starting point for each successive iteration. In the example given above, CT will include three nested transformations (pivot point, rotation, and translation). The pivot point transformation will remain unchanged, while the second and third transformations will be calibrated in each iteration.

[0053] In another example (where the camera is mounted on a rearview mirror), the CT would include a first transformation representing the mirror's pivot point and at least one transformation representing a rotation about the pivot point. If appropriate, there may be a separate transformation for each rotational degree of freedom.

[0054] Those skilled in the art realize that the present invention is by no means limited to the embodiments described above. On the contrary, many modifications and variations are possible within the scope of the appended claims. For example, the system may include more than one camera to be calibrated.

Claims

1. A method for calibrating a current coordinate transformation (CT) between a reference coordinate system (12) and a camera coordinate system (13), wherein the reference coordinate system is related to the interior of a vehicle in which a camera (2) is installed, the method comprising: Obtaining a 3D model of the vehicle interior, the 3D model comprising geometric data of the vehicle interior represented in the reference coordinate system (12), the geometric data including at least one physical structure (14) identifiable in the image; Using the camera (2) to acquire an image (20) of the interior of the vehicle; identifying at least one feature (21) in the image (20) corresponding to one of the physical structures (14), and selecting a set of pixel coordinates (22) of the feature, each pixel coordinate being associated with a particular point in the 3D model; forming a 2D projection (23) of the specific point on a camera image plane by applying the coordinate transformation and a model of the camera optics (6); as well as calibrating the current CT based on the relationship between the set of selected pixel coordinates (22) and the 2D projection (23), Wherein the coordinate transformation is represented as a series of nested transformations such that the coordinate transformation models the mechanical mounting of the camera.

2. The method according to claim 1, wherein The calibration steps include: Creating a set of equations where each equation defines an element of the error vector as the difference between one of the pixel coordinates and the associated 2D projection, and The error vector is minimized, for example in a least squares sense.

3. The method according to claim 1 or 2, wherein: The camera is mounted on a steering column, and wherein the series of nested transformations includes a first transformation representing the position of a pivot point P of the column, a second transformation representing a rotation of the column, and a third transformation representing a translation along the column.

4. The method according to claim 1 or 2, wherein: The camera is mounted on a rear view mirror, and wherein the series of nested transformations includes a first transformation representing a pivot point of the mirror and at least one second transformation representing a rotation about the pivot point.

5. The method according to claim 4, wherein The at least one second transformation comprises a separate transformation for each rotational degree of freedom.

6. A method according to any one of the preceding claims, wherein The 3D model is a CAD model of the vehicle, and each feature to be identified is defined by specifying a 3D description of the corresponding structure in the CAD data.

7. A method according to any one of the preceding claims, wherein The at least one physical structure includes at least one of a B-pillar, a C-pillar, a center console, a portion of a rear seat, and a door handle.

8. A system for calibrating a current coordinate transformation (CT) between a reference coordinate system (12) and a camera coordinate system (13), wherein the reference coordinate system is related to a vehicle interior, the system comprising: a camera (2), the camera being mounted inside the vehicle, a controller (8) for controlling the camera (2) to acquire an image (20) of the interior of the vehicle, and A processing circuit (9), the processing circuit being configured to: Obtaining a 3D model of the vehicle interior, the 3D model comprising geometric data of the vehicle interior represented in the reference coordinate system (12), the geometric data including at least one physical structure (14) identifiable in the image (20); identifying at least one feature (21) in the image (20) corresponding to one of the identifiable physical structures, and selecting a set of pixel coordinates (22) of the feature, each pixel coordinate being associated with a particular point in the 3D model; forming a 2D projection (23) of the specific point on the camera image plane by applying the coordinate transformation CT and a model of the camera optics (6); as well as calibrating the current coordinate transformation CT based on the relationship between the set of selected pixel coordinates (22) and the 2D projection (23), Wherein the coordinate transformation is represented as a series of nested transformations such that the coordinate transformation models the mechanical mounting of the camera.

9. The system according to claim 8, wherein: The processing circuit is configured to calibrate the coordinate transformation by: Creating a set of equations where each equation defines an element of the error vector as the difference between one of the pixel coordinates and the associated 2D projection, and The error vector is minimized, for example in a least squares sense, so that .

10. The system according to claim 8 or 9, wherein: The camera is mounted on a steering column, and wherein the series of nested transformations includes a first transformation representing the position of a pivot point P of the column, a second transformation representing a rotation of the column, and a third transformation representing a translation of the column.

11. The system according to claim 8 or 9, wherein: The camera is mounted on a rear view mirror, and wherein the series of nested transformations includes a first transformation representing a pivot point of the mirror and at least one second transformation representing a rotation about the pivot point.

12. The system according to claim 11, wherein The at least one second transformation comprises a separate transformation for each rotational degree of freedom.

13. The system according to one of claims 8 to 12, wherein: The 3D model is a CAD model of the vehicle, and each feature to be identified is defined by specifying a 3D description of the corresponding structure in the CAD data.

14. The system according to one of claims 8 to 13, wherein: The at least one physical structure includes at least one of a B-pillar, a C-pillar, a center console, a portion of a rear seat, and a door handle.

Images