Method for determining the orientation of a camera unit of a detection device, detection device and vehicle

Abstract

Method for determining the orientation of a camera unit (3) of a detection device (2) of a vehicle (1) with respect to a horizontal surface (14) in a vehicle environment (5) of the vehicle (1), wherein - the camera unit (3) captures an image (4) of the vehicle's surroundings (5), whereby - the environment image (4) has image points (6) that are arranged at respective image point coordinates (7) of the environment image (4), - by the detection device (2) in the environment image (4) a reference area (16) is detected according to a predetermined recognition procedure, which represents the horizontal surface (14), - by means of the detection device (2) image point depths (18) of at least some of the image points (6) in the reference area (16) are determined according to a predetermined depth determination procedure, which describe a respective distance to the camera unit (3), - the respective spatial positions of the image points (6) in relation to the camera unit (3) are determined by the detection device (2) from the image point coordinates (7) and the image point depths (18) according to a predetermined transformation procedure, and - by means of the detection device (2) a spatial camera position (15) of the horizontal surface (14) is determined from the spatial positions of the image points (6) according to a predetermined topography determination method, characterized in that - the pixel depths (18) are determined in the predetermined depth estimation procedure using a mono-depth method, wherein the mono-depth method is a machine learning method that enables the estimation of the pixel depth (18) in a single two-dimensional image.

Description

[0001] The invention relates to a method for determining the orientation of a camera unit of a detection device of a vehicle, a detection device and a vehicle.

[0002] Vehicle detection systems are required, for example, for advanced driver assistance systems that can support a driver in controlling the vehicle. Such driver assistance systems include adaptive cruise control, collision avoidance systems, lane change and lane keeping assist systems, and parking assistance systems. In particular, for autonomous driving systems of levels 2 to 5, accurate detection of the vehicle's surroundings by such systems is essential.

[0003] The detection systems incorporate various types of sensors capable of monitoring the vehicle's surroundings. These sensors can include, for example, lidar, radar, ultrasound, or camera units. Camera units are the most commonly used sensors in detection systems for advanced driver assistance systems. This is because camera units provide data for various systems that can be processed relatively easily.

[0004] For the use of camera units in detection systems, calibration is essential. Calibration involves linking the position of a camera unit to the vehicle. This allows the positions captured by the camera unit to be transferred from the camera unit's coordinate system to the vehicle's coordinate system. The vehicle's coordinate system is also known as the world coordinate system. To enable this transformation between the two coordinate systems, a rotation between the camera coordinate system and the world coordinate system must be defined. This rotation can be described using rotation matrices or Euler angles. When using Euler angles, the roll, pitch, and yaw angles are commonly employed. These relate to the longitudinal, transverse, and vertical axes of the camera and the vehicle, respectively.A so-called static calibration is typically performed at the factory at the end of the vehicle's manufacturing process. This calibration is carried out, for example, using either a static calibration or a short-track calibration. Static calibration involves calibration using markers in the vehicle's environment, while the vehicle remains stationary. Knowing the position of the markers and the vehicle allows for the determination of the transformation that converts the camera coordinate system into the world coordinate system.

[0005] Short-range calibration is a calibration in which the vehicle is driven at a speed of approximately 10 km / h along a straight, predetermined route with markings.

[0006] Calibration can be achieved by determining and tracking the coordinates of the markers in the world coordinate system and the camera coordinate system.

[0007] During vehicle operation, recalibration may be necessary. This means that a transformation converting the pixel coordinates into world coordinates must be adapted. This may be required, for example, due to changes in the rotation component and / or the camera's position caused by vehicle dynamics. This can be due to chassis movement caused by uneven road surfaces or sudden braking, or due to changes in the camera's angle and position resulting from varying loads within the vehicle. In the case of so-called highly dynamic calibrations, the greatest change in the orientation of a front camera involves the tilt angle. In a second case, also known as dynamic calibration, the greatest change in the front camera's orientation involves both the tilt angle and the roll angle.In both cases, essentially no change in camera position is expected. Therefore, only an adjustment of the rotation component is necessary.

[0008] Several methods are known in the art that enable highly dynamic or dynamic calibration of camera units. For example, a common calibration method is based on the detection of features on a road and their tracking across multiple images. These features could be, for example, road markings.

[0009] US Patent 2009 / 0290032 A1 discloses a method for calibrating a camera on a vehicle while it is in motion. During the process, the camera captures a sequence of images as the vehicle moves. The vehicle identifies at least two feature points, for example, along lane markings, in at least two camera images. The method determines a camera translation between two camera positions and a ground plane in camera coordinates based on the corresponding features from the images and the camera translation direction. The method also determines the camera's height above the ground and its rotation in vehicle coordinates. This calibration requires the capture of at least two images.

[0010] In DE 10 2019 008 081 A1 a method is disclosed in which the camera calibration is carried out by detecting the blurriness of the camera.

[0011] This requires recognizing objects in multiple images in order to calculate the blur and perform the calibration.

[0012] DE 10 2014 118 989 A1 discloses a method for calibrating a camera system. One camera of the camera system is configured as a plenoptic camera.

[0013] DE 10 2016 104 729 A1 discloses a method for the extrinsic calibration of a camera.

[0014] DE 11 2016 006 213 T5 discloses a method for fusing measurements from sensors that have different resolutions.

[0015] DE 10 2020 131 323 A1 discloses an automatic calibration and validation method to estimate and evaluate the accuracy of extrinsic parameters of a camera-to-LiDAR coordinate transformation.

[0016] DE 10 2021 113 105 A1 discloses techniques for modifying and training a neural network.

[0017] It is an object of the invention to provide a method and a device for calibrating a camera unit of a vehicle, enabling faster and more robust calibration of the camera unit.

[0018] The problem is solved by the subject matter of the independent patent claims. Advantageous further developments of the invention result from the features of the dependent patent claims, the following description, and the figures.

[0019] The invention relates to a method for determining the orientation of a camera unit of a vehicle's detection device with respect to a horizontal surface in the vehicle's surroundings. In other words, the method provides for determining the orientation of the camera unit of the vehicle's detection device with respect to the horizontal surface located in the vehicle's surroundings. The horizontal surface can be, for example, a surface on which the vehicle is positioned, in particular a road surface. The camera unit can be arranged at a predetermined position on the vehicle.

[0020] In the first step, the camera unit of the detection device captures an image of the vehicle's surroundings. In other words, the camera unit records an image of the vehicle's environment, depicting a specific area. The camera unit could be, for example, a front or rear camera positioned at a predetermined location on the vehicle. The image contains pixels arranged at specific coordinates within the image. In other words, the image is a representation of pixels located at specific coordinates within the image. These coordinates could, for example, represent a row and a column within the image.The system is designed to use a predefined recognition method to identify a reference area within the surrounding image, representing the horizontal surface. In other words, the system is designed to determine the reference area within the surrounding image that contains the horizontal surface located in the vehicle's environment. This recognition method can be a semantic segmentation technique that uses state-of-the-art deep learning algorithms to identify reference areas, such as roads or other horizontal surfaces.

[0021] The detection device determines the pixel depths of at least some of the pixels in the reference area using a predetermined depth estimation method. These pixel depths describe the respective distance of the pixel from the camera unit. In other words, the distance of a pixel from the camera unit is determined according to the predetermined depth estimation method. This predetermined depth estimation method can be a state-of-the-art monocular depth estimation method based on deep learning algorithms.

[0022] The acquisition device uses a predetermined transformation procedure to determine the spatial positions of the pixels relative to the camera unit from the pixel coordinates and pixel depths. In other words, the acquisition device performs a predetermined transformation procedure that determines the spatial positions of the pixels in the vehicle's surroundings relative to the camera unit from the pixel coordinates and pixel depths of the respective pixels. The predetermined transformation procedures can include projection equations that incorporate ray paths and lens curvatures to determine the pixel coordinates in the surroundings relative to the camera. The respective spatial positions of the pixels can then be output, for example, in a Cartesian coordinate system relative to the camera unit.In one step, the acquisition device determines the spatial camera position of the horizontal surface from the spatial positions of the image points using a predetermined topography determination method. In other words, the position of the horizontal surface is determined from the positions of the image points located within the reference area that represents the horizontal surface, according to the topography determination method. This topography determination method can, for example, be configured to determine the horizontal surface by simply connecting the respective image points. Alternatively, the horizontal surface can be determined using a fitting method or known methods for connecting point clouds. The invention offers the advantage that the position of the horizontal surface can be determined from a single image.This makes it possible to derive the position of the camera unit in relation to its horizontal surface and to output the spatial camera position using a single image.

[0023] The pixel depths are determined using a monodepth method in the predetermined depth estimation procedure. In other words, the predetermined depth estimation procedure incorporates a monodepth method. A monodepth method is a machine learning technique that enables the estimation of depth in a single two-dimensional image. The machine learning algorithm is typically trained using stereo images. It is also possible to train the algorithm using data from another 3D sensor, such as a LiDAR, or using data from structure-from-motion methods. The machine learning algorithm can be supervised or unsupervised. Preferably, the machine learning algorithm is a neural network. At the end of the training, the machine learning algorithm is able to output depth values ​​for each pixel of a single two-dimensional image.The advantage of this method is that the depth information is calculated from only a two-dimensional image. Another advantage of such a machine learning method is that no data from a three-dimensional sensor is required.

[0024] The invention also includes further developments that result in additional advantages.

[0025] A further development of the invention provides that the reference area comprises a predetermined reference pattern, which is predefined with respect to the camera unit or the vehicle. In other words, the pixels to be used are specified by the predetermined reference pattern. The predetermined reference pattern can define points, lines, and / or areas, which can be defined with respect to the camera unit or the vehicle. It can therefore be provided that the reference pattern defines a predetermined area in front of the vehicle. The predetermined area can be transformed into the pixel coordinates of the image, so that the corresponding area in the image is used as the reference area. This further development offers the advantage that at least some of the pixels are specified by the reference pattern. This eliminates the need for a computationally intensive acquisition method for selecting the pixels, at least for these pixels.

[0026] A further development of the invention provides that the predetermined reference pattern includes at least one reference line running parallel to a transverse direction of the vehicle. In other words, the predetermined reference pattern has a reference line that is aligned parallel to the transverse direction of the vehicle in world coordinates. The reference line running parallel to the transverse direction of the vehicle can be transformed into pixel coordinates and displayed as a horizontal line in the image. This further development offers the advantage that it is possible to detect a curvature of a roadway perpendicular to the vehicle.

[0027] A further development of the invention provides that the predetermined reference pattern includes at least one reference line running parallel to a longitudinal direction of the vehicle. In other words, the predetermined reference pattern has a reference line defined in world coordinates with respect to the longitudinal direction of the vehicle. In the image, the reference line can be represented as a line extending to a vanishing point of the image. This has the advantage that an incline or decline in the direction of travel of the vehicle can be detected.

[0028] A further development of the invention provides that the horizontal surface is determined as a plane. In other words, it is provided that the horizontal surface is approximated as a plane. For example, it can be provided that at least three image points are acquired and, based on the coordinates of these at least three image points, the horizontal surface is simplified and approximated as a plane. The topography determination method can include known methods of linear algebra for determining the position of the plane from the positions of points in the plane. For example, it can be provided that a plane is determined from the positions of the image points, which is then approximated to the positions of the image points using a predetermined fitting procedure. The plane can be described with respect to its position relative to the camera unit, so that a spatial camera position of the horizontal surface relative to the camera unit can be described.This offers the advantage of eliminating the need for complex fitting procedures.

[0029] A further development of the invention provides that the determined spatial camera position is compared with a reference camera position, and a predetermined error signal is output if a predetermined error criterion is met due to a relationship between the camera position and the reference camera position. In other words, it is provided that the spatial camera position of the camera unit, determined from the surrounding image with respect to the horizontal surface, is compared with the reference camera position. The reference camera position can describe a predetermined spatial position of the camera unit with respect to the horizontal surface, which may have been determined in a predetermined calibration procedure. This further development offers the advantage that the method can detect a change in the camera position with respect to the horizontal surface.

[0030] A further development of the invention provides that the predetermined error criterion includes a deviation of a rotation component of the camera position from a rotation component of the reference camera position. In other words, it is provided that the predetermined error criterion is fulfilled if the rotation component of the camera position deviates from the rotation component of the reference camera position by a certain range of values.

[0031] A further development of the invention provides that the predetermined error criterion includes a deviation of a roll angle and / or a pitch angle of the camera position from a rotation of the reference camera position. In other words, the predetermined error criterion is fulfilled when the roll angle and / or the pitch angle of the camera unit has changed by a certain range of values ​​with respect to the position of the horizontal surface.

[0032] A further development of the invention provides that the reference camera position is replaced by the determined camera position when the predetermined error criterion is met. In other words, it is provided that the reference camera position is overwritten by the determined camera position if the error criterion is met.

[0033] A further development of the invention provides that the reference area of ​​the environment image is detected in the predetermined recognition method by means of a predetermined semantic segmentation of the environment image. In other words, it is provided that the environment image is evaluated by the detection device using the semantic segmentation method, whereby the reference area of ​​the environment image is determined according to the predetermined semantic segmentation method, which comprises pixels that are assigned to the horizontal surface.

[0034] A further development of the invention provides that the method includes capturing respective environmental images from the respective camera units of the capture device. In other words, the capture device is provided for to have several camera units, with environmental images from the respective camera units being captured during the method. It can be provided that the determination of the camera positions of the respective camera units is based on the respective environmental images.

[0035] The invention comprises a detection device configured to determine the orientation of a camera unit of the detection device with respect to a horizontal surface in the vehicle's surroundings. The camera unit of the detection device is configured to capture an image of the vehicle's surroundings, wherein the image contains pixels arranged at respective pixel coordinates of the image. The detection device is configured to detect a reference area within the image, representing the horizontal surface, according to a predetermined recognition method.The acquisition device is configured to determine, according to a predetermined depth determination procedure, the pixel depths of at least some of the pixels in the reference area, which describe a respective distance to the camera unit, and, from the pixel coordinates and the pixel depths, to determine the respective spatial positions of the pixels in relation to the camera unit using a predetermined transformation procedure. The acquisition device is also configured to determine, from the spatial positions of the pixels, a spatial camera position on the horizontal surface using a predetermined topography determination procedure.

[0036] The pixel depths are determined using a monodepth method in the predetermined depth estimation procedure. In other words, the predetermined depth estimation procedure incorporates a monodepth method. A monodepth method is a machine learning technique that enables the estimation of depth in a single two-dimensional image. The machine learning algorithm is typically trained using stereo images. It is also possible to train the algorithm using data from another 3D sensor, such as a LiDAR, or using data from structure-from-motion methods. The machine learning algorithm can be supervised or unsupervised. Preferably, the machine learning algorithm is a neural network. At the end of the training, the machine learning algorithm is able to output depth values ​​for each pixel of a single two-dimensional image.The advantage of this method is that the depth information is calculated from only a two-dimensional image. Another advantage of such a machine learning method is that no data from a three-dimensional sensor is required.

[0037] The detection device can include a data processing device or a processor unit configured to perform an embodiment of the method according to the invention. For this purpose, the processor unit can include at least one microprocessor and / or at least one microcontroller and / or at least one FPGA (Field Programmable Gate Array) and / or at least one DSP (Digital Signal Processor). Furthermore, the processor unit can include program code configured to perform the embodiment of the method according to the invention when executed by the processor unit. The program code can be stored in a data memory of the processor unit.

[0038] The invention also includes a vehicle having a detection device. The vehicle according to the invention is preferably designed as a motor vehicle, in particular as a passenger car or truck, or as a passenger bus or motorcycle.

[0039] The invention also includes further developments of the detection device and the vehicle according to the invention, which have features already described in connection with the further developments of the method according to the invention. For this reason, the corresponding further developments of the detection device and the vehicle according to the invention are not described again here.

[0040] The invention also includes combinations of the features of the described embodiments. The invention therefore also includes realizations that each exhibit a combination of the features of several of the described embodiments, provided that the embodiments have not been described as mutually exclusive.

[0041] The following are exemplary embodiments of the invention described. This is illustrated by: Fig. 1 a schematic representation of a vehicle that has a detection device, Fig. 2 a schematic representation of an environment image taken by a camera unit of a vehicle; Fig. 3. Another schematic representation of an environment; and Fig. 4. A schematic representation of the sequence of a procedure.

[0042] The exemplary embodiments described below are preferred embodiments of the invention. In these exemplary embodiments, the described components each represent individual features of the invention, which can be considered independently of one another and each further develops the invention independently. Therefore, the disclosure is intended to include combinations of features of the embodiments other than those shown. Furthermore, the described embodiments can also be supplemented by further features of the invention already described.

[0043] In the figures, identical reference symbols denote functionally equivalent elements.

[0044] Fig. Figure 1 shows a schematic representation of a vehicle 1 having a detection device 2. The detection device 2 can have two camera units 3, which can be configured to capture respective environmental images 4 of the vehicle's surroundings 5. The detection device 2 can have a data processing device 19 or a processor unit configured to carry out an embodiment of the method according to the invention. The environmental images 4 can have pixels 6, which can have respective pixel coordinates 7 in the environmental image 4. It can be provided that the detection device 2 is configured to provide the environmental images 4 captured by the camera units 3 to a driver assistance system 8, which is configured to at least partially control the vehicle 1 or to issue predetermined signals to the driver.In order to detect objects 9 and their position relative to vehicle 1, it is necessary to calibrate the camera units 3 relative to vehicle 1. Calibration is performed by generating a reference camera position 11, which describes the position of each camera unit 3 relative to a world coordinate system 12 of the vehicle's environment 5. Using the reference camera position 11, the detected objects 9 can be transformed from a camera coordinate system 13 of the camera unit 3 into the world coordinate system 12. This may be necessary, for example, if a recording of the surroundings 4 with guide lines indicating a path to be followed is to be displayed on a screen of vehicle 1. The calibration may have been performed during the manufacturing of vehicle 1.It can be assumed that the vehicle 1 is located on a horizontal surface 14, which may have a predetermined position relative to the vehicle 1. However, during operation of the vehicle 1, the position of the horizontal surface 14 relative to the vehicle 1 may change. This can occur, for example, when the vehicle 1 accelerates, brakes, and / or steers. Due to these driving maneuvers, the position of the camera unit 3 relative to the surface 14 may change. This change may include, for example, a rotation, such that a rotation matrix, pitch angle, or roll angle of the horizontal surface 14 may change relative to the camera unit 3. Therefore, it may be necessary to acquire a new camera position 15 and use this instead of the reference camera position 11.To determine the position of the horizontal surface 14, the detection device 2 may be configured to capture a reference area 16 in which the horizontal surface 14 is visible in the surrounding image 4. Additionally or alternatively, the reference area 16 may include a predetermined reference pattern 24, which is predefined with respect to the camera unit 3 or the vehicle 1. The predetermined reference pattern 24 may, for example, describe points, reference lines 22, or surfaces whose positions relative to the camera unit 3 or the vehicle 1 may be predefined in world coordinates 17. The positions relative to the camera unit 3 or the vehicle 1 in the world coordinates 17 may be transformed into pixel coordinates 7 to describe their position in the surrounding image 4.This offers the advantage that at least some of the pixels 6 do not need to be selected using complex procedures but can be predetermined by the reference pattern 24. The reference pattern 24 can, for example, include at least one reference line 22 running parallel to a transverse direction of the vehicle 1. This makes it possible, for instance, to detect a camber in the road perpendicular to a direction of travel. The reference line 22 running parallel to a transverse direction of the vehicle 1 can be represented in the environment image 4 as a horizontal reference line 22, on which at least some of the pixels 6 can be arranged. The predetermined reference pattern 24 can also include at least one reference line 22 running parallel to a longitudinal direction of the vehicle 1.In the environment image 4, at least one reference line 22 running parallel to the longitudinal direction of the vehicle 1 can appear as a line that converges to a vanishing point of the environment image 4. This allows, for example, the determination of whether the horizontal surface 14 rises, falls, or is curved in a direction of travel of the vehicle 1. The detection device 2 can select certain image points 6 that lie within the reference area 16 and determine the image depths 18 of the respective image points 6 according to a predetermined image point depth determination procedure. According to a predetermined transformation procedure, the detection device 2 can determine the position of the image points 6 as world coordinates 17 in the world coordinate system 12 and approximate the horizontal surface 14 to the selected image points 6 according to a predetermined topography determination procedure.In a simple case, the topography determination method can be designed as a procedure configured to determine the horizontal surface 14 as a plane that approximates the world coordinates 17 of the image points 6. In this embodiment, only a planar plane is determined, which can enable a simple and fast determination of the horizontal surface 14. Further embodiments of the topography determination method can include fitting procedure steps to, for example, determine polynomials for describing the horizontal surface 14. The position of the horizontal surface 14 relative to the camera unit 3 can be determined. It can be provided that the camera position 15 determined in this way is checked against the reference camera position 11 for the presence of a predetermined error criterion.

[0045] If the predetermined error criterion is met, the detection device 2 can output a predetermined error signal. Alternatively, a new calibration of the camera unit 3 can be performed, whereby the reference camera position 11 can be replaced by the determined camera position 15. The predetermined error criterion can describe a deviation of the reference position from the detected camera position 15. This can, for example, involve rotation and / or translation. For instance, the predetermined error criterion can be met if the pitch or roll angle of the determined camera position 15 differs from that of the reference camera position 11.

[0046] Fig. Figure 2 shows a schematic representation of an environment image 4 captured by a camera unit 3 of a vehicle 1. The camera unit 3 can capture a two-dimensional environment image 4 of the vehicle's surroundings 5. The vehicle's surroundings 5 ​​comprise a road surface or carriageway, a drivable area, and optionally one or more objects 9 at their respective positions 10. The drivable area is a sub-area of ​​the road surface. Hereinafter, we refer to the drivable area as the reference area 16 or ROI (Region of Interest). The orientation of the camera unit 3 with respect to the road surface, which can be the horizontal surface 14, can be assumed to be arbitrary. The vehicle 1 can move on the road surface. The reference area 16 can comprise the reference pattern 24, which may have two reference lines 22 along which predetermined pixels 6 may be arranged.

[0047] In a preferred embodiment, the traversable area can be determined by a semantic segmentation method. The two-dimensional reference area 16 can be used as input for the predetermined depth determination method, which can be a machine learning algorithm. The depth determination method assigns respective pixel depths 18 to at least a subset of the pixels 6 of the two-dimensional reference area 16. In other words, a depth map 20 is estimated by assigning the distance of the pixel 6 to the camera unit 3 as a respective pixel depth 18 to at least a subset of the pixels 6 in the reference area 16. In a preferred embodiment, the machine learning algorithm assigns the respective pixel depth 18 to each pixel 6 of the two-dimensional reference area 16.

[0048] From the pixel depths 18 of the pixels 6 of the original two-dimensional reference area 16, a depth map 20 can be formed, which can describe an area or a depth profile. It is then possible to fit a plane, which can describe the horizontal surface 14, to the pixels 6 of the reference area 16 in the world coordinate system 12 and to determine the tilt and roll angle of the camera unit 3 as the relative orientation of the world coordinate system 12 to the camera coordinate system 13.

[0049] The relationship between image points 6 in the camera coordinate system 13 and image points 6 in the world coordinate system 12 is given by; Pc=R(Pw−T) PC pixel in camera coordinate system Pw image point to be transformed in world coordinate system R Rotation matrix T translation matrix

[0050] The rotation matrix can be written as follows. R=(sinαpsinαr−cosαr−cosαpsinαrsinαpcosαrsinαr−cosαrcosαpcosαp0sinαp)

[0051] where αp is the tilt angle and αr is the roll angle.

[0052] The values ​​of the pitch and roll angles determined in this way are then compared with the static values ​​for pitch and roll angles: If the two values ​​do not match, a miscalibration is detected and a calibration is performed if necessary.

[0053] However, if a camera unit 3 is incorrectly calibrated, the camera data is unreliable and cannot be used to provide usable information to the ADAS systems. For these reasons, camera calibration is an essential task in ADAS systems. A vehicle 1 typically has several camera units 3: a front camera, one or more side cameras, and a rear camera. In many detection systems 2, the front camera is a monocular camera. Although we will focus on calibration for a front camera, specifically a monocular camera, the concept can also be applied to other camera units 3, particularly a rear-view camera.

[0054] The calibration consists of evaluating an angular rotation which can be expressed in the form of a matrix between a camera coordinate system 13, which is assigned to the camera unit 3, and a world coordinate system 12, which refers to the world / vehicle environment 5.

[0055] This rotation can be subdivided into three elementary rotations, each defined by the angles pitch, roll, and yaw. Initial calibration of camera unit 3 can be performed at the factory at the end of the manufacturing process using a static or short-track calibration. "Static calibration" refers to camera calibration performed using markers while the vehicle 1 is stationary.

[0056] "Short distance" refers to a calibration that takes place while vehicle 1 slowly travels at a speed of approximately 10 km / h along a straight road marked with road markings, starting from a line assembly and ending at a parking space. By capturing the markers during the calibration, the reference camera position 11 can be determined, which establishes a relationship between the camera coordinate system 13, assigned to the camera unit 3, and a world coordinate system 12, which relates to the world / vehicle environment 5.

[0057] It may also be necessary to perform a recalibration during driving, for example due to changes in the angles and position of camera unit 3 due to vehicle dynamics, such as movements of the chassis due to uneven road surfaces or sudden braking, or due to changes in the angles and position of camera unit 3 due to different loads in vehicle 1.

[0058] In the first case, highly dynamic calibration, the greatest change in the front camera's tilt angle is related to the tilt angle. In the second case, dynamic calibration, the greatest change in the front camera's tilt angle is related to both the tilt angle and the roll angle. In both cases, essentially no change in the camera's position on vehicle 1 is to be expected.

[0059] Fig. Figure 2 shows an environment image captured by a camera unit 3 of a vehicle 1. The environment image 4 is two-dimensional and shows a road bounded by curbs. The environment image 4 shows examples of objects 9, such as trees, parked cars, and buildings. The drivable area is defined as the free space in front of the vehicle. The pixels 6 in the reference area 16 are the pixels 6 for which a depth value is determined. These pixels 6 can then be used to fit a two-dimensional surface 14 or plane into the XYZ world coordinate system 12.

[0060] A determined horizon line 21 defines the horizon as calculated using the previously explained method, while horizon line 23 represents the statically calibrated horizon. The reference lines 22 on the roadway represent alignment lines: it is clear that the determined horizon line 21 is compatible with the alignment lines.

[0061] The horizon does not need to be visible to perform the procedure described above: it can be obscured by other objects 9 without affecting the results of this method. This means it is still possible to determine the roll and pitch angles of the camera unit 3. This method works particularly well for miscalibrations caused by sudden braking.

[0062] The term surface 14 or ground plane refers to a real plane parallel to the roadway. The term image plane refers to a two-dimensional space provided as output by camera unit 3, which views a real three-dimensional space. The term plane at infinity refers to all points at infinity and to a plane perpendicular to the ground plane. The term horizon line 21, 23 refers to a straight line where the ground plane and a plane parallel to the ground plane intersect at infinity. The term vanishing point refers to a point where parallel lines in the ground plane appear to converge in an image plane. If camera unit 3 is centered between two parallel lines in the ground plane that are parallel to the optical axis of camera unit 3, the point of intersection of the two parallel lines is called the central vanishing point.The term "vanishing line" describes a location with estimated vanishing points.

[0063] Fig. Figure 3 shows a further schematic representation of an environment image 4, which was captured by a camera unit 3 of a vehicle 1. In the environment image 4, the two reference lines 22a, 22b can be seen, which run parallel to the longitudinal axis of the vehicle 1. In the environment image 4, these lines converge on a common vanishing point. The pixels 6 are arranged along the two reference lines 22a, 22b. In the depth map 20, the respective elevation profiles 25a, 25b of the surface along the reference lines 22a, 22b are shown. The elevation profile 25a results from a static calibration, the elevation profile 25b from the described procedure.

[0064] Fig.Figure 4 shows a schematic representation of the sequence of a procedure. The procedure can be used to determine the orientation of a camera unit 3 of a detection device 2 of a vehicle 1 with respect to a horizontal surface 14 in a vehicle environment 5 of the vehicle 1.

[0065] In step P1, the camera unit 3 of the detection device 2 can capture an environmental image 4 of the vehicle environment 5, wherein the environmental image 4 has pixels 6 that are arranged at respective pixel coordinates 7 of the image.

[0066] In step P2, the detection device 2 can detect a reference area 16 in the environment image 4 according to a predetermined recognition procedure, which represents the horizontal surface 14.

[0067] In step P3, the detection device 2 can determine pixel depths 18 of at least some of the pixels 6 in the reference area 16 according to a predetermined depth determination procedure, which describe a respective distance to the camera unit 3.

[0068] In step P4, the respective spatial positions of the image points 6 in relation to the camera unit 3 can be determined by the acquisition device 2 from the image point coordinates 7 and the image point depths 18 according to a predetermined transformation procedure.

[0069] In step P5, the acquisition device 2 can determine a spatial camera position 15 of the horizontal surface 14 from the spatial positions of the image points 6 according to a predetermined topography determination procedure.

[0070] Overall, the examples show how the camera position relative to the road surface can be determined from a single two-dimensional image of the surroundings.

Claims

[1] Method for determining the orientation of a camera unit (3) of a detection device (2) of a vehicle (1) with respect to a horizontal surface (14) in a vehicle environment (5) of the vehicle (1), wherein - the camera unit (3) captures an image (4) of the vehicle's surroundings (5), whereby - the environment image (4) has image points (6) that are arranged at respective image point coordinates (7) of the environment image (4), - by the detection device (2) in the environment image (4) a reference area (16) is detected according to a predetermined recognition procedure, which represents the horizontal surface (14), - by means of the detection device (2) image point depths (18) of at least some of the image points (6) in the reference area (16) are determined according to a predetermined depth determination procedure, which describe a respective distance to the camera unit (3), - the respective spatial positions of the image points (6) in relation to the camera unit (3) are determined by the detection device (2) from the image point coordinates (7) and the image point depths (18) according to a predetermined transformation procedure, and - by means of the detection device (2) a spatial camera position (15) of the horizontal surface (14) is determined from the spatial positions of the image points (6) according to a predetermined topography determination procedure, characterized by , that - the pixel depths (18) are determined in the predetermined depth estimation procedure using a mono-depth method, wherein the mono-depth method is a machine learning method that enables the estimation of the pixel depth (18) in a single two-dimensional image. [2] Method according to claim 1, characterized by, that the reference area (16) includes a predetermined reference pattern (24) which is predetermined with respect to the camera unit (3) or the vehicle (1). [3] Method according to claim 2, characterized by , that the predetermined reference pattern (24) includes at least one reference line (22) running parallel to a transverse direction of the vehicle (1). [4] Method according to claim 2 or 3, characterized by , that the predetermined reference pattern (24) includes at least one reference line (22) running parallel to a longitudinal direction of the vehicle (1). [5] Method according to any one of the preceding claims, characterized by , that the horizontal surface (14) is determined to be a plane. [6] Method according to any one of the preceding claims, characterized by, that the spatial camera position (15) is compared with a reference camera position (11), and a predetermined error signal is output if a predetermined error criterion is met by a relation between the camera position (15) and the reference camera position (11). [7] Method according to claim 6, characterized by , that the predetermined error criterion includes a deviation of a rotation component of the camera position (15) from a rotation component of the reference camera position (11). [8] Method according to claim 7, characterized by , that the predetermined error criterion includes a deviation of a roll angle and / or a pitch angle of the camera position (15) from a rotation of the reference camera position (11). [9] Method according to any one of claims 6 to 8, characterized by , that the reference camera position (11) is replaced by the spatial camera position (15) when the predetermined error criterion is met. [10] Method according to any one of the preceding claims, characterized by , that the reference area (16) of the environment image (4) is captured in the predetermined recognition procedure by means of a predetermined semantic segmentation of the environment image (4). [11] Method according to any one of the preceding claims, characterized by , that the procedure includes the acquisition of respective environmental images (4) by at least two camera units (3) of the acquisition device (2). [12] Detection device (2), configured to determine the orientation of a camera unit (3) of a detection device (2) of a vehicle (1) with respect to a horizontal surface (14) in a vehicle environment (5) of the vehicle (1), wherein - the camera unit (3) of the detection device (2) is configured to capture an image (4) of the vehicle's surroundings (5), wherein - the surrounding image (4) has image points (6) that are arranged at respective image point coordinates (7) of the image, - the detection device (2) is configured to detect a reference area (16) in the environment image (4) according to a predetermined detection procedure, which represents the horizontal surface (14), - the detection device (2) is set up to determine pixel depths (18) of at least some of the pixels (6) in the reference area (16) according to a predetermined depth determination procedure, which describe a respective distance to the camera unit (3), - the detection device (2) is configured to determine, from the image point coordinates (7) and the image point depths (18) according to a predetermined transformation procedure, the respective spatial positions of the image points (6) in relation to the camera unit (3), and to determine, from the spatial positions of the image points (6) according to a predetermined topography determination procedure, a spatial camera position (15) of the horizontal surface (14), characterized by , that - the pixel depths (18) are determined in the predetermined depth estimation procedure using a mono-depth method, wherein the mono-depth method is a machine learning method that enables the estimation of the pixel depth (18) in a single two-dimensional image. [13] Vehicle (1) comprising a detection device (2) according to claim 12.

Images