Method and system for calculating a vehicle trailer angle

Abstract

The invention relates to a method for determining a yaw angle (YA) of a trailer (2) relative to a longitudinal axis (LAV) of a towing vehicle (1) on the basis of at least one feature (F1, F2) contained in a plurality of captured images by providing a plurality of algorithms for calculating at least one angle estimate (α1, α2).

Description

TECHNICAL FIELD

[0001] The present invention relates generally to the field of vehicle assistance systems. More specifically, the present invention relates to a method and a system for calculating a yaw angle of a trailer associated with a towing vehicle based on image information provided by a camera of the vehicle. BACKGROUND

[0002] Methods for calculating a yaw angle of a trailer with respect to a towing vehicle based on image information provided by a camera of the vehicle are known.

[0003] In particular, a first type of method is known which provides a reliable approximation of the yaw angle of the trailer without taking into account the position of the pivot point of the trailer. However, the accuracy of the yaw angle approximation of the first type of method is low in the case of large yaw angles.

[0004] Furthermore, a second type of method is known which takes into account the position of the pivot point of the trailer. However, the second type of method is affected by high noise in certain cases. SUMMARY

[0005] It is an object of embodiments of the present invention to provide a method for calculating a yaw angle of a trailer which has a higher robustness and a higher reliability. This task is solved by the features of the independent claims. Preferred embodiments are specified in the dependent claims. Embodiments of the present invention can be freely combined with each other as long as nothing else is explicitly stated.

[0006] According to an invention, the present invention relates to a method for determining a yaw angle of a trailer with respect to a longitudinal axis of a towing vehicle. The method comprises the following steps:

[0007] First, at least a first and a second image of the trailer are taken by means of a camera. The method of taking the first and the second image is such that the trailer is different with respect to the direction of the vehicle in at least two images.

[0008] After taking the images, at least a first feature of the trailer is determined which has to be visible in the first and the second image.

[0009] Furthermore, based on the at least first feature, at least a first and a second algorithm for calculating an angle estimate are provided. The first algorithm can be an algorithm which investigates the geometry of the rotation of the trailer, in particular, which takes into account the position of the pivot point of the trailer. The second algorithm can be an algorithm which approximates the yaw angle of the trailer by considering two or more features of the trailer in different rotational positions. The second algorithm can not investigate the geometry of the rotation of the trailer.

[0010] Based on the first and / or the second algorithm, a first angle estimate is provided for the at least first feature.

[0011] According to a first embodiment, the provision of the first angle estimate is implemented based on one or more criteria per feature deciding whether to use the first or the second algorithm to calculate the first angle estimate and based on the algorithm selected for the first feature to calculate the first angle estimate. In other words, the decision which algorithm to use is made before the angle estimate is calculated.

[0012] According to a second embodiment, the provision of the first angle estimate is implemented in such a way that a first angle estimate is calculated for at least one feature based on the first and the second algorithm and that for each feature it is decided based on one or more criteria whether the first angle estimate obtained by the first algorithm or the first angle estimate obtained by the second algorithm is to be used for further processing. In other words, in the second embodiment, the first angle estimate is calculated based on a plurality of algorithms and the decision which angle estimate to use is made by taking into account the results of the calculation of the angle estimates. In contrast to the first embodiment described above, the decision which algorithm to use is made after the first angle estimate is calculated.

[0013] Finally, a yaw angle of the trailer is calculated based on at least the first angle estimate.

[0014] The method is advantageous in that by making a decision for each feature which of a plurality of methods should be used to provide an angle estimate, the result of the determination of the yaw angle is very accurate and highly robust, since for each feature a suitable method can be selected which provides the most reliable result of the angle estimate.

[0015] According to an embodiment, an attempt is made to calculate an angle estimate for each feature based on the first algorithm, i.e. the algorithm which makes use of the geometry of the rotation of the trailer. If the first algorithm fails due to one or more criteria not being met, in particular predefined criteria, the second algorithm is used.

[0016] According to an embodiment, the first algorithm is configured to perform the following steps:

[0017] - a first ray between the camera device and the first feature determined on the first image is set and projected onto a horizontal plane, thereby obtaining a first projected feature position. Likewise, a second ray between the camera device and the first feature determined on the second image is set and projected onto the horizontal plane, thereby obtaining a second projected feature position.

[0018] - a first perpendicular bisector is set up between the first projected feature position and the second projected feature position. In more detail, the first perpendicular bisector can be a perpendicular line through the center of the connecting line of the first projected feature position and the second projected feature position. Thereby, the first perpendicular bisector can be set up on the horizontal plane.

[0019] - after setting the first perpendicular bisector, setting a first intersection point of the first perpendicular bisector with a reference axis or another perpendicular bisector.

[0020] - calculating a first angle estimate based on the first perpendicular bisector. The first angle estimate is an angle in the horizontal plane between a first line from the first projected feature position to the first intersection point and a second line from the second projected feature position to the first intersection point.

[0021] It is worth mentioning that the term "position of the first feature on the first / second image" refers to a two-dimensional planar image coordinate of the image feature, or a corresponding ray (e.g. given as a three-dimensional unit vector or azimuth / elevation angle). The two-dimensional planar image coordinate can be converted to a ray using camera calibration information.

[0022] The first angle estimate can open in a direction from the towing vehicle towards the trailer.

[0023] According to an embodiment, a second feature of the trailer is determined to be visible on the first and second images, wherein the second feature is arranged at another trailer position different from the first feature. The first algorithm uses two or more trailer features for providing an angle estimate. In particular, the first algorithm is configured to:

[0024] - project a ray between the camera and the determined second feature on the first image onto the horizontal plane, thereby obtaining a third projected feature position, and project a ray between the camera and the determined second feature on the second image onto the horizontal plane, thereby obtaining a fourth projected feature position.

[0025] - establish a second perpendicular bisector between the third projected feature position and the fourth projected feature position;

[0026] - determine a second intersection point of the second perpendicular bisector with the reference axis, the first perpendicular bisector or another perpendicular bisector; and

[0027] - calculate a second angle estimate, wherein the second angle estimate is an angle in the horizontal plane between a first line from the third projected feature position to the second intersection point and a second line from the fourth projected feature position to the second intersection point.

[0028] Using two or more features can result in multiple angle estimates based on different features, thereby reducing the influence of noise and mismatches.

[0029] According to an embodiment, the second algorithm is configured to calculate a first angle estimate. The first angle estimate characterizes a swing angle in the horizontal plane between the first feature on the first image and the first feature on the second image with respect to a fixed point of the towing vehicle.

[0030] According to an embodiment, a second trailer feature visible on the first and second images is determined, wherein the second feature is at a different trailer position than the first feature, and a second algorithm is configured to calculate a second angle estimate, wherein the second angle estimate characterizes a roll angle between the second feature on the first image and the second feature on the second image with respect to a fixed point of the towing vehicle on a horizontal plane. Using two or more features in the second algorithm, multiple angle estimates can be calculated based on different features, thereby reducing the influence of noise and mismatches.

[0031] According to an embodiment, the step of determining based on one or more criteria comprises determining a length of a baseline between a feature on the first image and the feature on the second image, and comparing the length to a length threshold. The baseline length can be determined after the first and second features detected on the first and second images are transferred into a common map system or coordinate system, which can further comprise information about the position of one or more fixed points of the vehicle, such as the position of the camera. If the baseline length is below the length threshold, the angle estimate is preferably provided by the second algorithm rather than the first algorithm. In this way, inaccurate angle estimates due to high noise can be avoided.

[0032] According to an embodiment, the step of determining based on one or more criteria comprises determining a roll angle and / or a pitch angle of the trailer, and comparing the roll angle to a roll angle threshold and / or the pitch angle to a pitch angle threshold. If the roll angle is above the roll angle threshold and / or the pitch angle is above the pitch angle threshold, the angle estimate is preferably provided by the second algorithm rather than the first algorithm. In this way, inaccurate angle estimates due to high noise can be avoided.

[0033] According to an embodiment, the step of determining based on one or more criteria comprises determining a perpendicular distance of a feature with respect to a horizontal reference plane, and comparing the perpendicular distance to a distance threshold. The perpendicular distance can be determined after at least one feature detected on the first and second images is transferred into a common map system or coordinate system. The coordinate system can further comprise information about the height level of one or more fixed points of the vehicle in the vertical direction, such as the height level of the camera, and in particular, for example, can comprise information about the height level of the camera with respect to the horizontal reference plane. If the perpendicular distance is below the distance threshold, the angle estimate is provided by the second algorithm rather than the first algorithm. In this way, inaccurate angle estimates due to high noise can be avoided.

[0034] According to an embodiment, the yaw angle of the trailer with respect to the vehicle is zero on the first or second image. This image can thus be used as a "zero-attitude image", i.e. as a reference in which the longitudinal axis of the vehicle is exactly aligned with the longitudinal axis of the trailer. However, if the further yaw angle is known, another yaw angle value can also be used as a reference value.

[0035] According to an embodiment, in the second algorithm, the calculation of the at least one angle estimate comprises determining a line of sight between the fixed point and the at least one feature on the first and second images. The line of sight refers to a straight line between the fixed point and the respective feature. Based on the line of sight, the computational effort for determining the current yaw angle can be reduced, e.g. based on a geometrical approach or the like.

[0036] According to an embodiment, especially in the second algorithm, camera calibration information is used for converting the position of the first and / or second feature into a line of sight. For example, in case the camera position is known using camera calibration information, the position of a certain feature on the image can be transmitted in the position information depending on the camera position or in association with the camera position.

[0037] According to an embodiment, camera calibration information is used for converting the position of the first and / or second feature from a local domain of the image into a local domain of the vehicle or into a certain fixed point of the vehicle. For example, in case the camera position is known using camera calibration information, the position of a certain feature on the image can be transmitted in the position information depending on the position of a camera comprised in or fixed to the vehicle or in association with the position of this camera.

[0038] According to an embodiment, at least one further feature of the trailer is used for calculating the yaw angle in addition to the at least one feature. The at least one further feature is arranged at a further position of the trailer different from the first feature position. For example, the first feature can be a distinct first feature at a first position and the second feature can be a distinct second feature at a second position of the trailer. Using two or more features results in a further angle estimate which further improves the robustness and reliability of the determination of the yaw angle.

[0039] According to an embodiment, the yaw angle is calculated by setting a median value based on the at least two angle estimates provided by the first and second algorithm. Thereby, a very stable determination of the yaw angle can be obtained.

[0040] According to other embodiments, the calculation of the yaw angle is implemented by setting a mean value of the at least two angle estimates or by using a statistical approach to the angle estimates provided by the first and second algorithm.

[0041] According to an embodiment, the method further comprises the step of determining an angle window. The angle window can comprise an upper and a lower limit of the yaw angle. Furthermore, a set of features is determined, wherein a feature of the set of features leads to an angle estimate that lies within the angle window. The determined set of features, preferably only the features comprised in the set of features, is used for future yaw angle calculations. In other words, information from previous yaw angle determinations is used to determine two or more features of the trailer that lead to an angle estimate that is in close proximity to the determined yaw angle (i.e. within the angle window), while features that lead to an angle estimate that is significantly deviating from the determined yaw angle (i.e. outside the angle window) are not tracked. The computational complexity and accuracy requirements of the angle estimate are thus significantly reduced.

[0042] According to an embodiment, the calculated yaw angle value is increased by a certain fraction or percentage to compensate for underestimation. For example, the calculated yaw angle can be increased by a factor of 5% to 15%, in particular by a factor of 10%, to compensate for underestimation of the calculation result.

[0043] According to an embodiment, in the first algorithm the reference axis is the longitudinal axis of the towing vehicle, if the camera and the towing hitch of the vehicle are arranged in a plane that comprises the vertical of the longitudinal axis of the towing vehicle.

[0044] According to another embodiment, in the first algorithm the reference axis is a straight line between the camera and the towing hitch, if the camera and / or the towing hitch have a lateral offset with respect to the longitudinal axis of the towing vehicle. Thereby, the lateral offset between the camera and the towing hitch can be compensated for.

[0045] According to an embodiment, the camera is a rear view camera of the vehicle. Based on a rear view camera, less technical effort is spent for taking the trailer image.

[0046] According to another aspect, a system for determining a yaw angle of a trailer with respect to a longitudinal axis of a towing vehicle is disclosed. The system comprises a camera for taking an image of the trailer and a processing entity for processing the taken image. Furthermore, the system is configured to perform the following steps:

[0047] - detecting at least a first and a second image of the trailer using the camera, wherein the orientation of the trailer with respect to the vehicle is different in at least two images.

[0048] - determining at least a first feature of the trailer that is visible in the first and the second image;

[0049] - providing at least a first and a second algorithm for calculating at least one angle estimate based on the at least first feature.

[0050] - providing at least one first angle estimate for the first feature by

[0051] o deciding, based on one or more criteria for each feature, whether to use the first algorithm or the second algorithm for calculating the first angle estimate, and calculating the first angle estimate based on the algorithm selected for the first feature; or

[0052] o calculating a first angle estimate for the first feature based on the first algorithm and the second algorithm, and deciding, based on one or more criteria, whether to use the first angle estimate obtained by the first algorithm or the first angle estimate obtained by the second algorithm for further processing; and

[0053] calculating a yaw angle based on the first angle estimate.

[0054] Any of the above described features described as an embodiment of the method can also be used as system features of the system according to the patent application published herein.

[0055] According to another embodiment, a vehicle comprising the system according to any of the above described embodiments is published.

[0056] The term "vehicle" used in the present invention can refer to a car, a truck, a bus, a rail vehicle or any other means of transportation.

[0057] The term "yaw angle" used in the patent application published herein can refer to the yaw angle between the longitudinal axis of the vehicle and the longitudinal axis of the trailer.

[0058] The term "median" used in the patent application published herein can refer to the value separating the higher half from the lower half of a data sample or a probability distribution.

[0059] The term "substantially" or "approximately" used in the present invention refers to a deviation of + / - 10%, preferably + / - 5% from the exact value and / or a deviation which is not important for the function and / or for the traffic rules. BRIEF DESCRIPTION OF DRAWINGS

[0060] The different aspects of the invention, including its particular features and advantages, will become more readily apparent from the following detailed description, including the appended claims, taken in conjunction with the accompanying drawings, in which:

[0061] Figure 1 An exemplary top view of a vehicle towing a trailer is shown;

[0062] Figure 2 A schematic view illustrating angle estimates based on the first feature detected by the camera image at different yaw angles between the trailer and the towing vehicle according to the first algorithm is shown schematically;

[0063] Figure 3schematic diagram of two angle estimations according to a first algorithm, based on the first and second features detected by the camera at different yaw angles between the trailer and the towing vehicle, and

[0064] Figure 4 schematic diagram of two angle estimations according to a second algorithm, based on the first and second features detected by the camera images at different yaw angles between the trailer and the towing vehicle; and

[0065] Figure 5 a schematic block diagram showing the individual steps of a method for determining the yaw angle of a trailer relative to the longitudinal axis of a towing vehicle. DETAILED DESCRIPTION

[0066] The application will be described in more detail with reference to the drawings, which show example embodiments. The embodiments in the drawings are not necessarily to scale, and certain features can be shown in somewhat schematic form, and the embodiments relate to preferred embodiments, while all elements and features described in connection with the embodiments can be used in any other embodiment and in combination with any other embodiment discussed herein, in particular in connection with any other embodiment discussed further above. However, the application should not be construed as being limited only to the embodiments described herein. In all descriptions below, like reference numerals are used to denote like elements, parts, items or features, if applicable.

[0067] The features of the application disclosed in the description, the claims, the examples and / or the drawings can both be combined with each other and with any other feature described herein.

[0068] Figure 1 A top view of a vehicle 1 towing a trailer 2 is shown. The vehicle 1 comprises a longitudinal axis LAV running through the center of the vehicle 1. Likewise, the trailer 2 has a longitudinal axis LAT running through the center of the trailer 2. The trailer 2 is connected to the vehicle 1 by means of a trailer coupling comprising a drawbar 4.

[0069] In certain driving situations, the longitudinal axis LAV of the vehicle 1 and the longitudinal axis LAT of the trailer 2 can not be aligned parallel to each other, or coincide with each other, but the two longitudinal axes can define a yaw angle YA. In other words, the yaw angle YA defines the angular deviation of the longitudinal axis LAT of the trailer 2 relative to the longitudinal axis LAV of the vehicle 1. The yaw angle YA can be measured in a horizontal plane comprising the longitudinal axis LAT of the trailer 2 and the longitudinal axis LAV of the vehicle 1.

[0070] Knowledge of the yaw angle YA can also be advantageous, for example in a trailer assistance system.

[0071] In order to determine the yaw angle YA, a plurality of images of at least a portion of the trailer 2 are taken by means of the camera 3. The camera 3 can for example be a rear view camera of the vehicle 1 which is also used to take images of the environment of the vehicle 1 when reversing. One of the taken images can refer to a known angular setting of the trailer 2 with respect to the towing vehicle 1. This image can be used as a reference for calculating the yaw angle YA. In this known angular setting of the trailer 2 with respect to the towing vehicle 1, the yaw angle YA can be 0 degrees or any other angular value.

[0072] Figure 2 Shown is a schematic diagram showing the angular relationship of a first feature of the trailer 2 at different points in time at which the trailer 2 has different yaw angles YA with respect to the towing vehicle 1.

[0073] The camera 3 can take two or more images of the trailer 2 at different angular positions with respect to the vehicle 1 at different points in time. For example, a series of images can be taken.

[0074] In the present example, the second image can show the direction of the trailer with respect to the vehicle at a yaw angle YA = 0 degrees.

[0075] Due to the angular movement of the trailer 2 over time, the specific feature detected at the trailer can appear at different positions in the first and second images. Figure 2 The first feature is indicated by a square.

[0076] The upper part of the representation of the first feature (associated with the solid light ray R connecting the feature with the camera 3) is identified in the first image, and the lower part of the representation of the first feature (associated with the dashed light ray R connecting the feature with the camera 3) is identified in the second image at another point in time. In order to relate the position of the first feature in the respective images to the position of the vehicle 1, in particular to a specific fixed point of the vehicle 1, calibration information of the camera 3 can be used. In particular, in order to determine the light ray R connecting the first feature with the camera 3, calibration information of the camera 3 can be used to convert the position of the first feature in the image coordinates to a light ray. In other words, in order to relate the camera position to the feature position, based on the calibration information of the camera 3, the feature position on the image is related to the position of a fixed point of the vehicle 1.

[0077] The feature on the trailer is located and matched using a feature detection and matching algorithm. For example, a Harris Corner Detector, a Scale-Invariant Feature Transform (SIFT) algorithm, a Speeded-Up Robust Features (SURF) algorithm, a Binary Robust Invariant Scalable Keypoints (BRISK) algorithm, a Binary Robust Independent Elementary Features (BRIEF), an Oriented FAST and Rotated BRIEF (ORB) algorithm or another applicable feature detection and matching algorithm can be used.

[0078] The feature detection and matching algorithm can detect image features on the trailer or not on the trailer. To separate the trailer features from the non-trailer features, several different approaches can be used. For example, while driving straight ahead, the trailer features can be separated from the non-trailer features by looking for features that remain in the same position over time. Alternatively, the motion of the background features can be modeled using the known motion of the vehicle over time. This can be extracted from the CAN (Controller Area Network) data on speed and steering. Features that do not comply with the epipolar line constraint of the fundamental matrix can be considered as trailer features.

[0079] Depending on the specific situation or conditions, it can appear beneficial to select a specific algorithm from a set of algorithms, instead of using only a single algorithm for determining at least one angle estimate a1, a2representative for the yaw angle YA. In the following, a first algorithm and a second algorithm for determining at least one angle estimate a1, a2are provided, as well as a framework for deciding which algorithm to use for determining said at least one angle estimate a1, a2.

[0080] Figure 2 The determination of the yaw angle YA based on the first algorithm is explained in more detail in the following. The change in position of at least one feature between the first and the second image is used to determine at least a first angle estimate a1.

[0081] After identifying the features in the respective images, the first features of the first and the second image are projected onto a common horizontal plane. In more detail, the ray between the camera 3 and the determined first feature on the first image is projected onto a horizontal plane, resulting in a first projected feature position PFP1a. Furthermore, the ray between the camera 3 and the determined first feature on the second image is projected onto the same horizontal plane, resulting in a second projected feature position PFP1b. It is worth mentioning that the projection is done in the vertical direction, thus only the elevation angle of the light ray is changed, not the azimuth angle.

[0082] After determining the first and the second projected feature position PFP1a, PFP1b, a first perpendicular bisector B1 is set based on the first and the second projected feature position PFP1a, PFP1b. As shown in Figure 2 The first perpendicular bisector B1 is a line perpendicular to the connecting line of the first and the second projected feature position PFP1a, PFP1b. Furthermore, the first perpendicular bisector B1 passes through the center of the connecting line. The first perpendicular bisector B1 intersects a reference axis, in the present embodiment the vehicle longitudinal axis LAV. The intersection of the first perpendicular bisector B1 and the reference axis provides a rotation point around which the trailer 2 rotates. In more detail, the intersection indicates the position of the tow bar 4.

[0083] The first angle estimate α1 is calculated based on the first vertical bisector B1. The first angle estimate α1 refers to the angle between the first line L1 connecting the first projected feature position PFP1a and the intersection of the first vertical bisector B1 and the reference axis, and the second line L2 connecting the second projected feature position PFP1b and the intersection of the first vertical bisector B1 and the reference axis. The intersection point indicates the position of the tow bar 4. More specifically, the first angle estimate α1 characterizes the angle of swing around the first intersection point IP1 (at least the approximate position of the tow bar 4) between the projected position of the first feature of the trailer 2 in the first image on the horizontal plane and the projected position of the first feature in the second image on the horizontal plane.

[0084] In this embodiment, the reference axis is the longitudinal axis LAV of the tractor 1, because the camera device 3 and the tow bar 4 are located on the longitudinal axis LAV of the vehicle 1. In other embodiments, if the camera device 3 or the tow bar 4 has a lateral offset relative to the longitudinal axis LAV of the vehicle 1, or if the lateral offsets of the camera device 3 and the tow bar 4 relative to the longitudinal axis LAV of the vehicle 1 are different, the reference axis may be formed by a straight line connecting the camera device 3 and the tow bar 4.

[0085] The first angle estimate α1 represents the yaw angle YA of trailer 2 around its actual rotation point.

[0086] Figure 3 What is shown is with Figure 2 Similarly, using the first algorithm implementation, the first and second features (i.e. two different features) of the trailer 2, captured at different time points (at which the trailer 2 has a different yaw angle YA relative to the tractor 1), are used to determine the yaw angle YA.

[0087] Multiple different features can be identified in the images captured by camera device 3. For example... Figure 3 As shown, the features are identified at different angular positions relative to a fixed point on vehicle 1. The first feature is represented by a square, and the second feature by a triangle. The fixed point can be the location of the camera device 3 or the location of the tow bar 4.

[0088] Figure 3 In the first image, the upper pair of first and second features (represented by PFP1a and PFP2a and associated with the solid line light connecting the feature to the camera device 3) are identified, and the lower pair of first and second features F1 and F2 (represented by PFP1b and PFP2b and associated with the dashed line light connecting the feature to the camera device 3) are identified at different time points in the second image.

[0089] The method for determining the yaw angle YA and Figure 2The illustrated embodiments are similar. The main difference is that two angle estimates a1, a2 are set and the yaw angle YA of the trailer is established based on the two angle estimates a1, a2. In more detail, as described above, a first perpendicular bisector B1 is set and a first angle estimate a1 is obtained.

[0090] Furthermore, by setting a third projected feature position PFP2a and a fourth projected feature position PFP2b, a second perpendicular bisector B2 is set to obtain a second intersection point IP2 and the third projected feature position PFP2a and the fourth projected feature position PFP2b are connected with the second intersection point IP2 to obtain a second angle estimate a2. The third projected feature position PFP2a is obtained by projecting the second feature in the first image onto the horizontal plane and the fourth projected feature position PFP2b is obtained by projecting the second feature in the second image onto the horizontal plane. The second intersection point IP2 can be the intersection of the second perpendicular bisector B2 and a reference axis which in this embodiment is the longitudinal axis LAV of the vehicle. The second angle estimate a2 is the angle between a first line connecting the third projected feature position PFP2a and the intersection point IP2 and a second line connecting the fourth projected feature position PFP2b and the intersection point IP2.

[0091] In Figure 3 In the illustrated embodiment, the reference axis is also the longitudinal axis LAV of the towing vehicle 1 since the camera 3 and the drawbar 4 are on the longitudinal axis LAV of the vehicle 1. In other embodiments, where the camera 3 or the drawbar 4 has a lateral offset from the longitudinal axis LAV of the vehicle 1 or where the camera 3 and the drawbar 4 have different lateral offsets from the longitudinal axis LAV of the vehicle 1, the reference axis can be constituted by a straight line connecting the camera 3 and the drawbar 4.

[0092] It is worth mentioning that the above two features of the trailer 2 can be determined and tracked on multiple images. Furthermore, it is preferred to take more than two images at different points in time to obtain a better result of the yaw angle estimate. Thus, more than two angle estimates a1, a2 can be set to improve the quality of the yaw angle determination.

[0093] In the following, a second algorithm for determining at least one angle estimate a1, a2 is described. Figure 4 A second algorithm for determining at least one angle estimate a1, a2 is described.

[0094] Figure 4 A schematic diagram illustrating the angle relationship of the first and second features F1, F2 of the trailer 2 is shown, wherein the features F1, F2 are identified at different points in time and at different angular positions relative to a fixed point of the vehicle 1.

[0095] The camera 3 can take two or more images of the trailer 2 at different angular positions relative to the vehicle 1 at different points in time. For example, a series of images can be taken. The series of images can comprise three or more images, in particular five or more images.

[0096] In the present example, the second image can show the orientation of the trailer 2 relative to the vehicle at a yaw angle YA = 0 degrees. However, according to other embodiments, the yaw angle YA can be any other reference yaw angle known beforehand and can be used to determine the current yaw angle.

[0097] On the images taken by the camera 3, a number of different features can be identified. Figure 4 The features Fl and F2 are illustrated in the middle, which are identified at different angular positions relative to the position of the camera 3 or the reference axis of the vehicle 1. The first feature Fl is indicated by a square and the second feature F2 is indicated by a triangle. It is worth mentioning that more than two features and more than two images can be used for the yaw angle estimation. In addition, the yaw angle can also be estimated using only one feature.

[0098] Thus, the pair of first and second features Fl, F2 above (associated with the solid light rays connecting the features Fl and F2 with the camera 3) is identified in the first image and the pair of first and second features Fl, F2 below (associated with the dashed light rays connecting the features Fl and F2 with the camera 3) is identified in the second image at another point in time.

[0099] In the second algorithm, the features on the trailer 2 can also be located and matched using feature detection and matching algorithms. For example, the Harris Corner Detector, the Scale-Invariant Feature Transform (SIFT) algorithm, the Speeded-Up Robust Features (SURF) algorithm, the Binary Robust Invariant Scalable Keypoints (BRISK) algorithm, the Binary Robust Independent Elementary Features (BRIEF), the Oriented FAST and Rotated BRIEF (ORB) algorithm or other applicable feature detection and matching algorithms can be used.

[0100] The feature detection and matching algorithms can detect image features on the trailer or not on the trailer. To separate the trailer features from the non-trailer features, a number of different approaches can be used. For example, when driving straight ahead, the trailer features can be separated from the non-trailer features by looking for features that remain in the same position over time. Alternatively, the motion of background features can be modeled using the known motion of the vehicle over time. This can be extracted from the CAN (Controller Area Network) data on speed and steering. Features that do not comply with the epipolar line constraint of the fundamental matrix can be considered as trailer features.

[0101] It is worth mentioning that the first and second algorithm can use the same detected feature to set the angle estimation a1, a2. In other words, only one feature detection has to be performed for both algorithms.

[0102] To determine the angle estimation a1, a2, the rays R connecting the features F1 and F2 to the camera 3 are used. To relate the features F1 and F2 of the captured images to the camera 3 position, the calibration information of the camera 3 can be used to transform the feature positions F1, F2 in the image coordinates into the spatial domain of the camera 3, so that the rays R relating the position of each respective feature F1, F2 to the camera position can be provided. In other words, for relating the camera position and the feature position, the feature positions F1, F2 on the image are transformed based on the calibration information of the camera 3 into the local domain of the vehicle 1 or into the local domain of the camera 3 of the vehicle 1, respectively.

[0103] After determining the rays R between the camera position and one or more features in the first and second image, the swing angle of the first feature F1 and / or the second feature F2 is determined. Figure 4 In the above, a1 denotes an angle estimation of the swing angle of the first feature F1 between the two captured images and a2 denotes an angle estimation of the swing angle of the second feature F2 between the images. According to different embodiments, only one or more than two trailer features are determined and tracked over a plurality of images, respectively. Furthermore, it is preferred to capture more than two images at different points in time to improve the result of the yaw angle estimation.

[0104] As mentioned above, one of the captured images can provide a reference image, wherein the angle position of the trailer 2 relative to the vehicle 1 is known. In this known angle setting of the trailer 2 relative to the towing vehicle 1, the yaw angle YA can be 0 degrees or any other angle value. Thus, based on the at least one angle estimation a1, a2, the yaw angle YA can be calculated. Again with reference to Figure 4 For example, the angle setting of the rays R, which are represented by dashed lines, can be known, because the trailer 2 has a known reference direction relative to the vehicle 1 when the images are captured with reference to the rays R.

[0105] The above second algorithm has a very high robustness, that is, the second algorithm can provide an angle estimation even in case of poor image quality and low accuracy of the angle estimation.

[0106] According to the content of the present patent document, for each feature F1, F2, one of the algorithms (first or second algorithm) can be selected and the selected algorithm is used to provide the at least one angle estimation a1, a2. The selection can be performed by checking one or more conditions.

[0107] According to a preferred embodiment, a first algorithm using at least one perpendicular bisector B1, B2 ( Figure 2 , Figure 3 The algorithm can be preferably used to determine at least one angle estimate α1, α2. More specifically, in order to attempt to calculate at least one angle estimate α1, α2 based on the first algorithm, multiple checks are performed. Depending on the different results of the checks, the first or second algorithm is used to determine the angle estimate for a specific feature.

[0108] The first condition to check is the length L of the baseline BL used to establish the perpendicular bisector (refer to...). Figure 2 The baseline BL can be a straight line connecting the locations of specific features F1 and F2 in the first and second images. If the length L is below a certain length threshold, the second algorithm is used instead of the first algorithm because the first algorithm may be severely affected by noise.

[0109] The second condition to be checked is whether there are changes in the pitch and / or roll angles of trailer 2 due to uneven ground in a series of captured images. If the changes in pitch and / or roll angles exceed preset pitch / roll angle thresholds, the second algorithm is used instead of the first algorithm, because the first algorithm may be severely affected by noise.

[0110] Furthermore, the third condition to be checked is the vertical distance of features F1 and F2 relative to the aforementioned horizontal plane, which forms a reference plane in which the features are transformed before determining at least one angle estimate α1, α2. If the vertical distance of a particular feature relative to the reference plane is less than a certain distance threshold, the second algorithm is used instead of the first algorithm, because the first algorithm may be severely affected by noise.

[0111] Therefore, if one or more of the above criteria or conditions to be checked are met, the first algorithm is skipped and the second algorithm is used to estimate the at least one angle α1, α2.

[0112] It is worth mentioning that for each feature F1, F2, a decision can be made independently on which algorithm to use to determine at least one angle estimate α1, α2. In other words, the angle estimates α1, α2 can be calculated based on the same or different algorithms.

[0113] Ideally, if multiple angle estimates are established based on a specific algorithm, the first angle estimate α1 and at least another angle estimate α2 should be equal (α1 = α2), and the yaw angle YA should be specified. However, due to noise and mismatch, the values ​​of angle estimates α1 and α2 may differ. It is worth noting that to improve the quality of yaw angle determination, more than two angle estimates can be established.

[0114] For determining the yaw angle YA on the basis of the plurality of angle estimates a1, a2 having different values, a statistical measure can be used. According to a first embodiment, for determining the yaw angle YA, the median of the two or more angle estimates a1, a2 can be used. According to other embodiments, a statistical method can be used for determining the yaw angle YA on the basis of the two or more angle estimates a1, a2. The statistical method can be, for example, a RANSAC algorithm (RANSAC: random sample consensus) or a least squares method.

[0115] It appears that not all features visible on the captured images are suitable for the calculation of the yaw angle YA. In order to reduce the computational complexity and to increase the robustness, those features are selected which provide a yaw angle a1, a2 which is very close to the actual yaw angle and are further used for the determination of the yaw angle YA. For selecting the features, only those features are tracked in future images which provide a yaw angle a1, a2 in a certain window around the actual yaw angle. The window can be defined, for example, by an upper limit and a lower limit, wherein the upper limit and the lower limit define an angle window around the actual yaw angle. For example, the window can cover a range of 2 degrees to 10 degrees, in particular a range between 3 degrees and 5 degrees. In the last two or more steps of determining the yaw angle, all features which lead to a yaw angle within the window are further tracked in the next captured images.

[0116] If the tracking of a certain feature of the trailer 2 is to be performed on a plurality of images due to the movement of the trailer 2, samples of the feature can be arranged on an arc segment. The center of the arc segment represents the position of the hitch 4. Thereby, by tracking a certain trailer feature on a plurality of images, the position of the hitch 4 can be derived.

[0117] For reducing noise, the determination of the position of the hitch 4 can take into account the tracking of a plurality of trailer features on a plurality of images over a period of time. Each trailer feature can correspond to an arc segment with a certain center estimate. By applying a statistical method on the plurality of center estimates, the actual position of the hitch 4 can be determined. The statistical method can be, for example, a RANSAC (random sample consensus) algorithm or a least squares method.

[0118] Figure 5 Shown is a block diagram illustrating method steps of a method for determining a yaw angle YA of a trailer 2 relative to a longitudinal axis LAV of a towing vehicle 1.

[0119] In a first step, first and second images of the trailer are captured (S10).

[0120] After the capturing of the images, at least one visible trailer feature on the first and second images is determined (S11).

[0121] Furthermore, at least a first algorithm and a second algorithm for calculating at least one angle estimate are provided (S12).

[0122] At least a first angle estimate is provided for the first feature. According to a first option, at least one angle estimate is provided, i.e. by deciding based on one or more criteria for the first feature whether to use the first algorithm or the second algorithm for calculating the angle estimate, the first angle estimate is calculated based on the algorithm selected for the first feature (S13A).

[0123] According to a second option, at least one angle estimate is provided, which is calculated based on the first and the second algorithm for the first feature, and based on one or more criteria it is decided whether to use the angle estimate obtained by the first algorithm or the angle estimate obtained by the second algorithm for further processing (S13B).

[0124] Finally, a yaw angle is calculated based on the at least one angle estimate (S14).

[0125] It should be noted that the description and drawings merely illustrate the principles of the proposed application. As such, those skilled in the art will be able to devise various arrangements that, although perhaps not explicitly described or shown herein, embody the principles of the application.

[0126] List of reference signs

[0127] 1 vehicle

[0128] 2 trailer

[0129] 3 camera arrangement

[0130] 4 drawbar

[0131] a1 first angle estimate

[0132] a2 second angle estimate

[0133] B1 first perpendicular bisector

[0134] B2 second perpendicular bisector

[0135] BL baseline

[0136] F1 first feature

[0137] F2 second feature

[0138] IP1 first intersection point

[0139] IP2 second intersection point

[0140] L length

[0141] LAT trailer longitudinal axis

[0142] LAV vehicle longitudinal axis

[0143] R light

[0144] YA yaw angle

Claims

1. Method of determining a yaw angle of a trailer (2) relative to a longitudinal axis of a towing vehicle (1), wherein, The method comprises the steps of: detecting at least first and second images of the trailer (2) using the camera (3), wherein the orientation of the trailer (2) relative to the vehicle (1) is different in at least two images (S10); determining at least a first feature of the trailer (2) visible in the first and second images (S11); providing at least first and second algorithms for computing at least one angle estimate based on the at least first feature (S12); providing at least one first angle estimate for the first feature by deciding for each feature based on one or more criteria whether to use the first or the second algorithm for computing the first angle estimate and computing the first angle estimate based on the algorithm selected for the first feature (S13A) or computing a first angle estimate for the first feature based on the first and second algorithms and deciding based on one or more criteria whether to use the first angle estimate obtained by the first algorithm or the first angle estimate obtained by the second algorithm for further processing (S13B); and computing a yaw angle based on the first angle estimate (S14), wherein the step of deciding based on one or more criteria comprises determining a length of a baseline between a feature in the first image and the feature in the second image and comparing the length to a length threshold.

2. The method of claim 1, wherein, The first algorithm is configured for: projecting a ray between the camera (3) and the determined first feature in the first image onto a horizontal plane to obtain a first projected feature position and projecting a ray between the camera (3) and the determined first feature in the second image onto the horizontal plane to obtain a second projected feature position; setting a first perpendicular bisector between the first projected feature position and the second projected feature position; determining a first intersection of the first perpendicular bisector with a reference axis or another perpendicular bisector (S14); and computing the first angle estimate, wherein the first angle estimate refers to an angle between a first line from the first projected feature position to the first intersection and a second line from the second projected feature position to the first intersection on the horizontal plane.

3. The method of claim 2, wherein, determining a second feature of the trailer (2) visible in the first and second images, wherein the second feature is arranged at another trailer position different from the first feature, wherein the first algorithm is used for two or more features of the trailer (2) for providing angle estimates, wherein the first algorithm is configured for: projecting a ray between the camera (3) and the determined second feature in the first image onto a horizontal plane to obtain a third projected feature position and projecting a ray between the camera (3) and the determined second feature in the second image onto the horizontal plane to obtain a fourth projected feature position; setting a second perpendicular bisector between the third projected feature position and the fourth projected feature position; determining a second intersection of the second perpendicular bisector with the reference axis, the first perpendicular bisector or another perpendicular bisector; and computing the first angle estimate, wherein the first angle estimate refers to an angle between a first line from the third projected feature position to the second intersection and a second line from the fourth projected feature position to the second intersection on the horizontal plane. calculating a second angle estimate, wherein the second angle estimate is an angle between a first line from a third projected feature position to the second intersection point and a second line from a fourth projected feature position to the second intersection point on the horizontal plane.

4. The method of any one of claims 1 to 3, wherein, The second algorithm is configured to calculate a first angle estimate, wherein the first angle estimate characterizes a roll angle of the trailer (2) relative to the fixed point of the towing vehicle (1) between the first feature on the first image and the first feature on the second image on the horizontal plane.

5. The method of claim 4, wherein, determining a second feature of the trailer (2) visible on the first and second images, wherein the second feature is arranged at another trailer position different from the first feature, wherein the second algorithm is configured to calculate a second angle estimate, wherein the second angle estimate characterizes a roll angle of the trailer (2) relative to the fixed point of the towing vehicle (1) between the second feature on the first image and the second feature on the second image on the horizontal plane.

6. The method of any one of claims 1 to 3, wherein, The step of determining based on one or more criteria comprises determining a roll angle and / or a pitch angle of the trailer (2) and comparing the roll angle to a roll angle threshold value and / or comparing the pitch angle to a pitch angle threshold value.

7. The method of any one of claims 1 to 3, wherein, The step of determining based on one or more criteria comprises determining a perpendicular distance of at least one feature detected on the first and second images relative to a horizontal reference plane and comparing the perpendicular distance to a distance threshold value.

8. The method of claim 5, wherein, In the second algorithm, calculating at least one angle estimate comprises determining a ray between the fixed point and the at least one feature on the first and second images.

9. The method of claim 8, wherein, Camera calibration information is used to convert a position of the at least one feature from a local domain of the images to a local domain of the vehicle (1) in order to determine the ray.

10. The method of claim 3, wherein, At least one further feature of the trailer (2) is used for the calculation of the yaw angle in addition to the first feature and the second feature.

11. The method of claim 3, wherein, The calculation of the yaw angle is performed by establishing a median value based on at least two angle estimates, by establishing a mean value of at least two angle estimates or by using a statistical method on the angle estimates.

12. The method of any one of claims 1 to 3, further comprising the step of determining an angular window, wherein, The angle window comprises an upper limit and a lower limit around the yaw angle, a set of features resulting in angle estimates within the angle window is determined and the determined set of features is used for future calculations of the yaw angle.

13. System for determining the yaw angle of a trailer (2) relative to the longitudinal axis of a towing vehicle (1), wherein The system comprises a camera (3) for taking images of the trailer (2) and a processing entity for processing the taken images, and in addition, the system is configured to perform the following steps: detecting at least a first and a second image of the trailer (2) using the camera (3), wherein an orientation of the trailer (2) relative to the vehicle (1) is different on at least two images (S10); determining at least a first feature of the trailer (2) visible on the first and second images (S11); providing at least a first and a second algorithm for calculating at least one angle estimate based on the at least first feature (S12); providing at least one first angle estimate for said first feature by deciding for each feature based on one or more criteria whether to use a first algorithm or a second algorithm for calculating said first angle estimate, and calculating said first angle estimate based on the algorithm selected for said first feature (S13A), or calculating a first angle estimate for said first feature based on said first and second algorithm, and deciding based on one or more criteria whether to use the first angle estimate obtained by the first algorithm or the first angle estimate obtained by the second algorithm for further processing (S13B); and calculating a yaw angle based on said first angle estimate (S14), wherein said step of deciding based on one or more criteria comprises determining a length of a baseline between a feature on a first image and said feature on a second image, and comparing said length to a length threshold.

14. Vehicle comprising a system according to claim 13.

Images