Attitude angle calibration of vehicle camera during travel

Abstract

The invention relates to a system for determining an attitude angle of a camera unit (1) of a vehicle (3) during travel of the vehicle (3), comprising a camera unit (1) and a computing unit (5) connected to the camera unit (1), the computing unit (5) being configured to: acquire current image data of the camera unit (1) during travel of the vehicle (3), the present invention relates to a method for determining the attitude angle of a camera unit (1) of a vehicle (3) and checking the suitability of the current image data of the camera unit (1) for determining the attitude angle of the camera unit (1), and executing a pre-trained artificial neural network when the image data has sufficient suitability, with the image data captured by the camera unit (1) during the travel of the vehicle (3) as the input of the artificial neural network, the output quantity of the artificial neural network comprises the current attitude angle of the camera unit (1).

Description

Technical Field

[0001] The present invention relates to a system for determining the attitude angles of a vehicle camera unit relative to a reference coordinate system during vehicle travel, a vehicle equipped with such a system, and a method for training an artificial neural network to implement such a system. Background Art

[0002] Autonomous vehicles, like human drivers, typically rely on obtaining information about the surrounding environment. For example, in order to be able to perform optical lane recognition with a camera, high-precision external calibration of the camera is required. The result of this calibration is to provide the attitude angles of the camera relative to a reference coordinate system to describe the orientation of the camera. That is, only when these attitude angles are determined accurately enough can the coordinate system information of the camera data be transformed into the reference coordinate system of a driving assistance system or an autonomous driving control system (such as automatic lateral control, especially automatic lane keeping).

[0003] External camera calibration systems are known in the prior art. In this regard, WO 2019 / 067298 A2 relates to an apparatus for an external camera calibration system (ECCS). The apparatus has an external calibration processor for receiving images of a plurality of cameras mounted on a vehicle, where each image contains features collected from a calibration model; and further determines one or more external calibration parameters (ECPs) of one or more features collected by the camera, where the one or more ECPs map the coordinates of the one or more collected features to one or more camera coordinates. Thus, WO 2019 / 067298 A2 describes the external calibration of surround cameras. Here, the relative external positions of the cameras are calibrated with respect to each other, but this is not done in the driving direction.

[0004] In addition, US10,298,910B1 relates to a system including a vehicle, the system comprising: a plurality of cameras arranged on the vehicle body such that the fields of view of at least two cameras at least partially overlap; one or more processors; and a computer-readable medium storing instructions executable by the one or more processors, wherein the instructions include: receiving a set of images from the plurality of cameras, wherein the set of images includes images acquired by the plurality of cameras that substantially represent the environment at a first time point; determining a plurality of point pairs of the set of images, wherein each point pair includes a first point of one image in the set of images and a second point of another image in the set of images, and wherein the first point and the second point correspond to the same image feature; determining a first error associated with the plurality of point pairs, wherein the first error at least partially includes a first distance between the first point and an epipolar line corresponding to the second point; determining a first subset of the plurality of point pairs at least partially based on the first error; determining a first correction function representing the estimated relative pose misalignment of the cameras from the first subset of the plurality of point pairs; determining a second error associated with the plurality of point pairs, wherein the second error at least partially includes a second distance between the first point and the reprojection of the second point estimating the point depth; determining a second subset of the plurality of point pairs at least partially based on the second error; determining a second correction function representing the estimated relative pose misalignment of the cameras from the second subset of the plurality of point pairs; and calibrating the plurality of cameras at least partially based on the first correction function and the second correction function. Thus, US10,298,910B1 relates to the external calibration of cameras with respect to each other.

[0005] In addition, DE 10 2015 209 764 A1 relates to a device for externally calibrating an image acquisition device for a vehicle, in particular a passenger car, comprising: a storage device; at least one image acquisition device, in particular at least one camera, designed to acquire at least one environmental image of the vehicle; an identification device; a comparison device; and a calibration device; wherein at least one vehicle feature and at least one target position of the at least one vehicle feature are stored in the storage device; the identification device is designed to identify the actual position of the at least one vehicle feature in at least one environmental image of the vehicle; the comparison device is designed to compare the actual position and the target position; and the calibration device determines the external parameters of the at least one image acquisition device based on the deviation between the actual position and the target position. Thus, DE 10 2015 209 764 A1 describes the external calibration of an image acquisition device, but is only applicable to stationary vehicles. Summary of the Invention

[0006] The object of the present invention is to improve the camera-based environmental detection of an autonomous vehicle such that during driving, the attitude angles of the camera unit relative to a specified reference coordinate system can be determined based on the corresponding camera images of the camera unit itself, in particular without the need for other separate devices, such as additional sensors or physical calibration targets (such as calibration curtains).

[0007] The present invention results from the features of the independent claims. Advantageous improvements and design options are the subject of the dependent claims.

[0008] A first aspect of the present invention relates to a system for determining the attitude angles of a vehicle camera unit relative to a reference coordinate system during vehicle driving, having a camera unit and a computing unit connected to the camera unit, wherein the computing unit is designed to detect the current image data of the camera unit during vehicle driving and check the suitability of the current image data of the camera unit for determining the attitude angles of the camera unit, and when the suitability of the image data is sufficient, execute a pre-trained artificial neural network with the image data acquired by the camera unit during vehicle driving as the input quantity of the artificial neural network, wherein the output quantity of the artificial neural network corresponding to the image data of each image includes the current attitude angles of the camera unit relative to the specified reference coordinate system.

[0009] The reference coordinate system is preferably assumed to be arranged on the vehicle body or other components in a rigidly fixed manner. The advantage of this definition of the reference coordinate system is that when the orientation of the camera unit relative to the vehicle, especially the vehicle body, remains unchanged, the relative attitude angles between the camera unit and the vehicle body remain constant and are not affected by the kinematic driving state of the vehicle. Here, the relationship represented by the obtained attitude angles of the camera unit is limited to the relative orientation between the rigidly fixed coordinate system of the camera unit and the rigidly fixed coordinate system of the vehicle body. However, alternative definitions of the reference coordinate system are possible, so that the reference coordinate system can also be the result of further coordinate transformations, but in another alternative embodiment, the reference coordinate system can also be determined based on kinematic quantities, especially aligned with the current movement direction of the vehicle. Through this kinematic definition, a closer logical connection with vehicle guidance control can be achieved for the vehicle, but this is more complex than using only a rigidly fixed coordinate system definition.

[0010] The pitch angle refers to the angle of the camera unit generated when the camera unit rotates around the transverse axis. Therefore, the change in the pitch angle is generated by the camera unit tilting up and down in a vertical plane. Therefore, the change in the pitch angle also always means a change in the detection range in the vertical direction with respect to the horizontal line. In contrast, the yaw angle refers to the change in the azimuth angle of the detection range of the camera unit. Therefore, the yaw angle represents an angle in a compass plane.

[0011] The artificial neural network is trained as described below and then applied to the system according to the invention. Since, in particular, image data suitable for the training scenario are used as the specified input quantity when training the artificial neural network, when applying the trained artificial neural network, it is possible to obtain the attitude angle result of the camera unit relative to the reference coordinate system from the image data based on the current image data during vehicle travel. The key feature implicitly used by the artificial neural network for this purpose is the objects in the image data that converge to a common vanishing point, which makes it possible to determine the position of the vanishing point relative to the rest of the image data. Thus, for example, image data of a straight road is more suitable for applying the artificial neural network than an image of a road bend.

[0012] Preferably, the artificial neural network is a convolutional neural network.

[0013] Therefore, the system is particularly used for externally determining the attitude angle of the vehicle camera unit and particularly for externally calibrating the camera unit arranged on the vehicle, for example when the attitude angle is used to calibrate the camera unit for a driving assistance system or a fully automatic vehicle guidance system.

[0014] The camera unit is particularly used to detect the environment in front of the vehicle through its detection range, so it is usually aligned with the main driving direction of the vehicle. Preferably, a camera unit having only a single camera or a single stereo camera is used.

[0015] An advantage of the first aspect of the invention is that determining the attitude angle of the camera unit and thus calibrating the camera unit only requires the camera image of the camera unit itself, so no other devices are required, especially laser measurement devices arranged on the wheels or chassis components, and external physical calibration targets (such as specially printed calibration curtains). In addition, the determination of the attitude angle is advantageously carried out during vehicle travel. Since the system can be used during travel, the vehicle does not need to go to a repair shop separately, for example, after updating the windshield, replacing the camera unit, repairing after an accident, or performing other relevant modifications to the vehicle. Once there is suitable image data, the currently assumed attitude angle can also be repeatedly verified during vehicle travel. This is particularly important for ensuring the safety of fully autonomous vehicles.

[0016] The above vehicle system uses the artificial neural network trained as described below. Therefore, the characteristics generated by the training process also characterize the artificial neural network used in the system. The artificial neural network is implemented in such a way that data about the artificial neural network is stored in the permanent memory of the computing unit, that is, the artificial neural network can be executed by the computing unit during vehicle travel.

[0017] The image data for training the artificial neural network and the image data in subsequent system operation applications are preferably image data that includes the interesting part of the complete camera image. Such an interesting part is also referred to as the "Region of Interest" or simply ROI.

[0018] According to a preferred embodiment, the attitude angles only include the pitch angle and the yaw angle. Therefore, the roll angle of the camera unit is not considered.

[0019] According to another preferred embodiment, the calculation unit is designed to check the applicability of the image data of the camera unit for determining the attitude angles of the camera unit by analyzing the image data of the camera unit. Therefore, advantageously, no other sensors are required, so that the applicability can be checked from the image data of the camera unit, and the attitude angles can be derived when the applicability is sufficient.

[0020] According to another preferred embodiment, the calculation unit is designed to check the applicability of the image data of the camera unit by executing another pre-trained artificial neural network with the current image data as the input quantity. Therefore, the other pre-trained artificial neural network is preferably only used to check the applicability of the attitude angle determination. Therefore, the other pre-trained artificial neural network evaluates the straight road direction in front of the vehicle as more suitable than a curved road, because as described above, the vanishing point can be derived on a straight road and the attitude angles can be derived therefrom.

[0021] According to another preferred embodiment, the calculation unit is designed to check the applicability of the image data of the camera unit for determining the attitude angles of the camera unit by analyzing the current kinematic data of the vehicle.

[0022] As an alternative to or as a supplement to checking the current image data of the camera unit for determining the attitude angles of the camera unit, for example, through a common metric, vehicle kinematic data, especially compared with limit values, can be used. The limit values are especially the minimum vehicle speed and the maximum kinematic quantity values of the vehicle inertial measurement unit (such as the rotational speed around the vertical axis, also known as the vehicle yaw rate), or the translational acceleration of the vehicle. Because it is subsequently assumed that the vehicle is driving straight at a relatively high speed, which is only possible on a straight road section. This excludes traffic situations that are not suitable for determining the attitude angles of the camera unit, especially driving in curves, urban traffic driving, parking lot driving, etc.

[0023] According to another preferred embodiment, the calculation unit is designed to execute an artificial neural network with the image data of the camera unit collected during vehicle driving as the input quantity only when the applicability is sufficient, or to use the output quantity of the artificial neural network that is continuously determined regardless of the applicability. If the artificial neural network is only executed when the applicability is determined, computing power can be saved.

[0024] According to another preferred embodiment, the computing unit is designed to calibrate the camera unit during vehicle travel using the determined attitude angles of the camera unit and / or to verify the existing calibration of the camera unit during vehicle travel.

[0025] A calibrated camera unit means that the assumed vanishing point corresponds to the actual vanishing point in the image data. In ideal straight-line driving, all (approximate) infinitely long straight environmental objects mapped into the image data, such as lane markings, guardrails, etc., terminate at this vanishing point.

[0026] Another aspect of the present invention relates to a vehicle equipped with the system described above and below.

[0027] Another aspect of the present invention relates to a method for training an artificial neural network, wherein, in iterations of multiple training processes, the input quantities of the artificial neural network include image data in respective translational displacements, and the output quantities include the attitude angles of the camera unit that are hypothetically or actually related to the image data and related to the respective translational displacements, such that the training of the artificial neural network designs the parameters of the artificial neural network, i.e., the artificial neural network maps the relationship between the attitude angles and the translational displacements of the image data, so as to obtain, as the output quantity, the attitude angles of the camera unit that generates the image data through the actual or virtual displacement of the image data as the input quantity during subsequent operation of the artificial neural network.

[0028] Preferably, image data with multiple displacements is used when training the artificial neural network, particularly preferably independent combinations of displacements and image data on the order of 10 4 to 10 7 The training of the artificial neural network is preferably carried out offline, i.e., in a fixed computing unit rather than during vehicle travel, because this facilitates the user-assisted selection of images with appropriate image data. Once the artificial neural network has been sufficiently trained, it can be used for subsequent operation of the vehicle.

[0029] Since the artificial neural network receives image data of a real scene or an approximately real virtual scene during training, there is no need to manually select specific features from the image data and specifically transmit them to the artificial neural network through the input quantities. Instead, the complete image data set (as the input quantity) is transmitted to the artificial neural network. The artificial neural network autonomously and without further intervention considers the relevant features because the artificial neural network essentially uses specific patterns and features to obtain specific output quantities as the internal calculation results of the artificial neural network. It can be expected therefrom that the artificial neural network will not only consider lane markings but also other objects mapped in the image data, such as direction arrows, guardrails, especially all objects that perspectively point to a common vanishing point and reach or do not reach the common vanishing point in the image data.

[0030] As a result of the artificial neural network training process, at least the parameter values of the artificial neural network are obtained thereby, and if a variable structure of the artificial neural network (e.g., having a variable number of layers) is also used, the structure parameters are also obtained. The aforementioned parameter values particularly represent factors for enhancing the corresponding signal paths.

[0031] Here, the translational displacement simulates the pitch angle change or the yaw angle change. The horizontal displacement of the image data corresponds to the yaw angle change. The vertical displacement of the image data corresponds to the pitch angle change. Therefore, among multiple displacements, in some cases only horizontal displacement may be performed, in other cases only vertical displacement may be performed, and in some or all cases horizontal and vertical displacements may be applied in combination. Therefore, the translational displacement of the image data means an intentional deviation from the correct camera fixation vanishing point. Therefore, the translational displacement is equivalent to the pitch angle change and / or the yaw angle change of the camera unit relative to the reference coordinate system (depending specifically on whether the translational displacement includes vertical and / or horizontal displacements). The artificial neural network particularly generates exactly one set of pose angles for an image containing image data.

[0032] An advantageous effect of the second aspect of the present invention is that the training of the artificial neural network can be highly automated, for example, by automatically performing the translational displacement of the image data, such as using a random generator. The selection of image data suitable for training the artificial neural network can be performed automatically on the one hand, for example, by image analysis to check whether the straight road alignment is captured in the image data, but can also be alternatively or additionally performed by a person for the selection of appropriate image data.

[0033] According to another preferred embodiment, the translational displacement of the image data is performed using a random generator, wherein the random generator determines the direction and magnitude of the displacement.

[0034] By analogously transposing in meaning the content previously described for the proposed system, the advantages and preferred improvements of the proposed method are obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Other advantages, features, and details result from the following description that perhaps details at least one embodiment with reference to the drawings. Identical, similar, and / or functionally identical components carry the same reference numerals, wherein:

[0036] Figure 1 A vehicle is shown having a system for determining the pose angles of a camera unit according to an embodiment of the present invention.

[0037] Figure 2 A typical camera image that can be captured during vehicle travel is shown.

[0038] Figure 3 Shown according to Figure 2 the method of training an artificial neural network with the camera image in

[0039] The illustrations in the drawings are schematic and not drawn to scale. Detailed implementation mode

[0040] Figure 1 The towing head of a semi-trailer truck is shown as vehicle 3. The camera unit 1 oriented in the driving direction detects the environment in front of vehicle 3 through its detection range, including lane markings, signs, guardrails, asphalt road edges, etc.; in Figure 1 the upper part of the figure, vehicle 3 is shown in side view, and in the lower part of the figure, the same vehicle 3 is shown in top view. With the help of these views, the attitude angles of the camera unit 1 to be concerned about relative to the rest of vehicle 3 can be illustrated, including: the pitch angle N involves the downward and upward angles of the camera unit 1 relative to the horizontal line, and the yaw angle G is defined around the vertical axis of vehicle 3 and is associated with the left and right viewing angles from the cab of vehicle 3. If the camera unit 1 rotates by the pitch angle N, the change is manifested as an apparent translational displacement of the camera image upward and downward, while when rotating by the yaw angle G, it is manifested as a lateral displacement of the camera image. The pitch angle N and the yaw angle G are attitude angles that need to be determined by the calculation unit 5 in the following manner: The system for determining the attitude angle has a camera unit 1 and a calculation unit 5 connected to the camera unit 1. The calculation unit 5 receives the current image data of the camera unit 1 during the driving of vehicle 3 and checks whether it is suitable for determining the attitude angle of the camera unit 1 based on the image data itself. Only when the image data is sufficiently applicable, the calculation unit 5 will execute a pre-trained artificial neural network with the currently acquired image data of the camera unit 1 during the driving of vehicle 3 as the input quantity of the artificial neural network. The output quantity of the artificial neural network is a vector containing the attitude angles of "pitch angle" and "yaw angle", as Figure 1 described. How to pre-train the artificial neural network before permanently putting vehicle 3 into use, and how to use it for repeatedly recalibrating the pitch angle N and the yaw angle G during operation, see Figure 2 and Figure 3 for the description.

[0041] Figure 2 The complete camera image of the camera unit 1 is shown with an external border. There is a central area of particular interest, namely the "Region of Interest" (RoI), which contains the road alignment. The lane markings or road edges and their possible extensions intersect at a common vanishing point, namely the "Focus of Expansion". In addition, Figure 2Horizontal lines are also marked, and the converging road markings captured by the camera image extend to the horizontal lines. The camera image is automatically selected and / or selectively selected manually because it has a straight road orientation and is thus particularly suitable for training an artificial neural network. The applicability of the camera image lies in the fact that a straight road orientation can be recognized, from which the center vanishing point FoE can be derived. The ability to derive such a vanishing point FoE is an important feature for training an artificial neural network.

[0042] Figure 3 Shows the Figure 2 Artificial modification of an actual camera image. Here, a translational displacement generated by a computer random generator is used to randomly move the central region of interest RoI laterally and up / down to RoI' manually to simulate the pitch angle change and yaw angle change of the camera unit 1. If multiple suitable images are generated through multiple randomly generated displacements, a large dataset for training an artificial neural network can be created. Here, the artificial neural network receives the translation converted to the pitch angle and yaw angle related to the camera unit 1 as the specified output quantity, and the shifted image data as the input quantity. Here, the parameters of the artificial neural network are determined in such a way that the artificial neural network can also determine the specified output quantity based on the input quantity. The artificial neural network trained in this way can then be applied to suitable and currently determined image data during the subsequent operation of the vehicle 3 to determine the current pitch angle and current yaw angle of the camera unit 1 relative to the body of the vehicle 3 based on the current image data.

[0043] Although the details of the present invention have been described and explained in detail by means of preferred embodiments, the present invention is not limited to the disclosed examples, and those skilled in the art can derive other variations therefrom without exceeding the protection scope of the present invention. Therefore, it is obvious that there are many possibilities for variation. It is also obvious that the exemplified embodiments are actually only examples and should in no way be construed as a limitation on the protection scope, application possibilities or configuration of the present invention. On the contrary, the foregoing description and the description of the drawings enable those skilled in the art to specifically implement the exemplary embodiments, wherein those skilled in the art can make various modifications, such as regarding the functions or arrangements of the various components mentioned in the embodiments, without exceeding the protection scope defined by the claims and their legal equivalents such as the specific description in the specification.

[0044] List of reference numerals

[0045] 1 Camera unit

[0046] 3 Vehicle

[0047] 5 Computing unit

[0048] N Pitch angle

[0049] G Yaw angle

[0050] FoE Vanishing point

[0051] RoI Region of Interest

[0052] RoI' Region of Interest with artificial translation displacement

Claims

1. A system for determining the attitude angle of a camera unit (1) of a vehicle (3) relative to a reference coordinate system during travel of the vehicle (3), the system comprising the camera unit (1) and a computing unit (5) connected to the camera unit (1), wherein, The computing unit (5) is configured to: acquire current image data of the camera unit (1) during the travel of the vehicle (3), check the applicability of the current image data of the camera unit (1) for determining the attitude angle of the camera unit (1), and execute a pre-trained artificial neural network when the image data has sufficient applicability, using the corresponding image data acquired by the camera unit (1) during the travel of the vehicle (3) as the input quantity of the artificial neural network, wherein the output quantity associated with the image data of each image of the artificial neural network includes the current attitude angle of the camera unit (1) relative to a specified reference coordinate system.

2. The system according to claim 1, wherein The attitude angle only includes the pitch angle and the yaw angle.

3. The system according to one of the preceding claims, wherein, The computing unit (5) is configured to check the applicability of the image data of the camera unit (1) for determining the attitude angle of the camera unit (1) by analyzing the image data of the camera unit (1).

4. The system according to claim 3, wherein, The computing unit (5) is configured to perform the applicability evaluation of the image data of the camera unit (1) by executing an additional pre-trained artificial neural network with the current image data as the input quantity.

5. The system according to any one of the preceding claims, wherein, The computing unit (5) is configured to check the applicability of the image data of the camera unit (1) for determining the attitude angle of the camera unit (1) by analyzing the current kinematic data of the vehicle (3).

6. The system according to one of the preceding claims, wherein, The computing unit (5) is configured to: only execute the artificial neural network with the image data acquired by the camera unit (1) during the travel of the vehicle (3) as the input quantity when the applicability is sufficient, or only use the output quantity continuously determined by the artificial neural network independent of the applicability when the applicability is sufficient.

7. The system according to any one of the preceding claims, wherein, The computing unit (5) is configured to use the determined attitude angle of the camera unit (1) to calibrate the camera unit (1) during the travel of the vehicle (3) and / or to verify the existing calibration of the camera unit (1) during the travel of the vehicle (3).

8. A vehicle (3) having a system as described in one of the preceding claims.

9. A method for training an artificial neural network, wherein, In the iterations of multiple training processes, each input quantity of the artificial neural network includes image data with respective translation displacements, and each output quantity includes the attitude angle of the camera unit (1) associated with each translation displacement and hypothetically or actually associated with the image data, so that the training of the artificial neural network configures the parameters of the artificial neural network to: enable the artificial neural network to map the relationship between the attitude angle and the translation displacement of the image data, so that in the subsequent operation of the artificial neural network, the attitude angle of the camera unit (1) that generates the image data is obtained as the output quantity by using the actual or virtual displacement of the image data as the input quantity.

10. The method according to claim 9, wherein, Each translation displacement of the image data in each input quantity is generated by means of a random generator, wherein the random generator determines the direction and amplitude of the displacement.

Images