Method for calibrating a vehicle vision system of a motor vehicle, vision system for a motor vehicle, electronic vehicle guidance system, motor vehicle and computer program product

Abstract

The present invention relates to a method for calibrating a vehicle vision system (20) for a motor vehicle (10), the vehicle vision system (20) including a calibration module (42) and at least one vehicle-mounted camera (14) having an image processor (48), wherein a known light pattern projected by the vehicle headlights (22) onto a projection surface within the field of view of the at least one vehicle-mounted camera (14) is used as a calibration target.

Description

Technical Field

[0001] This invention relates to a method for calibrating a vehicle vision system for a motor vehicle, and more particularly to a computer-implemented method, the vehicle vision system including a calibration module and at least one onboard camera having an image processor. Other aspects of the invention relate to a vision system for a motor vehicle, an electronic vehicle guidance system having such a vision system, and a motor vehicle. Yet another aspect of the invention relates to a computer program and a computer-readable storage medium having the computer program stored thereon. Background Technology

[0002] Modern motor vehicles are typically equipped with a vehicle vision system, which usually includes at least one onboard camera, such as a front-facing camera, which may be positioned on the windshield or front grille. The system may include additional cameras, such as rearview mirror cameras and / or rear-facing cameras. From the captured images of the vehicle's surrounding environment, the distance to objects can be inferred, whereby, here and below, the term "object distance" refers to the distance between the vehicle equipped with the camera and objects in its surrounding environment, i.e., within the camera's field of view. Information from the camera can be used in various driver assistance functions, such as in autonomous or semi-autonomous driver assistance systems or other electronic vehicle guidance systems, such as parking assistance systems and / or automatic distance control systems. Camera calibration helps to accurately interpret the distance to objects from the vehicle, thereby providing accurate collision time estimates and avoiding potential collisions.

[0003] Compared to inherent camera parameters, such as focal length or other lens parameters, extrinsic camera parameters describe the camera's pose in the real world—its position and heading. Therefore, extrinsic camera parameters can be affected by vehicle motion. In other words, extrinsic calibration parameters can also vary due to vehicle movement and / or vehicle load. For example, when a vehicle is going uphill, the rear suspension lowers, and the front camera can tilt more upwards towards the sky, so the tilt angle of the extrinsic camera parameters can differ from the initial calibration value. As a result, the accuracy of interpreting distances to objects from the vehicle can be affected. Therefore, camera calibration, or the calibration of extrinsic camera parameters, is crucial for accurately interpreting any image information that can be derived from camera images.

[0004] Current external calibration methods rely on calibration targets with fixed, known shapes or on known patterns within the camera's field of view. These methods work with fixed patterns provided in a specified environment (e.g., a factory) or with known dimensions of objects on the road (e.g., known vehicle model dimensions, known road sign dimensions, etc.). Alternatively, the vanishing point of lane markings on a straight road can be evaluated. In vanishing point techniques, it is assumed that a vehicle is traveling in a lane with two parallel lane markings, which can be seen by the camera. From the two parallel lane markings, the camera's vanishing point can be derived. This vanishing point can then be used as a calibration target. Using the known vanishing point, the tilt angle relative to the road surface can be derived. Calibration parameters can be derived accordingly.

[0005] When calibrating external camera parameters, current methods rely on knowing the dimensions of known objects on the road or in a factory. For factory calibration (factory calibration), the camera is typically fixed in a designated mounting position on a vehicle. One or more target objects or calibration targets with known dimensions are placed in front of the camera. Due to the known dimensions of these objects, their three-dimensional (3D) real-world coordinates can be mapped to their corresponding two-dimensional (2D) image coordinates. For this mapping, a transformation matrix is ​​generated, which describes the relationship between these 3D and 2D points. In other words, the transformation matrix is ​​a result of the calibration. It can be used in any future context to interpret the 2D information of the camera image and convert it into real-world 3D information.

[0006] Calibration parameters include intrinsic and extrinsic parameters. While intrinsic values ​​are typically provided by the camera manufacturer, extrinsic values ​​can be determined based on a 3D-to-2D mapping or transformation. Extrinsic parameters are factory-calibrated when the vehicle is stationary. If the extrinsic calibration values ​​(i.e., the transformation matrix) determined at the factory are applied to camera images captured in real-life conditions, they may produce inaccurate results, such as regarding object distances to the vehicle. This could be due to road conditions, irregularities (such as lane grooves), and / or loads or vehicles on the front and / or rear axles.

[0007] The current calibration method used on roads requires an algorithm for object recognition. When an object known to the vision system is identified, the corresponding 2D point in the captured image is compared with a known 2D point in the vision system for matching. As the vehicle moves, more consecutive images of the known object are captured. Using known intrinsic parameters from the camera manufacturer, vehicle motion information based on odometer readings (distance traveled), and the 2D point matching results, the camera's extrinsic parameters can be calculated. Calibration can only be performed on roads when a known object is identified. Timeliness of calibration cannot be guaranteed.

[0008] Using vanishing point techniques based on lane markings on the road to calibrate extrinsic camera parameters is also a common method. Lane marking-based calibration can only be performed when the road is straight, and only when the vehicle with the camera to be calibrated is traveling along a straight road.

[0009] US 2024 / 0107001 A1 describes a camera calibration method based on vanishing point technology.

[0010] US 2021 / 0241492 A1 also relates to a method for calibrating a vehicle vision system. It mentions providing a known three-dimensional (3D) target in front of the vision system's camera and evaluating edges and vanishing lines from captured images to calculate extrinsic camera parameters. The known 3D target must be positioned in front of the camera. This means the vehicle must remain stationary. Therefore, the extrinsic parameters may deviate from those evaluated when the vehicle is stationary as the car moves.

[0011] A similar method is known from CN 114627190 A. Here, a known and fixed rectangular grid pattern is used in front of the camera to be calibrated. This pattern can be, for example, fixed as multiple strips arranged on the road surface in front of the vehicle equipped with the camera. The camera position is defined as the origin of a coordinate system, and the actual coordinates of the grid points in this coordinate system are determined. The camera then captures an image of the pattern, and the image pixel coordinates of the grid points are obtained. A transformation model or transformation matrix between the image pixel coordinates and the actual coordinates is established. The transformation model is used when operating a vehicle on the road, for example, to determine the actual coordinates of points in the vehicle environment based on the image pixel coordinates of the points.

[0012] DE 10 2022 206 865 A1 describes using the projected light pattern as a calibration target.

[0013] These methods cannot provide real-time, on-demand calibration that ensures camera extrinsic parameters are available along with the latest values. Summary of the Invention

[0014] Therefore, the object of the present invention is to provide up-to-date external calibration parameters for vehicle vision systems with at least one onboard camera during driving, particularly for applications in autonomous driving.

[0015] This objective is achieved through the subject matter of the independent claims. Advantageous embodiments are described by the dependent claims, the following description, and the accompanying drawings.

[0016] This invention relates to a novel camera calibration method utilizing vehicle headlights, particularly intelligent matrix LED lighting for vehicles. The invention is based on the understanding that modern vehicle headlights, especially intelligent matrix LED lighting, can generate different lighting patterns on the road. These patterns can be known to the vehicle's calibration module to calibrate the camera's extrinsic parameters. Because patterns from the intelligent matrix LED lighting can be generated on demand, the camera calibration process can be performed at any time when needed.

[0017] The present invention relates to a method for calibrating a vehicle vision system for motor vehicles, the vehicle vision system including a calibration module and at least one on-board camera having an image processor.

[0018] The method according to the invention includes equipping a motor vehicle with a vehicle vision system, wherein at least one onboard camera is disposed at a camera position on the motor vehicle and arranged relative to the vehicle's headlights, such that the field of view of the at least one onboard camera at least partially overlaps with the projection area of ​​the vehicle's headlights. In other words, at least one onboard camera is mounted on the vehicle, for example, mounted behind the vehicle's windshield in the rearview mirror area and / or on the vehicle's front grille and / or on the vehicle's rearview mirrors and / or on the vehicle's rear bumper. Alternatively, the vehicle vision system may include multiple onboard cameras, which may be mounted on the vehicle such that their respective fields of view at least partially overlap with the projection area of ​​the vehicle's headlights. The headlights may be the vehicle's front headlights and / or rear headlights.

[0019] Another step of the method according to the invention includes detecting calibration indication conditions and activating the calibration module of the vehicle vision system based on the detected calibration indication conditions. In other words, the calibration module may not be constantly running, but can be activated when needed. Various examples of calibration indication conditions are described in further detail below.

[0020] The activated calibration module can generate control signals for the headlight control unit of a motor vehicle, wherein the control signals describe the light pattern to be projected by the vehicle's headlights onto a projection surface within the projection area. The projection area can be in front of and / or behind the vehicle. It can include the ground of the road on which the vehicle is currently traveling. It can also include walls in front of or behind the vehicle, particularly vertical walls.

[0021] The light pattern comprises object coordinates describing the size of the light pattern and / or its position relative to the camera position in an object coordinate system. The camera position can be defined as being at the origin of the object coordinate system. In other words, an object coordinate system can be established, either centered at a known position of the camera or with its origin at the known position of the camera. The object coordinate system can be aligned with the x, y, and z axes of the vehicle coordinate system, where the x-axis is the vehicle's vertical axis, the y-axis is the vehicle's horizontal axis, and the z-axis is the vehicle's vertical axis. Within this object coordinate system, the real-world coordinates of the light pattern can be known and described by control signals used by the headlight control unit.

[0022] The headlight control unit executes control signals to cause the vehicle's headlights to project a light pattern onto a projection surface within the projection area. The headlights may include multiple light-emitting diodes (LEDs), which can be individually controlled by the headlight control unit to project a light pattern onto the projection surface within the projection area.

[0023] Another step of the method according to the invention includes activating at least one vehicle-mounted camera and capturing image data using the at least one vehicle-mounted camera, wherein the image data represents the field of view of the vehicle-mounted camera and describes an image of the projected light pattern. In other words, the at least one vehicle-mounted camera captures an image of at least a portion of the projected light pattern using known object coordinates of the light pattern.

[0024] The captured image data is provided to an image processor, which assigns image coordinates to the captured image of the light pattern by processing the captured image data at the image processor. For example, the image processor can run an object recognition algorithm on the captured image data to identify at least one edge pixel of an edge of one of the light pattern elements. The image processor can then assign image coordinates to the identified edge pixels, where the image coordinates are given in an image coordinate system. This can be done for multiple edge pixels or other characteristic features of the captured image. As with the object coordinate system described above, the known camera position can be defined at the origin of the image coordinate system. Thus, the image coordinates describe the size of the light pattern in the image and / or the position of the light pattern relative to the camera position in the image coordinate system.

[0025] Another step of the method according to the invention includes a calibration module performing a current transformation matrix or transformation model, based on the current calibration parameters of the vehicle vision system, to transform image coordinates into desired object coordinates. The current transformation matrix or transformation model may have already been generated during the initial calibration of the vehicle vision system, for example, in the factory setup described above. By applying the current transformation matrix to the image coordinates, they can be mapped to corresponding real-world or object coordinates, referred to herein and hereinafter as the desired object coordinates. If the vehicle remains in the same state as in the factory settings, the desired object coordinates should be consistent with the light pattern object coordinates. However, if the vehicle condition has changed, it is likely that a difference or mismatch between the desired object coordinates and the light pattern object coordinates will be observed.

[0026] Therefore, a further step of the method according to the invention includes inferring updated calibration parameters of the vehicle vision system from the difference or mismatch between the desired object coordinates and the light pattern object coordinates, and generating an updated transformation matrix based on the updated calibration parameters. In other words, the calibration parameters of the current transformation matrix are adjusted such that applying the updated transformation matrix to the image coordinates produces known light pattern object coordinates.

[0027] Another step of the method according to the invention includes calibrating the vehicle vision system using updated calibration parameters.

[0028] The advantage of the method of the present invention is that it provides the latest external calibration parameters for vehicle vision systems with at least one vehicle camera, which can operate while driving without performing complex object recognition algorithms or further regulations such as driving on straight roads with parallel lane markings.

[0029] Further embodiments of the present invention offer additional advantages.

[0030] According to an embodiment of the method, object coordinates of the camera position are defined at the origin of the object coordinate system, and image coordinates of the camera position are defined at the origin of the image coordinate system. By performing a current transformation matrix, the image coordinates describing the size of the light pattern in the image and / or the position of the light pattern relative to the camera position in the image coordinate system are transformed into desired object coordinates, which are also defined in the object coordinate system. In other words, all coordinates are transferred to a common coordinate system, the object coordinate system, to facilitate easy and rapid analysis of any mismatch between the desired object coordinates and the object coordinates of the light pattern.

[0031] According to another embodiment of the method, the image processor includes an object detection module that detects at least one edge pixel of at least one light pattern element and assigns image coordinates to the at least one edge pixel. For example, the light pattern may include multiple light pattern elements arranged in a predetermined known pattern. The light pattern elements may include, for example, two-dimensional geometries such as triangles, rectangles, circles, etc. These light pattern elements can be projected onto a projection surface in a projection area at predetermined sizes and / or predetermined distances between each other and / or predetermined distances to a camera position, according to control signals. The object detection module may, for example, detect edge pixels on one side of a rectangular light pattern element and another edge pixel on the opposite side of the same rectangular light pixel. Based on the image coordinates of these two pixels, a calibration module can determine the size of the rectangular light pattern element. The true or real-world size of the rectangular light pattern element is known to the calibration module based on the known light pattern object coordinates. As previously described, a current transformation matrix or transformation module based on current calibration parameters can be applied to the image coordinates of the two edge pixels of the light pattern element, thereby transforming them into desired object coordinates. If a mismatch is detected between the known object coordinates and the desired object coordinates of the light pattern element, updated calibration parameters can be derived from this mismatch.

[0032] According to another embodiment, the detection calibration indication condition includes receiving a time signal indicating the maximum permissible time elapsed since the last calibration of the vehicle vision system. The permissible time can be manually set by the system user or automatically set according to the system's intended use. The permissible time or update interval can be set to a few milliseconds to a few seconds, such that calibration runs every few milliseconds to a few seconds, particularly automatically.

[0033] Alternatively or additionally, the detection calibration indication condition includes receiving a tilt signal from at least one tilt sensor of the motor vehicle, wherein the signal describes a change in the vehicle's tilt relative to the calibration tilt. In other words, the calibration module can be activated when an abnormal tilt of the vehicle—preferably along its longitudinal axis—is detected. An abnormal tilt or tilt angle indicates a change in the camera's tilt parameters that is large enough to cause inaccuracies in object distance determination based on camera data, exceeding the permissible level of inaccuracy. Therefore, calibration is automatically initiated when an abnormal tilt is detected, which could be due to the vehicle driving steeply uphill or downhill or due to a heavy load on one of the vehicle's axles, such as when the vehicle is towing a trailer or has a heavy load in the trunk.

[0034] Alternatively or additionally, detecting calibration indication conditions includes receiving a load signal from at least one load sensor of the motor vehicle, wherein the signal describes a change in the motor vehicle's load conditions relative to calibrated load conditions. The described embodiments relate to situations where the vehicle is heavily loaded, wherein the load conditions preferably describe a load in kilograms, particularly in the trunk of the motor vehicle.

[0035] Alternatively or additionally, the detection calibration indication conditions include receiving acceleration signals from at least one acceleration sensor of the motor vehicle, wherein changes in the vehicle's tilt are inferred from the acceleration signals. Sudden acceleration or deceleration of the vehicle may, respectively, cause changes in tilt upward or downward along the vehicle's longitudinal axis. This may again alter the tilt parameters of at least one onboard camera—such as the vehicle's front-facing camera—making recalibration of external camera parameters necessary to avoid intolerable inaccuracies in object distance detection based on camera data.

[0036] Alternatively or additionally, the detection calibration indication conditions include receiving an ignition signal from the vehicle's ignition sensor, wherein the calibration module is activated upon receiving the ignition signal. In other words, the recalibration of the vehicle vision system can be performed every time the vehicle engine is started.

[0037] Another embodiment of the method includes placing at least one vehicle-mounted camera at the front of the vehicle, particularly integrated into the windshield and / or front grille of the vehicle.

[0038] Alternatively or additionally, the method includes mounting at least one vehicle-mounted camera at the rear of the motor vehicle, particularly integrating it into the rear bumper of the motor vehicle.

[0039] Preferably, the headlight control unit controls the vehicle's front headlights and / or rear headlights. In other words, the headlight control unit can operate the vehicle's front headlights to calibrate the vehicle's front camera, and the headlight control unit can operate the rear headlights to project a known light pattern onto a projection surface that is at least partially within the field of view of the respective camera to be calibrated in order to calibrate the vehicle's rear camera.

[0040] Another embodiment of the method includes: a projection surface arranged at an angle relative to the vehicle's current orientation plane—particularly relative to the vehicle's xy plane—wherein this angle, and particularly its distortion effects, are considered when calibrating the vehicle's vision system. For example, the projection surface may be the ground on which the vehicle rests, and the angle may be zero degrees. The projection surface may also be or include a wall in front of or behind the vehicle, such that the angle is 90 degrees. Any angle in between is also possible. In other words, the angle of the projection surface relative to the vehicle's current orientation plane may vary and affect the projected light pattern in terms of distortion. For example, the angle may also be varied within the projection surface, such that a first portion of the light pattern is projected onto a first portion of the projection surface, which has a different tilt or angle relative to the vehicle's current orientation plane than a second portion of the projection surface, onto which a second portion of the light pattern may be projected. In other words, the projection surface may be uneven or wavy, or include at least one kink. This non-uniformity will affect the distortion of the light pattern, as can be explained according to the described embodiment.

[0041] For any use cases or situations that may occur in the method according to the invention and are not explicitly described herein, it may be specified that, according to the computer-implemented method or the method according to the invention, error messages and / or user feedback prompts and / or default settings and / or predetermined initial states are output.

[0042] Another aspect of the present invention relates to a vision system for a motor vehicle, comprising:

[0043] - Calibration module,

[0044] - At least one vehicle-mounted camera with an image processor,

[0045] -Object detection module

[0046] The vision system is designed to be calibrated according to any of the methods described in the embodiments.

[0047] Another aspect of the present invention relates to an electronic vehicle guidance system, comprising:

[0048] - A processing device having at least one computing unit.

[0049] - Headlight control unit

[0050] - At least one tilt sensor

[0051] -At least one load sensor,

[0052] -At least one accelerometer sensor, and

[0053] - Ignition sensor,

[0054] The vision system according to the present invention is integrated into the electronic vehicle guidance system.

[0055] Electronic vehicle guidance systems can be understood as electronic systems configured to guide vehicles in a fully automated or fully autonomous manner, and in particular, without requiring manual intervention or control by the vehicle's driver or user. The vehicle automatically performs all necessary functions, such as steering, deceleration, and / or acceleration, as well as monitoring and recording road traffic and corresponding reactions. Specifically, electronic vehicle guidance systems can achieve fully automated or fully autonomous driving modes according to Level 5 of SAE J3016 classification. Electronic vehicle guidance systems can also be implemented as Advanced Driver Assistance Systems (ADAS) that assist the driver in partially automated or partially autonomous driving. Specifically, electronic vehicle guidance systems can achieve partially automated or partially autonomous driving modes according to Levels 1 to 4 of SAE J3016 classification. Here and below, SAE J3016 refers to the corresponding standard dated April 2021.

[0056] Therefore, at least partially automated guided vehicles may include Level 5 vehicles, classified according to SAE J3016, guided in fully automated or fully autonomous driving modes. At least partially automated guided vehicles may also include Levels 1 to 4, classified according to SAE J3016, guided in partially automated or partially autonomous driving modes.

[0057] Electronic guidance signals may be provided, for example, to one or more actuators of the vehicle, including, for example, one or more brake actuators and / or one or more steering actuators and / or one or more propulsion motors. One or more actuators may influence the longitudinal and / or lateral control of the vehicle to automatically guide the vehicle, at least in part, based on the electronic guidance signals.

[0058] Auxiliary information can be output using the vehicle's output devices, such as displays and / or audio output systems and / or haptic output systems.

[0059] Another aspect of the present invention relates to a motor vehicle having a vision system according to the invention and an electronic vehicle guidance system according to the invention, the motor vehicle comprising:

[0060] - Vehicle headlights and / or vehicle taillights, wherein the headlights and / or taillights are preferably intelligent matrix LED headlights.

[0061] Another aspect of the present invention relates to a computer program comprising instructions which, when executed by a processing device, particularly by a device that:

[0062] -The calibration module and / or image processor and / or object detection module of the vehicle vision system according to the present invention, and / or

[0063] - A processing device having at least one computing unit of the electronic vehicle guidance system according to the present invention

[0064] The processing device is instructed to perform computational steps according to any of the methods described in the embodiments.

[0065] Another aspect of the present invention relates to a computer-readable storage medium having a computer program according to the present invention stored thereon. The computer-readable storage medium and computer program according to the present invention can be described as a computer program product.

[0066] In other words, according to another aspect of the invention, a data processing system is provided, which is configured to perform a computer-implemented method according to the invention.

[0067] The terms "data processing system" and "data processing device" may be used interchangeably in this disclosure. In this disclosure, a data processing device (also referred to as a computing device) may be understood, for example, as a device having processing circuitry for processing data. Therefore, a data processing device can perform computational operations to process data. Index access to a data structure (e.g., a lookup table (LUT) or a database) can also be considered a computational operation. Data processing implemented partially or entirely in hardware can also be considered a computational operation.

[0068] Specifically, the data processing device may include one or more computers, one or more microcontrollers, and / or one or more integrated circuits, such as one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), and / or one or more system-on-a-chip (SoCs). The data processing device may also include one or more processors, such as one or more microprocessors, one or more central processing units (CPUs), one or more graphics processing units (GPUs), and / or one or more signal processors, particularly one or more digital signal processors (DSPs). The data processing device may also include a physical or virtual cluster of computers or other said devices.

[0069] The data processing device may also include one or more hardware and / or software interfaces, for example, for receiving and / or providing data.

[0070] The data processing device may also include one or more memory devices. These memory devices may be implemented as volatile memory, such as dynamic random access memory (DRAM) or static random access memory (SRAM), or as non-volatile data memory, such as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or flash EEPROM, ferroelectric random access memory (FRAM), magnetoresistive random access memory (MRAM), or phase-change random access memory (PCRAM).

[0071] According to another aspect of the present invention, a computer program comprising instructions is provided. When executed by a data processing device, the instructions cause the data processing device to perform a computer-implemented method according to the present invention.

[0072] For example, instructions can be provided as program code. Program code can be provided, for example, as binary code or assembler and / or as source code of a programming language (such as C) and / or as a program script (such as Python).

[0073] According to another aspect of the invention, a further computer program including additional instructions is provided. When the further instructions are executed by the calibration module and / or image processor and / or object detection module of the vehicle vision system according to the invention and / or the electronic vehicle navigation system according to the invention, particularly by the data processing device of the electronic vehicle navigation system, the further instructions cause the calibration module and / or image processor and / or object detection module and / or electronic vehicle navigation system according to the invention to perform the method for calibrating the vehicle vision system according to the invention.

[0074] For example, further instructions can be provided as program code. The program code can be provided, for example, as binary code or assembler and / or as source code of a programming language (such as C) and / or as a program script (such as Python).

[0075] According to another aspect of the invention, a computer-readable storage medium is provided for storing a computer program and / or another computer program according to the invention.

[0076] A computer program, another computer program, and a computer-readable storage medium are corresponding computer program products that include instructions and / or additional instructions.

[0077] Further embodiments of the vision system and / or electronic vehicle navigation system and / or motor vehicle according to the invention directly follow various embodiments of the method according to the invention, and vice versa. Specifically, the various features, corresponding explanations, and advantages associated with the various embodiments of the method according to the invention can be similarly transferred to corresponding embodiments of the vision system and / or electronic vehicle navigation system and / or motor vehicle according to the invention. In particular, the electronic vehicle guidance system according to the invention is designed or programmed to perform the method according to the invention. In particular, the data processing apparatus of the electronic vehicle guidance system according to the invention performs the method according to the invention.

[0078] If this disclosure refers to information, data, or datasets being received by a receiving entity, particularly from a sending entity, this can be understood as the information, data, or dataset being received by the corresponding hardware receiving port, hardware receiving terminal, or hardware receiving interface of the receiving entity. For example, the information, data, or dataset may be transmitted, for instance, from a sending hardware port, sending hardware terminal, or sending hardware interface of the sending entity to a hardware receiving port, hardware receiving terminal, or hardware receiving interface via wired or wireless communication. Attached Figure Description

[0079] The invention will be explained in detail below with reference to specific exemplary embodiments and corresponding schematic diagrams. In the drawings, the same or functionally identical elements may be denoted by the same reference numerals. Descriptions of the same or functionally identical elements are not necessarily repeated in different drawings. In the drawings:

[0080] Figure 1 This demonstrates the impact of deviated camera extrinsic parameters on the distance estimation of vehicles in front of the vehicle;

[0081] Figure 2 A schematic diagram of a motor vehicle having a vision system according to an embodiment of the present invention is shown;

[0082] Figure 3 A schematic diagram illustrating three different possible light patterns that can be generated by a vision system according to an embodiment of the present invention;

[0083] Figure 4 A schematic diagram of an electronic vehicle guidance system in which a vision system according to an embodiment of the present invention is integrated is shown; and

[0084] Figure 5 A schematic diagram of a method for calibrating a vehicle vision system according to an embodiment of the present invention is shown. Detailed Implementation

[0085] Figure 1 This illustrates the effect of deviated camera extrinsic parameters on the distance estimation of the vehicle 10 in front. On the left side, Figure 1The calibration status is shown, where the horizontal dashed line 12 indicates the vanishing line of the camera 14 mounted at the front of the vehicle 10 (not shown, see reference). Figure 2 (For the mounting position of camera 14 on vehicle 10). When the load and / or movement of vehicle 10 (e.g., due to acceleration, uphill / downhill driving, etc.) deviates minimally from the calibration conditions, there is no significant difference between the calibration parameters and the actual parameters. In this case, the distance to the vehicle 10 ahead can be accurately estimated based on the calibrated parameters.

[0086] However, in Figure 1 In the case shown on the right-hand side, vehicle 10 may have a high load at its rear or be going uphill, and the tilt angle of camera 14 has changed, making it point more upwards towards the sky. Therefore, the calibration parameters may indicate a vanishing line 12 at position 16, while the actual parameters indicate a vanishing line at position 18. In this case, the estimated distance to the vehicle 10 ahead will be incorrect because the actual external parameters deviate from the calibrated external parameters.

[0087] One or more embodiments of the present invention relate to using a smart LED headlamp to calibrate external camera parameters. Figure 2 A schematic diagram of a motor vehicle 10 with a vision system 20 according to an embodiment of the present invention is shown, wherein intelligent LED headlights 22 are used to project a known light pattern in front of the vehicle 10. The headlights 22 are connected to a headlight control unit 24, which can control a leveling controller 26 for the headlights 22. Figure 2 The illustrated vision system 20 may also include tilt detection sensors 28 and 30 and an acceleration sensor 32, which can be designed to detect or measure the current tilt angle of the vehicle 10. Using the detected tilt angle from the tilt sensors 28 and 30 and / or the acceleration sensor 32, the headlight control unit 24 can correct the illumination angle of the headlights 22 to ensure that their illumination range meets specifications and to prevent glare to oncoming vehicles. This mechanism ensures that the distance of the known light pattern is a known distance from the vehicle 10 when a specified known light pattern is projected onto the road by the headlights 22.

[0088] Figure 2 At least one vehicle-mounted camera 14 is also shown, which is mounted on the front of the vehicle 10. The field of view of the camera 14 and the projection area of ​​the headlight 22 at least partially overlap, such that the camera 14 can capture an image of a known pattern of projected light.

[0089] Figure 3 Three examples of known light patterns are shown, which can be projected onto the ground in front of vehicle 10 by headlight 22. In this example, camera 14 is positioned on the windshield of vehicle 10 in the area of ​​the rearview mirror.

[0090] like Figure 3Each of the exemplary light patterns shown includes a plurality of light pattern elements 34. These pattern elements 34 are not intended to be the only possible patterns for the calibration method described herein. In theory, all patterns are possible as long as the vision system 20 knows the size of the light pattern elements 34 and their position relative to the camera position.

[0091] When the headlight control unit 24 is instructed to project a known pattern onto a projection surface (e.g., the road in front of vehicle 10), the headlight 22 outputs the desired pattern. The camera 14 can then capture an image of the known pattern. Since the 3D object coordinates of the pattern on the road surface are known to the vision system 20, the corresponding mapping from real-world 3D object coordinates to 2D image coordinates can be used to calculate the extrinsic camera parameters as described herein. This allows the vision system 20 to perform camera calibration at any time.

[0092] The method described in this paper can also work in conjunction with other calibration methods to compensate for their weaknesses. For example, calibration at night using known objects or lane markings on a road can be challenging. The method described in this paper can complement these known methods and provide good calibration even at night.

[0093] Instead of using a static target and ensuring that the vehicle 10 is stationary, the method according to the invention can project a desired known target from the vehicle's headlights 22 (e.g., LED matrix headlights) in the form of a known light pattern, regardless of whether the vehicle 10 is moving or stationary. Therefore, the latest external parameters can be evaluated when needed.

[0094] For example, while vehicle 10 is still in the factory, the generated pattern from headlight 22 can be used to calibrate the external camera parameters, thus ensuring that camera 14 is calibrated before leaving the factory. When vehicle 10 is then driven on the road, it can be recalibrated at any appropriate time to ensure that the calibrated parameters match the conditions of vehicle 10's use on the road.

[0095] This method can be used on a vehicle 10 equipped with headlights 22, which illuminate patterns on the road surface, particularly LED headlights. Besides the aforementioned matrix LED headlights, there may be other upcoming technologies that could replace them. For example, laser headlights and laser + matrix LED headlights are already commercially available. Therefore, in the future, technologies that can replace matrix LED headlights may exist. However, as long as the headlights 22 can provide accurate pattern projection in front of the camera 14, the calibration method described herein can be applied.

[0096] Figure 4 A schematic diagram of an exemplary electronic vehicle guidance system 36 in which a vision system 20 is integrated is shown.

[0097] The electronic vehicle guidance system 36 includes a data processing device 38 having at least one computing unit 40 and a headlight control unit 24, which can control the vehicle's headlights 22. A calibration module 42 of the vehicle vision system 20 can also be integrated into the data processing device 38. The electronic vehicle guidance system 36 may also include an environmental sensor system for the vehicle 10, including at least one tilt sensor 28, 30, at least one load sensor 44, at least one acceleration sensor 32, and an ignition sensor 46. The vision system 20 may also include at least one onboard camera 14, which can also be integrated into the environmental sensor system of the vehicle 10. The camera 14 may include an image processor 48 and an object detection module 50. Communication between the described components can be achieved through the vehicle 10's onboard communication network, for example, via a communication bus. Additionally or alternatively, at least a portion of the communication can be achieved wirelessly, for example via Bluetooth or other wireless communication methods.

[0098] Figure 5 A schematic diagram illustrating a method for calibrating a vehicle vision system 20 according to an embodiment of the present invention is shown. The vehicle vision system 20 includes a calibration module 42 and at least one onboard camera 14 having an image processor 48. The method includes the following steps:

[0099] S1: Equip the motor vehicle 10 with an onboard vision system 20, wherein at least one onboard camera 14 is disposed at the camera position of the motor vehicle 10 and arranged relative to the onboard headlights 22 of the motor vehicle 10, such that the field of view of the at least one onboard camera 14 and the projection area of ​​the onboard headlights 22 at least partially overlap.

[0100] S2: Detect calibration indication conditions and activate the calibration module 42 of the vehicle vision system 20 based on the detected calibration indication conditions.

[0101] S3: The calibration module 42 generates a control signal for the headlight control unit 24 of the motor vehicle 10, wherein the control signal describes a light pattern to be projected by the vehicle headlight 22 onto a projection surface in the projection area, wherein the light pattern includes light pattern object coordinates, the light pattern object coordinates describing the size of the light pattern and / or the position of the light pattern relative to the camera position in the object coordinate system.

[0102] S4: The headlight control unit 24 executes the control signal, and the vehicle headlight 22 projects the light pattern onto the projection surface in the projection area.

[0103] S5: Capture image data using at least one vehicle-mounted camera 14, wherein the image data represents the field of view of the vehicle-mounted camera 14 and describes an image of the projected light pattern.

[0104] S6: The captured image data is provided to the image processor 48, and by processing the captured image data at the image processor 48, image coordinates are assigned to the captured image of the light pattern, wherein the image coordinates describe the size of the light pattern in the image and / or the position of the light pattern relative to the camera position in the image coordinate system.

[0105] S7: The calibration module 42 executes the current transformation matrix, which is based on the current calibration parameters of the vehicle vision system 20, and is used to transform the image coordinates into the desired object coordinates.

[0106] S8: Derive updated calibration parameters for the vehicle vision system 20 from the difference or mismatch between the desired object coordinates and the light pattern object coordinates, and generate an updated transformation matrix based on the updated calibration parameters.

[0107] S9: Calibrate the vehicle vision system with updated calibration parameters 20.

[0108] The present invention relates to a method for calibrating a vehicle vision system for a motor vehicle, the vehicle vision system including a calibration module and at least one on-board camera having an image processor, wherein a known light pattern projected by the vehicle's headlights onto a projection surface within the field of view of the at least one on-board camera is used as a calibration target.

[0109] In summary, these examples demonstrate how to provide up-to-date external calibration parameters for vehicle vision systems with at least one onboard camera during driving, particularly for applications in autonomous driving.

Claims

1. A method for calibrating a vehicle vision system (20) for a motor vehicle (10), the vehicle vision system (20) comprising a calibration module (42) and at least one onboard camera (14) having an image processor (48), the method comprising the steps of: a) Equipping a motor vehicle (10) with a vehicle vision system (20) (S1), wherein at least one vehicle-mounted camera (14) is disposed at a camera position on the motor vehicle (10) and arranged relative to the headlights (22) of the motor vehicle (10), such that the field of view of at least one vehicle-mounted camera (10) and the projection area of ​​the headlights (22) at least partially overlap. b) Detect (S2) calibration indication conditions and activate the calibration module (42) of the vehicle vision system (20) according to the detected calibration indication conditions. c) A control signal generated by the calibration module (42) for the headlight control unit (24) of the motor vehicle (10), wherein the control signal describes a light pattern to be projected by the vehicle headlight (22) onto a projection surface within the projection area, wherein the light pattern includes light pattern object coordinates, which describe the size of the light pattern and / or the position of the light pattern relative to the camera position in the object coordinate system. d) The headlight control unit (24) executes the (S4) control signal, and the vehicle headlights (22) project a light pattern onto the projection surface within the projection area. e) Capture (S5) image data using at least one vehicle-mounted camera (14), wherein the image data represents the field of view of the vehicle-mounted camera (14) and describes an image of the projected light pattern. f) The captured image data is provided (S6) to the image processor (48), and image coordinates are assigned to the captured image of the light pattern via processing of the captured image data at the image processor (48), wherein the image coordinates describe the size of the light pattern in the image and / or the position of the light pattern relative to the camera position in the image coordinate system. g) The calibration module (42) performs (S7) a current transformation matrix based on the current calibration parameters of the vehicle vision system (20) to transform the image coordinates into the desired object coordinates. h) Derive (S9) updated calibration parameters for the vehicle vision system (20) from the difference between the desired object coordinates and the light pattern object coordinates, and generate an updated transformation matrix based on the updated calibration parameters, and i) Use updated calibration parameters to calibrate (S9) the vehicle vision system (20).

2. The method according to claim 1, wherein, - Define the object coordinates of the camera position at the origin of the object coordinate system. - Define the image coordinates of the camera position at the origin of the image coordinate system, and - By performing the current transformation matrix according to step g), the image coordinates describing the size of the light pattern in the image and / or the position of the light pattern relative to the camera position in the image coordinate system are transformed into the desired object coordinates, which are also defined in the object coordinate system.

3. The method according to any one of the preceding claims, wherein, The image processor (48) includes an object detection module (50), wherein the object detection module (50) detects at least one edge pixel of at least one light pattern element (34) in step f) and assigns image coordinates to at least one edge pixel.

4. The method according to any one of the preceding claims, wherein, According to step b), the calibration indication condition includes receiving a time signal that indicates the maximum permissible time elapsed since the last calibration of the vehicle vision system (20).

5. The method according to any one of the preceding claims, wherein, Step b) Detecting calibration indication conditions includes receiving a tilt signal from at least one tilt sensor (28, 30) of the motor vehicle (10), wherein the signal describes the change in the tilt of the motor vehicle (10) relative to the calibration tilt.

6. The method according to any one of the preceding claims, wherein, Step b) Detecting calibration indication conditions includes receiving a load signal from at least one load sensor (44) of the motor vehicle (10), wherein the signal describes the change in the load conditions of the motor vehicle (10) relative to the calibration load conditions.

7. The method according to claim 6, wherein, The load condition describes the load in kilograms, particularly the load in the trunk of a motor vehicle (10).

8. The method according to any one of the preceding claims, wherein, According to step b), the detection calibration indication conditions include receiving an acceleration signal from at least one acceleration sensor (32) of the motor vehicle (10), wherein the change in tilt of the motor vehicle (10) is derived from the acceleration signal.

9. The method according to any one of the preceding claims, wherein, According to step b), the detection calibration indication conditions include receiving an ignition signal from the ignition sensor (46) of the motor vehicle (10), wherein the calibration module (42) is activated when the ignition signal is received.

10. The method according to any one of the preceding claims, wherein, Step a) includes placing at least one vehicle-mounted camera (14) at the front of the motor vehicle (10), particularly integrated into the windshield of the motor vehicle (10) and / or the front grille of the motor vehicle (10).

11. The method according to any one of the preceding claims, wherein, Step a) includes setting at least one vehicle-mounted camera (14) on the rear side of the motor vehicle (10), particularly integrating it into the rear bumper of the motor vehicle (10).

12. The method according to any one of the preceding claims, wherein, The headlight control unit (24) controls the vehicle's front headlights (22) and / or the vehicle's rear headlights (22).

13. The method according to any one of the preceding claims, wherein, The projection surface is arranged at an angle relative to the current orientation plane of the vehicle (10), particularly relative to the xy plane of the vehicle (10), wherein the angle is taken into account when calibrating the vehicle vision system (20).

14. A vision system (20) for a motor vehicle (10), comprising: - Calibration module (42); - At least one vehicle-mounted camera (14) with an image processor (48). -Object detection module (50) The vision system (20) is designed to be calibrated according to any one of the preceding claims.

15. An electronic vehicle guidance system (36), comprising: - A data processing device (38) having at least one computing unit (40); - Headlight control unit (24); - At least one tilt sensor (28, 30); -At least one load sensor (44); -At least one accelerometer sensor (32); and - Ignition sensor (46). The vision system (20) according to claim 14 is integrated into the electronic vehicle guidance system (36).

16. A motor vehicle (10) having a vision system (20) according to claim 14 and an electronic vehicle guidance system (36) according to claim 15, the motor vehicle (10) comprising: - Vehicle front headlights (22) and / or vehicle rear headlights (22), wherein the front headlights and / or the rear headlights (22) are preferably intelligent matrix LED headlights.

17. A computer program comprising instructions, said instructions, when executed by a data processing device (38), particularly by... - The calibration module (42) and / or image processor (48) and / or object detection module (50) of the vehicle vision system (20) according to claim 14, and / or - Data processing device (38) having at least one computing unit (40) of the electronic vehicle guidance system (36) according to claim 15. During execution, the data processing device (38) is instructed to perform the calculation steps of the method according to any one of claims 1 to 13.

18. A computer-readable storage medium having a computer program as claimed in claim 17 stored thereon.

Images