Method for compensating for a sensor blockage

Abstract

The invention relates to a method in which the following steps are carried out to compensate for a blockage of a sensor: -Detection of a blockage at the sensor by evaluating the image area captured by the sensor, -Determining the position of the obstruction, the dimensions of the obstruction in the sensor's field of view based on its position in the vehicle, and determining the sensor's orientation relative to its vehicle environment. -Determination of the sensor's image area corresponding to the obstruction in a synthetic 2D view using a satellite image-to-street image transformation, which is determined from images of a satellite system based on the position of the obstruction, the dimensions of the obstruction and the orientation of the sensor, and preferably the optical data of the sensor. -Superimposition of the image area corresponding to the blockage in the synthetic 2D view onto the image area corresponding to the blockage, as displayed in a display device and detected by the sensor.

Description

[0001] The invention relates to a method for compensating for a blockage of a sensor of a vehicle and a vehicle for carrying out the method.

[0002] Modern vehicles are equipped with vision assistance systems that scan the driver's field of vision and display it on a screen. For example, a surround-view system assists the driver in their driving task. Rain, driving through muddy conditions, driving in traffic, etc., can cause dirt to accumulate on the cameras. This dirt can sometimes remain on the cameras for extended periods and be perceived as distracting by the driver.

[0003] To solve this problem, an additional secondary vehicle camera can be positioned so that its field of view captures the portion of the original field of view of the vision assistance system obscured by dirt. By superimposing the images from the secondary vehicle camera and the images from the vision assistance system, the original field of view can be restored. The disadvantage of this solution is that an additional camera is required.

[0004] US Patent 2021 / 0201464A1 discloses a system for detecting sensor degradation, comprising multiple sensors configured to capture image data from an environment. The image data is compared, and discrepancies between corresponding image areas captured by different sensors indicate a blockage in one of the sensors. The operation of a vehicle or other system can be controlled, at least partially, based on the detection of degradation in the image data captured by the sensors.

[0005] Document US 10,701,300 B2 describes a display system for generating a composite view of an area behind a vehicle towing a trailer. A first camera is designed to output initial image data corresponding to a first image and is configured to be mounted on the vehicle facing backwards. A second camera is designed to output secondary image data corresponding to a second image and is configured to be mounted on the trailer facing backwards. An image processor receives the initial and secondary image data. The image processor is configured to combine the initial and secondary image data to generate composite image data corresponding to a composite image.

[0006] The publication “HIROHASHI, Yoshihiro. et al. Removal of image obstacles for vehicle-mounted surrounding monitoring cameras by real-time video inpainting. In: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops. 2020. pp. 214-215” describes a method for removing obstacles such as raindrops, dust, or dirt from the images of vehicle-mounted surrounding monitoring cameras (SMCs) in real time. The proposed approach is based on video inpainting, in which obscured image areas are replaced with matching, unobstructed pixels from previous video frames. A convolutional neural network (CNN) is used to estimate the optical flow between the frames, i.e., to track the movement of objects and scenery.

[0007] German patent DE 10 2013 016 247 A1 describes a method for augmented display in vehicles that ensures the stable and positionally accurate overlay of virtual additional information onto a video image of the vehicle's surroundings. The core of the invention is a two-stage hybrid process. First, an approximate preliminary position of the information is determined using a coarse positioning method (e.g., GPS). This position is then precisely adjusted using a fine, image-based method (e.g., markerless feature tracking) to find a stable target position. This approach combines the advantages of different techniques to compensate for instabilities during driving maneuvers and to firmly "anchor" the virtual information in the real image.

[0008] The publication “GAO, Wenjie, et al. Complementing onboard sensors with satellite maps: a new perspective for HD map construction. In: 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE. 2024. pp. 11103-11109” describes how to supplement data from vehicle sensors with cloud-based satellite maps. For this purpose, a hierarchical fusion module is presented, consisting of two stages: feature-level fusion and BEV-level fusion.

[0009] The invention is based on the objective of providing an alternative method for compensating a vehicle environment image in a blockage or obscuration area caused by dirt or damage.

[0010] The problem is solved according to the invention by a method having the features of claim 1.

[0011] Advantageous embodiments of the invention are the subject of the dependent claims.

[0012] According to one aspect of the present invention, a method is proposed in which the following steps are carried out to compensate for a blockage of a sensor: - Detection of a blockage at the sensor by evaluating the image area captured by the sensor, - Determining the position and dimensions of the obstruction within the sensor's field of view based on its position in the vehicle, and determining the sensor's orientation relative to its vehicle environment. - Determination of the sensor's image area corresponding to the obstruction in a synthetic 2D view using a satellite image-to-street image transformation, which is determined from images of a satellite system based on the position of the obstruction, the dimensions of the obstruction and the orientation of the sensor, and preferably the optical data of the sensor. - Overlay of the image area corresponding to the blockage in the synthetic 2D view with the image area corresponding to the blockage, as displayed in a display device and captured by the sensor.

[0013] A camera sensor obstruction is determined by analyzing the camera image. The camera's position and orientation within a vehicle coordinate system are known from the design or CAD data. An obstruction detected in the camera's field of view can be transferred to the vehicle coordinate system using the sensor's optical or geometric data. In this context, an obstruction is defined as dirt on the sensor lens or the lens being covered by an object that blocks the camera's field of view. The sensor's viewing direction is determined by assessing its orientation relative to the vehicle's surroundings. This position and orientation of the camera, as well as the position and dimensions of the obstruction, are preferably transformed into a coordinate system common to a satellite system. Additionally, the camera's optical data is preferably transmitted.A synthetic 2D view of the vehicle's surroundings is retrieved, derived from satellite imagery using a satellite image-to-street image transformation based on the position and dimensions of the obstruction, the sensor's orientation, and preferably the sensor's optical data. The optical data includes extrinsic camera parameters such as lens diameter, lens curvature, focal length, angle of view, and / or field of view. These parameters are necessary to determine the precise position of the obstruction on the camera lens as well as the field of view covered by the obstruction.

[0014] Based on satellite imagery of the Earth's surface acquired through overflights, data—that is, image data—of the vehicle's surroundings are determined using the vehicle's geolocation. This image data includes at least a top-down view of the surroundings and dimensions, i.e., at least elevation data of objects within the image.

[0015] Using a satellite image-to-street image transformation such as Sat2Cam, synthetic 2D views of the image area corresponding to the obstruction are generated from the image data. Based on the sensor's position and orientation, as well as the position and dimensions of the obstruction in the coordinate system shared with the satellite system, the synthetic 2D view of the sensor's image area corresponding to the obstruction is determined from the satellite system's image data.

[0016] Methods for satellite image-to-street image transformation are known from the prior art, for example from the publications by Y. Shi, D. Campbell, X. Yu and H. Li, “Geometry-Guided Street-View Panorama Synthesis From Satellite Imagery,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 12, pp. 10009-10022, 1 Dec. 2022, doi: 10.1109 / TPAMI.2022.3140750, Weijia Li, Jun He, Junyan Ye, Huaping Zhong, Zhimeng Zheng, Zilong Huang, Dahua Lin, Conghui He, "CrossViewDiff: A Cross-View Diffusion Model for Satellite-to-Street View Synthesis", https: / / doi.org / 10.48550 / arXiv.2408.14765 and in Aysim Toker, Qunjie Zhou, Maxim Maximov, Laura Leal-Taixe, “Coming Down to Earth: Satellite-to-Street View Synthesis for Geo-Localization,” Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 6488-6497.

[0017] Another method for synthesizing streetscapes from satellite imagery is described in the publication "Geometry-Guided Street-View Panorama Synthesis from Satellite Imagery" by Yujiao Shi, Dylan Campbell, Xin Yu and Hongdong Li, see https: / / arxiv.org / pdf / 2103.01623. The methods described in this document allow for rendering a streetscape from satellite imagery within 0.2s to 0.02s.

[0018] The image area of ​​the synthetic 2D view corresponding to the sensor's line of sight is superimposed and adjusted to the image area of ​​the blockage displayed on a screen, which was detected by the sensor. This adjustment can include geometric adjustments and / or adjustments to color and brightness values, minimizing edge distortions ("seams") at the transition between the vehicle's surroundings image and the synthetic sub-image. In other words, the image area of ​​the synthetic 2D view is displayed on the screen at the exact position corresponding to the image area of ​​the blockage detected by the sensor.

[0019] The method involves supplementing an image area of ​​the vehicle's surroundings, captured by the sensor and displayed in the display device, with blockages caused by dirt, i.e., obscurations, so that the user's view of the vehicle's surroundings is seemingly no longer obstructed.

[0020] Advantageously, the method does not require additional camera systems and is therefore particularly robust and easy to integrate.

[0021] In a further embodiment of the method, the blockage is determined based on discontinuities in the optical flux within the image stream captured by the sensor. When the vehicle is in motion, the optical flux in the blockage area detected by the sensor deviates significantly from the area detected by the sensor that is not affected by the blockage; that is, virtually no optical flux can be detected in the blockage area. Therefore, the blockage area can advantageously be detected at the sensor with high accuracy. Due to the known geometric data of the sensor and its known arrangement in the vehicle, the blockage area can be determined in the vehicle coordinate system and transformed into the coordinate system common to the satellite system.

[0022] In another embodiment, a dynamic object in the vehicle's vicinity is detected by means of an additional sensor, and a corresponding position is determined on the display device. Dynamic objects in the vehicle's environment can be detected using a radar sensor or a lidar sensor. This allows the position and distance to the camera sensor to be determined within the vehicle's coordinate system. By projecting the object onto the camera sensor, it is possible to determine where it would appear on the display device. A representation of a detected dynamic object, or a symbol associated with that object, is superimposed on the image area of ​​the synthetic 2D view on the display device, provided its position lies within the area of ​​the obstruction displayed on the display device or within the synthetic 2D view superimposed on the obstruction area. In other words, the representation of the dynamic object is displayed at the position of the obstruction on the display device, which is superimposed on the synthetic 2D view. Preferably, the dynamic object detected by radar or lidar is classified, i.e., identified, for example, as a bicycle, pedestrian, or vehicle, and a symbol corresponding to the object classification is determined and displayed on the display unit as described above. Advantageously, this allows dynamic objects to be displayed within the display area and prevents them from being overlooked.

[0023] In a further additional embodiment, the method is triggered when the driver's gaze is directed towards the display device. Advantageously, the method and associated calculations are only executed when the user is looking at the display device, thus preventing unnecessary heat generation and saving energy.

[0024] In one embodiment of the method, the superimposition of the synthetic 2D view is adapted to the image area captured by the sensor using image registration and / or contrast adjustment. Image registration ensures that the superimposed image area of ​​the synthetic 2D view aligns precisely with the occlusion area on the display unit, even in irregularly shaped image areas. Additional contrast adjustment achieves seamless transitions to the image areas on the display unit that are not affected by the occlusion.

[0025] In a further additional or alternative embodiment, the position of the obstruction, its dimensions, and the sensor's orientation are transformed into a coordinate system common to a satellite system. Having all data in a common coordinate system allows for efficient performance of all calculations, particularly the satellite image-to-street image transformation.

[0026] In another embodiment of the method, the World Geodetic System 1984 (WGS84) is chosen as the common coordinate system. Advantageously, WGS84 is suitable for the position and orientation of the sensor and the satellite system as a common coordinate system, since many satellite systems already provide data in WGS84, and thus, ideally, only the vehicle data needs to be transformed.

[0027] A vehicle equipped to carry out the aforementioned procedure includes a camera for capturing an image stream of the vehicle's surroundings, the image stream being transmitted to a display unit, such as a screen. A control unit determines the position and dimensions of the camera's sensor obstruction within a vehicle coordinate system, based on known optical data from the camera and the sensor's position within the vehicle. The camera's orientation relative to its surroundings is determined, for example, by GPS. The control unit then transforms the obstruction's position, dimensions, and sensor orientation into a coordinate system common to a satellite system.The control unit receives a synthetic 2D view of the area obscured by the obstruction, generated from satellite imagery using position coordinates, dimensions of the obstruction, sensor orientation, and optical data from the sensor's satellite imagery. This synthetic 2D view is created in the sensor's line of sight. The satellite image-to-street-image transformation is performed on and retrieved from an external server. The control unit then overlays this synthetic 2D view onto the corresponding area of ​​the obstruction on the display unit.

[0028] Exemplary embodiments of the invention are explained in more detail below with reference to drawings.

[0029] This shows: Fig. 1 View of a vehicle cockpit with a display unit encompassing a blockage, Fig. 2 View from Fig. 1 with a synthetic 2D view superimposed on the blockage, Fig. 3. A schematic diagram of the procedure.

[0030] Corresponding parts are marked with the same reference symbol in all drawings.

[0031] Fig. Figure 1 shows a purely schematic perspective representation of a vehicle cockpit 1 in a vehicle environment U. The vehicle cockpit 1 is assigned to a vehicle-related vehicle coordinate system 3.

[0032] The pose (that is, the position and orientation) of the vehicle comprising the vehicle cockpit 1 in relation to the vehicle environment U is specified by reference to a vehicle-independent coordinate system 4, which may, for example, be selected in a unified form as World Geodetic System 1984 (WGS84).

[0033] The vehicle 1 includes a camera 5 that captures an image of the vehicle's surroundings U and displays it on a display device 7. Due to a blockage of the camera 5 or its lens, a portion of the surroundings U cannot be displayed on the display device 7. This image area on the display device 7, hereinafter referred to as the display blockage 9, correlates with the image area captured by the sensor, i.e., the area corresponding to the blockage. The vehicle further includes a lidar or radar sensor 6, which is likewise configured to detect the surroundings U, in particular dynamic objects.

[0034] Display blockage 9 represents the area not detectable due to the blockage at camera 5 as a black area. According to Fig. 1. The building 11 and the vehicle 13 are not visible on the display device 7; a vehicle user who is not shown only sees the display blockage 9.

[0035] Based on the Fig. 2 and Fig. Figure 3 shows how a display blockage 9 is removed. In a first step S1, a blockage at the sensor is detected by evaluating the image area captured by the sensor.

[0036] In a further step S3, a position and dimension of the blockage in the image area of ​​the sensor is determined based on a position in the vehicle relative to the vehicle coordinate system 3 and a determination of a sensor orientation with respect to its vehicle environment U.

[0037] In a subsequent optional step S5, the position of the blockage, dimensions of the blockage and orientation of the sensor are transformed into a coordinate system common to a satellite system 4.

[0038] In a further step S 7, the image area of ​​the sensor corresponding to the blockage is determined in a synthetic 2D view 15 by means of a satellite image-street image transformation, which is determined from images of a satellite system 17 based on the position of the blockage, the dimensions of the blockage, the optical data of the sensor and the orientation of the sensor.

[0039] In a subsequent step S9, the image area corresponding to the blockage of the synthetic 2D view 15 is superimposed on the image area corresponding to the blockage displayed in the display device 7 and detected by the sensor, i.e., a superimposition of the synthetic 2D view 15 of the display blockage 9. A vehicle user can thus recognize the building 11 of the surroundings U on the display device 7.

[0040] Dynamic objects such as the vehicle 13 may still not be visible on the display device 7, as they are not present in the synthetic 2D view 15 due to the low refresh rate of images from the satellite system 17.

[0041] In a further optional step S11, a dynamic object in the form of the vehicle 13 is detected in the environment U by means of another sensor, and a corresponding position is determined on the display device 7. A presentation of the vehicle 16 is then superimposed on the image area of ​​the synthetic 2D view on the display device 7, provided its position lies within the area of ​​the synthetic 2D view superimposed on the display device 7 by the blocking area, i.e., within the area of ​​the display blocking 9. In other words, the vehicle 13 detected by the radar or lidar sensor 6 is displayed on the display device 7 if its position is within the area of ​​the display blocking 9.

[0042] In display area 7, the visibility of building 11 as well as the dynamically moving vehicle 13 is thus restored despite the blockage of sensor 5.

Claims

[1] Method for compensating for a blockage of a sensor of a camera of a vehicle characterized by the following steps; - Detection of a blockage at the sensor by evaluating the image area captured by the sensor (S1), - Determining the position and dimensions of the obstruction in the sensor's field of view based on a position in the vehicle, as well as determining the sensor's orientation relative to its vehicle environment (S3), - Determination of the sensor's image area corresponding to the obstruction in a synthetic 2D view using a satellite image-to-street image transformation, which is determined from images of a satellite system based on the position of the obstruction, the dimensions of the obstruction and the orientation of the sensor (S7), - Overlay of the image area corresponding to the blockage in the synthetic 2D view with the image area corresponding to the blockage displayed in a display device and detected by the sensor (S9). [2] Method according to claim 1, characterized by , that the blockage is determined based on discontinuities in the optical flow in the image stream captured by the sensor. [3] Method according to claim 1 or 2, characterized by , that a dynamic object in the vicinity of the vehicle is detected by means of a further sensor and a corresponding position is determined on the display device, whereby a presentation of the detected dynamic object is superimposed on the image area of ​​the synthetic 2D view on the display device (S11), provided that its position is in the area of ​​the synthetic 2D view superimposed on the blockage area in the display device. [4] Method according to claim 3. characterized by that the additional sensor is designed as a radar sensor or a lidar sensor. [5] Method according to any one of the preceding claims, characterized bythat the procedure is triggered when the driver's gaze is directed towards the display device. [6] Method according to any of the preceding claims characterized by , that the overlay of the synthetic 2D view is adapted to the image area captured by the sensor using a method of image registration and / or contrast adjustment. [7] Method according to any of the preceding claims characterized by , that transformation of the position of the blockage, dimensions of the blockage and orientation of the sensor into a coordinate system common with a satellite system (S5). [8] Method according to one of the preceding claims, wherein the World Geodetic System 1984 (WGS84) is chosen as the common coordinate system.

Images