Method for determining impairment information describing an impairment of at least one camera for a vehicle due to contamination and / or glare

Abstract

The invention relates to a method for determining impairment information (24) that describes an impairment of at least one camera (2) for a vehicle (1) due to contamination (13) and / or glare (14). The method comprises providing a camera image (10, 11) that is captured by the at least one camera (2). It also comprises determining a contamination mask image (20) that describes parts of the camera image (10, 11) that are influenced by contamination (13) and a glare mask image (21) that describes parts of the camera image (10, 11) that are influenced by glare (14) by applying a mask determination algorithm (15) to the provided camera image (10, 11). In addition, the method comprises determining the impairment information (24) by applying an impairment determination algorithm (23) to the determined contamination mask image (20) and glare mask image (21).

Description

[0001] The invention relates to a method for determining impairment information describing an impairment of at least one camera for a vehicle due to contamination and / or glare. The invention also relates to a control device for a vehicle, a vehicle with such a control device, and a computer program product for implementing such a method.

[0002] A vehicle may include at least one camera configured to capture at least a portion of the vehicle's surroundings. The camera may include a lens and / or a lens cover that are at least partially exposed to the vehicle's surroundings. Therefore, at least a portion of the lens and / or the lens cover may be affected by contamination. The contamination may be caused, for example, by mud, water splashes, or other dirt. In addition, the camera may be affected by glare, in particular solar glare or glare caused by direct artificial light. Both contamination and glare typically result in at least portions of an image captured by the camera failing to provide usable information about the surroundings.

[0003] A vehicle function, such as a driver assistance system, may rely on information about the surroundings described by the camera images captured by the camera. Therefore, it is relevant to determine at least which camera of the vehicle, and in particular which parts of that camera's image, are affected by contamination and / or glare. The vehicle function can then take action to process or compensate for the unreliable information due to the contamination and / or glare.

[0004] It is the object of the invention to provide a solution to reliably detect an impairment of at least one camera for a vehicle due to contamination and / or glare.

[0005] The independent claims solve the problem.

[0006] A first aspect of the invention is a method for determining impairment information. The impairment information describes an impairment of at least one camera for a vehicle due to contamination and / or glare. For example, if a specific area or part of a camera image captured by the camera is affected by contamination and / or glare, the impairment information can describe the camera and / or the affected area or part. Therefore, it can describe all pixels of the camera image that are affected by contamination and / or glare.

[0007] The method according to the invention comprises providing a camera image that is captured by at least one camera. The camera is preferably mounted in the vehicle. Therefore, the camera can be a vehicle camera of the vehicle. The at least one camera is, for example, a front camera, a rear camera, and / or a side camera of the vehicle. The side camera can be arranged in side mirrors of the vehicle and / or in mounting devices for side cameras. The camera image describes an environment of the vehicle if the camera is mounted in the vehicle. Alternatively or additionally, the at least one camera can be arranged within a vehicle cabin of the vehicle. In this case, the captured camera image describes the environment corresponding to the vehicle cabin. Preferably, the camera comprises a fisheye lens, so that the captured camera image is a fisheye image.

[0008] A control device of the vehicle preferably carries out the method. Therefore, the camera that captures the camera image can send or transmit the captured camera image to the control device. The control device can be a control device of the camera. Alternatively or additionally, it can be a control device of the vehicle. The control device can be arranged outside the camera. The method is therefore preferably a computer-implemented method that is carried out by the control device alone. Furthermore, it is possible to divide the following individual steps of the method so that they are carried out by several different control devices and / or control units of the vehicle and / or the camera. The capture of the camera image by the at least one camera can preferably not be part of the method according to the invention.The method may begin after the camera image has been captured, with the camera image being made available for further processing by the control device.

[0009] The method comprises determining a contamination mask image and a glare mask image. The control device can determine the contamination mask image and the glare mask image. To do so, it applies a mask determination algorithm to the provided camera image. The mask determination algorithm comprises at least one rule that describes how the contamination mask image and the glare mask image are to be generated based on the camera image. Preferably, the mask determination algorithm is based on machine learning techniques. The contamination mask image describes parts of the camera image that are affected by contamination. It describes at least a part of the camera image if only a part is affected by contamination.For example, if a group of pixels in the camera image indicate that the corresponding parts of a camera lens and / or lens cover are covered with, for example, dirt or mud, the contamination mask image describes these pixels and therefore those parts of the camera image that are affected by contamination. The glare mask image describes parts of the camera image that are affected by glare. It describes at least a part of the camera image when only a part is affected by glare. For example, if some pixels in the camera image lack detail due to excessive exposure to sunlight, these pixels and therefore these parts of the camera image are considered to be affected by glare and are therefore described by the glare mask image.The contamination mask image and the glare mask image preferably show at least one area or section of the camera image that is affected by contamination or glare, respectively. This area or section is referred to herein as parts of the camera image.

[0010] The method includes determining the impairment information by applying an impairment determination algorithm to the determined contamination mask image and glare mask image. The impairment determination algorithm includes at least one rule applied to the contamination mask image and the glare mask image to determine whether the camera is at least partially affected by contamination and / or glare. The impairment information is therefore processed information determined based on the glare mask image and the contamination mask image. For example, the impairment information may describe that a certain percentage of the camera image is affected by contamination and another percentage of the camera image is affected by glare.The interference information can also describe which camera, in the case of multiple cameras, and / or which part of the camera image is neither affected by contamination nor by glare. The method thus provides reliable detection of interference with at least one camera for a vehicle due to contamination and / or glare.

[0011] One embodiment comprises determining a clean mask image by applying the mask determination algorithm to the provided camera image. It also comprises applying the impairment determination algorithm to the determined clean mask image to determine the impairment information. The clean mask image describes parts of the camera image that are neither affected by contamination nor by glare. Ideally, the clean mask image is a negative image of the contamination mask image and the glare mask image. It is possible to have an individual clean mask image that describes parts of the camera image that are unaffected by contamination. The clean mask image is then preferably a negative version of the contamination mask image. Furthermore, an individual clean mask image is determined that describes parts of the camera image that are unaffected by glare.The clean mask image is then preferably a negative version of the glare mask image. However, the clean mask image can combine parts that are unaffected by contamination and glare. It then describes all parts of the camera image that are unaffected by contamination and glare.

[0012] However, the clean image is determined by applying the mask determination algorithm to the provided camera image. Therefore, it is not necessarily a negative version of the contamination mask image and / or the glare mask image, but may describe parts of the camera image as clean, even though the contamination mask image and / or the glare mask image describe the same parts as affected by contamination or glare, respectively. Therefore, it is helpful to consider the clean mask image, for example, to determine how reliable the information described by the contamination mask image and / or the glare mask image is. Consequently, the impairment information is preferably determined not only based on the contamination mask image and the glare mask image, but also based on the clean mask image.This increases the reliability of the method as it provides a possible internal verification of the determined contamination mask image and glare mask image by comparing them with the clean mask image.

[0013] A preferred embodiment includes the mask determination algorithm comprising an artificial neural network. Preferably, the artificial neural network has an encoder-decoder architecture. The mask determination algorithm has been trained to identify parts affected by contamination and / or glare. Therefore, it has been trained to determine the contamination mask image and the glare mask image, and in particular, the clean mask image. This enables precise determination of the mask images due to the task-specific use of an artificial neural network.

[0014] The artificial neural network can be understood as software code or a compilation of multiple software code components, wherein the software code can comprise multiple software modules for different functions, for example, one or more encoder modules and one or more decoder modules. The artificial neural network can be provided, for example, in a computer-readable format, for example, stored on a storage medium of the vehicle, in particular the control device. The control device can operate the artificial neural network.

[0015] An artificial neural network can be understood as a nonlinear model or nonlinear algorithm that maps an input to an output. The input is given by an input feature vector or an input sequence, and the output can be an output category for a classification task or a predicted sequence. Here, the input is the camera image, and the output can be the contaminant mask image, the glare mask image, and, in particular, the clean mask image.

[0016] The artificial neural network may comprise multiple modules, including the encoder module and the decoder module. These modules may be understood as software modules or respective parts of the artificial neural network. A software module may be understood as software code that is functionally connected and combined into a single unit. A software module may comprise or implement multiple processing steps and / or data structures.

[0017] The modules can, in particular, represent artificial neural networks or subnetworks themselves. A module of the artificial neural network can be understood as a trainable and, in particular, as a trained module of the artificial neural network. The artificial neural network, and therefore all of its trainable modules, can, for example, be trained in a consistent manner before the method according to the invention is carried out. In other implementations, however, different modules can be individually trained or pre-trained. Consequently, the method described above corresponds to a development phase of the artificial neural network.

[0018] Furthermore, one embodiment comprises training the mask determination algorithm and therefore the artificial neural network. Training the mask determination algorithm comprises feeding at least one training camera image, one training contamination mask image, and one training glare mask image into the mask determination algorithm. In particular, it may comprise feeding a clean training mask image into the mask determination algorithm. Preferably, multiple training camera images, as well as multiple training contamination mask images, training glare mask images, and in particular, clean training mask images are used during training. The training contamination mask image, the training glare mask image, and in particular, the clean training mask image correspond here to the training camera image. They are generated in advance for the training camera image, for example.During training, the artificial neural network therefore learns which type of mask images are generated based on a specific camera image, allowing it to learn how to generate the respective mask image for any camera image, at least in a motor vehicle context. Therefore, it is possible to use a specially trained mask determination algorithm to receive precise contamination mask images and glare mask images, and especially clean mask images.

[0019] Furthermore, one embodiment comprises determining the training contamination mask image and the training glare mask image by applying a training mask determination algorithm to the at least one training camera image. The training mask determination algorithm is, for example, a further determination algorithm designed to determine the respective training mask images. In particular, the clean training mask image is also determined by applying the training mask determination algorithm to the at least one training camera image. The training mask determination algorithm comprises at least one rule for specifying how the training contamination mask image and the training glare mask image, and in particular the clean training mask image, are to be determined based on the training camera image.

[0020] The mask determination algorithm may be at least partially identical to the training mask determination algorithm. Preferably, the training mask determination algorithm differs at least slightly from the mask determination algorithm. The training mask determination algorithm may alternatively be referred to as a pseudomask generation module. The aforementioned training mask images may alternatively be referred to as respective pseudomask images. The individual training mask images are therefore generated before being used during the training procedure to train the mask determination algorithm. Therefore, the training is particularly precise and reliable.

[0021] Another embodiment includes determining the training contamination mask image and / or the clean training mask image comprising dividing the training camera image into multiple sections. In the case of the training contamination mask image, it comprises determining a contamination annotation for each of the multiple sections. Each contamination annotation describes how likely it is that the respective section is affected by contamination. Determining the training contamination mask image then comprises composing the multiple sections with the determined contamination annotations to generate the training contamination mask image. The contamination annotation may distinguish between the classes "clean," "transparent," "semi-transparent," and "opaque." Other, fewer, or more classes are possible.For example, the contamination annotation describes a percentage for all pixels in the section that are considered "clean," "transparent," "semi-transparent," and "opaque." For example, the contamination annotation for one of the sections of the training camera image might be 70 percent "opaque," 20 percent "semi-transparent," 5 percent "transparent," and 5 percent "clean." This section as a whole is then considered to be affected by contamination due to the high percentage of "opaque" pixels. However, another section that might have an contamination annotation of 70 percent "clean," 20 percent "transparent," 5 percent "semi-transparent," and 5 percent "opaque" might indicate a clean or mostly clean section that is not affected by contamination, for example.Various combinations of the four classes can exist, each indicating that a section is affected by contamination or that it is not affected by contamination. For example, in the training contamination mask image, the section with the high percentage of "opaque" pixels is marked here as "affected by contamination," and the section with the high percentage of "clean" pixels is marked here as "not affected by contamination." The sections with their contamination annotations are then assembled to form the training contamination mask image. This annotates each section with its contamination annotation, so that the resulting training contamination mask image shows all sections of the camera image classified as affected by contamination and all sections classified as not affected by contamination.This can be done analogously to determining the clean training mask image, taking into account the impurity annotation of each section of the training camera image. This describes in detail how the training impurity mask image and / or the clean training mask image should be generated.

[0022] A further embodiment includes determining the training glare mask image and / or the clean training mask image comprising dividing the training camera image into multiple sections and determining a glare annotation for each of the multiple sections. The glare annotation describes how likely it is that the respective section is affected by glare. It also includes composing the multiple sections with the determined glare annotation. Analogous to the manner described above for determining the training contamination mask image and / or the clean training mask image taking into account the contamination annotations of the sections, the training glare mask image is generated taking into account the glare annotations of the individual sections. Analogously, the clean training mask image can also be based on the glare annotations.The method also includes composing the multiple sections with their determined glare annotation to determine the training glare mask image and / or the clean training mask image. Here, the glare annotation can distinguish between the classes “clean” and “glare.” For example, if a section has the glare annotation 40 percent “clean” and 60 percent “glare,” 60 percent of this section is therefore affected by glare, whereas the remaining pixels or parts of the section are clean and consequently not affected by glare. This section can be understood as a section affected by glare and is therefore described as such in the determined training glare mask image and / or clean training mask image. This describes in detail how the clean training mask image and / or the clean training mask image are to be generated.

[0023] To generate the clean training mask image covering both contamination and glare, it is possible to determine the contamination annotation as well as the glare annotation for the respective section, and composite the multiple sections with the determined contamination annotation and glare annotation to receive the clean training mask image.

[0024] Composition here means that a respective training mask image is generated by recombining the individual sections with the respective impurity annotations or the glare annotation to the respective training mask image, i.e., the training impurity mask image, the training glare mask image, or the clean training mask image.

[0025] Another embodiment involves generating the multiple sections by dividing the camera image into grids or tiles. Each grid or tile comprises multiple pixels of the camera image. For example, it is possible to divide the training camera image into an 8-by-8 grid and therefore into 64 individual sections. Other grids are possible. This makes it fast and requires little computational power to determine the respective training mask image, since the training mask images are determined not at the pixel level but at a grid level by considering all pixels from a portion of the grid or tile simultaneously.

[0026] Alternatively, one embodiment includes each of the multiple sections comprising a single pixel of the camera image. Therefore, it is possible to perform a pixel-by-pixel determination of the contamination annotation and / or the glare annotation to determine the training contamination mask image, the training glare mask image, and / or the clean training mask image. Consequently, particularly precise training mask images are determined with respect to contamination, glare, and, in particular, clean parts of the training camera image.

[0027] A preferred embodiment includes the camera image being a bird's-eye view. Furthermore, the camera image can be a composite of several individual camera images, captured, for example, by the front camera, the rear camera, and the side cameras of the vehicle. Preferably, the camera image shows the entire surroundings of the vehicle, and therefore the surroundings at a front, a rear, and the sides of the vehicle. The bird's-eye view image can include height information of at least one object in the surroundings.

[0028] The mask determination algorithm includes an image encoder to determine a two-dimensional feature map of the camera image. The image encoder includes convolutional layers that map input data, and therefore the camera image, to various, often lower-dimensional and compressed feature representations. The deeper the encoder, the better the features extracted. The type of encoder can be selected based on the constraints of the embedded platform. Standard encoder families such as ResNet, VGG, Inception, EfficientNet, and so on can be used for this task.

[0029] The mask determination algorithm also includes a bird's-eye view projection module to convert the two-dimensional feature map into a voxel map. A voxel is a three-dimensional coordinate, which here not only comprises two dimensions of length and width as a two-dimensional feature map, but also includes a third dimension. The third dimension can describe a height. For example, it can describe the height of an object in the surrounding area. Alternatively, the height information can be depth information. The height information can be determined by considering additional sensor data from the vehicle. The additional sensor data can be lidar data acquired by a lidar device or a lidar sensor on the vehicle. Alternatively or additionally, other sensor data can be considered, such as radar data or ultrasonic sensor data.

[0030] The bird's-eye view projection module accepts two-dimensional image features provided by the image encoder and projects each feature point into three-dimensional space. The resulting voxel or three-dimensional pixel map contains the height, width, and depth information of the objects described by the input feature map and, therefore, by the camera image. The resulting voxel map provides an enhanced representation of each object in top-view space compared to the camera image. The projection utilizes intrinsic and extrinsic parameters of the camera that captured the image. Therefore, the type of optical components the camera includes, as well as its position and orientation within the vehicle, are taken into account.

[0031] The mask determination algorithm includes a bird's-eye view encoder to determine a bird's-eye view feature map of the voxel map and thus the three-dimensional map. The bird's-eye view encoder includes a set of convolutional layers that convert the bird's-eye view input data into bird's-eye view features, particularly those of low dimension. Standard encoder families such as ResNet, VGG, EfficientNet, and so on can be used as bird's-eye view encoders.

[0032] The three described parts of the mask determination algorithm can be understood as the encoder section of the mask determination algorithm. These parts of the mask determination algorithm are based on the observation that it is possible to detect the areas around the vehicle that are degraded and therefore affected by impairment due to contamination and glare. They are assumed to be provided in the bird's-eye view space. For this purpose, surround fisheye images are considered as camera images and therefore as input for the mask determination algorithm. Camera features are then projected into the bird's-eye view space, and the detection of degraded areas in this top-view space is performed. The degraded areas here refer to the parts of the camera image that are affected by contamination and / or glare.First, the camera image is fed as input to the image encoder, which consists of convolutional layers and extracts the features present in the camera image. It also generates the two-dimensional feature maps. These feature maps are then passed to the bird's-eye projection module, which transforms the two-dimensional feature points into voxels. The voxel here is the three-dimensional representation of each pixel, which contains additional information such as depth or height. The voxel therefore describes how dense each two-dimensional feature point is in the real world. This is followed by the bird's-eye encoder to extract deeper features in the voxel map. The bird's-eye encoder outputs the bird's-eye features and, hence, the three-dimensional feature map. The bird's-eye features are passed to a decoder of the mask determination algorithm.This decoder can comprise two detection branches, one dedicated to detecting contamination areas and the other dedicated to detecting glare areas. This explains in detail how the camera image should be prepared so that reliable contamination mask images, glare mask images, and, in particular, clean mask images can be generated as the output of the mask determination algorithm.

[0033] A further embodiment comprises determining at least one control command for a camera cleaning system if the determined impairment information describes an impairment of the camera due to contamination. If it has been determined that the camera is at least partially affected by contamination, it is possible to generate the control commands for an automatic camera cleaning system. The automatic camera cleaning system can clean the lens and / or the lens cover of the camera using a water jet or another cleaning unit. Therefore, it is possible to take a measure to automatically reduce at least the parts of the camera image that are affected by contamination by cleaning the camera.

[0034] Another embodiment comprises determining a message for a vehicle function. The vehicle function performs processing of the camera image. The vehicle function may be a driver assistance system. The message is determined if the determined impairment information describes an impairment of the camera due to glare. In particular, the message describes which parts of the camera image are affected by glare. This means that, since it is not possible to actively counteract the effect of glare on the camera image, details about the affected areas and therefore the parts of the camera image that unreliably describe the surroundings are provided to the vehicle function. The parts affected by glare can then be discarded or ignored by the vehicle function.Therefore, the specific information about the parts of the camera image affected by glare is further utilized by the vehicle function. This makes it possible to respond advantageously to the specific impairment information.

[0035] The method described above is based, at least in part, on the following observations. Cameras are an essential part of the sensor suite in autonomous driving. Surround-view cameras can be directly exposed to the environment and are degraded by mud, water, snow, glare, and so on. Cameras exhibit higher performance degradation due to contamination or glare compared to other sensors. Consequently, it is necessary to detect the degraded area and, for example, trigger an automatic cleaning system in case of contamination or track the faulty area in case of glare. Currently, the camera degradation problem is handled by regressing the degraded area in each tile of the camera image. Here, the camera image is typically a fisheye image.The number of tiles required to represent an image depends on the overall performance and the embedded constraints. Typically, the image is divided into 4 by 4 or 8 by 8 individual tiles. Glare detection was a standalone solution, whereas contamination was predicted as part of a multitask network. Now, these two degradations should be combined so that they are predicted individually, as part of a software application, specifically as part of a decoder. The inventive method detects the degradation in the reference world to generate a holistic synopsis of the surround view affected by degradation.

[0036] There are several advantages of this feature over existing solutions. The inventive method performs a one-time degradation identification. Instead of working on individual cameras and with separate solutions for contamination and glare, the proposed invention is capable of directly regressing the degradation by only executing it once in a bird's-eye view. The inventive method also provides a one-time system-level degradation output. According to prior art methods and current product-level practices, contamination and glare are both predicted in isolation, and outputs are passed to the upper layer. There is some hand-crafted heuristic logic enforced at the system level to summarize both outputs for all cameras.Instead, the inventive method provides a new uncertainty-guided learnable degradation summary that generates a unique system-level output. This also helps eliminate the individual post-processing modules for contamination and glare. Furthermore, the inventive method performs a one-time vision algorithm degradation estimation. Since the method can be easily integrated into a multi-task network (MTN) setup, other vision-based outputs for a parking pipeline or general driving, such as object detection, semantic segmentation, depth estimation, motion estimation, and so on, can be derived together. Consequently, the algorithm outputs coming from the degraded region can be easily filtered out while being forwarded to the upper layer, as they should not be trusted.These one-time filtering algorithm outputs due to degradation significantly improve current practices, where all these modules run in isolation.

[0037] In this invention, degradation is defined as the effect caused by either contamination or glare. If it is contamination, the cleaning system must be invoked, and other algorithm outputs from the contaminated area must be filtered out. Whereas for glare, it cannot be cleaned, so it must be tracked, and other outputs from vision algorithms must be discarded. Therefore, it makes sense to combine both corruptions but identify them differently. In the case of four cameras, the method should be considered as four fisheye images that are, for example, partially contaminated. Then, their corresponding contamination mask is generated in the bird's-eye view, which is used in training. The method further generates an inference image at a bird's-eye view for the same four fisheye images.To precisely isolate the contaminated area, contours from the ground truth are overlaid on the same inference image.

[0038] Another aspect of the invention relates to a control device for a vehicle. The control device is configured to carry out the method described above. The control device carries out the method described above. The control device comprises, for example, a processor unit. The processor unit can comprise at least one microprocessor, a microcontroller, an FPGA (field programmable logic array), and / or a DSP (digital signal processor). Furthermore, it can comprise program code, which can alternatively be referred to as a computer program product. The program code can be stored in a data memory of the processor device.

[0039] Another aspect of the invention relates to a vehicle with a control device. The vehicle is configured to carry out the method described above. The vehicle may comprise at least one camera so that the vehicle can capture a camera image with the camera. The vehicle may be a motor vehicle, such as a passenger car, a truck, a bus, a motorcycle, and / or a moped.

[0040] Another aspect relates to a computer program product. The computer program product is a computer program. The computer program product comprises instructions that, when executed by a computer, such as the control devices, cause the computer to perform the method described above.

[0041] The embodiments described in connection with the method apply individually and in combination with one another, as applicable, to the control device, the vehicle, and the computer program product. The invention encompasses combinations of the described embodiments.

[0042] The figures show in: Fig. 1 a schematic representation of a vehicle with several cameras, Fig. 2 a schematic representation of a method for determining impairment information, Fig. 3 a schematic representation of a procedure to prepare a training process, and Fig. 4 a schematic representation of an impairment determination algorithm.

[0043] Fig. 1 shows a vehicle 1. The vehicle 1 comprises at least one camera 2. The vehicle 1 here comprises a front camera 3, a rear camera 4 and two side cameras 5. The respective side camera 5 is positioned in a side mirror 6 of the vehicle 1. Further cameras 2 or combinations of cameras 2 are possible. The camera 2 of the vehicle 1 is preferably configured to generate a camera image 10, 11 (see reference numerals 10, 11 in Fig. 2) that describes the surroundings 7 of the vehicle 1. Preferably, all camera images 10, 11 captured by the outlined cameras 2 can be combined to show a top-view image or bird's-eye view of the surroundings 7. Each of the cameras 2 is preferably a camera with a fisheye lens. The camera image 10, 11 can thus be a fisheye image.

[0044] The vehicle 1 may include a control device 8. The control device 8 may perform the steps of the method described below: Fig. 2 shows a method for determining impairment information 24. The impairment information describes an impairment of the camera 2 of the vehicle 1 due to contamination 13 and / or glare 14. It is assumed that the camera 2 can capture a camera image 10, 11 and transmit it to the control device 8. The camera images 10, 11 are provided here by the control device 8. Therefore, the method can be considered a computer-implemented method since it is carried out solely by the control device 8.

[0045] Two camera images 10, 11 are shown here as an example. Due to the fisheye lens of camera 2, the camera images 10, 11 show distortions. A first camera image 10 shows the surroundings 7 with a zebra crossing. Here, part of the camera image 10 is influenced by contamination 13. For example, mud or dirt is partially arranged on the lens or a lens cover of camera 2, so that part of the surroundings 7 is not described by the first camera image 10 due to contamination 13. Analogously, the second camera image 11, which here shows the same surroundings 7 as the first camera image 10, is partially influenced by glare 14. Glare 14 can be caused by direct exposure to sunlight or artificial light. Both camera images 10, 11 therefore have parts of the camera image 10, 11 that are influenced by impairment due to contamination 13 and / or glare 14.

[0046] The method comprises applying a mask determination algorithm 15 to the provided camera image 10, 11 to determine a contamination mask image 20 and a glare mask image 21. The contamination mask image 20 describes parts of the camera image 10, 11 that are influenced by contamination 13. The glare mask image 21 describes parts of the camera image 10, 11 that are influenced by glare 14. Here, the contamination mask image 20 is based on the first camera image 10, and the glare mask image 21 is based on the second camera image 11. Furthermore, it is possible to determine a clean mask image 22 that describes parts of the camera image 10, 11 that are neither influenced by contamination 13 nor by glare 14.

[0047] The method comprises determining the impairment information 24 by applying an impairment determination algorithm 23 to the determined contamination mask image 20 and glare mask image 21. It can also be applied to the determined clean mask image 22. Thereafter, at least one control command 25 for a camera cleaning system of the vehicle 1 can be determined if the determined impairment information 24 describes an impairment of the camera 2 due to a contamination 13. Alternatively or additionally, a message 26 can be determined for a vehicle function. The vehicle function can process the camera image 10, 11. If the determined impairment information 24 describes an impairment of the camera 2 due to glare 14, the message 26 can be generated and can describe which parts of the camera image 10, 11 are affected by the glare 14.

[0048] The camera image 10, 11 is preferably a bird's-eye view. It can be a composite of several individual camera images 10, 11 or a single camera image 10, 11 captured by a camera with a fisheye lens. The combined camera image 10, 11 can describe a bird's-eye view of the surroundings 7 of the vehicle 1.

[0049] The mask determination algorithm 15 can be based on machine learning methods. It can comprise an artificial neural network, in particular having an encoder-decoder architecture. Here, it can comprise an image encoder 16 for determining a two-dimensional feature map of the camera image 10, 11, a bird's-eye view projection module 17 for converting the two-dimensional feature map into a voxel map, and a bird's-eye view encoder for determining a bird's-eye view feature map of the voxel map. Furthermore, the mask determination algorithm 15 can comprise a guided multi-space degradation decoder 19.

[0050] Fig. Figure 3 shows in more detail how the mask determination algorithm 15, and more specifically, the guided multi-space degradation decoder 19, is trained. This is done by first providing a training camera image 30. Here, as an example, the camera images 10, 11 are assumed to be training camera images 30. The training camera image 30 is provided to a pseudomask generation module 31 to generate training masks on an image plane. Here, it generates the image plane contamination mask 32 and the image plane glare mask 33. These are both present on an image plane or pixel plane. The respective image plane masks 32, 33 are then provided to the bird's-eye view projection module 17 to generate a training contamination mask image 34 and a training glare mask image 35, both present in a bird's-eye view. It can also generate a clean training mask image 36.

[0051] The generated training mask images 34, 35 and 36 are then provided to a training procedure 37 for the guided multi-space degradation decoder 19. The training of the mask determination algorithm 15 and therefore the training process 37 comprises feeding the at least one training camera image 30, the training contamination mask image 34, the training glare mask image 35 and in particular the clean training mask image 36 into the mask determination algorithm 15 or more precisely into the multi-space guided degradation decoder 19 of the mask determination algorithm 15. This comprises determining the training contamination mask image 34 and the training glare mask image 35 and in particular the clean training mask image 36 by applying a training mask determination algorithm 38 to the at least one training camera image 30. The training mask determination algorithm 38 is in Fig. 3 outlined.

[0052] Determining the training contaminant mask image 34 and / or the clean training mask image 36 may include dividing the training camera image 30 into a plurality of sections, determining a contaminant annotation for each of the plurality of sections, each describing how likely it is that the respective section is affected by a contaminant 13, and composing the plurality of sections with the determined contaminant annotations to generate the training contaminant mask image 34.Analogously, determining the training glare mask image 35 and / or the clean training mask image 36 may comprise dividing the training camera image 30 into a plurality of sections, determining a glare annotation for each of the plurality of sections, each describing how likely the respective section is to be affected by glare, and composing the plurality of sections with the determined glare annotation to generate the training glare mask image 35 and / or the clean training mask image 36. The plurality of sections may be generated by dividing the training camera image 30 into grids or tiles. Each grid or tile comprises a plurality of pixels of the training camera image 30. Alternatively, each of the plurality of sections comprises a single pixel of the training camera image 30.

[0053] In other words, Fig. Figure 3 shows how binary degradation masks (image plane contamination mask 32 and image plane glare mask 33) are to be generated from the available degradation annotation. For illustration purposes, the image plane masks 32, 33 from multiple cameras 1 with contaminated data and sun glare data are used. If the annotations for the grid level are available, that is, the input training camera image 30, which is divided into 8 by 8 grids, and labels for these grids are available, contaminated annotations can consist of four classes such as "clean," "transparent," "semi-transparent," and "opaque," with a corresponding percentage of occupancy. As an example, for a single grid annotation, {Clean: 0.4, Transparent: 0.3, Semi-transparent: 0.2, Opaque: 0.1} is possible. Other values for the individual classes are possible. Of these, clean samples belong to the uncontaminated class and the rest belong to the contaminated class.For example, a binary 1 can be assumed for contaminated pixels and a binary 0 for non-contaminated / clean pixels for binary mask generation. Here, a certain contaminated threshold value such as 0.3 or 0.4 can be assumed. If the sum of contaminated samples (0.3 + 0.2 + 0.1 in the above example) is greater than the threshold, the pixels falling under the special grid belong to the contaminated class, or these pixels are assigned a binary 1. Otherwise, a binary 0 can be assigned. In this way, the binary contaminated image-plane mask, i.e., the image-plane contaminant mask 32, is generated.

[0054] To generate image-plane glare masks, i.e., the image-plane glare mask 33, the procedure described above is applied with different annotations. For a single grid, the annotation is {Clean: 0.4, Sun Glare: 0.6}, which means that 60 percent of the grid area is covered with glare, and the remaining pixels are clean pixels. If the annotations are available at the pixel level, the entire image is considered a single grid. Then the degradation masks are very accurate. In this method, both the contamination and glare annotations are also the same, but annotations are assigned per pixel, not per grid.

[0055] Fig. Figure 4 shows the impairment determination algorithm 23 in more detail. This algorithm can alternatively be referred to as a post-processing module. The idea here is to overlay the impurity mask image 20 and the glare mask image 21 with the clean mask image 22 to assess their match. In an ideal situation, all of these masks, i.e., the impurity mask image 20 and the glare mask image 21, both closely match the clean mask image 22. If pixels of the impurity mask image 20 and the glare mask image 21 pass thresholds T1 and T2, respectively, then the resulting mask is generated using a bitwise operation between the impurity mask image 20 and the glare mask image 21. If the impurity mask image 20 and the glare mask image 21 cannot pass the thresholds T1, T2, then they are declared clean.If the resulting masks pass the contamination thresholds T3, T4 or glare thresholds T5, T6, they are declared as contaminated or glared, respectively, and sent, for example, to a camera view separation module to determine the camera view from which the contamination 13 or glare 14 originates. This means that the "contamination" or "glare" declaration can be described by the impairment information 24. The goal of the camera view separation module can be to activate the single or multiple camera cleaning system when the camera 2 is affected by a contamination 13. From camera parameters, it can be calculated how much area in the environment 7 is affected by this. A region of interest and a convolutional neural network prediction can be performed to determine information about the area affected by the contamination 13.These two inputs can be fed into the threshold module, which checks whether the number of contaminated pixels in each image exceeds the threshold. If so, the corresponding camera cleaning system can be activated; otherwise, it is not activated.

[0056] If the resulting mask fails to pass a threshold, it is declared unknown 41. It is then sent for human intervention 42. Here, the "clean" declaration is indicated by reference numeral 43.

[0057] Fig. Figure 4 shows an intersection module 40 for determining differences between the two masks, which here are the contamination mask image 20 and the glare mask image 21. After each verification step, which includes the threshold values T1, T2, T3, T4, T5, T6, a continuation line indicates the result "yes" and a downward-sloping line indicates the result "no". The number of mask pixels is denoted as n. mThe thresholds T1, T2, T3, T4, T5, T6 mentioned may be situation-dependent, task-dependent, vehicle-dependent, and / or camera-dependent.

[0058] In summary, the invention presents a method for detecting camera degradation in the real world. Camera degradation here refers to impairment due to contamination 13 and / or glare 14.

[0059] Further examples are described below, whereby the described features can be considered individually or in combination with each other: In real-world tasks, the data distribution typically deviates over time. Robustness under distributional deviance (OOD) is the ability of a machine learning model to perform well on data it hasn't seen before. In real-world scenarios, self-driving cars may encounter unforeseen events or environments that are not well represented in the training data. Without proper OOD robustness, the model may make incorrect or unsafe decisions in such situations, leading to accidents or other undesirable outcomes. To achieve this OOD robustness, we propose deep ensembles, but in a new way with real-time degradation. The model used is the guided multi-space degradation decoder 19 (MSG decoder). It takes three inputs and produces three outputs corresponding to contamination, glare, and clean, respectively.

[0060] The difference between the guided multi-space degradation decoder 19 and state-of-the-art deep ensemble methods is that recent advances in deep ensembles have demonstrated notable achievements in achieving robustness to OOD data and improving uncertainty estimation, all while minimally amplifying model parameters. However, the state-of-the-art deep ensemble methods have proven impractical due to their significant prediction time and larger model size. The basic concept underlying the guided multi-space degradation decoder 19 is to split the task into two equal and opposite subtasks, then combine and evaluate their results.This is possible due to the fact that degraded regions (affected by contamination 13 or glare 14) are always surrounded by clean regions, so the same clean masked output can be compared to both the contamination mask image 20 and the glare mask image 21. This method eliminates the need for multiple ensemble networks, as only two networks are required to accomplish the goal of ensemble learning.

[0061] Research has found that artificial neural networks have more capacity than necessary. Overparameterization of artificial neural networks and sparsity insights allow for pruning up to 70 to 80 percent of connections without affecting performance. The technique enables the simultaneous training of independent subnetworks or auxiliary networks within a single network. These subnetworks coexist without separation, facilitating concurrent evaluation and exploiting ensemble advantages. Only two architectural changes are required: input layer replacement with N data points and output layer replacement with N heads. Each head predicts its corresponding input, and at test time, N heads produce independent predictions on the same input, forming a robust ensemble in a single pass.The guided multi-space degradation decoder 19 may comprise multiple auxiliary ConvNets within a network.

[0062] For training in a typical setting, a forward pass of a guided multi-space degradation decoder 19 consists of feeding three encoded bird's-eye view images—polluted, blinded, and clean—with their respective semantic masks, which outputs predictions from each input given the data. During backpropagation, the model attempts to optimize the loss between predictions and ground truths. The loss (LN) used for the guided multi-space degradation decoder 19 is the sum of the negative logarithmic probabilities of the predictions and the regularization term, where N is the number of inputs and heads (N = 3 for the guided multi-space degradation decoder 19): LN(θ)=E(xN,y¨N)∈X(z1,y1)∈X[∑n=1N−logpθ(yn|x1,…,.xN)]+R(θ)

[0063] Regarding the inference phase: During the evaluation phase, three copies of a single query image are fed to the artificial neural network, which outputs three different semantic masks (contaminated mask image 20, glare mask image 21, and clean mask image 22). All generated outputs are passed through a post-processing module, which evaluates them and shares the final decision with the system.

[0064] The dataset used may include existing degraded camera images 10, 11, 30 to train the bird's-eye view-based segmentation network. Ground-truth degradation masks may be generated using thresholding methods.

Claims

[1] Method for determining impairment information (24) describing an impairment of at least one camera (2) for a vehicle (1) due to contamination (13) and / or glare (14), comprising: - Providing a camera image (10, 11) which is captured by means of at least one camera (2); - by applying a mask determination algorithm (15) to the provided camera image (10, 11), determining a contamination mask image (20) describing parts of the camera image (10, 11) that are influenced by a contamination (13), and a glare mask image (21) describing parts of the camera image (10, 11) that are influenced by a glare (14); and - determining the impairment information (24) by applying an impairment determination algorithm (23) to the determined contamination mask image (20) and glare mask image (21). [2] The method of claim 1, comprising determining a clean mask image (22) describing parts of the camera image (10, 11) that are not affected by contamination (13) or glare (14) by applying the mask determination algorithm (15) to the provided camera image (10, 11), and applying the impairment determination algorithm (23) likewise to the determined clean mask image (22) to determine the impairment information (24). [3] Method according to one of the preceding claims, wherein the mask determination algorithm (15) comprises an artificial neural network. [4] Method according to claim 3, wherein training of the mask determination algorithm (15) comprises feeding at least one training camera image (30), one training contamination mask image (34) and one training glare mask image (35), and in particular a clean training mask image (36) into the mask determination algorithm (15). [5] Method according to claim 4, comprising determining the training contamination mask image (34) and the training glare mask image (35), and in particular the clean training mask image (36) by applying a training mask determination algorithm (38) to the at least one training camera image (30). [6] The method of claim 5, wherein determining the training contamination mask image (34) and / or the clean training mask image (36) comprises dividing the training camera image (30) into a plurality of sections, determining a contamination annotation for each of the plurality of sections, each describing how likely it is that the respective section is affected by a contamination (13), and assembling the plurality of sections with their determined contamination annotation. [7] The method of claim 5 or 6, wherein determining the training glare mask image (35) and / or the clean training mask image (36) comprises dividing the training camera image (30) into a plurality of sections, determining a glare annotation for each of the plurality of sections, each describing how likely it is that the respective section is affected by glare, and composing the plurality of sections with their determined glare annotation. [8] Method according to claim 6 or 7, wherein the plurality of sections are generated by dividing the training camera image (30) into grids each having a plurality of pixels of the training camera image (30). [9] The method of claim 6 or 7, wherein each of the plurality of sections comprises a single pixel of the training camera image (30). [10] Method according to one of claims 3 to 9, wherein the camera image (10, 11) is in a bird's eye view and the mask determination algorithm (15) comprises: - an image encoder (16) for determining a two-dimensional feature map of the camera image (10, 11), - a bird's-eye view projection module (17) for converting the two-dimensional feature map into a voxel map, and - a bird's eye view encoder (18) to determine a bird's eye view feature map of the voxel map. [11] Method according to one of the preceding claims, comprising determining at least one control command (25) for a camera cleaning system if the determined impairment information (24) describes an impairment of the camera (2) due to contamination (13). [12] Method according to one of the preceding claims, comprising determining a message (26) for a vehicle function which carries out the processing of the camera image (10, 11) if the determined impairment information (24) describes an impairment of the camera (2) due to glare (14), wherein the message (26) describes in particular which parts of the camera image (10, 11) are influenced by glare (14). [13] Control device (8) for a vehicle (1) which is designed to carry out a method according to one of the preceding claims. [14] Vehicle (1) with a control device (8) according to claim 13. [15] A computer program product comprising instructions which, when executed by a computer, cause the computer to perform a method according to any one of claims 1 to 12.

Images