CORRECTION OF CAMERA IMAGES IN RAIN, LIGHT INJURY AND DIRT

Description

[0001] The invention relates to a machine learning method, a method and a device for correcting images from a camera in the event of rain, light incidence or contamination, for example a vehicle-mounted environmental detection camera.

[0002] Today's and future vehicles are equipped with camera-based driver assistance systems that detect objects to avoid collisions and recognize road boundaries to keep the vehicle in its lane. These systems use, for example, forward-facing cameras, which perform detection or display functions either alone or in combination with surround-view cameras.

[0003] There are concepts for forward-facing cameras that include a rain detection function. Combined with camera-based daytime running light detection for controlling the vehicle's headlights, a single camera can be used to create a so-called Rain Light Detector that detects rain on the windshield and, for example, activates the windshield wipers.

[0004] The recognition algorithms based on these camera systems already partially combine approaches from classical image processing with approaches from machine learning, especially deep learning. Classical approaches to object or structure recognition in image processing are based on manually selected features, while deep learning-based approaches determine and optimize relevant features themselves during the training process.

[0005] The camera systems mentioned above show a degradation in both the detection of objects and the representation of the environment or objects, however, as soon as visibility is impaired by rain, light incidence or dirt in a situation.

[0006] If the view of the front camera, for example, installed in the base of the rearview mirror, is obstructed by water droplets on the windshield or dirt, the view can be restored by activating the windshield wipers. This requires that the camera be installed within the wiper's path.

[0007] With the increasing automation of vehicles, the number of forward-facing cameras is rising. These can be mounted not only centrally in the mirror base, but also near the A-pillar in the upper corners of the windshield. These areas are more critical for detection functionality, as they are outside the wiper range. Visibility obstructed by raindrops or dirt negatively impacts detection capabilities.

[0008] With increasing levels of vehicle automation, from SAE Level 2+ to SAE Level 4 / Level 5, or with automated parking or visualization systems, cameras mounted on the sides of vehicles are expected to be used in the future. These cameras will not only display the surroundings but also detect objects approaching from the side. If the lenses become wetted with water droplets or dirt, the display or detection functionality can be severely limited. Due to the lack of cleaning options such as windshield wipers, this can lead to system degradation or failure.

[0009] Finally, consider reversing cameras, which are usually mounted above the license plate and get dirty very quickly. Here, too, rain or dust can cause condensation, making a clear display difficult.

[0010] While CNN-based object recognition methods are largely able to compensate for contamination or wetting of the lenses by water droplets to at least some extent, object recognition methods based on image features, such as optical flow or structure-from-motion, experience severe degradation due to contamination.

[0011] Algorithmic methods are known for detecting dirt or precipitation on the outer lens of a camera or on the windshield of a vehicle.

[0012] WO 2013 / 083120 A1 discloses a method for evaluating image data from a vehicle camera, in which information about raindrops on a windshield within the camera's field of view is taken into account during the image data evaluation. The information about raindrops can, in turn, be derived from the image data. As an example of image data evaluation, object recognition is given, which then specifically considers the information. For instance, the influence of the rain on the edges seen by the camera (light / dark or color transitions) can be estimated from the detected rainfall intensity.

[0013] Accordingly, edge-based evaluation methods can have their threshold values ​​adjusted. In particular, a quality criterion for the image data can be derived from the information, which is then taken into account during the evaluation of the image data.

[0014] H. Porav et al., in "I Can See Clearly Now: Image Restoration via De-Raining", 2019 IEEE Int. Conference on Robotics and Automation (ICRA), Montreal, Canada, pp. 7087-7093, accessed on July 13, 2020 at: http: / / www.robots.ox.ac.uk / ~mobile / Papers / iGCRA19 porav.pdf, demonstrate a method for improving segmentation tasks on images affected by overlying raindrops or streaks. For this purpose, a stereo dataset was generated in which one lens was affected by real water droplets and the other lens was clear of any impairment. This dataset was used to train a "denoising generator" to remove the effect of the water droplets in the context of image reconstruction and road marking segmentation.

[0015] The present invention can be understood as a further development of the prior art, as known, for example, from H. Porav et al.

[0016] The object of the present invention is to provide solutions for this.

[0017] The problem is solved by the subject matter of the independent patent claims. Advantageous embodiments are the subject matter of the dependent claims, the following description, and the figures.

[0018] A machine learning method according to the invention relates to image correction of input image data from a camera, which is affected by rain, light exposure, and / or dirt, into output image data using an artificial neural network. The learning process is carried out with a plurality of training image pairs such that the input to the artificial neural network consists of a first image affected by rain, light exposure, and / or dirt, and a second image of the same scene without such impairment is provided as the target output image. The artificial neural network is designed to determine a safety factor c, which depends on the degree of wetting by water, light exposure, and / or dirt for an input image. The network design can be achieved, for example, through a corresponding design or architecture of the artificial neural network.After the machine learning process is complete, the artificial neural network can determine and output the confidence level c for a new input image. The confidence level c therefore depends on the degree of impairment caused by rain or water exposure, light exposure, and / or dirt, and, when the trained network is used, characterizes the certainty that an image correction is correct.

[0019] The confidence level c characterizes, in a sense, the "uncertainty" with which image correction is performed by the trained neural network.

[0020] In other words, the security measure c depends on the impairment of the input image data and characterizes the security of the network that the network's image correction is accurate.

[0021] The artificial neural network can, for example, be a Convolutional Neural Network (CNN).

[0022] The conversion to output image data "without impairment" usually includes the conversion to output image data with reduced impairment.

[0023] The camera can, for example, be a monocular camera mounted in or on a vehicle that captures the vehicle's surroundings. An example of such a vehicle-mounted camera is one positioned behind the windshield inside the vehicle, capable of capturing and displaying the area in front of the vehicle through the windshield.

[0024] The effect of a camera image being affected by rain, sunlight, or protection is similar in that it leads to (local) blurring in the image. In all these cases, image correction is desirable to reduce or eliminate this blurring.

[0025] According to one embodiment, at least one factor d is determined as a measure of the difference between the corrected output image and the impaired input image and provided to the artificial neural network during training. The artificial neural network takes the factor d into account during learning, for example, by training the neural network to associate the input image, the output image, and the factor d. This allows the trained network to later estimate or determine a factor d for a currently captured impaired camera image and generate (or reconstruct) a correspondingly corrected output image. After training is complete, a factor d can, for example, be specified to the trained neural network, thereby controlling the degree of correction of the currently captured camera image.

[0026] The factor d can be determined, for example, by a local comparison of an undisturbed image with an image affected by rain or dirt.

[0027] The factor d can be determined using 2D filters, which can be represented, for example, in the input layers of an artificial neural network.

[0028] In a simple embodiment, the factor d can be represented as the variance of a 2D low-pass filter. Alternatively, more complex contrast values ​​(structural similarity) or correlations (sum of absolute distances - SAD, sum of squared distances - SSD, zero-means normalized cross correlation - ZNCC), calculated from the two images with local filters, are also conceivable.

[0029] For example, a camera lens contaminated by rain or dirt produces a distorted image, which can make object recognition difficult. Within the machine learning process, a factor d can be determined by comparing the target output image with the corresponding impaired input image. This determination can be done in advance, meaning that a factor d is already available for each training image pair. Alternatively, the factor d can be determined solely based on the training image pairs within the learning process.

[0030] The factor d provides a value indicating the degree of possible reconstruction of the corrected image, which is then passed to subsequent image processing or display functions. For example, a low value can indicate a high degree of correction, and a high value a low degree of correction for further processing stages. This value, along with the safety factor c, is taken into account when determining the quality of the generated object data.

[0031] In one embodiment, the training image pairs are generated by capturing a first image with rain, light incidence and / or dirt impairment (in the optical path of the camera) and a second image without impairment simultaneously or immediately following each other with different exposure times.

[0032] According to one embodiment, the training image pairs contain at least one sequence of consecutive input and output images (as image data). In other words, image sequences (video sequences) are used as image data. For machine learning, at least one input video sequence and one target video sequence are required in this case.

[0033] Using image sequences allows for the advantageous consideration of temporal aspects and relationships in reconstruction (or image correction). For example, consider raindrops or dirt particles that move over time. This creates areas in the image that, at time t, had a clear view, while at time t+1, the view was obscured by rain. By using image sequences, information from the clear image areas can be used to reconstruct the view in the areas obscured by rain or dirt.

[0034] The temporal aspect can be particularly helpful in reconstructing a clear image, especially in areas covered by dirt. For example, imagine that some parts of the lens are covered by dirt, while other areas are clear. At time t, an object can be seen completely, but at another time t+1, dirt prevents the object from being fully captured. By moving the object and / or the camera while driving, the information gathered about the object in the image at time t can be used to reconstruct the image at time t+1.

[0035] In one embodiment, the artificial neural network has a common input interface for two separate output interfaces. The common input interface has shared feature representation layers. Corrected (i.e., converted) image data is output at the first output interface. ADAS-relevant detections from at least one ADAS detection function are output at the second output interface. ADAS stands for Advanced Driver Assistance Systems. ADAS-relevant detections are, for example, objects, road users, and other road users that represent important input variables for ADAS / AD systems. The artificial neural network includes ADAS detection functions such as lane detection, object detection, depth detection (3D estimation of image components), semantic recognition, and so on.As part of the training, the outputs of both output interfaces are optimized.

[0036] A procedure for correcting input image data from a camera that is affected by rain, light exposure and / or dirt includes the following steps: a) Input image data captured by the camera, affected by rain, light exposure, and / or dirt, is provided to a trained artificial neural network; b) the trained artificial neural network is configured to convert the input image data affected by rain, light exposure, and / or dirt into output image data without impairment and to determine a safety measure c, which depends on the degree of wetting by water, light exposure, and / or dirt for an image (or for each image) of the input image data and characterizes the network's confidence that the network's image correction is correct; and c) the trained artificial neural network is configured to output the output image data and the determined safety measure(s) c.

[0037] The corrected output image data advantageously enables better machine object recognition on the output image data, e.g., conventional lane / object or traffic sign detection.

[0038] According to one embodiment, the factor d is estimated, and the impairment of the currently acquired input image data is taken into account during the estimation. Cumulatively or alternatively, the estimation of the factor d of the currently acquired input image data can take into account the factor(s) d of the previously acquired image data.

[0039] According to one embodiment, the temporal development of the factor d can be taken into account when determining or estimating it. For this purpose, the temporal development of the factor d and a sequence of input images are included in the estimation.

[0040] In one embodiment, the camera is a vehicle-mounted environment detection camera.

[0041] According to one embodiment with a vehicle-mounted environmental sensing camera, information about the vehicle's current surroundings is taken into account when determining the factor d. This information can include, for example, rain sensor data, external (V2X data or data from a navigation system, e.g., GPS receiver with digital map) spatially resolved weather and / or sun position information, and driving situation information (country road, city, highway, tunnel, underpass). This information can also be (at least partially) obtained from the camera image data via image processing.

[0042] For example, the current factor d can be estimated based on environmental situation information and from the temporal sequence of images as well as from the history of the factor d.

[0043] The estimation of the factor d can therefore be performed dynamically when using a trained artificial neural network.

[0044] In one embodiment, the corrected image data from the vehicle-mounted surround-view camera and the determined safety measure(s), and optionally also the factor d, are output to at least one ADAS detection function, which detects and outputs ADAS-relevant detections. ADAS detection functions can include known edge or pattern recognition methods, as well as recognition methods that can detect and optionally classify relevant image objects using an artificial neural network.

[0045] In an alternative embodiment, the approach can be extended, and the artificial neural network for correcting the image data can be combined with a neural network for ADAS detection functions, such as lane detection, object detection, depth detection, and semantic recognition. This results in minimal additional computational overhead. After training, the (first) output interface for the converted (corrected) image data can be eliminated, so that when used in the vehicle, only the (second) output interface for the ADAS detections remains.

[0046] In another embodiment, the learned method can be used conversely, instead of reconstructing unclear or impaired image data, to artificially add rain or dirt to recorded image data from the learned reconstruction profile for a simulation to ensure accuracy.

[0047] In another embodiment, the learned reconstruction profile can also be used to evaluate the quality of an artificial rain simulation in recorded image data.

[0048] According to another embodiment, the method can be applied to augmented reality and in the field of dash cam and accident recordings.

[0049] The invention further relates to a device with at least one data processing unit configured for correcting input image data from a camera, which is affected by rain, light exposure and / or dirt, into output image data. The device comprises: an input interface, a trained artificial neural network and a (first) output interface.

[0050] The input interface is configured to receive input image data that is affected by rain, light exposure, and / or dirt captured by the camera. The trained artificial neural network is configured to convert the affected input image data into unaffected output image data and to determine a confidence level c, which depends on the degree of wetting by water, light exposure, and / or dirt for an image or for each image of the input image data. This confidence level characterizes the network's confidence that the image correction performed by the network is accurate.

[0051] The (first) output interface is configured to output the converted (corrected) image data and the determined safety measure(s) c.

[0052] According to one embodiment, the input image data contains at least one sequence of successively captured input images as input image data, and the artificial neural network has been trained using at least one sequence of successive input and output images as image data.

[0053] The device or data processing unit may in particular include a microcontroller or microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) and the like, as well as software for carrying out the corresponding process steps.

[0054] According to one embodiment, the data processing unit is implemented in a hardware-based image signal processor (ISP).

[0055] In one embodiment, the trained artificial neural network for image correction is a component of a vehicle-side ADAS detection neural network, e.g. for semantic segmentation, lane detection or object detection, with a shared input interface (input or feature representation layers) and two separate output interfaces (output layers), wherein the first output interface is configured to output the converted output image data and the second output interface is configured to output the ADAS detections (image recognition data).

[0056] The invention further relates to a computer program element which, when used to program a data processing unit, instructs the data processing unit to perform a method for correcting input image data from a camera into output image data.

[0057] The invention further relates to a computer-readable storage medium on which such a program element is stored.

[0058] The invention further relates to the use of a machine learning method for image correction of input image data from a camera into output image data for training an artificial neural network of a device with at least one data processing unit.

[0059] The present invention can therefore be implemented in digital electronic circuits, computer hardware, firmware or software.

[0060] Key advantages include: Enabling object recognition in case of impairment (e.g. fogging) of cameras. Generation of an image data stream for human and computer vision from a neural network for optimized correspondence search (search for feature match).

[0061] Besides its use in motor vehicles, there are many other areas of application: Assistance systems in buses, trains, airplanes, and robotics systems. All applications of feature-based detection, e.g., detection methods based on optical flow, structure-from-motion, etc., which experience a dramatic degradation due to missing features when exposed to contamination / rain / light impairment; thus, in particular, assistance systems that rely on optical flow for feature search. Online calibration of cameras, which also loses performance dramatically when wet or dirty due to missing features or low contrast.

[0062] The following section describes exemplary embodiments and figures in more detail. Fig. 1: schematically a vehicle with an environment detection camera; Fig. 2: a system for correcting impaired camera images; Fig. 3: a system with a first neural network for image correction and a downstream second neural network for detection functions; Fig. 4: a system with combined image correction and detection functions; and Fig. 5: a modified system in which the image correction is only calculated and output during training.

[0063] Fig. 1 Figure 1 schematically shows a vehicle F with an environment-sensing camera K, which is located inside the vehicle behind the windshield and captures the vehicle's surroundings. In darkness, the vehicle F's headlights S illuminate the area in front of the vehicle, which is then captured by camera K. Camera K can be a monocular camera. Camera K captures a sequence of images of the vehicle's surroundings. As the vehicle F moves, the area of ​​the environment captured by camera K changes continuously.

[0064] If rain or dirt is present on the windshield or an external camera lens, or if stray light, e.g. sunlight when the sun is low or there is strong reflection, enters the optical path of camera K, this leads to serious impairments in the images captured by camera K.

[0065] Fig. 2 The diagram schematically shows a general overview of a system for correcting camera images in the event of rain, light exposure, and / or dirt. A key component is an artificial neural network, CNN1, which, during a training phase, learns to match a set of training input images, In (In1, In2, In3, ...), with a set of corresponding corrected training (target) output images, Out (Out1, Out2, Out3, ...). Matching here means that the neural network, CNN1, learns to generate a corrected image. An input image (In1, In2, In3, ...) might, for example, contain a street scene in the rain, in which only larger objects, such as a large lane marking representing a bicycle and the sky, are visible to the human eye. The corresponding corrected image (Out1, Out2, Out3, ...) additionally shows the contours of a street intersection, a traffic light, a tree, and a street lamp.Realistic photographic illustrations for color images with and without impairment by raindrops are shown, for example, in the illustration. Fig. 1 to be seen by Porav et al.

[0066] Optionally, a factor d serves as an additional input for the neural network CNN1. Factor d is a control parameter that determines the degree of correction applied to image impairments (rain, light exposure, or dirt). During training, factor d for a pair of images (In1, Out1; In2, Out2; In3, Out3; ...) consisting of a training image and a corrected image can be determined beforehand or during training and provided to the neural network CNN1. This allows factor d to be learned along with the image.

[0067] When using the trained neural network CNN1, the degree to which it corrects a currently captured image can be controlled by specifying a factor d. This factor d can also be thought of as an external regression parameter (with arbitrary increments). Since the factor d can be subject to fluctuations of approximately ±10%, this is taken into account during training. The factor d can be made noisier by approximately ±10% during training (e.g., during the different training epochs of the neural network) to ensure robustness against miscalculations of the factor d within the range of approximately ±10% during inference in the vehicle. In other words, the required accuracy of the factor d is in the range of ±10% – thus, the neural network CNN1 is robust against deviations in its estimates of this parameter.

[0068] Alternatively or additionally, the factor d for any image correction performed by the trained neural network CNN1 can be output. This provides downstream image recognition functions with information about the extent to which the originally captured image was corrected.

[0069] The artificial neural network CNN1 is designed to determine a safety measure c, which depends on the degree of wetting by water, light exposure, and / or soiling of an input image. The network's design can be achieved, for example, through appropriate architecture design. After machine learning is complete, CNN1 can determine and output the safety measure c for a new input image. This safety measure c depends on the degree of impairment caused by rain or water, light exposure, and / or soiling, and, when the trained network is used, characterizes the certainty that an image correction is accurate.

[0070] In Fig. 2 Three image pairs, In1 + Out1, In2 + Out2, and In3 + Out3, are schematically represented. Accordingly, the trained neural network CNN1 determines and outputs a safety measure c1, c2, and c3 for each image pair.

[0071] Since the quality of image correction for rain, light exposure, or pollution depends on numerous factors (such as the presence of a similar case in the training data, sufficient exposure for a possible correction, avoidance of ambiguous scenarios, etc.), the network outputs a measure of the confidence with which it makes its decision, in addition to the image correction. This confidence measure c can take one of the following forms or a combination thereof: A confidence measure c_Prob: Here, the network's output is calibrated so that it can be interpreted probabilistically as the probability that the network makes the correct decision. Values ​​for this are normalized to a range between [0,1] and correspond to the spectrum from 0% probability to 100% probability that the network has calculated a correct correction for an image. This calibration can be performed after completion of the actual machine learning process using a training image dataset by subsequently verifying the quality of the learning using a validation image dataset. The validation image dataset also contains image pairs: each pair consists of a first image with rain, light exposure, and / or dirt impairment and, as the corresponding target output image, a second image of the same scene without impairment. In practice, a portion of the input and target output images can be retained, i.e.,not used for the machine learning process and subsequently used for validation. A measure of dispersion similar to a standard deviation, c_Dev: Here, an uncertainty of the network output is estimated in such a way that it describes the dispersion of the network output. This can be implemented in different ways. One possibility is the division into measurement and model uncertainty. Measurement uncertainty refers to uncertainties caused by the input data, e.g., slight disturbances. These can be added to the network via an additional output and trained by changing the error function. Model uncertainty refers to uncertainties caused by the limited mapping accuracy and generalizability of a network. This relates to factors such as the size of the training data and the architecture of the network design. Model uncertainty can be estimated, e.g., through Monte Carlo dropout or network ensembles. Model uncertainty and measurement uncertainty can be additively combined. A combination of confidence and dispersion measures.

[0072] The safety factor c can be calculated for the entire image, image areas, or the individual pixels of the image.

[0073] Based on the safety measure c, the following decisions can be made: c_Prob Low: The network has low confidence in its estimate – misestimations occur more frequently. High c_Prob: The network has high confidence in its estimate – the image correction is correct in most cases. Low c_Dev: The variance of the network's image correction is low – thus, the network predicts a very accurate image correction. High c_Dev: The estimated variance of the image correction, similar to a standard deviation, is high, and the network's output is less accurate / less sharp – small changes to the input data or in the network model would cause deviations in the image correction. Combinations: o High c_Prob and low c_Dev: a very reliable and accurate image correction that can be accepted with a high degree of certainty o Low c_Prob and high c_Dev: a very uncertain and inaccurate image correction that would likely be rejected o High c_Prob and high c_DevLow c_Prob and low c_Dev: these corrections are subject to uncertainties, and careful use of image corrections is recommended here.

[0074] The inclusion of safety measures is particularly relevant for safety-critical functions.

[0075] One way to generate the training data (training images (In1, In2, In3, ...) and associated corrected images (Out1, Out2, Out3, ...)) is to acquire image data with a "stereo camera setup" as described in Porav et al. using Fig. 8: a two-part chamber with transparent discs is arranged in front of two identical camera modules at a small distance from each other; the chamber in front of the right stereo camera module, for example, is sprayed with water droplets, while the chamber in front of the left stereo camera module is kept free of obstructions.

[0076] To simulate the effects of light exposure in an analogous way, a light source can, for example, be directed only onto one chamber. Or, in the case of dirt, it can also be applied only to one chamber.

[0077] Alternatively, one can use unaffected images to generate the training image pairs and then degrade them using rendering techniques that simulate the effects of rain, light exposure, or dirt in the image.

[0078] Once the neural network CNN1 is trained, image correction is performed according to the following scheme: Input image → CNN1 Optional: Factor d → CNN1 CNN1 → corrected output image + safety measure c.

[0079] The Fig. 3 bis 5 show exemplary implementations of possible combinations of a first network for image correction with one or more networks of functions for driver assistance functions and automated driving, ordered according to the consumption of computing resources.

[0080] Fig. 3 Figure 1 shows a system with a first neural network, CNN1, for image correction and a downstream second neural network, CNN2, for detection functions (fn1, fn2, fn3, fn4). These detection functions (fn1, fn2, fn3, fn4) are image processing functions that detect objects, structures, and properties (generally: features) relevant to ADAS or AD functions in the image data. Many such detection functions (fn1, fn2, fn3, fn4), which are based on machine learning, have already been developed or are currently under development (e.g., traffic sign classification, object classification, semantic segmentation, depth estimation, lane marking detection and localization). On corrected images (Opti), the detection functions (fn1, fn2, fn3, fn4) of the second neural network, CNN2, deliver better results than on the original input image data (Ini) under poor visibility conditions.

[0081] Once the two neural networks CNN1 and CNN2 are trained, a process can proceed according to the following scheme: Input image (Ini), optional factor d → CNN1 → corrected output image (Opti) + safety factor c → CNN2 for detection functions (fn1, fn2, fn3, fn4) → output of the detections: objects, depth, trace, semantics, ...

[0082] Fig. 4 Figure 1 shows a neural network CNN10 for image correction of an input image (Ini), optionally controlled by a factor d, which shares feature representation layers (as input or lower layers) with the network for the detection functions (fn1, fn2, fn3, fn4). In the feature representation layers of the neural network CNN10, common features for image correction and for the detection functions are learned.

[0083] The neural network CNN10 with split input layers and two separate outputs has a first output CNN 11 for outputting the corrected output / output image (Opti) and the safety measure c, and a second output CNN 12 for outputting the detections: objects, depth, trace, semantics, etc.

[0084] Because the feature representation layers are optimized during training with regard to both image correction and detection functions (fn1, fn2, fn3, fn4), optimizing image correction simultaneously improves the detection functions (fn1, fn2, fn3, fn4).

[0085] If outputting the corrected image (Opti) is not desired or necessary, the approach can be further varied, as shown by... Fig. 5 will be explained.

[0086] Fig. 5 shows one on the system of Fig. 4This approach is based on neural network-based image correction through feature optimization. To save computation time, the features for the detection functions (fn1, fn2, fn3, fn4) are optimized during training with respect to image correction and with respect to the detection functions (fn1, fn2, fn3, fn4).

[0087] At runtime, i.e., when using the trained neural network (CNN10, CNN11, CNN12), no corrected images (Opti) are calculated.

[0088] Nevertheless, the detection functions (fn1, fn2, fn3, fn4) - as already explained - are improved by the joint training of image correction and detection functions compared to a system with only one neural network (CNN2) for detection functions (fn1, fn2, fn3, fn4), in which only the detection functions (fn1, fn2, fn3, fn4) were optimized in the training.

[0089] During the training phase, an additional output interface (CNN11) outputs the corrected image (Opti) and compares it to the ground truth (the corresponding corrected training image). In the test phase, or at runtime, this output (CNN11) can be used further or, to save computation time, truncated. The weights for the detection functions (fn1, fn2, fn3, fn4) are modified during this training with the additional output (CNN11) to account for the image corrections applied to these functions. Thus, the weights of the detection functions (fn1, fn2, fn3, fn4) implicitly learn the information about the image correction.

[0090] Further aspects and implementations of an assistance system that algorithmically converts the image data from the underlying camera system into a representation that corresponds to a recording without these impairments, despite impairments caused by rain, light exposure, or dirt, are described below. The converted image can then serve either for pure display purposes or as input for feature-based recognition algorithms. 1) In an initial implementation, the calculation in a system is based, for example, on a neural network that, upstream of a detection or display unit, converts an input image with condensation, dirt, or water droplets and low contrast and color information into a cleaned representation. For this task, the neural network was trained with a dataset consisting of "condensed input images" and the corresponding "cleaned images." 2) In particular, the use of cleaned images trains the neural network to preserve, and ideally even enhance, features present in the image pairs to be improved, despite condensation or dirt, for subsequent correspondence searches or object recognition.3) When training the image correction / enhancement network, feature-based methods for display and object detection can be considered, allowing the method to be specialized for the features to be recognized and to explicitly highlight these features for subsequent processing. 4) In a further implementation, the image enhancement or correction method can be integrated into a hardware-based image preprocessing stage (ISP). This ISP is augmented on the hardware side with a neural network that performs the conversion and makes the processed information available to possible detection or display methods along with the original data. 5) In another application, information on image quality can be provided to the training network in addition to information on soiling or condensation.The system and the process can therefore be optimized to calculate image data optimized for object recognition and human vision.

[0091] In addition to correcting images caused by condensation or water droplets, the system detects water droplets or dirt, for example, to activate windshield wipers or to prompt a satellite camera to clean. Combined with brightness detection, this enables a rain light detection function alongside image correction.

Claims

1. A method for machine learning of image correction of input image data of a camera (K), which are impaired by rain, incident light, and / or soiling, into output image data by means of an artificial neural network (CNN1, CNN10, CNN11, CNN12), wherein the learning is carried out using a plurality of training image pairs (In1, Out1; In2, Out2; In3, Out3; ...) in such a way that in each case a first image (In1, In2, In3, ...) having rain, incident light, and / or soiling impairment is provided at the input of the artificial neural network (CNN1, CNN10) and a second image (Out1, Out2, Out3, ...) of the same scene without impairment is provided as the target output image, characterised in that the artificial neural network (CNN1, CNN10, CNN11, CNN12) is designed in such a way that it determines a certainty measure c, which depends on the degree of wetting by water, incident light, and / or soiling for an input image, and characterises the certainty of the network that the image correction of the network is accurate, wherein the artificial neural network (CNN1, CNN10, CNN11, CNN12) can, after completion of the machine learning, determine and output the certainty measure c for a new input image, and wherein at least one factor d is determined as a measure of the difference between the target output image (Out1, Out2, Out3, ...) and the impaired input image (In1, In2, In3, ...) of a training image pair (In1, Out1; In2, Out2; In3, Out3; ...) and is provided to the artificial neural network as an additional input variable.

2. The method according to Claim 1, wherein the training image pairs (In1, Out1; In2, Out2; In3, Out3; ...) are generated by in each case recording a first image (In1, In2, In3, ...) with rain, incident light, and / or soiling impairment, and a second image (Out1, Out2, Out3, ...) without impairment, simultaneously or in immediate succession.

3. The method according to any one of the preceding claims, wherein the training image pairs (In1, Out1; In2, Out2; In3, Out3; ...) contain at least one sequence of successive input and output images in the form of video sequences.

4. The method according to any one of the preceding claims, wherein the artificial neural network (CNN1, CNN10, CNN11, CNN12) has a common input interface for two separate output interfaces (CNN11, CNN12), wherein the common input interface has shared feature representation layers, wherein corrected image data (Opti) are output at the first output interface (CNN11), wherein ADAS-relevant detections of at least one ADAS detection function (fn1, fn2, fn3, fn4) are output at the second output interface (CNN12), and wherein the outputs of both output interfaces (CNN11, CNN12) are optimized in the context of the training.

5. A method for correcting input image data of a camera (K) that are impaired by rain, incident light, and / or soiling, comprising the following steps: a) input image data (Ini), which are captured by the camera (K) and are impaired by rain, incident light, and / or soiling, and a factor d as an additional input variable are provided to a trained artificial neural network (CNN1, CNN10, CNN11, CNN12), b) the trained artificial neural network (CNN1, CNN10, CNN11, CNN12) is configured to convert the input image data (Ini) with rain, incident light, and / or soiling impairment into output image data (Opti) without impairment and to determine a certainty measure c, which depends on the degree of wetting by water, incident light, and / or soiling for an image of the input image data and characterizes the certainty of the network that the image correction of the network is accurate, wherein the conversion is controlled depending on the factor d, and c) the trained artificial neural network (CNN1, CNN10, CNN11, CNN12) is configured to output the output image data (Opti) and the determined certainty measure c.

6. The method according to Claim 5, wherein the input image data contain at least one sequence of successively captured input images as input image data.

7. The method according to any one of Claims 5 and 6, wherein the camera (K) is a vehicle-related surroundings capture camera.

8. The method according to Claim 7, wherein the converted image data (Opti) and the determined certainty measure c are output to at least one ADAS detection function, which determines and outputs ADAS-relevant detections on the basis of the converted image data.

9. A device having at least one data processing unit configured to correct input image data (Ini) of a camera (K), which are impaired by rain, incident light, and / or soiling, into output image data (Opti), comprising: - an input interface configured to receive the input image data (Ini) with rain, incident light, and / or soiling impairment from the camera (K), - a trained artificial neural network (CNN1, CNN10, CNN11, CNN12) configured to convert the input image data (Ini) into output image data (Opti) without impairment depending on a factor d that is provided to the neural network (CNN1, CNN10, CNN11, CNN12) as an additional input variable, and to ascertain a certainty measure c which is dependent on the degree of wetting by water, incident light, and / or soiling for an image of the input image data and characterizes the certainty of the network in that the image correction of the network is accurate, and - a first output interface (CNN11) configured to output the converted output image data (Opti) and the determined certainty measure c.

10. The device according to Claim 9, wherein the data processing unit is implemented in a hardware-based image pre-processing stage.

11. The device according to Claim 9 or 10, wherein the camera (K) is a vehicle-based surroundings capture camera, and the trained artificial neural network (CNN1, CNN10, CNN11) for image correction is part of a vehicle-side ADAS-detection neural network (CNN2, CNN12) having a shared input interface and two separate output interfaces, wherein the first output interface (CNN11) is configured to output the corrected output image data (Opti) and the second output interface (CNN12) is configured to output the ADAS-relevant detections.

12. The device according to any one of Claims 9 to 11, wherein the input image data contain at least one sequence of successively captured input images as input image data, and the artificial neural network has been trained on the basis of at least one sequence of successive input and output images.

13. A computer program element which, when it is used to program a data processing unit, instructs the data processing unit to carry out a method according to any one of Claims 5 to 8.