Correcting images from a surround view camera system in rainy, backlit, or dirty conditions

Abstract

The present invention relates to a machine learning method, a method and an apparatus for correcting input image data (Ini) of multiple cameras (2-i) of a surround view system, e.g., a vehicle-mounted surround capture camera, that have been corrupted by rain, backlight and / or dirt. The method for correcting input image data corrupted by rain, backlight, and / or dirt from a plurality of cameras (2-i) of the surround view system comprises the following steps: a) providing input image data (Ini) captured by cameras (2-i) that are corrupted by rain, backlight and / or dirt to trained artificial neural networks (CNN1, CNN10, CNN11, CNN12); b) the trained artificial neural network (CNN1, CNN10, CNN11, CNN12) is configured to convert input image data (Ini) affected by rain, backlighting and / or dirt into lossless output image data (Opti) and to characterize the safety of the network depending on the degree of water coverage, backlighting and / or dirtiness of the input image data and to determine a safety measure c corresponding to the image modification of the network; and c) The trained artificial neural network (CNN1, CNN10, CNN11, CNN12) is configured to output the output image data (Opti) and the determined safety measure c. The present method has the advantage of enabling the generation of image data streams for human and computer vision applications from networks for object recognition and optimized correspondence search when the camera is overcast.

Description

[Technical field]

[0001] The present invention relates to a machine learning method, a method and an apparatus for correcting image data of multiple on-board cameras of a surround view system, for example a surround view camera system connected to a vehicle, in the event of rain, light shining (backlighting) or dirt. [Background technology]

[0002] Today's and future vehicles are equipped with camera-based assistance systems whose role is to recognize objects in order to avoid collisions and to recognize road boundaries in order to keep the vehicle in its lane. For this purpose, for example, forward-facing camera(s) are used. In addition to forward-facing cameras, surround view cameras and satellite cameras, which can be mounted on the vehicle in any position and provide detection functions for driving and parking or visualization of the 360° surroundings (or parts of it) of the vehicle in the near and far distances, are also used.

[0003] There are also concepts for forward facing cameras that also include detection capabilities for detecting rain.

[0004] By combining camera-based daylight recognition with a camera for controlling the vehicle headlights, so-called rain-light detection can be realized, which recognizes rain on the windshield and activates the wipers, for example.

[0005] Recognition algorithms based on such camera systems in part combine approaches from traditional image processing with approaches from machine learning, especially deep learning: whereas traditional approaches to object and structure recognition in the field of image processing are based on manually selected features, deep learning-based approaches identify and optimize relevant features themselves within the training method.

[0006] As soon as the camera systems described above find themselves in conditions where visibility is impaired by rain, backlight or dirt, they immediately show a degradation in both object recognition and peripheral and object depiction.

[0007] For example, if the field of view of a front camera integrated into the mounting of the rearview mirror becomes restricted by water droplets or dirt on the windshield, the field of view can be restored by activating the wipers, provided that the camera is mounted within the wiper's wiping area.

[0008] Considering the increasing degree of automation of vehicles and vehicle functions, it is advantageous to increase the number of cameras facing forward. However, since it is not possible to centralize them at the rearview mirror mounting, satellite cameras are also installed at the upper corners of the windshield, near the A-pillars. However, this area is problematic for detection purposes, since it is outside the wiper wiping area. The limited field of view due to raindrops and dirt affects the detection function of these satellite cameras.

[0009] As the degree of automation in vehicles increases to SAE Level 4 / Level 5, more and more cameras are installed on both sides of the vehicle, which not only display the surrounding area but also detect objects to the side. These cameras are often installed externally to the vehicle, for example in the area of ​​the side mirrors. If water droplets or dirt get on the (external) lens of the camera, the display and detection functions can be significantly limited. However, since they do not have a means of cleaning, such as a wiper, this can lead to degradation or failure of the system's functions.

[0010] An example of the latter is the reversing camera located above the license plate, which is very susceptible to getting dirty. Here too, rain and dust can accumulate, making it difficult to provide a clear image.

[0011] While convolutional neural network (CNN)-based object recognition methods can compensate, at least to some extent, for dirty or rain-soaked lenses, methods based on image features to recognize objects, such as optical flow and structure-from-motion, suffer from significant degradation in performance due to contamination.

[0012] Algorithmic methods are known for detecting dirt or precipitation on the exterior lens of a camera or on the windshield of a vehicle using image processing.

[0013] WO 2013 / 083120 A1 discloses a method for evaluating image data of a car-mounted camera, where raindrops on a glass in the field of view of the car-mounted camera are taken into account when evaluating the image data. Here, information about the raindrops is also determined from the image data. An example of the evaluation of image data is a method of recognizing objects, where the information is appropriately taken into account. From the recognized rain intensity, for example, the influence of edges (light and dark or color transitions) as seen by the camera can be inferred. That is to say, the edge-based evaluation method can be adapted appropriately by means of thresholds. In particular, from the information, it is also possible to determine a quality criterion of the image data, which is taken into account when evaluating the image data.

[0014] On July 13, 2020, H. Porav et al. http: / / www.robots.ox.ac.uk / ~mobile / Papers / ICRA19_porav.pdf Taken from: "I Can See Clearly Now: Image Restoration via De-Raining", 2019 IEEE Int. Conference on Robotics and Automation (ICRA), Montreal, Canada, p. 7087-7093, they show a method to improve segmentation tasks in images that are affected by raindrops, streaks, etc.

[0015] In the method, stereo datasets of lenses with realistic raindrops and clear lenses are generated, which are then used to train a "Denoising-Generator" to remove the effects of water droplets in the context of image reconstruction and road marking segmentation.

[0016] What is needed is a system that uses algorithms to enhance images even in the presence of dirt, backlight, or water droplets, for example, to improve subsequent object recognition, and additionally allows for rain and light detection. [Prior art documents] [Patent documents]

[0017] [Patent Document 1] International Publication No. 2013 / 083120 (WO2013 / 083120A1) [Non-patent literature]

[0018] [Non-Patent Document 1] H. Porav et al., "I Can See Clearly Now: Image Restoration via De-Raining", 2019 IEEE Int. Conference on Robotics and Automation (ICRA), Montreal, Canada, p. 7087-7093, (accessed July 13, 2020) http: / / www.robots.ox.ac.uk / ~mobile / Papers / ICRA19_porav.pdf Summary of the Invention [Problem to be solved by the invention]

[0019] The object of the present invention is therefore to provide a solution for this. [Means for solving the problem]

[0020] According to the invention, this object is achieved by the subject matter set forth in the independent claims, with advantageous embodiments being the subject matter of the dependent claims, the following description as well as the figures.

[0021] The machine learning method according to the invention concerns image correction of rain, backlight and / or dirt affected input image data of a plurality of cameras of a surround view system into corrected output image data by means of an artificial neural network. The training with a plurality of training image pairs is performed by providing the input side of the artificial neural network with a first image (or a first plurality of simultaneously captured images) affected by rain, backlight and / or dirt and a second image (or a second plurality of simultaneously to be achieved ideal images) of the same scene that is not affected by rain, backlight and / or dirt, respectively, as an ideal output image. The artificial neural network is configured to determine a safety measure c for the input image that depends on the degree of wetness, backlight and / or dirt. The specification of the network is achieved for example by a suitable design or a suitable architecture of the artificial neural network. After the machine learning, the artificial neural network is able to determine and output a safety measure c for a new input image (or input images captured by a plurality of cameras, each simultaneously). That is, the safety measure c characterizes the safety of image retouching when using the trained network depending on the degree of impairment due to wetness due to rain or water, backlighting and / or dirt.

[0022] In other words, the safety measure c characterizes to some extent the "(un)safety" of the image modification performed by the trained neural network. In practice, the safety measure c is a measure of the safety of the network in the computational task (i.e., image modification by the network).

[0023] The artificial neural network can be, for example, a Convolutional Neural Network (CNN).

[0024] In addition, the term "conversion to output / image data without impairment" as used herein includes "conversion to output / image data with reduced impairment."

[0025] The camera can be, for example, a (monocular) camera mounted inside or on the vehicle that captures the surroundings of the vehicle. An example of such a vehicle-connected camera can be a camera located inside the vehicle behind the windshield that captures and depicts the area of ​​the vehicle surroundings that is in front of the vehicle through the windshield.

[0026] The adverse effects on camera images caused by rain, backlighting, or dirt are similar in that they manifest as (local) blurring in the image. In all these cases, image modification that reduces or eliminates the blurring in the image is desirable.

[0027] According to an embodiment, at least one factor d is determined as a measure of the difference between the modified output image and the impaired input image and is provided to the artificial neural network during training. The factor d is taken into account by the artificial neural network during learning, for example by training the neural network on the link between the input image, the output image and the factor d. This allows the trained network to subsequently estimate or determine the factor d for the latest captured impaired camera image and to generate (or reproduce) an output image with a corresponding modification. After training, for example, the trained neural network can be provided with the factor d, thereby controlling the degree of modification of the latest captured camera image.

[0028] The factor d can be determined, for example, by a local comparison of an undamaged image and an image that is impaired by rain or dirt.

[0029] The factor d can then be determined, for example, by using a D2 filter, which can be mapped onto the input layer of an artificial neural network.

[0030] In a simple embodiment, the factor d can be expressed as the variance of a D2 low-pass filter. Alternatively, complex contrast values ​​(Structural Similarity) or correlations (Sum of absolute distances - SAD, Sum of squared distances - SSD, Zero-means normalized cross correlation - ZNCC) can be considered, which can be calculated using local filters from both images.

[0031] For example, a camera lens that is dirty due to rain or dirt may produce impaired images that may make object recognition difficult. In the context of machine learning methods, the factor d can be determined by comparing an ideal output image with its corresponding impaired input image. This determination can be performed in advance, i.e., for each training image pair, the factor d is already prepared. Alternatively, the factor d can be determined based only on the training image pairs in the context of the learning method.

[0032] The factor d indicates the degree of feasible reproduction of the corrected image and allows a value to be prepared for attachment to subsequent image processing or display functions, e.g., low values ​​indicate high correction and high values ​​indicate low correction to the next processing step, which can be taken into account - similar to the safety measure c - when determining the quality of the created object data.

[0033] In one embodiment, training image pairs are created by simultaneously capturing, or using multiple cameras to capture, a first image (in the optical path of the camera) that is affected by rain, backlight, and / or dirt, and a second image that is not affected by rain, backlight, and / or dirt, either simultaneously or in rapid succession at different exposure times.

[0034] In one embodiment, a single artificial neural network is trained jointly or simultaneously for all vehicle-mounted cameras.

[0035] A sequence of images for each single camera can be used for joint training.

[0036] Temporal correlation of multiple images can be advantageously taken into account during training and / or when using a pre-trained network.

[0037] Using information about the image features captured by the front camera at time t and the side and rear cameras at a later time point, as well as the ideal output image data, we can train objects with a particular image feature in all single camera output images to have the same brightness and saturation.

[0038] According to an embodiment, the training image pairs include at least a sequence of input and output images (image data). In other words, the image data can be a series of images (video sequences). In this case, at least one input video sequence is required for machine learning, as well as an ideal video sequence.

[0039] When using an image sequence, temporal aspects or correlations can preferably be taken into account during the reconstruction (or image correction). An example is raindrops or dust particles that move over time. This results in areas in the image that at a certain time t have a clear view and at a certain time t+1 have a view obstructed by rain. By using an image sequence, the information in the clear image areas can be used to reconstruct the view obstructed by rain or dirt.

[0040] The time perspective helps to recreate a clear image, especially in areas covered by dirt. In one example, one area of ​​the lens is covered by dirt, while other areas are not. At time t, an object is completely visible, but at another time t+1, the dirt prevents the object from being captured in its entirety. The information about the object in the image obtained at time t helps to recreate the image at time t+1, because the object has moved and / or the camera has moved due to the driving.

[0041] In one embodiment, the artificial neural network has a common input interface for two independent output interfaces. The common input interface has a joint feature representation layer. The first output interface outputs the corrected (i.e. transformed) image data. The second output interface outputs the ADAS-related detections of at least one ADAS detection function. ADAS is an acronym for Advanced Driver Assistance System. Thus, the ADAS-related detections represent, for example, objects, objects, traffic participants, which are important inputs for the ADAS / AD system. The artificial neural network includes the ADAS detection functions, for example, lane recognition, object recognition, depth recognition (3D estimation of image components), semantic recognition, and the like. In the training range, the outputs of both output interfaces are optimized.

[0042] A method for correcting input image data corrupted by rain, backlighting, and / or dirt from multiple cameras of a surround view system comprises the following steps: a) providing input image data captured by a camera, which may be corrupted by rain, backlight and / or dirt, to a trained artificial neural network; b) the trained artificial neural network is configured to convert input image data affected by rain, backlighting and / or dirt into loss-free output image data and to determine a safety measure c characterizing the safety (when using the trained network) as a function of the degree of water coverage, backlighting and / or dirt for an image or for each image of the input image data; and c) The trained artificial neural network is configured to output the output image data and the determined safety measure c. The modified output image data has the advantage that it can improve mechanical object recognition on the output image data, for example conventional lane recognition, object recognition, or traffic sign recognition, or can improve stitching (combining images captured by multiple cameras simultaneously) and the display of the combined image data.

[0043] According to one embodiment, the input image data comprises at least one sequence of input images captured by a number of cameras (a video sequence).

[0044] In one embodiment, the cameras are vehicle mounted perimeter capture cameras.

[0045] In one embodiment, in step a), a factor d is additionally provided to the trained artificial neural network, and in step b), the (degree or rate of) image modification or transformation is controlled in dependence on the factor d.

[0046] According to one embodiment, the factor d is estimated and the estimation takes into account impairments of the most recently captured input image data. In an additive or alternative manner, the factor d of the previously captured image data can be taken into account in the estimation of the factor d of the most recently captured input image data.

[0047] According to an embodiment, the determination or estimation of the factor d can also take into account the temporal evolution of the factor d, where the estimation takes into account the temporal evolution of the factor d as well as the sequence of input images. Information on the temporal evolution of the luminance can also be referenced for image regions with different factors d.

[0048] In one embodiment, a separate factor d is estimated or determined for each camera of the surround view system, which allows separate transformations for the image data of each (vehicle) camera depending, among other things, on the latest impairments of each camera's image.

[0049] In an embodiment with a vehicle-mounted perimeter capture camera, the latest vehicle perimeter information is taken into account when determining the factor d. The latest perimeter information may include, for example, rain sensor data, location-determined weather and / or sun position (direction and height) information from the outside (V2X data or navigation system data, e.g. GPS receiver with digital camera), driving conditions (road, city, highway, tunnel, underpass). This information (at least in part) may be obtained from the camera images by image processing.

[0050] For example, the latest factor d can be estimated not only from the surrounding situation information and the chronological order of the images, but also from the history of the factor d.

[0051] Thereby, the estimation of the factor d can be performed dynamically when using a trained artificial neural network.

[0052] In one embodiment, the output of the corrected image data of the vehicle mounted surroundings capture camera and the determined safety measure(s) c and, optionally, the factor d are provided to an ADAS detection function for determining and outputting detections related to the ADAS. The ADAS detection function may include a recognition system that can recognize and, optionally, classify image objects related to known edge or pattern recognition methods as well as artificial neural networks.

[0053] In an alternative embodiment, the approach can be extended and the artificial neural network for modifying the image data can be combined with neural networks for ADAS detection functions, e.g. lane recognition, object recognition, depth recognition, semantic recognition, which generates almost no additional burden in terms of computation time. After training, the (first) output interface for outputting the transformed (modified) image data can be eliminated, so that when used in a vehicle, only the (second) output interface for the ADAS detection functions is present.

[0054] According to yet another embodiment, the learned method is used in reverse to add artificial rain or dirt to the captured image data from the learned reproduction profile as a simulation to obtain authenticity, instead of reproducing a blurred or impaired image.

[0055] According to yet another embodiment, the learned reproduction profile can also be used to assess the quality of artificial rain simulation in captured image data.

[0056] According to a further embodiment, the Augmented Reality method can be applied in the context of Dash Cam and Accident Recording.

[0057] The invention further relates to an apparatus with at least one data processing unit configured to correct input image data of a plurality of cameras of a surround view system impaired by rain, backlight and / or dirt in output image data, the apparatus comprising an input interface, a trained artificial neural network and a (first) output interface.

[0058] The input interface is configured to receive input image data captured by the multiple cameras and impaired by rain, backlighting and / or dirt. The trained artificial neural network is configured to convert the impaired input image data into unimpaired output image data and to output, for an image or for each image of the input image data, a safety measure c that depends on the degree of water coverage, backlighting and / or dirtiness and that is a measure of the safety of the network or characterizes the safety of the network at the calculated output and is a measure of the adequacy of the image modification by the network.

[0059] The (first) output interface is configured to output the transformed (modified) image data as well as the determined safety measure(s) c.

[0060] According to one embodiment, the input image data includes at least one series of captured input images as input image data, and the artificial neural network is trained using the at least one series of input image data and the sequence of output image data as image data.

[0061] The apparatus or data processing unit particularly preferably includes a microcontroller or processor, a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and the like, and associated software for performing the method steps.

[0062] According to one embodiment, the data processing unit is implemented in a hardware-based image pre-processing stage (Image Signal Processor, ISP).

[0063] In one embodiment, the trained artificial neural network for modifying images is a component of an on-board ADAS detection neural network, e.g., a joint input interface (input or feature representation layer) for performing semantic segmentation, lane detection, and / or object detection, and two separate output interfaces (output layers), where the first output interface is configured to output the transformed output image data and the second output interface is configured to output the ADAS detection (image recognition data).

[0064] The invention further relates to a computer program element which, when programmed with it, causes a data processing unit to implement a method for modifying images of input image data of a plurality of cameras of a surround view system in output image data.

[0065] The present invention further encompasses a computer readable storage medium having corresponding program elements stored therein.

[0066] The invention also relates to the use of the method for machine learning image correction of input image data of multiple cameras of a surround view system in output image data for training an artificial neural network of an apparatus comprising at least one data processing unit.

[0067] Thus, the invention can be implemented in digital electronic circuitry, computer hardware, firmware, or software.

[0068] The main advantages include: - Enabling recognition of objects when individual, multiple or all cameras of a surround view system are obstructed (e.g., cloudy). - Improve image quality in side satellite cameras and reverse driving cameras for display purposes when driving through water / mud and the image is impaired. - Generate image data streams for human and computer vision from neural networks for optimized correspondence search (feature matching).

[0069] In addition to use in vehicles, they have many other areas of application: - Assistant systems in buses, trains, aircraft and robotics systems, - All applications that perform feature-based detection, e.g. optical flow or structure-from-motion based detection methods, which lose features and are significantly degraded when affected by dirt / rain / backlighting. - Specifically, assistant systems using optical flow for feature search; - Online calibration of cameras that, when wet or dirty, lose features, reduce contrast and experience significant performance degradation.

[0070] The following examples and figures are explained in detail. Figure Description: [Brief description of the drawings]

[0071] [Figure 1] FIG. 1: A first schematic representation of a device according to the invention in one embodiment; [Diagram 2] FIG. 2: is a second schematic representation of an on-board device according to the present invention in one embodiment; [Diagram 3] Figure 3: A system for correcting impaired camera images; [Figure 4] Figure 4: The system with the first neural network for image retouching; [Diagram 5] Figure 5: A system that combines image correction and detection functions; and [Figure 6] Figure 6: shows a modified system where image corrections are calculated and output only within the training range. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0072] As is evident from Fig. 1, the device 1 according to the invention for performing image correction of input image data of a plurality of cameras of a surround view system can comprise a plurality of units or circuit components. In the embodiment shown in Fig. 1, the image correction device 1 comprises a plurality of on-board cameras 2-i each producing a camera image or video data. In the embodiment shown in Fig. 1, the image correction device 1 comprises four on-board cameras 2-i for producing a camera image. The number of on-board cameras 2-i can be different for different applications. The device 1 according to the invention comprises at least two on-board cameras for producing a camera image. The camera images of adjacent on-board cameras 2-i typically have overlapping image areas.

[0073] The device 1 includes a data processing unit 3 for combining the camera images produced by the on-board cameras 2-i into a composite full-view image. As shown in FIG. 1, the data processing unit 3 comprises a system 4 for performing image correction or image transformation. The image transformation system 4 generates unimpaired and corrected output images or output image data (Opti) from the input image data (Ini) of the on-board cameras (2-i), at least partially impaired by rain, backlight and / or dirt. The optimized output image data of the individual on-board cameras 2-i are combined (so-called stitching) into a composite full-view image. The full-view image combined from the image data (Opti) corrected by the image processing unit 3 is then displayed for the user by a display unit 5. In one possible embodiment, the image correction system 4 is constituted by hardware circuits for performing independent image correction. In an alternative embodiment, the system executes program instructions implementing a method for image correction.

[0074] The data processing unit 3, which may comprise one or more image processors, transforms the camera images or video data received from the different on-board cameras 2-i and subsequently combines (stitches) a composite full-view image. In one possible embodiment, the image transformation system 4 consists of a dedicated processor that performs image retouching in parallel with the other processor(s) of the data processing unit 3. Due to the parallel data processing, the time required to process the image data is reduced.

[0075] Fig. 2 shows a further schematic depiction of one embodiment of the device 1 according to the invention. The device 1 shown in Fig. 2 is employed in a surround view system of a vehicle 10, in particular a passenger car or a motor vehicle for goods. Here, each of the on-board cameras 2-1, 2-2, 2-3, 2-4 is arranged on a different side of the vehicle 10 and has a corresponding field of view (dashed lines), i.e., forward=V, rear=H, left=L, and right=R of the vehicle 10.

[0076] For example, a first on-board camera 2-1 is arranged in front of the vehicle 10, a second on-board camera 2-2 in the rear of the vehicle 10, a third on-board camera 2-3 in the left side of the vehicle 10 and a fourth on-board camera 2-4 in the right side of the vehicle 10. The camera images of two adjacent on-board cameras 2-i have overlapping areas VL, VR, HL, HR. The on-board cameras 2-i can be so-called fish-eye cameras with a viewing angle of at least 185° in one possible embodiment. The on-board cameras 2-i can transmit camera images, or camera image frames, or video data, in one possible embodiment via an Ethernet connection to a data processing unit 3. The data processing unit 3 calculates from the camera images of the on-board cameras 2-i a synthesized surround view image for showing to the driver and / or passengers on a display means 5 of the vehicle 10. For example, the visibility conditions of the rear vehicle-mounted camera 2-2 may be different from those of the other cameras 2-1, 2-3, and 2-4 because the lens of the rear vehicle-mounted camera 2-2 is covered with raindrops or is dirty. In many cases, the visibility conditions of the cameras are different from each other.

[0077] By simultaneously or jointly training an artificial neural network with impaired images (e.g., from the rear camera 2-2) and non-impaired images (e.g., from the front camera 2-1 and the side cameras 2-3 and 2-4), the neural network learns optimal parameters for this situation.

[0078] In the joint training for multiple vehicle-mounted cameras 2-i, ground truth data with image quality that is preferably not affected by rain, backlight or dirt is used for all target cameras 2-1, 2-2, 2-3, 2-4 in the first application. With the ground truth data as a reference and the input data of the target cameras 2-1, 2-2, 2-3, 2-4 that may have such impairments, the neural networks CNN1, CNN10, CNN11, CNN12 are trained in terms of the optimal parameter set for the network.

[0079] In a further application, in case training data and ground truth data for a camera, e.g. a side camera 2-3 or 2-4, is missing, a neural network for a common camera 2-i can be trained such that the network can train and optimize parameters for the camera 2-3 or 2-4 for which data is missing based on the training data of the other cameras 2-1, 2-2 and 2-4 or 2-3.

[0080] In the last example, the neural network uses training data that are different in time and correlate to each camera 2-i, and ground truth data captured or photographed at different times by the different cameras 2-i. This can be information on features or objects and their ground truth data, e.g., captured by the front camera 2-1 at time t and by the side cameras 2-3, 2-4 at time t+n. These features or objects and their ground truth data can fill in missing information in the training or ground truth data of the other cameras, if they are present in the images of the other cameras 2-i and are used by the network as training data. In this way, the network can optimize the parameters for all the side cameras 2-3, 2-4 and possibly compensate for missing information in the training data.

[0081] When multiple on-board cameras 2-i are used, the above method allows the individual impairments in the entire network to be explicitly identified and trained, resulting in image corrections that are individually tailored for all on-board cameras 2-i.

[0082] Fig. 3 shows a schematic general overview of a system for correcting camera images in rain, backlight and / or dirt. The essential element is an artificial neural network CNN1 which, during a training phase, learns to assign a set of training input images In (In1, In2, In3, ...) to corresponding corrected training (ideal) output images Out (Out1, Out2, Out3, ...). "Assignment" means that the neural network CNN1 learns to generate corrected images. A single input image (In1, In2, In3, ...) may contain, for example, a rainy road scene where the human eye can only recognize the sky and large road markings that are faint or blurred, representing large objects, e.g. bicycles. In the corresponding corrected image (Out1, Out2, Out3, ...), the outlines of intersections, traffic lights, trees and street lights can additionally be recognized. A realistic photographic image of a color image, with or without impairment by raindrops, is shown, for example, in Fig. 1 of Porav et al. The term "input image" in this case refers to the input images captured simultaneously by multiple or all single cameras 2-i, since there are multiple cameras 2-i. Similarly, the term "output image" can encompass the ideal or output images of multiple or all single cameras 2-i.

[0083] Optionally, a factor d can be used as an additional input for the neural network CNN1. The factor d is a control parameter for controlling the degree of correction for the effects of (rain, backlight, dirt) on the images. During training, the factor d for an image pair consisting of a training image and a corrected image (In1, Out1; In2, Out2; In3, Out3; ...) is determined beforehand or during training from the image pair (In1, Out1; In2, Out2; In3, Out3; ...) and provided to the neural network CNN1. This allows the factor d to be learned together.

[0084] When using a trained neural network CNN1, the reference value of the factor d can control how much the neural network CNN1 corrects the latest captured image - i.e., the factor d can also be considered as an external regression parameter (with any gradation). Since the factor d can have a variation in the range of + / -10%, this is taken into account during training. The factor d can be made noisy during training (at various epochs of training the neural network) of about + / -10%, so that the factor d is robust to misestimation of the factor d in the range of about + / -10% during inference on the vehicle. In other words, the required accuracy of the factor d is within the range of + / -10% - i.e., the neural network CNN1 is robust to deviations in the estimation of this parameter.

[0085] Alternatively or additionally, the factor d can be output for the image modification performed by the trained neural network CNN1, so that subsequent image recognition and / or image display functions have information about the extent to which the originally captured image has been modified.

[0086] The artificial neural network CNN1 is configured to determine a safety measure c for an input image depending on the degree of wetness, backlighting and / or dirt. The specification of the network is achieved, for example, by designing a suitable architecture of the artificial neural network CNN1. After the machine learning, the artificial neural network CNN1 is able to determine and output a safety measure c for a new input image, i.e. the safety measure c characterizes the safety of the image correction when using the trained network depending on the degree of impairment due to wetness, backlighting and / or dirt. Insofar as the term "input image" refers to input images captured simultaneously by a plurality or all of the single cameras 2-i, the term "safety measure c" can also refer to the determination of individual (possibly different) safety measures c for different simultaneously captured input images.

[0087] Three image pairs In1+Out1, In2+Out2, In3+Out3 are shown diagrammatically in Fig. 3. In short, a neural network CNN1 is trained or designed such that for each input image of an image pair, one safety measure c1, c2, or c3 can be determined and output.

[0088] Since the quality of the rain, backlight or dirt correction of an image depends on many factors (e.g. the presence of similar cases in the training data, sufficient exposure to perform the correction, avoidance of ambiguous scenarios, etc.), in addition to the correction of the image, the network also outputs a safety measure on which the network relies to make decisions. This safety measure c includes one of the following embodiments or a combination of them: - confidence c_Prob: here the network output is calibrated so that it can be interpreted probabilistically as the probability that the network makes a correct decision. Its value is normalized in the range [0,1] and corresponds to the probability that the network has calculated the correct correction of the image, a spectrum from 0% to 100%. This calibration can be performed after the completion of the actual machine learning method using the training image dataset, by checking the quality of said learning using a validation image dataset. The validation image dataset also contains image pairs, each consisting of a first image affected by rain, backlight and / or dirt, and a second image of the same scene without impairment as the corresponding ideal output image. In practice, it is also possible to keep some of the input images and the ideal-output images, i.e. not to use them for the machine learning method, but to use them for later validation. - a measure of dispersion similar to the standard deviation, c_Dev: Here, the uncertainty of the network output values ​​is estimated as a value that indicates the variance of the network output. This can be done in various ways, i.e. it can be split into measurement uncertainty and model uncertainty. Measurement uncertainty concerns uncertainties due to the input data, e.g. minor disturbances. These are added to the network via further outputs and trained by modifying the error function. Model uncertainty is due to the limited representation accuracy and generalizability of the network. It concerns factors such as the size of the training data and the architecture of the network design. Model uncertainty can be estimated by Monte Carlo dropout or network ensembles. Model uncertainty and measurement uncertainty can be combined additively. - Combining reliability and variability measures

[0089] The safety measure c can be calculated for the entire image, for image regions, or for individual pixels of the image.

[0090] Based on the safety measure c, we can make the following decisions: - c_Prob·Low: The network has low confidence in its estimates - frequent misestimations are possible. - c_Prob·High: The network has high confidence in its estimation - image corrections will be correct in most cases. - c_Dev·Low: The network has low variability in image corrections - i.e., the network predicts very accurate image corrections. - c_Dev·High: The estimated variability of the image correction, similar to the standard deviation, is high and the output of the network is less accurate / sharp - small changes in the input data or the network modeling will cause deviations in the image correction. - combination: ○ c_Prob·High and c_Dev·Low: Very reliable and accurate image correction that is considered to be of high reliability ○ c_Prob·Low and c_Dev·High: Image corrections that are very uncertain and inaccurate to the point that they may be rejected. ○ c_Prob·High and c_Dev·High, or c_Prob·Low and c_Dev·Low: These combinations have uncertainty and careful use of image correction is recommended here. The adoption of safety measures is particularly important for safety-critical functions.

[0091] One way to generate training data (training images (In1, In2, In3, ...) and the corresponding rectified images (Out1, Out2, Out3, ...)) is to capture image data using the "stereo camera setup" described in Figure 8 of Porav et al.: two identical camera modules with a small distance between them are placed in a chamber divided into two parts with transparent glass, and the chamber of the right stereo camera module is sprayed with, for example, raindrops while keeping the chamber of the left stereo camera module free from any impairments.

[0092] Similarly, to simulate backlight impairment, for example, the light source can be directed towards only one chamber, and dirt can also be applied to only one chamber.

[0093] Alternatively, to generate the training image pairs, the unimpaired images can be used and then degraded using rendering techniques that simulate the effects of rain, backlighting, or dirt in the images.

[0094] After the neural network CNN1 is trained, image retouching is performed according to the following scheme: Input image → CNN1 Option: Factor d → CNN1 CNN1 → corrected output image / output image + safety measure c.

[0095] Fig. 4 shows a system with a trained neural network CNN1 for image correction. The trained neural network CNN1 contains as input the original input image data (Ini) of several cameras 2-i. Optionally, a factor d, which determines (controls) how much the input image data is corrected, can be given in advance or determined by the neural network CNN1 on the basis of the input image data (Ini). The neural network calculates the corrected unimpaired image data (Opti) of the several cameras 2-i and one or several safety measures c. The corrected unimpaired image data (Opti) of the several cameras 2-i and at least one safety measure c are output.

[0096] 5 and 6 show an embodiment of a combination of a first network for image correction with one or more networks for driver assistance systems and / or (detection) functions for automated driving.

[0097] FIG. 5 shows a neural network CNN10 for image correction of an input image (Ini) controlled by a factor d, optionally sharing a feature representation layer (as an input layer or lower layer) with a network for detection functions (fn1, fn2, fn3, fn4). The detection functions (fn1, fn2, fn3, fn4) are image processing functions that detect objects, structures, features (general: features) in the image data that are important for ADAS and AD functions. Many such machine learning based detection functions (fn1, fn2, fn3, fn4) have already been developed or are the subject of recent developments (e.g. traffic sign classification, object classification, semantic segmentation, depth estimation, lane recognition, lane locating). For the corrected image (Opti), the detection functions (fn1, fn2, fn3, fn4) of the second neural network CNN2 provide improved results over the original impaired input image data (Ini). The feature representation layer of the neural network CNN10 learns common features for image correction and detection functions.

[0098] A neural network CNN10 with a common input layer and two independent outputs has a first output CNN11 for outputting a modified output image / output image data (Opti) and a second output CNN12 for outputting detections: objects, depth, lane, meaning, etc.

[0099] During training, the feature representation layer is optimized not only from the perspective of image retouching but also from the perspective of detection functions (fn1, fn2, fn3, fn4), so the optimization of image retouching simultaneously helps improve the detection functions (fn1, fn2, fn3, fn4).

[0100] If output of a modified image (Opti) is not desired or necessary, a further variation of the approach is possible, as described in FIG.

[0101] Figure 6 shows a neural network-based approach for image rectification based on the system of Figure 5 by feature optimization. To save computation time, features for detection functions (fn1, fn2, fn3, fn4) are optimized in terms of image rectification as well as detection functions (fn1, fn2, fn3, fn4) during training.

[0102] At run-time, i.e. when using pre-trained neural networks (CNN10, CNN11, CNN12), the modified image (Opti) is not calculated.

[0103] However, the detection functions (fn1, fn2, fn3, fn4) are improved by joint training of the image correction and the detection functions - as described above - compared to a system with only one neural network (CNN2) for the detection functions (fn1, fn2, fn3, fn4) in which only the detection functions (fn1, fn2, fn3, fn4) are optimized during training.

[0104] During the training phase, the modified image (Opti) is output from the additional output interface (CNN11) and compared with the ground truth (corresponding modified training image). During the test phase or at run time, this output (CNN11) can be used continuously or can be cut to save computation time. The weights for the detectors (fn1, fn2, fn3, fn4) are modified during training with this additional output (CNN11) to take into account the image modifications for the detectors (fn1, fn2, fn3, fn4). In this way, the weights for the detectors (fn1, fn2, fn3, fn4) learn information about the luminance improvement implicitly.

[0105] Further aspects and embodiments of the assistant system are presented below that use an algorithm to convert the image data of the underlying camera system into a representation equivalent to a lossless shot, regardless of the effects of rain, backlight, or dirt. The converted image can be used for pure display purposes or as input for feature-based recognition algorithms.

[0106] 1) In a first embodiment, the calculation is based on a system, for example a neural network, which is placed in front of the detection and / or display unit and which converts an input image, which is cloudy, dirty or has water droplets and has low contrast and color information, into a cleaned representation. For this task, the neural network is trained with a data set consisting of "cloudy input images" and the corresponding "cleaned images".

[0107] 2) In particular, by using the cleaned images, the neural network is trained to obtain and, in good cases, even enhance the features present in the image pair to be improved, despite the presence of cloudiness and smudges, for subsequent matching and object recognition.

[0108] 3) In training the network, image correction / enhancement takes into account feature-based methods for representation and object detection, and specializes the methods on the features to be recognized, thereby explicitly highlighting those features for subsequent processing.

[0109] 4) According to yet another embodiment, the method for image enhancement or correction can be embedded in a hardware-based image pre-processing stage, e.g., an ISP, which is supplemented on the hardware side with neural networks that perform the transformations and provide the pre-processed information together with the original data to the detection or display method.

[0110] 5) According to yet another embodiment, the system with the neural network can be trained to additionally use information from clear cameras, such as side cameras, to further improve the classification of cloudy areas, where the network is trained as a whole system made up of multiple cameras, rather than being trained individually with individual images for each camera.

[0111] 6) According to yet another embodiment, for training purposes the network is provided with information on image quality in addition to information on dirt and haze, so that the system and method are optimized for object recognition as well as for computing image data optimized for human vision.

[0112] 7) In a further application, the cameras are differently dirty. For example, a satellite camera on the side of the vehicle is more dirty than a satellite camera on the front of the vehicle. In such a case, an artificial neural network is designed to be trained and optimized so that the image information and image features of the clear satellite camera can be used for a clear display of the image of the cloudy camera. The image calculated in this way can be used for display purposes, but also for feature recognition. In the following example, the corrected image of the cloudy camera is used not only for feature recognition for optical flow, but also for SfM (Structure from Motion) and display purposes.

[0113] 8) In one application, the method is designed to simultaneously learn optimal parameters for all satellite cameras by joint training of an artificial neural network with images with different levels of dirt (e.g., cloudy side cameras) and clear images (e.g., front and rear view cameras).

[0114] In the joint training of multiple vehicle-mounted cameras, ground truth data with the image quality used for all target cameras is preferably used in the first application. In other words, the ground truth data for all target cameras is balanced so that no brightness differences are perceptible in the ground truth, for example in a surround view application. Using this ground truth data as a reference and the input data of the target cameras, which may have different cloudiness, the neural network is trained in terms of the optimal parameter set for the network. Similarly, data with differently illuminated side areas, for example when the vehicle is next to a street lamp or when there is an additional light source on the side, is also conceivable.

[0115] 9) In a further application, in case training data and ground truth data for a camera, e.g. a side camera, are missing, a network for a common camera can be trained so that the network can train and optimize the parameters for the missing camera based on the training data of the other cameras. This can be achieved, for example, by making assumptions in the training of the network, e.g. corrections and training, that the image quality of the satellite cameras is similar and therefore always the same.

[0116] 10) In the last example, the network uses training data that are temporally different and correlated to the cameras, and ground truth data taken by different cameras at different times. This can be done using feature information and their ground truth data, e.g., taken by the front camera at time t and by the side camera at time t+n. These features and their ground truth data can fill in missing information in the training data and ground truth data of other cameras, if they are present in the images of the other cameras and used by the network as training data. In this way, the network can optimize the parameters for all side cameras and possibly compensate for missing information in the training data.

[0117] When multiple on-board cameras 2-i are used, the above method allows the individual outside-cabin fogging profiles across the network to be explicitly determined and trained, resulting in tailored image quality for every on-board camera 2-i.

[0118] 11) In addition to correcting images when it is cloudy or there are water droplets, the system can also recognize water droplets or dirt to trigger, for example, a display to activate the wipers or to prompt the user to clean the satellite camera. In this way, by combining with brightness detection, it is possible to realize not only image correction but also rain-light detection functions. The present application also includes the following aspects as other aspects. 1. A machine learning method for image correction of input image data of a plurality of cameras (2-i) of a surround view system, comprising: The method comprises: Transforming input image data, at least partially impaired by rain, light interference (backlighting) and / or dirt, into corrected output image data using an artificial neural network (CNN1, CNN10, CNN11, CNN12); The learning is carried out using a number of image pairs (In1, Out1; In2, Out2; In3, Out3; ...) by providing the input of an artificial neural network (CNN1, CNN10) with first images (In1, In2, In3, ...) affected by rain, backlight and / or dirt, and second images (Out1, Out2, Out3, ...) of the same scene as ideal output images, The artificial neural networks (CNN1, CNN10, CNN11, CNN12) are designed to be able to determine a safety measure c that depends on the degree of water coverage, backlighting and / or dirtiness of the input image, and The artificial neural network (CNN1, CNN10, CNN11, CNN12) can determine and output a safety measure c for a new input image after the machine learning process is completed. A method comprising: 2. 2. The method according to claim 1, characterized in that the training image pairs (In1, Out1; In2, Out2; In3, Out3; ...) are created by capturing a first image (In1, In2, In3, ...) affected by rain, backlight and / or dirt and a second image (Out1, Out2, Out3, ...) that is not affected simultaneously or immediately after each other. 3. 3. The method according to claim 1 or 2, characterized in that one artificial neural network (CNN1; CNN10, CNN11, CNN12) is trained jointly for all cameras (2-i). 4. 4. A method according to any one of 1 to 3 above, characterized in that the training image pairs (In1, Out1; In2, Out2; In3, Out3; ...) are a series of input images and at least one sequence of output images. 5. A method for correcting input image data corrupted by rain, backlight and / or dirt from a plurality of cameras (2-i) of a surround view system, comprising: The method comprises the steps of: a) providing input image data (Ini) captured by a camera (2-i), which may be corrupted by rain, backlight and / or dirt, to a trained artificial neural network (CNN1, CNN10, CNN11, CNN12); b) the trained artificial neural network (CNN1, CNN10, CNN11, CNN12) is configured to convert input image data (Ini) affected by rain, backlighting and / or dirt into lossless output image data (Opti) and to characterize the safety of the network depending on the degree of water coverage, backlighting and / or dirtiness of the input image data and to determine a safety measure c corresponding to the image modification of the network, c) the trained artificial neural network (CNN1, CNN10, CNN11, CNN12) is configured to output the output image data (Opti) and the determined safety measure c; A method comprising: 6. 6. The method according to claim 5, characterized in that the input image data (Ini) comprises at least one sequence of a series of captured input images as the input image data (Ini). 7. 7. The method according to claim 5 or 6, wherein the camera (2-i) is a surrounding capture camera mounted on a vehicle. 8. 8. The method according to claim 7, characterized in that the output of the transformed image data (Opti) and the determined safety measure c is performed to at least one ADAS detection function for determining and outputting detections that are important for the ADAS based on the transformed image data. 9. 1. An apparatus comprising at least one data processing unit adapted to modify input image data (Ini) of a plurality of cameras (2-i) of a surround view system, the input image data (Ini) being impaired by rain, backlight and / or dirt, into modified output image data (Opti), The device comprises: - an input interface configured to receive input image data (Ini) of a camera (2-i) impaired by rain, backlight and / or dirt; - a trained artificial neural network (CNN1, CNN10, CNN11, CNN12) configured to transform input image data (Ini) into lossless output image data (Opti) and to characterize the safety of said network depending on the degree of water coverage, backlighting and / or dirtiness of the input image data and to determine a safety measure c corresponding to the image modification of said network; - an output interface (CNN11) configured to output the transformed output image data (Opti) and the determined safety measure c; An apparatus comprising: 10. 10. The apparatus according to claim 9, characterized in that the data processing unit is implemented within a hardware-based image pre-processing stage. 11. 11. The device according to claim 9 or 10, characterized in that the camera (2-i) is a surroundings camera mounted on a vehicle and the trained artificial neural network (CNN1, CNN10, CNN11) for image correction is a component of a vehicle-side neural network for ADAS detection (CNN2, CNN12) having a common input interface and two independent output interfaces, the first output interface (CNN11) being configured to output the corrected image data (Opti) and the second output interface (CNN12) being configured to output detections important for the ADAS. 12. 12. The apparatus according to any one of claims 9 to 11, characterized in that the input image data comprises at least one series of sequences of captured input images as input image data (Ini), and the artificial neural network (CNN1, CNN10, CNN11, CNN12) is trained using at least one series of sequences of input images and output images as image data. 13. A computer program element comprising: The computer program element When a data processing unit is programmed by The computer program element 9. A computer program element comprising instructions for carrying out a method according to any one of 5 to 8 above. 14. 14. A computer readable storage medium having stored thereon a program element according to claim 13. 15. Use of the method according to any one of claims 1 to 4 for training an artificial neural network (CNN1, CNN10, CNN11, CNN12) of an apparatus according to any one of claims 9 to 12.

Claims

1. A method for machine learning image correction of input image data of a plurality of cameras (2-i) of a surround view system, comprising: The method comprises: converting input image data, at least partially corrupted by rain, light interference (backlighting) and / or dirt, into modified output image data using artificial neural networks (CNN1, CNN10, CNN11, CNN12); The learning is carried out using a number of image pairs (In1, Out1; In2, Out2; In3, Out3; ...) by providing at the input of an artificial neural network (CNN1, CNN10) first images (In1, In2, In3, ...) affected by rain, backlight and / or dirt, and second images (Out1, Out2, Out3, ...) of the same scene as ideal output images, said artificial neural networks (CNN1, CNN10, CNN11, CNN12) are designed to be able to determine a safety measure c that depends on the degree of water coverage, backlighting and / or dirtiness of the input image; a security measure c characterizes the security of the image correction performed by the trained neural network; and The artificial neural networks (CNN1, CNN10, CNN11, CNN12) are able to determine and output a safety measure c for new input images after completing machine learning. A method comprising:

2. 2. The method according to claim 1, characterized in that the training image pairs (In1, Out1; In2, Out2; In3, Out3; ...) are created by capturing a first image (In1, In2, In3, ...) affected by rain, backlight and / or dirt and a second image (Out1, Out2, Out3, ...) that is not affected by rain, backlight and / or dirt simultaneously or in close succession.

3. 3. The method according to claim 1 or 2, characterized in that one artificial neural network (CNN1; CNN10, CNN11, CNN12) is trained jointly for all cameras (2-i).

4. 4. The method according to claim 1, wherein the training image pairs (In1, Out1; In2, Out2; In3, Out3; ...) are a series of input images and at least one sequence of output images.

5. A method for correcting input image data corrupted by rain, backlight and / or dirt from a plurality of cameras (2-i) of a surround view system, comprising: The method comprises the steps of: a) providing input image data (Ini) captured by a camera (2-i), corrupted by rain, backlight and / or dirt, to trained artificial neural networks (CNN1, CNN10, CNN11, CNN12); b) the trained artificial neural network (CNN1, CNN10, CNN11, CNN12) is adapted to transform input image data (Ini) affected by rain, backlighting and / or dirt into lossless output image data (Opti) and to characterize the safety of the network depending on the degree of water coverage, backlighting and / or dirtiness of the input image data and to determine a safety measure c corresponding to the image modification of the network, c) the trained artificial neural networks (CNN1, CNN10, CNN11, CNN12) are configured to output the output image data (Opti) and the determined safety measure c; A method comprising:

6. 6. A method according to claim 5, characterized in that the input image data (Ini) comprises at least one sequence of a series of captured input images as input image data (Ini).

7. Method according to claim 5 or 6, characterized in that the camera (2-i) is a vehicle-mounted surroundings camera.

8. 8. The method according to claim 7, characterized in that the output of the transformed image data (Opti) and the determined safety measure c is implemented to at least one ADAS detection function for determining and outputting detections of interest to the ADAS based on the transformed image data.

9. An apparatus comprising at least one data processing unit configured to modify input image data (Ini) of a plurality of cameras (2-i) of a surround view system, the input image data (Ini) being impaired by rain, backlight and / or dirt, into modified output image data (Opti), The device comprises: an input interface adapted to receive input image data (Ini) of the cameras (2-i) impaired by rain, backlight and / or dirt; a trained artificial neural network (CNN1, CNN10, CNN11, CNN12) configured to transform input image data (Ini) into lossless output image data (Opti) and to characterize the safety of the network depending on the degree of water coverage, backlighting and / or dirtiness of the images of the input image data and to determine a safety measure c corresponding to the image modification of the network; an output interface (CNN11) configured to output the transformed output image data (Opti) and the determined safety measure c; An apparatus comprising:

10. 10. The apparatus of claim 9, wherein the data processing unit is implemented in a hardware-based image pre-processing stage.

11. 11. Apparatus according to claim 9 or 10, characterized in that the cameras (2-i) are vehicle-mounted peripheral cameras and the trained artificial neural networks (CNN1, CNN10, CNN11) for image correction are components of a vehicle-side neural network for ADAS detection (CNN2, CNN12) with a common input interface and two independent output interfaces, the first output interface (CNN11) being arranged to output corrected image data (Opti) and the second output interface (CNN12) being arranged to output detections of interest to the ADAS.

12. 12. Apparatus according to any one of claims 9 to 11, characterized in that the input image data comprises at least one series of sequences of captured input images as input image data (Ini), and the artificial neural networks (CNN1, CNN10, CNN11, CNN12) are trained using at least one series of sequences of input images and output images as image data.

13. A computer program element comprising: A computer program element which, when a data processing unit is programmed with the computer program element, commands the data processing unit to carry out a method according to any one of claims 5 to 8.

14. 14. A computer readable storage medium having stored thereon a program element according to claim 13.

15. Use of the method according to any one of claims 1 to 4 for training an artificial neural network (CNN1, CNN10, CNN11, CNN12) of an apparatus according to any one of claims 9 to 12.

Images