Method for classifying blind spots of optical images

Abstract

The invention relates to a method for classifying blind areas of optical images. A method for classifying blind areas of an optical image of an environment of a mobile platform is suggested, the method having the following steps: providing a digital optical image of an environment of a mobile platform; forming a plurality of regions for dividing the digital image; classifying each region of the plurality of regions of the digital image by means of a trained machine learning system using at least two different blind area attributes and using the digital image as an input signal for the machine learning system, wherein the machine learning system has been trained by means of a plurality of digital reference images and classified regions of the digital reference images using the machine learning system have been classified with respect to the at least two blind area attributes; and assigning the at least two blind area attributes to the respective regions of the digital image.

Description

Technical Field

[0001] The present invention relates to a method for classifying blind spots in an optical image, a method for improving an optical image, a method for determining at least one image plane of an optical image, and a method for generating a trained machine learning system, and relates to a corresponding device, computer program, and machine-readable storage medium. Background Technology

[0002] The automation of driving has emerged with the equipping of vehicles with increasingly extensive and powerful sensor systems for detecting their surroundings. Machine learning methods are employed to interpret sensor data for classification and detection tasks. In the video domain, convolutional neural networks are used, for example. Video sensors can degrade due to extremely different disturbances along the shooting path, such as dirt on the camera lens or the windshield, camera damage, or other environmental influences such as rain, fog, snow, or splashes.

[0003] In order to detect an environmental representation of at least partially automated driving using optical sensors, sufficiently clear images from the optical sensors are particularly important. Summary of the Invention

[0004] Typical schemes for object detection and classification fail to recognize their degraded state in these cases due to the ineffectiveness of false positive and false negative identification. Errors in the detection of the surrounding environment by video sensors can lead to functional misjudgments in higher-level systems. Depending on the functions implemented, this can result in safety risks or loss of comfort. Therefore, video sensors in vehicles can be equipped with separate blind spot detection. Through blind spot detection mechanisms, the functionality implemented on sensor data throughout the system may degrade in a manner appropriate for the current sensor availability.

[0005] In other words, data is generated by the sensor system to enable system functionality. If the availability of the sensor system is known, for example, through reliable blind zone detection, then the availability of system functionality can be derived from this. That is, the system can diagnose or identify appropriate system functionality degradation simply by well identifying sensor degradation. This means that, for example, in environments where only partial detection is possible, due to the system's local blind zone, the corresponding functionality can only be provided in a correspondingly limited manner.

[0006] Current blind spot detection has significant weaknesses, particularly in recognizing complex degradation effects such as lightsaber effects or distortion. The false positive rate of blind spot detection is also too high, making its adoption for levels of automation where the driver is no longer responsible and may take over vehicle guidance in cases of suspicion.

[0007] Similar to object recognition, a central approach for identifying blind spots in video images is to classify images or image segments about the blind spots. The classifier is typically trained in a supervised manner. The key to this is an appropriate amount, variety, and quality of ground truth reference data.

[0008] The prerequisite for generating reference data is a suitable label schema, according to which the data can be appropriately characterized as blind or non-blinded. Unlike objects such as cars, pedestrians, or road surfaces, it is meaningful to determine where the boundaries of this "blind zone" label should be for a wide variety of blind zone causes, blind zone effects, and blind zone densities.

[0009] For example, many causes in an image can no longer be distinguished, such as dust or clumps; some causes cannot be confused, such as water and leaves. Some causes may lead to complete local degradation, such as bird droppings; while others may be translucent and partially reveal the background, such as thin dust. Some effects are difficult to define boundaries, such as streaks or scattering effects; and other effects scatter in small parts of the image, such as sand; while other effects appear planar, such as clumps. These effects can be distinguished greatly depending on the focal length, angle of attack, and distance from the contaminated surface (e.g., windshield or lens). At the same time, the purpose of classifying images according to blind spot labels, for example, always depends on the functional aspect.

[0010] If, for example, the only requirement is to identify blind spots along a degradation path, then cause differentiation is unnecessary. However, for potential cleaning, as accurate a distinction as possible regarding the cause is helpful in selecting appropriate cleaning parameters / procedures. Other examples involve traffic light identification, such as using only a single sensor where the entire image is available, as the traffic light can appear across the entire imaging area, thus causing complete sensor / system failure due to partial occlusion of the imaging area. In other cases, such as the "Tele-Kamera-Nach-Hinten" scenario mentioned above, data from partially occluded sensors can certainly be used, thus imposing different requirements on the spatial resolution of blind spot identification depending on this functional aspect.

[0011] According to aspects of the present invention, a method for classifying blind spots of an optical image, a method for improving an optical image, a method for determining at least one image plane of an optical image, a method for generating a trained machine learning system to classify blind spots of an optical image, a method for providing control signals, an apparatus, a computer program product, and a machine-readable storage medium are proposed.

[0012] According to one aspect of the invention, a method for classifying blind zones of an optical image of an environment of a mobile platform is proposed, the method comprising the following steps: In a first step, a digital optical image of the environment of the mobile platform is provided. In a subsequent step, a plurality of regions are formed for dividing the digital image. In another subsequent step, each region of the plurality of regions of the digital image is classified using at least two different blind zone attributes by means of a trained machine learning system, and the classification is performed using the digital image as input signal to the machine learning system, wherein the machine learning system has been trained with reference to a plurality of digital reference images and the classified regions of the corresponding digital reference images to classify the regions of the digital reference images with respect to at least two blind zone attributes, and at least two blind zone attributes are assigned to the corresponding regions of the digital image.

[0013] Here, an example of a machine learning system is a neural convolutional network, which may be combined with a fully connected neural network when necessary. Classical schemes such as support vector machines, boosting, decision trees, and random forests may also be used as machine learning systems for the described method, depending on the application of classical regularization and stabilization layers, such as batch normalization and training drop-outs, and different activation functions such as sigmoid and ReLU.

[0014] Throughout the description of the invention, the method steps are shown in sequence to facilitate understanding of the method. However, those skilled in the art will recognize that many of these method steps can be traversed in a different order and result in the same or corresponding results. In this sense, the order of these method steps can be changed accordingly. Several features are equipped with counters to improve readability or make the assignment more explicit, but this does not imply the existence of definite features.

[0015] The reference image is one that has been taken specifically for the purpose of teaching a machine learning system, and has been manually selected and labeled, or synthesized, for example, in which multiple regions are labeled according to their classification. This labeling of regions can be performed manually based on pre-given classifications (such as at least two blind zone attributes).

[0016] In neural networks, the signals supplied to the connections of artificial neurons can be real numbers, and the output of an artificial neuron is calculated as a nonlinear function of the sum of its inputs. Connections in artificial neurons typically have weights that are adapted using progressive learning. These weights increase or decrease the strength of the signals supplied to the connections. Artificial neurons may have thresholds such that they only output a signal when the total signal exceeds that threshold.

[0017] Typically, multiple artificial neurons are combined into multiple layers. Different layers may perform different types of transformations on the inputs of these layers. The signal travels from the first layer (input layer) to the last layer (output layer); it may travel to the last layer after traversing these layers multiple times.

[0018] The architecture of this artificial neural network can be as follows: the neural network is extended with other differently constructed layers as needed, based on a multi-layer perceptron (MLP). Multi-layer perceptron (MLP) networks belong to the family of artificial feedforward neural networks. In principle, an MLP includes at least three neuron layers: an input layer, a hidden layer, and an output layer. This means that all neurons in the network are divided into multiple layers.

[0019] In this feedforward network, there are no connections to previous layers. Besides the input layer, different layers include neurons that can undergo non-linear activation functions and can connect to neurons in the next layer. Deep neural networks can have multiple such intermediate layers.

[0020] Such neural networks must be trained for specific tasks. Here, each neuron in the corresponding architecture of the neural network receives, for example, random initial weights. Then, input data is fed into the network, and each neuron can weight the input signal with its own weights, further contributing to the neurons in the next layer. The final result is then provided at the output layer. The magnitude of the error can be calculated, as well as the share of the error contributed by each neuron, so that the weights of each neuron are then changed in the direction that minimizes the error. Then, a recursive traversal is performed, remeasuring the error and adapting the weights until the error criterion is satisfied.

[0021] Such an error criterion could be, for example, the classification error on the test dataset, or it could be the current value of the loss function, such as the current value of the loss function on the training dataset. Alternatively or additionally, the error criterion could involve an interruption criterion as a step, in which overfitting might have been employed during training, or the available time for training might have expired.

[0022] Optical images are provided digitally as input signals to a trained neural network.

[0023] A mobile platform can be understood as a system that is at least partially automated, is mobile, and / or is understood as a driver assistance system. Examples can be at least partially automated vehicles or vehicles with driver assistance systems. That is, in this context, a system that is at least partially automated includes a mobile platform with at least partially automated functionality, but a mobile platform also includes vehicles including driver assistance systems and other mobile machines. Other examples of mobile platforms can be: driver assistance systems with multiple sensors, such as mobile multi-sensor robots like robotic vacuum cleaners or lawnmowers, multi-sensor monitoring systems, production machines, personal assistance devices, or access control systems. Each of these systems can be fully or partially automated.

[0024] Multiple regions can include a small number of regions, such as two or three regions, or they can include 100 regions or a much larger number of regions that can be classified.

[0025] By employing a method or concept of labeling regions of an optical image with two blind zone attributes, the shortcomings of current blind zone identification methods for video degradation / blind zones, as described above, can be at least partially overcome. The concept of labeling with at least two blind zone attributes allows for the annotation of all possible blind zone information on a single image and can be applied to all blind zone situations. Therefore, data records labeled in this way can be used for many image-based blind zone classification tasks.

[0026] Using methods including labeling concepts, it is possible to detect and classify the current state of sensor degradation using supervised machine learning schemes. This scheme can be implemented either using a standalone classifier (e.g., in the form of a convolutional network) or by integrating it into or combining it with another classifier (e.g., for objects).

[0027] This can achieve the following: a. Locating dirt, water, etc., in optical images. This contributes to greater system availability because, despite some limitations, at least some functions can be performed using the remaining images.

[0028] b. When the availability of the system that generates the digital image falls below a certain level, the system can be reliably, promptly, and in a controlled manner shut down. This is necessary to ensure security.

[0029] c. The system's retention functions that generate the digital image can be automatically triggered, such as windshield wipers, sensor heating devices, or sensor cleaning processes, to maintain system availability.

[0030] 2. The described labeling concept with at least two blind zone attributes enables labeling for multiple causes of degradation without discarding essential identifiable information. This creates a standard for labeling digital images with degraded portions for various applications. Digital images labeled in this way can thus be reused cost-effectively.

[0031] 3. In addition, it is important to understand the labeling concept simply, and in most cases, the labeling concept creates a well-defined labeling task.

[0032] Using the labeling concept included in this method, training and validation data records for camera-based degradation recognition can be generated based on machine learning. Degradation states encompass all damage the system may experience during its lifespan. Examples include: rain, snow, scratches, coatings, dust, fog, glare from low-lying sun, foliage, and other contaminants.

[0033] In particular, the labeling concept described herein can generate suitable labels that can be applied to all blind spots without discarding important identifiable information. The labeling concept is "clearly defined" to a specific and objective pre-given degree that explains the purpose of the label to the labeling machine.

[0034] High-performance blind-spot recognition can be achieved using these tagged images. High levels of security can be ensured with high sensor and system availability through accurate identification of the aforementioned effects.

[0035] One suggestion is that the surface density of the blind zone of an optical image is assigned to a first blind zone attribute, while the opacity of the blind zone of an optical image is assigned to a second blind zone attribute.

[0036] The labeling concept may specifically involve the properties described below. For such digitally described images, the terms "blind spot" or "degradation" can be understood as an area where the image is damaged and the scene's perception is reduced due to residues (such as dirt or water droplets) on the windshield or surface glass. Alternatively or additionally, scattering and absorbing substances in the atmosphere may be the cause of partial or complete blind spots in the digitally described image.

[0037] Blind spot / degradation labels can be understood as overlays, meaning that blind spot / degradation labels can be contained in a separate layer on top of any other semantic label. For example, such blind spots can be labeled overlays with pedestrian labels.

[0038] Here, blind spots / degradation can form due to various causes, such as dirt, water droplets, snow, ice, condensation, or even defects like cracks or scratches. Since the cause is often unreliable, it cannot be distinguished during the labeling process. Instead, the effect is categorized using multiple attributes. For this purpose, clearly identifiable causes can be additionally assigned attributes. Depending on the implementation, one or more of the attributes described below may be omitted.

[0039] To locate blind spots in a camera image, polygonal labeling can be used for the three attributes described above; that is, polylines are used to define image segments to which the attributes are assigned. Depending on the needs and possibilities, different numbers of line segments can be used. In particular, there are applications where the image region to which the attributes are assigned is bounded by a small number of line segments (e.g., 3-10), and other applications where pixel-precise boundary delineation is required. In other applications, additional edge parameterization can be used for the labeled region, such as using curved line segments.

[0040] The first blind zone attribute, "Density," describes how densely the marked area is spatially covered by the blind zone effect. This does not relate to the physical volume density of the material, but rather to the pixel or surface density of the blind zone effect—that is, how many pixels within a polygon are actually involved by the blind zone effect. In the following text, the associated blind pixel region will be referred to as a "spot."

[0041] 1. Low: The polygon contains small, segmented blind spots (small: starting from occlusion of only one pixel). The number of events is high enough that, for example, from an economic or algorithmic perspective, delineating individual boundaries for these spots may be meaningless.

[0042] 2. High: The polygon contains multiple blind spots that cover most of the pixels but have gaps between them.

[0043] 3. Full: In practice, this refers to all pixels within the polygon, meaning the polygon essentially corresponds to a single dot.

[0044] The second blind spot attribute, "obscurity," describes how strongly the blind spot effect reduces the perception of the actual scene behind the spots in the labeled area. The main metric is how much structure in the underlying scene can still be perceived.

[0045] 1. Low: The scene's structure or texture is still clearly visible despite blind spots. Furthermore, individual edges or lines can be identified.

[0046] 2. Medium: Identifying the structure or texture of the scene is difficult. It is possible to correctly identify large shapes and possible scene colors, but details are lost.

[0047] 3. High: It is impossible to identify the structure or texture of the scene in the blind spot.

[0048] According to one suggestion, the identifiable cause of the blind spot is assigned to the third blind spot attribute.

[0049] One suggestion was made that assigning a third blind zone attribute to a transparent surface covered with a substance different from water before the camera system generates a digital image could be used to make the surface recognizable.

[0050] The third blind spot attribute, Water Indication, describes whether a blind spot could potentially have been created by water. Possible values ​​are: 1. True: Blind spots may have been created by water.

[0051] 2. False: Clearly identified that the blind spot was not created by water.

[0052] Here, characterizing or classifying whether a blind zone has been created by water is functionally relevant information in many cases, such as when used in conjunction with a sensor cleaning system: a sensor surface covered by water is preferably cleaned by a dry cleaning process rather than a wet cleaning process. However, labeling machines cannot reliably distinguish whether a blind zone has been created by water. The purpose of the "water indication" label is to still describe the maximum amount of information that can be extracted from the image.

[0053] Since the strong defocus often shows coverage of transparent surfaces, it is advantageous to exclude the third blind zone attribute "water" as a reason when labeling, as this is simpler in practice than identifying water.

[0054] Alternatively, the "water indicator" can be replaced and / or supplemented by a liquid indicator or a volatile indicator.

[0055] One suggestion is that multiple regions are formed by the entire number of pixels in a digital image, meaning that each pixel is assigned at least one region, or each pixel is given attributes.

[0056] One suggestion is to form polygonal curves for determining the image planes that are classified differently in the corresponding digital image, in such a way that the polygonal curves extend between the regions of the digital image that are classified differently (with respect to blind spots).

[0057] Using this polygonal curve, regions that are classified differently can be clearly separated from each other regarding blind zone attributes.

[0058] A method for improving the optical image of a mobile platform's environment is proposed, wherein an optical image has been generated using a digital camera that is mechanically isolated from the environment by a transparent surface, and a cleaning process of the transparent surface is triggered based on a number of regions in the optical image, which have been classified using key blind zone attributes according to embodiments of the invention. Here, the transparent surface may also specifically refer to the lens of the camera system.

[0059] To determine key blind zone attributes, thresholds can be selected based on the application. In this context, the task is to trigger sensor cleaning only when (e.g., regarding effective distance, object recognition rate, or angular accuracy) a significant performance benefit for the sensor is expected and all boundary conditions for sensor cleaning (e.g., hardware availability, safety enable, velocity window, etc.) are met.

[0060] A method for determining at least one image plane of an optical image is proposed, the method being suitable for object detection, wherein the image plane is determined based on a number of regions of the optical image that have been classified using relevant blind zone attributes according to embodiments of the invention.

[0061] To determine the relevant blind spot attributes, the following two approaches can be considered: On the one hand, distance and object relevance can be considered here, where, for example, slight haze in an image can still identify large objects nearby, but cannot identify smaller objects further away.

[0062] On the other hand, ideally, when a perception algorithm can no longer reliably identify the object for a given requirement due to interference, an image region should be classified as blind for a particular detection. Qualitatively, this currently roughly corresponds to whether a person can still reliably identify the object. For example, this could mean that the object has been modified due to softening effects, distortion effects, and contrast loss effects, such that the essential identifying features of the object are no longer sufficiently represented in the digital image of the camera or video system.

[0063] A method for generating a trained machine learning system is proposed for classifying blind spots in optical images of a mobile platform environment using one of the methods described above. The method has multiple training epochs, each of which includes the following steps.

[0064] In one step, an optical image is provided having different blind zones in multiple regions of an optical image within the environment of a mobile platform. In other steps, an assigned reference image is provided for the optical image, wherein the multiple regions are labeled in the reference image according to at least two blind zone attributes. In other steps, the optical image is provided as an input signal to a machine learning system. In still other steps, the machine learning system is adapted to minimize deviations from the corresponding assigned reference image when classifying the blind zones of the regions of the optical image.

[0065] Using this method, a trained machine learning system can be generated, which can be employed in the method described above.

[0066] A method is proposed in which control signals for operating at least partially automated vehicles are provided based on a number of relevant and / or critical regions of an optical image, wherein the number of regions has been classified using blind spot attributes according to one of the methods described above; and / or alarm signals for warning vehicle passengers are provided based on a number of relevant and / or critical regions of an optical image, wherein the number of regions has been classified using blind spot attributes according to one of the methods described above.

[0067] The term "based on" should be understood broadly regarding the following characteristics: the control signal is provided based on a number of regions of the optical image, which have been classified using a blind zone attribute according to one of the methods described above. This term is to be understood such that each determination or calculation of the control signal is considered based on a number of regions of the optical image, where multiple regions have been classified using a blind zone attribute according to one of the methods described above; it is not excluded that additional input variables are also considered in determining the control signal. This correspondingly applies to the provision of alarm signals.

[0068] Using this method, sensor degradation can be addressed differently depending on the existing degree of degradation in the digital image. Thus, using this method, for example, when a retention or cleaning function is present, cleaning with varying intensities can be activated, such as using nozzles and windshield wipers targeting the sensor surface. Furthermore, highly automated systems may begin to transition to a safer state. For example, this could lead to a slow stop in an emergency lane in at least partially automated vehicles.

[0069] According to one aspect, an apparatus is described that has a machine learning system trained according to one of the methods described above. Using this apparatus, the corresponding methods can be easily integrated into different systems.

[0070] According to one aspect, a computer program is described, the computer program comprising instructions that, when executed by a computer, cause the computer to perform one of the methods described above. This computer program enables the described methods to be used in different systems.

[0071] A machine-readable storage medium is described, on which the computer program described above is stored. With the aid of this machine-readable storage medium, the computer program described above is portable. Attached Figure Description

[0072] Reference Figures 1 to 2 Embodiments of the invention are shown and described in more detail below. In the accompanying drawings: Figure 1 A graph is shown for classifying blind zones in optical images using two blind zone attributes; and Figure 2 Example images with different regions are shown, each categorized using two blind spot attributes. Detailed Implementation

[0073] Figure 1 The illustration schematically illustrates a combination of two blind zone attributes used to classify regions in corresponding images. Here, in each image, a region is characterized by having a first blind zone attribute and a second blind zone attribute, both schematically represented in different ways. The first blind zone attribute is the areal density of the blind zone, and the second blind zone attribute is the opacity of the blind zone. Here, DL, DH, and DF represent low, high, and full areal densities of the blind zone, respectively, while L, M, and H represent low, medium, and high opacities of the blind zone, respectively.

[0074] Figure 2 Example images with different regions are schematically shown, each categorized using two blind zone attributes. Here, region 210 is categorized, for example, due to raindrops, by low opacity and high areal density of the blind zone, and region 22 is categorized, for example, due to pollution, by high opacity and high areal density of the blind zone.

Claims

1. A method for classifying blind spots in optical images of an environment of a mobile platform, comprising the steps of: Provides digital optical images of the environment of the mobile platform; Multiple regions are formed to divide the digital image; Using a trained machine learning system with at least two distinct blind zone attributes (DL, DH, DF; L, M, H), and using the digital image as input signal, the machine learning system classifies each of the plurality of regions of the digital reference image, wherein the machine learning system has been trained with the aid of multiple digital reference images and the corresponding classified regions of the digital reference images to classify the regions of the digital reference images with respect to at least two blind zone attributes (DL, DH, DF; L, M, H); and Assign the at least two blind zone attributes (DL, DH, DF; L, M, H) to the corresponding regions of the digital image. The areal density of the blind zone of the optical image is assigned to a first blind zone attribute (DL, DH, DF), while the transparency of the blind zone of the optical image is assigned to a second blind zone attribute (L, M, H). The formation of polylines and / or curved line segments to define differently classified image planes in the corresponding digital image is achieved by the polylines and / or curved line segments extending between regions of the digital image that are differently classified with respect to the blind spot. The surface density of the blind zone describes how many pixels within the defined image plane are actually involved in the blind zone effect, and The transparency of the blind zone describes how strongly the blind zone effect reduces the perception of the actual scene behind the blind pixel area.

2. The method according to claim 1, wherein, The identifiable cause of the blind spot is assigned to the third blind spot attribute.

3. The method according to claim 2, wherein, The third blind zone attribute is assigned to the camera system before the camera system generates the digital image by covering a transparent surface with a substance different from water.

4. The method according to any one of claims 1-3, wherein, The plurality of regions are formed by the entire number of pixels of the digital image.

5. The method according to any one of claims 1-3, wherein, Polygonal curves are formed to determine image planes that are classified differently in a corresponding digital image, in such a way that the polygonal curves extend between regions of the digital image that are classified differently with respect to the blind zone.

6. The method according to any one of claims 1 to 3, wherein, A number of regions of an optical image are classified using relevant and / or key blind zone attributes, and control signals for operating at least partially automated vehicles are provided based on the number of regions of the optical image. And / or utilize blind zone attributes to classify a number of relevant and / or critical regions of an optical image, and provide an alarm signal for warning vehicle passengers based on that number of relevant and / or critical regions of the optical image.

7. A method for improving an optical image of an environment of a mobile platform, wherein the optical image has been generated using a digital camera mechanically separated from the environment by a transparent surface, and a cleaning process of the transparent surface is triggered based on a number of regions of the optical image: the number of regions has been classified using key blind zone attributes according to any one of claims 1 to 6.

8. A method for determining at least one image plane of an optical image, the method being adapted for object detection, wherein the image plane is determined based on a number of regions of the optical image: the number of regions have been classified using relevant blind zone attributes according to any one of claims 1 to 6.

9. A method for generating a trained machine learning system for classifying blind spots in an optical image of an environment of a mobile platform using the method according to any one of claims 1 to 6, the method having a plurality of training epochs, wherein each training epoch includes the steps of: An optical image is provided, the optical image having different blind zones in multiple regions (210, 220) of the optical image in the environment of a mobile platform; For the optical image, an assigned reference image is provided, wherein the plurality of regions in the reference image are labeled according to at least two blind zone attributes (DL, DH, DF; L, M, H), wherein the areal density of the blind zone of the optical image is assigned to the first blind zone attribute (DL, DH, DF), and the transparency of the blind zone of the optical image is assigned to the second blind zone attribute (L, M, H). The optical image is provided as the input signal for the machine learning system; and The machine learning system is adapted to minimize the deviation from the corresponding assigned reference image when classifying blind spots in regions of the optical image.

10. A computer program product comprising a computer program, the computer program including instructions that, when implemented by a computer, cause the computer to perform the method according to any one of claims 1 to 9.

11. A machine-readable storage medium on which a computer program is stored, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 9.

Images