Method for determining cleaning information, method for training a neural network algorithm, control device, camera sensor system, vehicle, computer program, and recording medium

Abstract

To provide a method and the like for determining cleaning information of a camera sensor that is at least partially obstructed, including obstruction on a transparent camera sensor component in an optical path of the camera sensor.SOLUTION: A method includes the steps of: controlling a camera sensor to capture at least one camera image; processing, by at least one computing device, by using a neural network algorithm; determining cleaning information according to a degree of a camera sensor blockage and a class of the camera sensor blockage; and transmitting the cleaning information to a cleaning device associated with the camera sensor, in order to clean the camera sensor in accordance with the cleaning information. The camera sensor is used, for example, as a camera sensor of a vehicle, in particular as a part of a driving assistance system, in order to capture at least a part of vehicle surroundings.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a method for determining cleaning information of a camera sensor that is at least partially obstructed, including obstructions on a transparent camera sensor component in the optical path of the camera sensor. The present invention also relates to a method for training a neural network algorithm, a control device, a camera sensor system, a vehicle, a computer program, and a recording medium.

Background Art

[0002] In recent vehicles, camera sensors are used to capture the surrounding environment of the vehicle, especially to supply the functions of advanced driver assistance systems to the driver of the vehicle. Therefore, the camera sensor is directed towards the surrounding environment of the vehicle, and thus at least a part of the camera sensor needs to be located in the external environment of the vehicle. Usually, this is a transparent part, for example, a camera lens or a transparent camera cover. Depending on the weather conditions of the surrounding environment of the vehicle and / or depending on the driving conditions of the vehicle, these parts of the camera sensor may be obstructed. In the obstructed state, the camera sensor is at least partially blocked, for example, by dirt or by precipitation such as rain, snow, etc.

[0003] Due to such obstructions of the camera sensor, its ability to detect feature points in the surrounding environment of the vehicle is reduced or hindered. Therefore, it is known to detect the state in which the camera sensor is obstructed and start cleaning the camera sensor to remove the obstruction.

[0004] In Patent Document 1, a processor is described that is configured to detect occlusions on a surface in the optical path of a vehicle sensor based on segmentation of sensor image data. A cleaning plan for this surface is selected according to the detected occlusion and according to map data and vehicle route data.

[0005] Patent Document 2 discloses a computer-implemented method for detecting a field-of-view obstruction that covers an image capture sensor. This method compares the captured scene of a first data set with a reference topology of the scene described in a second data set. An obstruction is detected if the difference between the classified scene elements of the first data set and the classified scene elements of the second data set exceeds a threshold value.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0007] When each of the presumed obstructions of the camera sensor is detected, there is a possibility that the camera sensor will be cleaned. Therefore, in order to avoid any unnecessary cleaning by the cleaning device assigned to the camera sensor, the detection of obstructions should be as accurate as possible.

[0008] Accordingly, the present invention is based on the objective problem of providing an improved method for supplying cleaning information, whereby, in particular, the detection of obstructions of the camera sensor and the cleaning of the obstructed camera sensor by the cleaning device assigned to the camera sensor are improved.

Means for Solving the Problems

[0009] According to the present invention, the above objective problem is solved by the method described at the beginning, which method comprises - controlling the camera sensor to capture at least one camera image, and - A step of processing at least one camera image by a computing device using a neural network algorithm, wherein the neural network algorithm determines, as an output, a camera sensor interference degree based on a segmentation of at least a part of the camera image or each camera image, and an interference class from a plurality of interference classes of camera sensor interference based on a classification of at least a part of the camera image. - A step of determining cleaning information according to the camera sensor interference degree and the class of camera sensor interference, wherein when a cleaning criterion is assigned to the determined class of camera sensor interference and at least one degree threshold is exceeded by the determined camera sensor interference degree, the cleaning information indicates that cleaning of the camera sensor is necessary. - A step of transmitting the cleaning information to a cleaning device associated with the camera sensor to clean the camera sensor according to the cleaning information.

[0010] The camera sensor may be, for example, a camera sensor of a vehicle, particularly a camera sensor used as part of a driving assistance system to capture at least a part of the vehicle's surrounding environment. The transparent camera sensor component may be, in particular, at least partially transparent to light in the visible spectrum and be a component provided in the optical path of the camera, such as a lens, a protective lens cover, an optical filtering element, etc. The transparent camera sensor component may be provided on the outer surface of the vehicle, such as the front or rear bumper, door mirror or any other part of the vehicle body.

[0011] In the first step of the method for determining cleaning information, the camera sensor is controlled to capture at least one camera image. When the transparent camera sensor component is obstructed, the obstruction is visible in the image because at least a part of the transparent camera component is covered by the obstruction, resulting in a part of the camera sensor's field of view being blocked by the obstruction.

[0012] At that time, at least one camera image is processed by a computing device. The camera image is input data for a neural network algorithm that processes the captured camera image. The neural network algorithm is implemented in the computing device. The neural network algorithm is configured to determine, for each processed camera image, a camera sensor interference level and an interference class from among a plurality of interference classes as outputs.

[0013] Hereinafter, the camera sensor interference level, which is also described as the interference level, is determined based on at least partial segmentation of the camera image. The interference level is a criterion for a part of the field of view of the camera sensor that is being interfered with or blocked by an interfering object. The interference level may vary, for example, between 0% of an unobstructed camera sensor and 100% of a completely obstructed camera sensor.

[0014] In addition to the interference level, the neural network algorithm determines an interference class of the camera sensor interference based on at least partial classification of the camera image. Hereinafter, the interference class of the camera sensor interference is also described as the interference class.

[0015] The interference class may be selected from among a plurality of predetermined interference classes, and the plurality of predetermined interference classes may include various types of precipitation that may cover a camera sensor component that is transparent during operation. The interference class may include at least one class indicating an unobstructed sensor, i.e., a case where there is no interference. Cleaning criteria may be assigned to at least some of the interference classes, i.e., at least some classes may be labeled as types of interference that may be removed by a cleaning procedure. The determination of the interference class may occur, for example, in parallel with or sequentially to the determination of the interference level in the neural network algorithm.

[0016] After determining the degree of interference and the interference class as the output of the neural network algorithm, cleaning information is determined according to these outputs. The cleaning information may indicate that the cleaning of the camera sensor is necessary, that is, an interference of the camera sensor is detected and this interference should be removed by a cleaning process. Therefore, to determine the cleaning information, it is checked whether an interference class to which a cleaning criterion is assigned is detected. Also, during the determination of the cleaning information, it is checked whether the degree of interference determined by the neural network algorithm exceeds a predetermined threshold. In other words, the cleaning information may indicate that the cleaning of the camera sensor is necessary when the type of interference to be cleaned is detected and a considerable degree of the field of view of the camera sensor is blocked by the interference.

[0017] If the cleaning of the camera sensor is necessary, the cleaning information is transmitted to a cleaning device associated with the camera sensor. The cleaning device may be any device that can clean the camera sensor or remove the detected interference from a transparent camera sensor component. The cleaning information may particularly include information on a cleaning process suitable for the detected type of interference and / or the detected degree of interference.

[0018] Detecting both the degree of interference and the interference class has the advantage that the cleaning information can be determined according to both the amount of interference and the type of interference. In particular, the camera sensor can be cleaned according to the type of interference medium and / or the severity of the interference. Also, since the cleaning of the sensor can be performed using an optimal amount of cleaning medium and / or energy, the efficiency of the cleaning process can be improved. Advantageously, unnecessary cleaning procedures, or waste of unnecessary cleaning procedure duration and / or cleaning medium can be prevented.

[0019] Also, by determining both the degree of interference and the type of interference, or by using both segmentation and classification, redundancy is created in determining the presence of interference. This redundancy can improve the detection of states requiring cleaning and reduce the amount of false detection that can lead to incorrect decisions regarding cleaning.

[0020] Also, this redundancy makes it possible to reduce the time required to detect the obstructed state of a camera sensor that requires cleaning. By reducing the time for interference determination, it is also possible to speed up the processing of the camera image in further algorithms, such as an image restoration algorithm for restoring the obstructed part of the camera image. Advantageously, this shortens the response time of the driving assistance function using the camera image supplied by the camera sensor.

[0021] Using a neural network algorithm to detect the obstructed state of a camera sensor to be cleaned has the advantage that automated cleaning adapted to the detected interference is possible without requiring user interaction. Also, this makes it possible to reliably determine interference in the initial image that would in principle occur when using a camera sensor that captures part of the vehicle's surrounding environment. Advantageously, since interference can be detected from a single camera image, the need for additional real-time frame calculations for a camera image stream of multiple camera images to detect interference is eliminated.

[0022] It is advantageous that the method according to the present invention can be used in a system having only a single sensor because sensor redundancy is not required to detect interference. On the other hand, the method according to the present invention can also be applied to a multi-sensor system of multiple camera sensors, even when the respective fields of view do not overlap, because the interference determination may occur based only on the camera image supplied by a specific camera sensor for each of the camera sensors.

[0023] Obstacle detection and optimized cleaning may be important, for example, from the perspective related to the safety of advanced driver assistance system (ADAS) functions in a vehicle, because a clean camera lens is a prerequisite for other functions, such as semantic segmentation of roads and the like. In particular, in an autonomous driving scenario, the method according to the present invention has the advantage that a reference image for determining an obstacle is not required. Comparing an image captured by a camera sensor with a reference image is difficult in a real-world driving environment because the scene captured in these images may be affected to some extent by changes due to the movement of the vehicle. In addition to vehicles, the method according to the present invention can also be used in any type of device supplied with camera images by at least one camera sensor that tends to be obstructed by cleanable obstacles before and during operation.

[0024] In a preferred embodiment of the present invention, the neural network algorithm is an algorithm that includes both a semantic segmentation algorithm and / or a binary segmentation and a classifier model, or includes an algorithm that includes both a semantic segmentation algorithm and / or a binary segmentation and a classifier model. The neural network algorithm may be, for example, a semantic segmentation algorithm (SemSeg algorithm) capable of multi-class segmentation that can determine the degree of obstruction and each obstruction class. Additionally or alternatively, the neural network algorithm may be, for example, a combined binary segmentation and classification algorithm (BinCla algorithm) that determines both the segments obstructed in the camera image, i.e., the obstruction segments, and the unobstructed regions based on segmentation and assigns an obstruction class to the obstruction segments.

[0025] Preferably, the plurality of interference classes includes at least one non-interference class to which no cleaning standard is assigned. This makes it possible to include the state where there is no interference with the camera sensor. In such a case, it may not be necessary to determine the cleaning information, or the cleaning information may include information indicating that cleaning is unnecessary.

[0026] In a preferred embodiment, the plurality of interference classes includes at least a dirt class, a droplet class, and / or a fog class to which cleaning standards are respectively assigned. Therefore, the neural network algorithm may be able to distinguish between interference caused by dirt, interference caused by droplets, and / or interference caused by fog. It is conceivable that there are additional classes to which cleaning standards are respectively assigned, and the additional classes may include additional types of precipitation and / or other interference types that occur in a specific environment of the camera sensor. As additional classes, for example, a snow class, an ice class, or a dust class may be used, and these interference classes respectively indicate interference with the transparent camera sensor components by snow, ice, or dust.

[0027] Preferably, the cleaning information is determined by a further algorithm, the cleaning information indicates a cleaning strategy for cleaning the camera sensor, and the cleaning strategy is determined by comparing, according to the determined interference class and / or two or more different degree thresholds assigned to each cleaning strategy, with the determined degree of interference of the camera sensor, from among a plurality of cleaning strategies. This makes it possible to define different cleaning strategies for different types of interference and / or different degrees of interference, that is, for one interference class, two or more cleaning strategies may be defined according to the degree of interference. For example, when dirt is determined as the interference class, different cleaning strategies can be selected for a degree of interference of 10% - 67% and a degree of interference exceeding 67%, and the cleaning information is not determined for a degree of interference less than 10%.

[0028] Further algorithms may receive, as input, the output of the neural network, i.e., the degree of interference and the interference class. In one embodiment, a decision tree is used as the further algorithm. The decision tree may assign various cleaning strategies to various interference classes and various interference thresholds. By using the decision tree, it becomes possible to quickly determine the cleaning strategy indicated by the cleaning information. It is advantageous that the cleaning strategy indicated by the cleaning information is adapted to the detected interference class and the detected degree of interference.

[0029] The cleaning strategy may include one or more instructions for the cleaning device, and the instructions may include cleaning medium discharge, the duration and / or amount of cleaning medium discharge, the temperature and / or composition of the cleaning medium discharge, the duration and / or intensity of the cleaning actuator operation, etc.

[0030] In a preferred embodiment, the cleaning strategy is additionally determined according to at least one cleaning device state information indicating the current state of the cleaning device. The cleaning device state information may be, for example, the water level of the cleaning fluid in the container. Additionally or alternatively, other types of cleaning device state information may be used. By also considering the current state of the cleaning device, it becomes possible to adapt the cleaning strategy for cleaning the camera sensor to the cleaning ability of the cleaning device in its current state, and as a result, unnecessary or infeasible instructions can be prevented.

[0031] According to the present invention, the cleaning information may include cleaning instructions for liquid-based cleaning, air-based cleaning, and / or actuator-based cleaning. The cleaning device may be configured to discharge a cleaning fluid, such as water and / or detergent and / or air, to clean the camera sensor, in particular to clean a transparent camera sensor component. Additionally or alternatively, the cleaning device may comprise an actuator for cleaning, for example, a movable wiper blade, an ultrasonic actuator, etc., thereby enabling mechanical removal of obstructions from the transparent camera sensor component.

[0032] Furthermore, the present invention relates to a method for training a neural network algorithm, the method comprising - supplying a plurality of camera images captured by at least one unobstructed camera sensor; - expanding at least some of the camera images by superimposing an obstruction mask on each camera image, wherein the obstruction mask is assigned to an obstruction class from a plurality of obstruction classes, the obstruction mask obstructs a part of the camera image according to the degree of obstruction of the obstruction mask, and the degree of obstruction is determined stochastically for each camera image; - associating a label with each camera image, the label indicating the obstruction class and the degree of obstruction of the obstruction mask that expands the camera image; - generating an output of the neural network algorithm for each expanded camera image by processing the expanded camera image through one or more network layers of the neural network algorithm according to parameters associated with the one or more network layers; - comparing the output generated for each expanded camera image with the label associated with the expanded camera image using an objective function; - updating the parameters based on the comparison.

[0033] In particular, the method for training the neural network algorithm according to the present invention may be used to train the neural network algorithm of the method for determining the cleaning information of a camera sensor that is at least partially affected by the present invention. Therefore, the neural network algorithm of the method for determining the cleaning information of a camera sensor that is at least partially affected by the present invention may be trained by the method for training the neural network algorithm according to the present invention.

[0034] The method for training a neural network algorithm uses a plurality of camera images captured by an unobstructed camera sensor. These images may in particular show various peripheral environments of the camera sensor that are expected to occur during normal use of the camera sensor. For example, when training a neural network for a camera sensor used to capture the peripheral environment of a vehicle, various scenes that may occur during the operation of the vehicle may be shown in the supplied camera images.

[0035] Since the supplied camera images are captured by an unobstructed camera sensor, there is initially no obstruction in these images. To train a neural network algorithm for determining an obstruction segment or degree of obstruction and various obstruction classes, at least some of the unobstructed camera images are augmented by overlaying an obstruction mask. The obstruction mask represents a certain type of obstruction that virtually covers a certain degree of the field of view of the camera sensor. The obstruction that virtually augments the camera image is assigned to an obstruction class selected from among a plurality of obstruction classes. The obstruction mask may virtually reproduce a certain type of obstruction by covering one or more portions of the camera image corresponding to a certain type of obstruction, for example, dirt, droplets, or fog.

[0036] To obtain as many different augmented images as possible, each occlusion mask covers a probabilistically or randomly determined portion of the camera image. Thus, the degree of occlusion of each mask is probabilistically determined, and the degree of occlusion of the occlusion mask corresponds to the degree of occlusion of the camera image overlaid with the occlusion mask. By probabilistically determining the degree of occlusion, it becomes possible to create a large number of different occluded augmented camera images, and thus it becomes possible to precisely and extensively train a neural network based on various degrees of occlusion and various occlusion classes. Therefore, the method for training a neural network according to the present invention is particularly suitable for the method for determining the cleaning information according to the present invention because it becomes possible to train a neural network based on a variety of different occluded images by automatically creating virtual occluded images.

[0037] Each augmented camera image is assigned a label indicating the occlusion class and degree of occlusion of the occlusion mask used to augment a particular camera image. The label is created, in particular, by the same computing device used to augment the camera image using the occlusion mask. This significantly reduces the effort required to create training data because manual labeling of the training data or the augmented camera images is not required.

[0038] Here, the augmented and labeled camera images are supplied to the neural network algorithm to be trained, and as a result, the output of the corresponding neural network algorithm is generated. Thus, the neural network algorithm processes the augmented and labeled camera images through one or more network layers according to the parameters associated with each layer.

[0039] Thereafter, the output of the neural network algorithm for each of the extended camera images, in particular the determined degree of interference and the determined interference class, is compared with the label assigned to each of the extended and labeled camera images used as input. To perform the comparison, an operating function may be used to quantify and / or qualify the match between the label assigned to each extended camera image and the output of the neural network algorithm. Based on the result of the comparison using the objective function, the parameters associated with one or more layers of the neural network algorithm are updated to increase the match between the label and the output of the neural network algorithm.

[0040] To also include the case of an unobstructed state of the camera sensor, it is conceivable that some of the camera images used to train the neural network algorithm are not extended by the interference mask. This makes it possible to train the neural network, for example, based on a determination of a 0% degree of interference and / or an interference class indicating an unobstructed camera sensor.

[0041] In a preferred embodiment, additionally, at least one further parameter regarding the appearance of the interference mask is probabilistically varied, and the at least one further parameter is in particular color, color distribution, transparency, transparency distribution, number of masking segments, distribution of masking segments, size of masking segments, and / or the contour of the masking segments of the interference mask is probabilistically varied. This makes it possible to further adapt the virtual interference created by the interference mask to the types of interference that may actually occur. By varying parameters such as color, transparency or their distribution, etc., various types of dirt may be represented. Also, the number, size, contour and / or distribution of the masking segments may be varied to include as many possible interference appearances as possible.

[0042] Preferably, at least one further parameter is varied within one or more intervals associated with the interference class assigned to the interference mask. This makes it possible to adapt the interference mask to the specific appearance of various interference classes. For example, in order to simulate a camera sensor component obstructed by raindrops, a number of segments having a relatively high transparency and having a circular or elliptical shape may be used, and for interference due to dirt, a few large segments having a high opacity and an indefinite shape may be used.

[0043] The control device according to the present invention includes a computing device, and this control device is configured to execute a method for determining cleaning information according to the present invention.

[0044] The camera sensor system according to the present invention includes at least one camera sensor, at least one cleaning device associated with the camera sensor, and a control device according to the present invention.

[0045] The vehicle according to the present invention includes a camera sensor system according to the present invention.

[0046] The computer program according to the present invention includes instructions for controlling a computer to execute a method for determining cleaning information according to the present invention.

[0047] The non-transitory recording medium according to the present invention includes a computer program according to the present invention.

[0048] All of the details and advantages described with respect to one of the methods according to the present invention are correspondingly applicable to the other methods according to the present invention. Also, these details and advantages are correspondingly applicable to the control device, camera sensor system, vehicle, computer program, and recording medium, and vice versa.

[0049] Further features and details of the present invention are discussed in connection with the figures. The figures are schematic drawings.

Brief Description of the Drawings

[0050]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Embodiments for Carrying Out the Invention

[0051] FIG. 1 shows an embodiment of vehicle 1. Vehicle 1 may be an electric or non - electric vehicle, such as a passenger car, a truck, a train, a trailer, etc. Vehicle 1 is provided with a camera sensor system 2, and the camera sensor system 2 includes a camera sensor 3, a control device 4, and a cleaning device 5. The camera sensor 3 and the cleaning device 5 communicate with the control device 4 via a communication link, for example, via a bus system, such as a CAN - Bus, and / or via one or more point - to - point connections.

[0052] The camera sensor 3 is provided on the vehicle body of vehicle 1, for example, on the bumper, side mirror or radiator grille of vehicle 1. The camera sensor 3 is configured to capture a camera image, in particular, a video stream consisting of a plurality of consecutively captured camera images. A part of the surrounding environment of vehicle 1 is captured by the camera sensor 3.

[0053] The camera sensor system 2 may be part of the driver assistance system of the vehicle 1. The driver assistance system is configured to supply at least one control signal to the actuators of the vehicle 1 and / or at least one optical and / or acoustic signal to the driver of the vehicle 1, in particular depending on the camera image supplied by the camera sensor 3, and more particularly depending on the interpretation of the content of the camera image supplied by the camera sensor 3.

[0054] The cleaning device 5 is assigned to the camera sensor 3 and is configured to clean the camera sensor 3, in particular to clean the transparent camera sensor component 6 provided in the optical path of the camera sensor 3, for example, the lens, optical filter and / or transparent cover of the camera sensor 3. The cleaning device 5 may be configured to discharge a cleaning fluid, such as water, detergent and / or air, in order to clean the transparent camera sensor component 6. Additionally or alternatively, the cleaning device 5 may include an actuator (not shown), for example, a movable wiper blade, an ultrasonic actuator, etc., for cleaning, thereby making it possible to mechanically remove obstructions such as droplets, dirt, fogging, etc. from the transparent camera sensor component 6.

[0055] The control device 4 is a computing device or includes a computing device. When the control device 4 is at least partially obstructed, that is, when the transparent camera sensor component 6 is at least partially covered by an obstruction that blocks the field of view of the camera sensor 3, the control device 4 is configured to execute a method for determining the cleaning information of the camera sensor 3.

[0056] FIG. 2 shows a block diagram of an embodiment of a method for determining the cleaning information of the camera sensor 3 that is at least partially obstructed.

[0057] In step S1, the camera sensor 3 is controlled to capture at least one camera image, for example, by the control device 4 sending an image acquisition trigger to the camera sensor 3. It is also conceivable to control the camera sensor 3 using a further control device of the vehicle 1 to capture one or more camera images. The camera sensor 3 captures at least one camera image and transmits it to the control device 4. It is conceivable that the camera sensor 3 captures a video stream consisting of a plurality of continuously captured camera images and continuously transmits a single camera image to the control device 4 for detecting interference and / or determining cleaning information.

[0058] In step S2, the camera image transmitted to the control device 4 is processed by the control device 4 using a neural network algorithm implemented in the control device 4. The neural network algorithm is configured to determine, as the output of the neural network algorithm, both the degree of camera sensor interference based on at least partial segmentation of the camera image and the interference class from a plurality of interference classes of camera sensor interference based on at least partial classification of the camera image or each camera image, which has been processed by the neural network algorithm. For example, an algorithm having both a semantic segmentation algorithm and / or a binary segmentation and a classifier model may be used as the neural network algorithm or as part of the neural network algorithm.

[0059] The degree of camera sensor interference indicates the portion of the field of view of the camera sensor 3 covered by interference with the transparent camera sensor component 6. The degree of camera sensor interference may vary from 0% to 100%, where 0% indicates a state where the field of view of the camera sensor 3 is not interfered with, 100% indicates a state where the field of view of the camera sensor 3 is completely interfered with, and values in between indicate that the field of view of the camera sensor 3 is partially interfered with.

[0060] The degree of obstruction is determined based on the segmentation of the camera image, and one or more portions of the camera image indicating obstruction and one or more portions of the camera image indicating the vehicle's surrounding environment are determined. The degree of obstruction may be determined, for example, as the ratio of the area of the segment indicating obstruction to the area of the unobstructed segment indicating the surrounding environment.

[0061] In addition to the degree of obstruction, the neural network algorithm determines the obstruction class from a plurality of obstruction classes. The obstruction class is determined based on the classification of at least a part of the camera image. Cleaning criteria are assigned to at least some of the obstruction classes, and the cleaning criteria indicate that cleaning of the camera sensor may be required due to this specific type of obstruction, or that cleaning of the camera sensor is conceivable due to this type of obstruction. Advantageously, both the segmentation and the classification may be performed independently of each other, so that a combined prediction is obtained and the reliability of the system is improved by fusing the predictions from the two models.

[0062] The plurality of obstruction classes preferably include at least a dirt class, a droplet class, and a fog class, and cleaning criteria are assigned to each of them. It is also conceivable to use additional obstruction classes indicating other precipitation such as ice and snow, and / or a specific type of dirt, for example, dirt caused by dust, mud, sand, insects, etc. The plurality of obstruction classes may particularly include at least one non-obstruction class to which no cleaning criteria are assigned. The non-obstruction class indicates a state in which the camera sensor 3 or its transparent camera sensor component 6 is not covered by an obstruction. Information regarding the various obstruction classes and the cleaning criteria assigned to the obstruction classes may be stored, for example, in the control device 4.

[0063] In step S3, cleaning information is determined according to the degree of camera sensor interference and the class of camera sensor interference determined by the neural network algorithm. If a cleaning criterion is assigned to the determined class of camera sensor interference and at least one degree threshold is exceeded by the determined degree of camera sensor interference, the cleaning information indicates that cleaning of the camera sensor 3 is necessary.

[0064] Preferably, the cleaning information is determined by a further algorithm implemented in the control device 4. The further algorithm may be, for example, a decision tree, which is used to determine a cleaning strategy from a plurality of cleaning strategies by comparing, according to the determined interference class and / or two or more different degree thresholds assigned to each cleaning strategy, with the determined degree of camera sensor interference. The determined cleaning information indicates a cleaning strategy for cleaning the camera sensor 3. The cleaning strategy may include, for example, cleaning instructions for liquid-based cleaning, air-based cleaning, and / or actuator-based cleaning according to the type of the cleaning device 5.

[0065] In step S4, the cleaning information is transmitted to the cleaning device 5 in order to clean the camera sensor 3 according to the determined cleaning strategy. If the cleaning information indicates that cleaning of the camera sensor 3 is not necessary, the transmission to the cleaning device 5 may be omitted. In this case, the method can be continued by controlling the camera sensor 3 to capture the next image of the plurality of camera images supplied by the camera sensor 3 (step S1) or by processing the next camera image (step S2).

[0066] FIG. 3 shows an embodiment of a decision tree used as a further algorithm for determining cleaning information. For the degree of interference shown as D, three thresholds of 10% or less, 66.6% or less, and 100% or less are used. For each degree of interference D in the corresponding interval, four different interference classes C1 to C4 distinguished by a neural network algorithm are exemplarily shown.

[0067] For example, no cleaning criteria are assigned to the interference class C1 indicating a non-interference class. The cleaning criteria are assigned, for example, to the interference class C2 which is a dirt interference class, for example, to the interference class C3 which is a clouding interference class, for example, to the interference class C4 which is a droplet interference class. In FIG. 3, the interference classes to which the cleaning criteria are assigned are shown in bold for distinction.

[0068] Since no cleaning criteria are assigned to the interference class C1, no cleaning strategy is assigned to the interference class C1 regardless of the threshold for the degree of interference D. In other words, when a non-interference class is determined as an interference class based on classification, the cleaning strategy is not executed regardless of the determination result of the degree of interference D based on classification. In such a case, the cleaning information may indicate, for example, that cleaning is unnecessary, or the cleaning information may not be determined.

[0069] When the degree of interference D is 0% to 10%, even if the interference is classified as one of the interference classes C2, C3, or C4, although the cleaning criteria are assigned to each of these interference classes, the cleaning of the camera sensor 3 does not occur. Thereby, the energy when the function of the camera sensor 3 is not affected by interference and / or the use of the cleaning detergent of the cleaning device 5 is reduced.

[0070] When the degree of interference is between 10% and 66.6%, the first cleaning strategy CS1 is assigned to each of the interference classes C3 and C4. The second cleaning strategy CS2 is assigned to the interference class C2. When the degree of interference D is between 66.6% and 100%, instead, the third cleaning strategy CS3 is associated with the interference class C2, and the second cleaning strategy CS2 is associated with the interference classes C3 and C4.

[0071] The decision tree shown in FIG. 3 is only an example, and it is conceivable that a different number of interference classes, a different number of cleaning strategies, a different number of degree thresholds, and / or different degree thresholds may be used.

[0072] The cleaning strategy may additionally be determined according to at least one cleaning device state information indicating the current state of the cleaning device. The cleaning device state information may be transmitted, for example, from the cleaning device 5 to the control device 4. The cleaning device state information may be, for example, the fluid level of the cleaning fluid in the fluid container of the cleaning device 5. Additionally or alternatively, other types of cleaning device state information may be used. By also considering the current state of the cleaning device, it becomes possible to adapt the cleaning strategy for cleaning the camera sensor 3 to the cleaning ability of the cleaning device 5 in its current state, and as a result, unnecessary or impossible commands for the cleaning device 5 can be prevented.

[0073] For example, when the fluid level of the cleaning fluid in the fluid container of the cleaning device 5 is 10% or more, a certain cleaning strategy is determined as cleaning information or as part of the cleaning information, while when the fluid level is less than 10%, a different cleaning strategy is considered to be selected. Additionally or alternatively, when the fluid level is low, for example, warnings and / or replenishment requests may be issued to the driver of the vehicle 1 as optical and / or acoustic signals.

[0074] FIG. 4 shows, in particular, a block diagram of an embodiment of a method for training a neural network algorithm in the control device 4 with respect to the neural network algorithm. This method enables efficient training of a neural network algorithm that determines both the degree of interference and the interference class of a camera sensor with respect to an interference that blocks at least a part of the field of view of the camera sensor.

[0075] In step T1, a plurality of camera images captured by at least one non-interfered camera sensor are supplied. These images may in particular show various peripheral environments of the camera sensor that are expected to occur during normal use of the camera sensor. For example, the training of the neural network algorithm of the control device 4 may be performed using camera images showing various scenes of the vehicle's peripheral environment, in particular various traffic situations and / or traffic environments. The camera images may be recorded, for example, by the camera sensor 3, or by a similar camera sensor 3 of the vehicle 1, and / or by a similar camera sensor 3 of another vehicle.

[0076] In step T2, at least some of the camera images are extended by superimposing an interference mask on each of these camera images, the interference mask being assigned an interference class from among a plurality of interference classes, the interference mask interfering with a part of the camera image according to the degree of interference of the interference mask, and the degree of interference being determined stochastically for each camera image. Also, a label is associated with each camera image, the label indicating the interference class and the degree of interference of the interference mask that extends the camera image. The process of extending and labeling the camera images will be described in more detail below with reference to FIG. 5.

[0077] In step T3, an output of the neural network algorithm is generated for each extended camera image by processing the extended camera image through one or more network layers of the neural network algorithm according to parameters associated with the one or more network layers.

[0078] In step T4, for each of the extended camera images, the output generated is compared with the label associated with the extended camera image using an objective function, and based on the comparison, the parameters of the neural network algorithm associated with the layer of the neural network algorithm are updated.

[0079] In FIG. 5, the process of extending and labeling camera images to train a neural network algorithm is described in detail. This process can be performed by a computer, in particular as an automated pipeline. The camera images to be extended are supplied, for example, using an unobstructed camera sensor 3. An unobstructed camera image 7 is schematically shown in FIG. 5.

[0080] To extend the unobstructed camera image 7, i.e., to add a virtual obstruction to the camera image 7 such that it appears as if the camera sensor used to capture the camera image 7 was obstructed by an obstruction, an obstruction mask 8 is used. For each camera image 7 to be extended, a related obstruction mask 8 is created. The obstruction mask generates a virtual obstruction of a certain obstruction class.

[0081] For each obstruction mask 8, the obstruction class can be deterministically or probabilistically selected from a plurality of predetermined obstruction classes. Also, each obstruction mask supplies a virtual obstruction overlay covering a part of the camera image 7 to the camera image 7 according to a probabilistically determined degree of obstruction. The degree of obstruction supplied by the obstruction mask may depend, for example, on the number of masking segments 9 and the size of each masking segment 9 in comparison to the size of the obstruction mask 8 or the camera image 7 to be extended. The degree of obstruction determines the degree of obstruction of the camera image 7 after extension by the obstruction mask 8.

[0082] Additionally, one or more further parameters regarding the appearance of the interference mask 8 may be probabilistically varied. As further parameters, for example, the color of the individual masking segments 9, the color distribution of the individual masking segments 9, the transparency of the individual masking segments 9, the transparency distribution of the individual masking segments 9, the number of masking segments 9, the distribution of the masking segments 9 in the interference mask 8, the size of the individual masking segments 9 and / or the contour of the individual masking segments 9 may be used.

[0083] In particular, one or more further parameters may be varied within one or more intervals associated with the interference class assigned to the interference mask 8. For example, if the interference class of the interference mask indicates interference due to dirt, the color may be varied between various brownish colors, and the transparency may be varied between zero transparency and 30% transparency. If the interference class of the interference mask 8 is, for example, raindrops, much higher transparency values and different colors can be used. Also, in order to reproduce the appearance of raindrops on the transparent camera component 6, the number and shape of the interference segments 9 can be varied in various ways.

[0084] Here, the interference mask 8 is used to expand the camera image 7. Accordingly, the interference mask 8 is superimposed on the camera image 7 to create an expanded camera image 10. Also, a label is associated with the expanded camera image 10, and the label indicates the interference class and the degree of interference of the interference mask superimposed on the camera image 7. It is conceivable that the label has a value of the degree of interference created by the interference mask, or, for example, a binary image label corresponding to the interference mask is supplied, and this binary image label can indicate both the area interfered with by the segment 9 and the area not interfered with. The label is created automatically, in particular, during the expansion of the camera image 7. This has the advantage that subsequent manual labeling is not required.

[0085] Subsequently, the enhanced camera image 10 is used for training, in particular, to train the neural network algorithm of the control device 4. In addition to the enhanced camera image 10, unobstructed camera images may also be used. These unobstructed camera images may be associated with an unobstructed interference class and thus may include a label indicating the unobstructed interference class.

[0086] Automatically generating the enhanced camera image 10 has the advantage of being able to create a large number of training images that encompass a wide variety of different interference conditions. This enables the neural network algorithm to be efficiently trained based on, in particular, the degree of interference and the interference class of the camera image 7 supplied from the obstructed camera sensor 3 during the operation of the vehicle 1.

[0087] In this case, the neural network algorithm trained based on the enhanced camera image 10 may be used to evaluate and predict the actual interference of the camera image. Therefore, the weights of the neural network algorithm learned during training using the enhanced dataset, i.e., the updated parameters, are used in the neural network algorithm to make predictions about the real-world dataset part regarding interference. It is advantageous that the trained neural network algorithm can be provided in this way without the need to supply images that are actually interfered with, which may be difficult to obtain for real vehicle driving scenarios. On the other hand, it is also conceivable to further fine-tune this neural network algorithm based on a real dataset comprising camera images that are actually interfered with and have the same degree of interference and the same interference class selected from a plurality of interference classes, thereby achieving a further improvement in the performance of the neural network algorithm. Although this application relates to the invention described in the claims, it also includes the following from other perspectives. 1. In a method for determining cleaning information of the camera sensor (3) that is at least partially obstructed, including obstruction on a transparent camera sensor component (6) in the optical path of the camera sensor (3), - Controlling the camera sensor (3) to capture at least one camera image; - Processing the at least one camera image (3) by a computing device using a neural network algorithm, wherein the neural network algorithm determines, as an output, a camera sensor obstruction degree based on at least partial segmentation of the camera image or each of the camera images, and an obstruction class from a plurality of obstruction classes of the camera sensor obstruction based on at least partial classification of the camera image; - Determining cleaning information according to the camera sensor obstruction degree and the class of the camera sensor obstruction, wherein when a cleaning criterion is assigned to the determined class of the camera sensor obstruction and at least one degree threshold is exceeded by the determined camera sensor obstruction degree, the cleaning information indicates that cleaning of the camera sensor (3) is necessary; - A method comprising transmitting the cleaning information to a cleaning device (5) associated with the camera sensor (3) to clean the camera sensor (3) according to the cleaning information. 2. The method according to item 1 above, wherein the neural network algorithm comprises an algorithm comprising both a semantic segmentation algorithm and / or a binary segmentation and a classifier model, or comprises an algorithm comprising both a semantic segmentation algorithm and / or a binary segmentation and a classifier model. 3. The method according to item 1 or 2 above, wherein the plurality of obstruction classes includes at least one non-obstruction class to which no cleaning criterion is assigned. 4. The method according to any one of the above 1 to 3, characterized in that the plurality of interference classes include at least a dirt class, a drip class and / or a fog class, each of which is assigned a cleaning criterion. 5. The cleaning information is determined by a further algorithm, in particular by a neural network. The cleaning information indicates a cleaning strategy for cleaning the camera sensor. The cleaning strategy is determined from among a plurality of cleaning strategies according to the determined interference class and / or by comparing two or more different degree thresholds assigned to each cleaning strategy with the determined degree of interference of the camera sensor. The method according to any one of the above 1 to 4. 6. The method according to any one of the above 1 to 5, characterized in that the cleaning strategy is additionally determined according to at least one cleaning device state information indicating the current state of the cleaning device. 7. The method according to any one of the above 1 to 6, characterized in that the cleaning information includes cleaning instructions for liquid-based cleaning, air-based cleaning and / or actuator-based cleaning. 8. In particular, in a method for training a neural network algorithm for use in the method according to any one of the above 1 to 7, - supplying a plurality of camera images (7) captured by at least one non-interfered camera sensor; - expanding at least some of the camera images (7) by superimposing an interference mask (8) on each camera image (7), the interference mask (8) being assigned an interference class from among a plurality of interference classes, the interference mask (8) interfering with a part of the camera image (7) according to the degree of interference of the interference mask, and the degree of interference being determined stochastically for each camera image (7); - associating a label with each camera image (7), the label indicating the interference class and the degree of interference of the interference mask (8) for expanding the camera image. - Processing the extended camera image (10) through one or more network layers of the neural network algorithm, according to parameters associated with the one or more network layers, to generate an output of the neural network algorithm for each said extended camera image (7); - Comparing the generated output for each said extended camera image (10) with the label associated with the extended camera image (10) using an objective function; - Updating the parameters based on the comparison. A method comprising these steps. 9. Additionally, at least one further parameter regarding the appearance of the interference mask (8) is probabilistically varied. The at least one further parameter is, in particular, color, color distribution, transparency, transparency distribution, number of masking segments, distribution of masking segments, size of masking segments, and / or the contour of the masking segments of the interference mask is probabilistically varied. The method according to item 8 above. 10. The method according to item 9 above, characterized in that the at least one further parameter is varied within one or more intervals associated with the interference class assigned to the interference mask (8). 11. A control device comprising a computing device, wherein the control device (4) is configured to execute the method according to any one of items 1 to 7 above. 12. A camera sensor system comprising at least one camera sensor (3), at least one cleaning device (5) associated with the camera sensor (3), and the control device (4) according to item 11 above. 13. A vehicle comprising the camera sensor system (2) according to item 12 above. 14. A computer program comprising instructions for controlling a computer to execute the method according to any one of items 1 to 7 above. 15. A non-transitory recording medium comprising the computer program according to item 14 above.

Claims

1. 1. A method for determining cleaning information of a camera sensor (3) that is at least partially obstructed, including an obstruction on a transparent camera sensor component (6) in the optical path of the camera sensor (3), comprising: - controlling said camera sensor (3) to capture at least one camera image; - processing said at least one camera image (3) by a computing device using a neural network algorithm, said neural network algorithm being adapted to determine as output from the or each camera image a degree of camera sensor obstruction based on a segmentation of at least a part of the camera image and a obstruction class from among a plurality of obstruction classes of camera sensor obstructions based on a classification of at least a part of the camera image; - determining cleaning information as a function of the camera sensor obstruction degree and the class of the camera sensor obstruction, the cleaning information indicating that cleaning of the camera sensor (3) is necessary if a cleaning criterion has been assigned to the determined class of camera sensor obstruction and if at least one degree threshold is exceeded by the determined camera sensor obstruction degree; - transmitting said cleaning information to a cleaning device (5) associated with said camera sensor (3) for cleaning said camera sensor (3) according to said cleaning information, The method for training the neural network algorithm comprises the steps of: - providing a plurality of camera images (7) captured by at least one unobstructed camera sensor; - expanding at least some of the camera images (7) by superimposing a disturbance mask (8) on each camera image (7), said disturbance mask (8) being assigned to a disturbance class from among a plurality of disturbance classes, said disturbance mask (8) disturbing a part of the camera image (7) according to a disturbance degree of said disturbance mask, said disturbance degree being determined probabilistically for each camera image (7); - associating to each camera image (7) a label, said label indicating said obstruction class and said obstruction degree of said obstruction mask (8) that augments the camera image; - processing the augmented camera images (10) through one or more network layers of the neural network algorithm and in accordance with parameters associated with said one or more network layers to generate an output of said neural network algorithm for each augmented camera image (7); - comparing said generated output for each augmented camera image (10) with said label associated with said augmented camera image (10) using an objective function; - updating said parameters based on said comparison; Including, said cleaning criteria being criteria which allow to decide whether or not to carry out cleaning with said cleaning device (5) in response to a particular disturbance class (C1, C2, C3, C4); A disturbance class to which said cleaning criteria is not assigned does not require cleaning; The interference class to which the cleaning criterion is assigned is cleaned according to the degree of interference with the camera sensor. A method comprising:

2. 2. The method of claim 1, wherein the neural network algorithm is or comprises a semantic segmentation algorithm and / or an algorithm comprising both a binary segmentation and a classifier model.

3. 2. The method of claim 1, wherein the plurality of disturbing classes includes at least one non-disturbing class that has no assigned cleaning criteria.

4. The method of claim 1 , wherein the plurality of disturbance classes includes at least a dirt class, a droplet class and / or a cloudy class, each of which is assigned a cleaning criterion.

5. 2. The method according to claim 1, characterized in that the cleaning information is determined by a further algorithm or a neural network, the cleaning information indicating a cleaning strategy for cleaning the camera sensor, the cleaning strategy being determined from among a plurality of cleaning strategies depending on the determined obstruction class and / or by comparing the determined camera sensor obstruction degree with two or more different degree thresholds assigned to each cleaning strategy.

6. 6. The method of claim 5, wherein the cleaning strategy is additionally determined in response to at least one cleaning device status information indicative of a current status of the cleaning device.

7. The method of claim 1 , wherein the cleaning information includes cleaning instructions for liquid-based cleaning, air-based cleaning, and / or actuator-based cleaning.

8. 2. The method according to claim 1, further comprising stochastically varying at least one further parameter related to the appearance of the obstruction mask (8), said at least one further parameter being a color, a color distribution, a transparency, a transparency distribution, a number of masking segments, a distribution of masking segments, a size of masking segments and / or stochastically varying a contour of a masking segment of the obstruction mask.

9. 9. The method according to claim 8, characterized in that the at least one further parameter is varied within one or more intervals which are associated with the disturbance classes assigned to the disturbance mask (8).

10. A control device comprising a computing device, said control device (4) being configured to carry out the method according to any one of claims 1 to 7.

11. A camera sensor system comprising at least one camera sensor (3), at least one cleaning device (5) associated with said camera sensor (3), and a control device (4) according to claim 10.

12. A vehicle comprising a camera sensor system (2) according to claim 11.

13. A computer program comprising instructions for controlling a computer to carry out the method according to any one of claims 1 to 7.

14. A non-transitory recording medium comprising the computer program of claim 13.

Images