Method for computer-aided monitoring of the light transmission of a lens of a camera system in a motor vehicle

Abstract

The following descriptions relate to a method for computer-aided monitoring of the light transmittance of a lens of a camera system in a motor vehicle, comprising the following steps: Determining (100) and providing (200) a plurality of limit values ​​for the illuminances measurable by the camera system using a pre-trained neural network; Measuring (300) the illuminance using the camera system and providing (400) measurement data; Evaluating (500) the measurement data using an evaluation unit connected to the neural network; Outputting (600) maintenance information if the evaluation of the measurement data reveals a deviation of at least one limit value.

Description

[0001] The following statements relate to a method for computer-assisted monitoring of the light transmittance of a lens of a camera system in a motor vehicle, which method enables cost-effective real-time monitoring of the degree of contamination of the lens. Furthermore, the following statements relate to a motor vehicle configured to carry out the above-described method.

[0002] DE 10 2015 117 612 A1 discloses a method for detecting the degree of contamination of a camera lens in a motor vehicle. The degree of contamination of the camera lens is recorded by tracking the exposure. If the contamination falls below a predetermined threshold, automated cleaning of the lens is activated.

[0003] There is a need to monitor the proper functioning of camera systems in motor vehicles in a simple, cost-effective and weather-independent manner.

[0004] Based on this situation, the present task is to propose a method by which the degree of contamination of a lens of a camera system in a motor vehicle can be monitored simply, cost-effectively and independently of the weather.

[0005] The present problem is solved by the features of the independent claim. Advantageous embodiments are specified in the subclaims, the description, and the drawings. To the extent technically feasible, the teachings of the subclaims can be combined arbitrarily with the teachings of the main and subclaims.

[0006] In particular, the object is achieved by a method for computer-assisted monitoring of the light transmission of a lens of a camera system in a motor vehicle, comprising the following steps: - Determining and providing a plurality of limit values ​​for the illuminances measurable by the camera system by means of a pre-trained neural network; - Measuring the illuminance using the camera system and providing measurement data; - Evaluation of the measurement data using an evaluation unit connected to the neural network; - Output of maintenance information in the event that the evaluation of the measured data shows a deviation from at least one limit value.

[0007] By using the pre-trained neural network, a large amount of measurement data can be easily and cost-effectively evaluated using computer-aided analysis. This allows the degree of lens contamination to be monitored over a long period of time and constant changes in the lens's light transmittance to be detected, independent of external influences such as weather conditions. The method thus enables fast and cost-effective monitoring of the light transmittance of the lens of the camera system in the motor vehicle.

[0008] Advantageous aspects are explained below, and preferred modified embodiments are described further below. Explanations, particularly regarding advantages and definitions of features, are essentially descriptive and preferred, but not limiting, examples. If an explanation is limiting, this will be expressly stated.

[0009] It is preferred that the sequence of process steps can be varied, unless technically required in an explicit order. However, the aforementioned sequence of process steps is particularly preferred.

[0010] Where ordinal numbers, such as "first", "second", etc., are used, for example to designate a component, an element, a method step, or a method action, these ordinal numbers are intended purely for differentiation in the designation and do not indicate any dependencies or sequences. This means, in particular, that, for example, a device does not have to have a "first component" in order to have a "second component." A device can also have a "first component" and a "third component" without necessarily having a "second component." Multiple units of the same ordinal number can also be provided, for example, multiple "first components."

[0011] If the degree of contamination of camera lenses increases, relevant information for the driver or the driving function of the vehicle becomes less visible / evaluable. In order to clean the lens only as needed, the degree of contamination of the lens must be checked. Checking with the aid of artificial intelligence enables the evaluation of large amounts of data. The artificial intelligence is based on a neural network. The limit values ​​determined by the neural network are defined independently of the weather conditions occurring during the measurement, allowing a constant change in the light transmittance of the lens to be detected even under external influences. The limit values ​​determined by the neural network can provide an indication of lens contamination if the limit value is exceeded by the measured data.Such an exceedance can occur particularly when water condenses on the lens. A fall below the limit can occur particularly when the lens is contaminated by, for example, sand, dust, dirt, Sahara dust, pollen, snow, or similar. The limit values ​​represent a specific range.

[0012] Illuminance describes the luminous flux per unit area that hits an illuminated object. In this specific case, the luminous flux hits an image sensor of the camera system. Before this happens, the luminous flux must pass through the lens and its outer lens. Consequently, the luminous flux can be at least partially reflected and / or absorbed by dirt on the lens, which in turn results in a lower illuminance at the image sensor. The SI unit of illuminance is lux.

[0013] The term “camera system” can be understood to mean, in particular, one or all of the cameras required for safe and comfortable vehicle control in a motor vehicle.

[0014] The method specifically involves degrading the camera system when a deviation from at least one threshold is detected. Due to contamination of the camera system, the measurement data received by the camera system cannot be considered, or at least to a lesser extent, during evaluation by, for example, a driver assistance system. This allows only measurement data from other sensor units or camera systems to be considered in the evaluation.

[0015] Alternatively or additionally, it can be provided that the neural network is trained with a large number of data of the illuminance of a clean lens and a dirty lens that can be measured by the camera system. In particular, it is provided that the data used for training include information about the ambient conditions of the motor vehicle and the measurement times. By analyzing a large number of data from the camera system, the neural network can monitor the light transmittance of the lens more precisely. In this case, a neural network is trained with a large number of image data or illuminances from a large number of camera systems with and without dirty lenses. The network learns to recognize typical changes in the exposure values ​​and image features that are caused by lens contamination or daytime-related lighting conditions.The neural network is designed, in particular, to receive data on the environmental conditions of the motor vehicle. The evaluation unit can be connected to a plurality of sensors arranged on the motor vehicle and can be designed to receive and transmit the data received from the sensors.

[0016] Training a neural network is therefore a process in which the network learns to recognize thresholds in data and makes predictions or decisions based on these thresholds. To do this, the neural network requires a large amount of input data and the corresponding target values ​​that represent the expected outputs. The data is divided into two groups: training data, for the actual training, and test data, to evaluate the model's performance.

[0017] The artificial intelligence used in the method therefore comprises at least one trained and / or pre-trained neural network. The neural network can be trained on a server using at least one processor. The neural network has an input layer, in particular a plurality of hidden layers, and an output layer, each with a plurality of interconnected neurons. The input layer of the neural network is the layer that receives the input data into the network and prepares it for processing by the network. The main task of the input layer is to convert the raw data into a form that can be understood and processed by the network. The hidden layers in a neural network are the layers between the input layer and the output layer.In contrast to the input layer, which receives the input data, and the output layer, which provides the network's outputs, the hidden layers perform the actual processing and pattern recognition. The output layer in a neural network is the last layer that produces the network's final output after the input data has been processed by the hidden layers. The main task of the output layer is to generate the network's outputs, which represent the solution to the underlying problem.

[0018] Alternatively or additionally, it can be provided that the neural network determines limit values, in particular dynamic limit values ​​depending on the ambient conditions of the motor vehicle, for maximum illuminance. The maximum illuminance refers in particular to a point at which the light is most strongly detected or to the entire image area on the image sensor of the camera system. If this value is unusually low, for example too dark, or too high, for example too bright, this may indicate contamination of the lens. The maximum illuminances are recorded and analyzed over time. If the illuminance deteriorates suddenly or gradually over a certain period of time, this indicates contamination.

[0019] Alternatively or additionally, the neural network can be configured to determine threshold values ​​for average illuminance. The average illuminance indicates how much light falls on the entire image sensor of the camera system on average. A dirty lens can distort these values, as dirt, dust, or water droplets can scatter or absorb the light unevenly. The average illuminance values ​​are recorded and analyzed over time. If the illuminance deteriorates suddenly or gradually over a period of time, this indicates contamination.

[0020] Alternatively or additionally, the illuminance measurement may be performed incrementally, particularly over a predetermined period of time. In particular, the illuminance measurement is provided as a real-time measurement. Incremental measurements are performed stepwise or in small increments, i.e., in successive short time periods.

[0021] Alternatively or additionally, it can be provided that the evaluation of the measurement data takes into account the ambient conditions and / or the measurement time. The illuminance measured by the camera system can vary depending on the ambient conditions. In particular, measurement data taken at the same time of day can differ from one another due to weather influences. Furthermore, the measurement data varies depending on the time of day. By analyzing a large amount of data, the neural network can take these influencing factors into account when determining the limit values ​​and, for example, determine limit values ​​for specific times of day, seasons and / or weather conditions. In particular, the neural network can access the ambient conditions via the evaluation unit and take these into account in the evaluation.

[0022] Alternatively or additionally, it can be provided that the maintenance information includes an instruction to carry out an automated cleaning of the lens of the camera system. In particular, it is additionally or alternatively provided that the maintenance information includes a request for the motor vehicle driver to carry out a manual cleaning of the lens of the camera system. Depending on whether the motor vehicle with the camera system in question has an automated cleaning system, the maintenance information can include instructions to switch the cleaning system on or off. Without the cleaning system, cleaning can only be carried out manually by the motor vehicle driver or by another authorized person. For this purpose, the motor vehicle driver receives a request from the evaluation unit in the form of the maintenance information. In particular, the maintenance information can include both the instruction and the note.This ensures that the driver is informed about the degree of contamination of the lens.

[0023] The object is further achieved by a motor vehicle which is designed to carry out a method which can be designed and further developed as described above, comprising: - at least one camera system having a translucent lens for recording the surroundings of the motor vehicle and for measuring the illuminance; - an evaluation unit for receiving and evaluating the measurement data provided by the camera system using a neural network; wherein the neural network determines a plurality of limit values ​​for the illuminances measurable by the camera system and provides them for the evaluation of the measurement data; wherein the evaluation unit outputs maintenance information if the evaluation of the measurement data reveals a deviation from at least one limit value.

[0024] The camera system has, in particular, an image sensor that converts the incoming light into electrical signals to generate a digital image. The lens is designed as part of the lens of the camera system and is arranged in the lens adjacent to the surroundings. This can lead to dirt, dust, condensate, or similar deposits on the lens. Contamination of the lens also reduces its light transmittance. By measuring the illuminance, in particular its rate of change, the degree of contamination can be determined. For this purpose, the evaluation unit is connected to a neural network that determines a plurality of limit values ​​for the illuminance levels measurable by the camera system. If the measured data determined by the camera system deviate from at least one limit value, the evaluation unit outputs maintenance information. In particular, the limit values ​​can also represent value ranges for the corresponding illuminance levels.The vehicle therefore features intelligent self-diagnosis for camera systems. It uses time-dependent exposure information and pattern recognition through artificial neural networks to detect when the camera system lens needs cleaning.

[0025] Alternatively or additionally, the motor vehicle may further comprise: a cleaning system for receiving the maintenance information and for performing automated cleaning of the lens of the camera system; and / or an output unit for receiving the maintenance information and for displaying a request for the motor vehicle driver to perform manual cleaning of the lens of the camera system. The cleaning system may, in particular, comprise a cleaning fluid-based, mechanical, thermal, and / or ultrasonic cleaning system. The use of water solutions for cleaning other motor vehicle components, such as the windshield, is already implemented in motor vehicles, whereby the water solution can be used to clean the lens in a correspondingly simple and cost-effective manner.Mechanically acting concepts, for example in the form of windshield wipers, can also be installed easily and cost-effectively. Thermal heating allows a lens fogged up by steam or frost to be cleaned quickly and cost-effectively. For this purpose, a heating element can be installed on the lens or in a surrounding area, which can be switched on and off by the evaluation unit. Ultrasonic cleaning, particularly in combination with a cleaning fluid, can remove dirt residue from the lens through vibration. The output unit can be designed, in particular, to output visual and / or acoustic signals within the motor vehicle.

[0026] Alternatively or additionally, the neural network can be implemented on a server connected to the evaluation unit. By outsourcing the neural network, the computing power required in the motor vehicle can be reduced and the costs for additional hardware can be saved. In particular, the neural network can be trained using a large amount of data from different motor vehicles. By outsourcing the neural network, the limit values ​​determined by the neural network can be made available to a large number of motor vehicles for monitoring the light transmittance of the lens of the camera system. This enables cost-effective determination of the limit values ​​for monitoring the light transmittance of the lens. The evaluation unit is connected to the server in such a way that data can be exchanged in both directions.The evaluation of the measured data can be carried out locally in the vehicle using the evaluation unit. This means that monitoring can also be carried out autonomously, without a permanent connection to the server. For this purpose, the evaluation unit only needs to transmit the numerous limit values ​​from the server once. In particular, the comparison of the measured data with the limit values ​​can take place on the server. This can save additional computing capacity in the form of hardware in the vehicle. The evaluation unit can transmit the measured data provided by the camera system to the server. The evaluation unit can then receive and process maintenance information from the server. Due to the large volume of data received, the neural network can be continuously trained and thus optimized. In principle, the more data the neural network processes, the more accurate the subsequent monitoring result will be.

[0027] A preferred technical solution is explained in more detail below with reference to the accompanying drawings using preferred embodiments. The term "figure" is abbreviated to "Fig." in the drawings.

[0028] The drawings show Fig. 1 is a diagram of a preferred embodiment of a method for computer-aided monitoring of the light transmission of a lens of a camera system in a motor vehicle; Fig. 2 a diagram of a modified embodiment of the method according to Fig. 1; Fig. 3 a diagram of a modified embodiment of the method according to Fig. 1; and Fig. 4 shows a diagram of a combination of the embodiments according to Fig. 2 and Fig. 3.

[0029] The described embodiments are merely examples that can be modified and / or supplemented in a variety of ways within the scope of the claims. Each feature described for a specific embodiment can be used independently or in combination with other features in any other embodiment. Each feature described for an embodiment of a specific claim category can also be used correspondingly in an embodiment of a different claim category.

[0030] Fig. 1 shows a diagram of a preferred embodiment of a method 10 for computer-assisted monitoring of the light transmittance of a lens of a camera system in a motor vehicle. In a first and second step 100, 200, a plurality of limit values ​​for the illuminances measurable by the camera system are determined and provided by means of a pre-trained neural network. In a third and fourth step 300, 400, the illuminance is measured by means of the camera system, and the measurement data is provided. In a fifth step 500, the measurement data is evaluated by means of an evaluation unit connected to the neural network. In a sixth step 600, maintenance information is output in the event that the evaluation 500 of the measurement data reveals a deviation from at least one limit value.

[0031] Fig. 2 shows a diagram of a modified embodiment of the method 10 according to Fig. 1. In an additional step 700, the maintenance information comprising an instruction to carry out an automated cleaning of the lens of the camera system is output to a cleaning system of the motor vehicle.

[0032] Fig. 3 shows a diagram of a modified embodiment of the method 10 according to Fig. 1. In an additional step 800, the maintenance information comprising a request to carry out a manual cleaning of the lens of the camera system is output to an output unit of the motor vehicle.

[0033] Fig. 4 shows a diagram of a combination of the embodiments according to Fig. 2 and Fig. 3. Steps 700 and 800 are output equally. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] DE 10 2015 117 612 A1

[0002]

Claims

[1] A method for computer-aided monitoring of the light transmission of a lens of a camera system in a motor vehicle, comprising the following steps: - determining (100) and providing (200) a plurality of limit values for the illuminances measurable by the camera system by means of a pre-trained neural network; - measuring (300) the illuminance by means of the camera system and providing (400) measurement data; - evaluating (500) the measurement data by means of an evaluation unit connected to the neural network; - Outputting (600) maintenance information in the event that the evaluation of the measurement data results in a deviation from at least one limit value. [2] Method according to claim 1, wherein the neural network is trained with a plurality of data of the illuminance of a clean lens and a dirty lens that can be measured by the camera system, wherein in particular the data used for the training comprise information about the ambient conditions of the motor vehicle and the measurement times. [3] A method according to claim 1 or 2, wherein the neural network determines threshold values for maximum illuminance levels. [4] Method according to one of the preceding claims, wherein the neural network determines threshold values for average illuminances. [5] Method according to one of the preceding claims, wherein the measurement (300) of the illuminance is carried out incrementally, in particular over a predetermined period of time, wherein in particular the measurement (300) of the illuminance represents a real-time measurement. [6] Method according to one of the preceding claims, wherein the evaluation (500) of the measurement data takes into account the ambient conditions and / or the measurement time. [7] Method according to one of the preceding claims, wherein the maintenance information comprises an instruction (700) for carrying out an automated cleaning of the lens of the camera system, and / or wherein the maintenance information comprises a request (800) for carrying out a manual cleaning of the lens of the camera system by the motor vehicle driver. [8] Motor vehicle designed to carry out a method according to one of the preceding claims, comprising: - at least one camera system having a translucent lens for recording the surroundings of the motor vehicle and for measuring the illuminance; - an evaluation unit for receiving and evaluating the measurement data provided by the camera system using a neural network; wherein the neural network determines a plurality of limit values for the illuminances measurable by the camera system and provides them for the evaluation of the measurement data; wherein the evaluation unit outputs maintenance information if the evaluation of the measurement data reveals a deviation from at least one limit value. [9] Motor vehicle according to claim 8, comprising: - a cleaning system for receiving the maintenance information and for performing automated cleaning of the lens of the camera system; and / or - an output unit for receiving the maintenance information and for displaying a request for the motor vehicle driver to carry out a manual cleaning of the lens of the camera system. [10] Motor vehicle according to claim 8 or 9; wherein the neural network is implemented on a server connected to the evaluation unit.

Images